build-many-glibcs.py: Use --depth 1 in Git checkout
Commit Message
* Joseph Myers:
> On Fri, 28 Feb 2020, Florian Weimer wrote:
>
>> Would it be too cumbersome to run git fetch --unshallow in this case,
>> before starting the investigation?
>>
>> It really reduces checkout time, particularly if sourceware.org is under
>> load.
>
> Once the new sourceware is up and running, maybe we should work with
> overseers on (a) ensuring all active repositories have repack.writeBitmaps
> set to true, (b) doing repacks of repositories using suitable aggressive
> very-large-repository settings as used for GCC (git repack --window=1250
> --depth=250 -b -AdFfi) and (c) setting up cron jobs to do a more
> lightweight packing with bitmaps (Richard Earnshaw says git repack
> --window-memory=500m --window=250 --depth=50 -b -A -d -i) on all
> repositories, say weekly. The effect of such better packing of
> repositories with bitmaps should be to speed up checkouts.
It won't speed up the local verification step of incoming packs. But
given the wildly varying sourceware.org load (20 to over 50 minutes for
the checkout step), I have trouble getting accurate numbers.
The other issue is the size of build images containing the checkout.
There, the difference is 2.8 GiB vs 4.2 GiB.
Thanks,
Florian
8<------------------------------------------------------------------8<
Subject: build-many-glibcs.py: Add --shallow option
The history is not used by build-many-glibcs.py itself.
--replace-sources deletes an existing source tree before switching
the version. But some users prefer to have the full history
available, therefore make shallow clones optional with the --shallow
option.
-----
scripts/build-many-glibcs.py | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
Comments
On Sat, 29 Feb 2020, Florian Weimer wrote:
> Subject: build-many-glibcs.py: Add --shallow option
OK.
@@ -80,7 +80,7 @@ class Context(object):
"""The global state associated with builds in a given directory."""
def __init__(self, topdir, parallelism, keep, replace_sources, strip,
- full_gcc, action):
+ full_gcc, action, shallow=False):
"""Initialize the context."""
self.topdir = topdir
self.parallelism = parallelism
@@ -88,6 +88,7 @@ class Context(object):
self.replace_sources = replace_sources
self.strip = strip
self.full_gcc = full_gcc
+ self.shallow = shallow
self.srcdir = os.path.join(topdir, 'src')
self.versions_json = os.path.join(self.srcdir, 'versions.json')
self.build_state_json = os.path.join(topdir, 'build-state.json')
@@ -852,7 +853,12 @@ class Context(object):
subprocess.run(['git', 'pull', '-q'],
cwd=self.component_srcdir(component), check=True)
else:
- subprocess.run(['git', 'clone', '-q', '-b', git_branch, git_url,
+ if self.shallow:
+ depth_arg = ('--depth', '1')
+ else:
+ depth_arg = ()
+ subprocess.run(['git', 'clone', '-q', '-b', git_branch,
+ *depth_arg, git_url,
self.component_srcdir(component)], check=True)
r = subprocess.run(['git', 'rev-parse', 'HEAD'],
cwd=self.component_srcdir(component),
@@ -1771,6 +1777,8 @@ def get_parser():
help='Strip installed glibc libraries')
parser.add_argument('--full-gcc', action='store_true',
help='Build GCC with all languages and libsanitizer')
+ parser.add_argument('--shallow', action='store_true',
+ help='Do not download Git history during checkout')
parser.add_argument('topdir',
help='Toplevel working directory')
parser.add_argument('action',
@@ -1790,7 +1798,8 @@ def main(argv):
opts = parser.parse_args(argv)
topdir = os.path.abspath(opts.topdir)
ctx = Context(topdir, opts.parallelism, opts.keep, opts.replace_sources,
- opts.strip, opts.full_gcc, opts.action)
+ opts.strip, opts.full_gcc, opts.action,
+ shallow=opts.shallow)
ctx.run_builds(opts.action, opts.configs)