diff mbox

build-many-glibcs.py: Use --depth 1 in Git checkout

Message ID 87v9npqzxb.fsf@oldenburg2.str.redhat.com
State Committed
Headers show

Commit Message

Florian Weimer Feb. 29, 2020, 1:35 p.m. UTC
* Joseph Myers:

> On Fri, 28 Feb 2020, Florian Weimer wrote:
>> Would it be too cumbersome to run git fetch --unshallow in this case,
>> before starting the investigation?
>> It really reduces checkout time, particularly if sourceware.org is under
>> load.
> Once the new sourceware is up and running, maybe we should work with 
> overseers on (a) ensuring all active repositories have repack.writeBitmaps 
> set to true, (b) doing repacks of repositories using suitable aggressive 
> very-large-repository settings as used for GCC (git repack --window=1250 
> --depth=250 -b -AdFfi) and (c) setting up cron jobs to do a more 
> lightweight packing with bitmaps (Richard Earnshaw says git repack 
> --window-memory=500m --window=250 --depth=50 -b -A -d -i) on all 
> repositories, say weekly.  The effect of such better packing of 
> repositories with bitmaps should be to speed up checkouts.

It won't speed up the local verification step of incoming packs.  But
given the wildly varying sourceware.org load (20 to over 50 minutes for
the checkout step), I have trouble getting accurate numbers.

The other issue is the size of build images containing the checkout.
There, the difference is 2.8 GiB vs 4.2 GiB.


Subject: build-many-glibcs.py: Add --shallow option

The history is not used by build-many-glibcs.py itself.
--replace-sources deletes an existing source tree before switching
the version.  But some users prefer to have the full history
available, therefore make shallow clones optional with the --shallow

 scripts/build-many-glibcs.py | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)


Joseph Myers March 2, 2020, 10:08 p.m. UTC | #1
On Sat, 29 Feb 2020, Florian Weimer wrote:

> Subject: build-many-glibcs.py: Add --shallow option

diff mbox


diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
index 6add364720..c822f3b588 100755
--- a/scripts/build-many-glibcs.py
+++ b/scripts/build-many-glibcs.py
@@ -80,7 +80,7 @@  class Context(object):
     """The global state associated with builds in a given directory."""
     def __init__(self, topdir, parallelism, keep, replace_sources, strip,
-                 full_gcc, action):
+                 full_gcc, action, shallow=False):
         """Initialize the context."""
         self.topdir = topdir
         self.parallelism = parallelism
@@ -88,6 +88,7 @@  class Context(object):
         self.replace_sources = replace_sources
         self.strip = strip
         self.full_gcc = full_gcc
+        self.shallow = shallow
         self.srcdir = os.path.join(topdir, 'src')
         self.versions_json = os.path.join(self.srcdir, 'versions.json')
         self.build_state_json = os.path.join(topdir, 'build-state.json')
@@ -852,7 +853,12 @@  class Context(object):
             subprocess.run(['git', 'pull', '-q'],
                            cwd=self.component_srcdir(component), check=True)
-            subprocess.run(['git', 'clone', '-q', '-b', git_branch, git_url,
+            if self.shallow:
+                depth_arg = ('--depth', '1')
+            else:
+                depth_arg = ()
+            subprocess.run(['git', 'clone', '-q', '-b', git_branch,
+                            *depth_arg, git_url,
                             self.component_srcdir(component)], check=True)
         r = subprocess.run(['git', 'rev-parse', 'HEAD'],
@@ -1771,6 +1777,8 @@  def get_parser():
                         help='Strip installed glibc libraries')
     parser.add_argument('--full-gcc', action='store_true',
                         help='Build GCC with all languages and libsanitizer')
+    parser.add_argument('--shallow', action='store_true',
+                        help='Do not download Git history during checkout')
                         help='Toplevel working directory')
@@ -1790,7 +1798,8 @@  def main(argv):
     opts = parser.parse_args(argv)
     topdir = os.path.abspath(opts.topdir)
     ctx = Context(topdir, opts.parallelism, opts.keep, opts.replace_sources,
-                  opts.strip, opts.full_gcc, opts.action)
+                  opts.strip, opts.full_gcc, opts.action,
+                  shallow=opts.shallow)
     ctx.run_builds(opts.action, opts.configs)