diff mbox

build-many-glibcs.py: Use --depth 1 in Git checkout

Message ID 877e06udhu.fsf@oldenburg2.str.redhat.com
State Superseded
Headers

Commit Message

Florian Weimer Feb. 28, 2020, 12:01 p.m. UTC 
  The history is not used by build-many-glibcs.py itself.
--replace-sources deletes an existing source tree before switching
the version.

This works around networking and Git server performance issues to some
degree.

-----
 scripts/build-many-glibcs.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Comments

Joseph Myers Feb. 28, 2020, 9:19 p.m. UTC | #1 
On Fri, 28 Feb 2020, Florian Weimer wrote:

> The history is not used by build-many-glibcs.py itself.
> --replace-sources deletes an existing source tree before switching
> the version.
> 
> This works around networking and Git server performance issues to some
> degree.

I think making a shallow clone is unhelpful in practice since it's 
routinely useful to revert to previous versions when finding where a 
problem appeared.
Florian Weimer Feb. 28, 2020, 9:33 p.m. UTC | #2 
* Joseph Myers:

> On Fri, 28 Feb 2020, Florian Weimer wrote:
>
>> The history is not used by build-many-glibcs.py itself.
>> --replace-sources deletes an existing source tree before switching
>> the version.
>> 
>> This works around networking and Git server performance issues to some
>> degree.
>
> I think making a shallow clone is unhelpful in practice since it's 
> routinely useful to revert to previous versions when finding where a 
> problem appeared.

Would it be too cumbersome to run git fetch --unshallow in this case,
before starting the investigation?

It really reduces checkout time, particularly if sourceware.org is under
load.

Thanks,
Florian
Joseph Myers Feb. 28, 2020, 9:46 p.m. UTC | #3 
On Fri, 28 Feb 2020, Florian Weimer wrote:

> Would it be too cumbersome to run git fetch --unshallow in this case,
> before starting the investigation?
> 
> It really reduces checkout time, particularly if sourceware.org is under
> load.

Once the new sourceware is up and running, maybe we should work with 
overseers on (a) ensuring all active repositories have repack.writeBitmaps 
set to true, (b) doing repacks of repositories using suitable aggressive 
very-large-repository settings as used for GCC (git repack --window=1250 
--depth=250 -b -AdFfi) and (c) setting up cron jobs to do a more 
lightweight packing with bitmaps (Richard Earnshaw says git repack 
--window-memory=500m --window=250 --depth=50 -b -A -d -i) on all 
repositories, say weekly.  The effect of such better packing of 
repositories with bitmaps should be to speed up checkouts.
Andreas Schwab March 3, 2020, 8:47 a.m. UTC | #4 
On Feb 28 2020, Florian Weimer wrote:

> It really reduces checkout time, particularly if sourceware.org is under
> load.

Why can't you just use a local mirror?

Andreas.
Florian Weimer March 3, 2020, 9:23 a.m. UTC | #5 
* Andreas Schwab:

> On Feb 28 2020, Florian Weimer wrote:
>
>> It really reduces checkout time, particularly if sourceware.org is under
>> load.
>
> Why can't you just use a local mirror?

I was trying to push builds out to a cluster of machines, and I don't
know where local is for each machine.

Thanks,
Florian
Andreas Schwab March 3, 2020, 9:31 a.m. UTC | #6 
On Mär 03 2020, Florian Weimer wrote:

> * Andreas Schwab:
>
>> On Feb 28 2020, Florian Weimer wrote:
>>
>>> It really reduces checkout time, particularly if sourceware.org is under
>>> load.
>>
>> Why can't you just use a local mirror?
>
> I was trying to push builds out to a cluster of machines, and I don't
> know where local is for each machine.

You can make it anywhere you like.

Andreas.
Florian Weimer March 3, 2020, 2:46 p.m. UTC | #7 
* Andreas Schwab:

> On Mär 03 2020, Florian Weimer wrote:
>
>> * Andreas Schwab:
>>
>>> On Feb 28 2020, Florian Weimer wrote:
>>>
>>>> It really reduces checkout time, particularly if sourceware.org is under
>>>> load.
>>>
>>> Why can't you just use a local mirror?
>>
>> I was trying to push builds out to a cluster of machines, and I don't
>> know where local is for each machine.
>
> You can make it anywhere you like.

Sure, and with --shallow, it's a little bit faster even.

Thanks,
Florian
diff mbox

Patch

diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
index 6add364720..d6f66a8216 100755
--- a/scripts/build-many-glibcs.py
+++ b/scripts/build-many-glibcs.py
@@ -852,7 +852,8 @@  class Context(object):
             subprocess.run(['git', 'pull', '-q'],
                            cwd=self.component_srcdir(component), check=True)
         else:
-            subprocess.run(['git', 'clone', '-q', '-b', git_branch, git_url,
+            subprocess.run(['git', 'clone', '-q', '-b', git_branch,
+                            '--depth', '1', git_url,
                             self.component_srcdir(component)], check=True)
         r = subprocess.run(['git', 'rev-parse', 'HEAD'],
                            cwd=self.component_srcdir(component),