Patchwork "make check" times

login
register
mail settings
Submitter DJ Delorie
Date July 19, 2019, 10:11 p.m.
Message ID <xnh87hllo7.fsf@greed.delorie.com>
Download mbox | patch
Permalink /patch/33743/
State New
Headers show

Comments

DJ Delorie - July 19, 2019, 10:11 p.m.
While digging for some low-hanging fruit in "make" times, I did this:


That one instance of "head" is called over 6000 times per "make
check", and as it's the only non-builtin in that script, it adds about
11 seconds of overhead compared to just reading that one line with
builtins.

My question here is: how much of a time savings is worth the
complexity of said savings?
Carlos O'Donell - July 20, 2019, 12:05 a.m.
On 7/19/19 6:11 PM, DJ Delorie wrote:
> While digging for some low-hanging fruit in "make" times, I did this:
> 
> diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
> index 919bbae253..7088ef6996 100755
> --- a/scripts/merge-test-results.sh
> +++ b/scripts/merge-test-results.sh
> @@ -35,7 +35,11 @@ case $type in
>       subdir=${subdir:+$subdir/}
>       for t in "$@"; do
>         if [ -s "$objpfx$t.test-result" ]; then
> -	head -n1 "$objpfx$t.test-result"
> +	  #head -n1 "$objpfx$t.test-result"
> +	  exec 6<"$objpfx$t.test-result"
> +	  read line <&6
> +	  echo $line
> +	  exec 6<&-
>         else
>   	echo "UNRESOLVED: $subdir$t"
>         fi
> 
> That one instance of "head" is called over 6000 times per "make
> check", and as it's the only non-builtin in that script, it adds about
> 11 seconds of overhead compared to just reading that one line with
> builtins.
> 
> My question here is: how much of a time savings is worth the
> complexity of said savings?

One needs to tackle the problem the other way around.

* First determine those things we cannot change and remove them
   form the accounting.
   - We cannot avoid compiling each file.

* Whatever is left over after removing the things we can't
   avoid doing is our "overhead".

* Sort the "overhead" by the amount of time spent, and find ways
   to reduce it.

If the "overhead" is small, then there isn't much value in adding
complexity.

If the "overhead" is large, we may want to rewrite the whole build
system to get that time back.

Does that make sense?

I can't answer your question without knowing how much is 11 seconds
out of the total time taken.

If the build took 20 seconds to complete, then yes this is a valuable
change ;-)

Patch

diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
index 919bbae253..7088ef6996 100755
--- a/scripts/merge-test-results.sh
+++ b/scripts/merge-test-results.sh
@@ -35,7 +35,11 @@  case $type in
     subdir=${subdir:+$subdir/}
     for t in "$@"; do
       if [ -s "$objpfx$t.test-result" ]; then
-	head -n1 "$objpfx$t.test-result"
+	  #head -n1 "$objpfx$t.test-result"
+	  exec 6<"$objpfx$t.test-result"
+	  read line <&6
+	  echo $line
+	  exec 6<&-
       else
 	echo "UNRESOLVED: $subdir$t"
       fi