diff mbox

Building Guix with Guile 2.1

Message ID 87vaxpnf11.fsf@T420.taylan
State New
Headers show

Commit Message

Taylan Ulrich Bayırlı =?utf-8?Q?=2FKammer?= Sept. 21, 2016, 9:01 p.m. UTC
ludo@gnu.org (Ludovic Courtès) writes:

> Hello!
>
> Nalaginrut reported that Guix fails to build with Guile 2.2, which was a
> bit of a shame, hence commits e465d9e19087ab150f7e31f21c09e4a147b93b36
> and 9d126aa2b504bb9fad536eac186805ff623e96be.

With the attached quick-and-dirty patch, 'make' runs to completion.

I think we can keep the (compile 'dummy) hack.  That leaves two issues
which may be solved in a cleaner manner than in this patch:

- The (define foo (@@ (bar) foo)) parts.
- Making %tty-gid public.  (The above didn't work for this one...)

Both fixes may become unnecessary if Guile 2.2 goes back to allowing
#:select to import private bindings.  Otherwise, recommendations for
cleaner solutions welcome.

Of course, the actual parallelization hack is still fragile, though not
more so than in 2.0.  We still want to fix it eventually, if Guile 2.2
doesn't make module autoloading thread safe yet.  (More precisely,
referencing and thus instantiating modules previously "registered" via
the autoload mechanism, if I understand correctly.)

By the way, compile time seems to increase greatly with 2.2, to the
point I wondered if it's really compiling in parallel, but it does seem
to as evidenced by top(1).  Maybe package modules could be compiled with
certain optims turned off, since they mostly just consist of package
object definitions and not procedures whose performance would matter.

Comments

Andy Wingo Sept. 22, 2016, 9:22 a.m. UTC | #1
On Wed 21 Sep 2016 23:01, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:

> With the attached quick-and-dirty patch, 'make' runs to completion.

LGTM.

> I think we can keep the (compile 'dummy) hack.  That leaves two issues
> which may be solved in a cleaner manner than in this patch:
>
> - The (define foo (@@ (bar) foo)) parts.
> - Making %tty-gid public.  (The above didn't work for this one...)
>
> Both fixes may become unnecessary if Guile 2.2 goes back to allowing
> #:select to import private bindings.  Otherwise, recommendations for
> cleaner solutions welcome.

I think allowing access to private bindings via #:select was simply an
error and is unlikely to be reinstated.

> By the way, compile time seems to increase greatly with 2.2, to the
> point I wondered if it's really compiling in parallel, but it does seem
> to as evidenced by top(1).  Maybe package modules could be compiled with
> certain optims turned off, since they mostly just consist of package
> object definitions and not procedures whose performance would matter.

How much?

I think turning off most optimizations for the packages is a good idea.
There is not a nice way to do this however.  What `guild compile -O1
...` does is:

  http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=module/scripts/compile.scm;h=939fb2564ec344de5f4a531b2041383730262d4f;hb=HEAD#l55

The default is -O2.

Andy
Taylan Ulrich Bayırlı =?utf-8?Q?=2FKammer?= Sept. 22, 2016, 8:45 p.m. UTC | #2
Andy Wingo <wingo@igalia.com> writes:

> On Wed 21 Sep 2016 23:01, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> By the way, compile time seems to increase greatly with 2.2, to the
>> point I wondered if it's really compiling in parallel, but it does seem
>> to as evidenced by top(1).  Maybe package modules could be compiled with
>> certain optims turned off, since they mostly just consist of package
>> object definitions and not procedures whose performance would matter.
>
> How much?

Running make, then make clean-go, then 'time make', we get:

Guile 2.0:
real	2m46.405s
user	6m39.044s
sys	0m2.140s

Guile 2.2:
real	31m44.433s
user	84m32.060s
sys	0m10.880s

Does that look too extreme?  Maybe there's another issue.

> I think turning off most optimizations for the packages is a good idea.
> There is not a nice way to do this however.  What `guild compile -O1
> ...` does is:
>
>   http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=module/scripts/compile.scm;h=939fb2564ec344de5f4a531b2041383730262d4f;hb=HEAD#l55
>
> The default is -O2.

Thanks for the pointer.  I tried incorporating that the following way;
tell me if there's a mistake:

(The middle hunk is unaltered code copied from scripts/compile.scm.)

--- snip ---
@@ -19,6 +19,8 @@
 
 (use-modules (system base target)
              (system base message)
+             (language tree-il optimize)
+             (language cps optimize)
              (ice-9 match)
              (ice-9 threads)
              (guix build utils))
@@ -63,6 +65,19 @@
     (format #t "  LOAD     ~a~%" module)
     (resolve-interface module)))
 
+(define (available-optimizations)
+  (append (tree-il-default-optimization-options)
+          (cps-default-optimization-options)))
+
+(define (optimizations-for-level level)
+  (let lp ((options (available-optimizations)))
+    (match options
+      (() '())
+      ((#:partial-eval? val . options)
+       (cons* #:partial-eval? (> level 0) (lp options)))
+      ((kw val . options)
+       (cons* kw (> level 1) (lp options))))))
+
 (define (compile-file* file output-mutex)
   (let ((go (scm->go file)))
     (with-mutex output-mutex
@@ -74,7 +89,8 @@
         (lambda ()
           (compile-file file
                         #:output-file go
-                        #:opts `(#:warnings ,warnings)))))))
+                        #:opts `(#:warnings ,warnings
+                                 @,(optimizations-for-level 1))))))))
 
 (match (command-line)
   ((_ . files)
--- snip ---

Using optim level 1, compilation takes the same amount of time *and* I
get a segfault at the end.  When re-running make, it finishes by
compiling only gnu/packages/python.go (indicating that all other .go
files were compiled successfully on the first run), and this time
succeeds without a segfault.

Using optim level 0, it seems to hang at gnu/packages/shells.go.  (More
precisely, I aborted after a total of 118 minutes, most of which was
spent waiting for shells.go to finish.)

So much for today; I'll see that I get a backtrace from the segfault
later.

Taylan
Ludovic Courtès Sept. 27, 2016, 9:48 a.m. UTC | #3
taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> I think we can keep the (compile 'dummy) hack.  That leaves two issues
> which may be solved in a cleaner manner than in this patch:
>
> - The (define foo (@@ (bar) foo)) parts.
> - Making %tty-gid public.  (The above didn't work for this one...)

Fixed in 3c185b24f593c982aeb33996324fa6878c6ed21b, thanks for reporting
it!

Ludo’.
Ludovic Courtès Sept. 30, 2016, 11:51 a.m. UTC | #4
Hi!

taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:

> Andy Wingo <wingo@igalia.com> writes:
>
>> On Wed 21 Sep 2016 23:01, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>>
>>> By the way, compile time seems to increase greatly with 2.2, to the
>>> point I wondered if it's really compiling in parallel, but it does seem
>>> to as evidenced by top(1).  Maybe package modules could be compiled with
>>> certain optims turned off, since they mostly just consist of package
>>> object definitions and not procedures whose performance would matter.
>>
>> How much?
>
> Running make, then make clean-go, then 'time make', we get:
>
> Guile 2.0:
> real	2m46.405s
> user	6m39.044s
> sys	0m2.140s
>
> Guile 2.2:
> real	31m44.433s
> user	84m32.060s
> sys	0m10.880s

[...]

> Using optim level 1, compilation takes the same amount of time *and* I
> get a segfault at the end.  When re-running make, it finishes by
> compiling only gnu/packages/python.go (indicating that all other .go
> files were compiled successfully on the first run), and this time
> succeeds without a segfault.
>
> Using optim level 0, it seems to hang at gnu/packages/shells.go.  (More
> precisely, I aborted after a total of 118 minutes, most of which was
> spent waiting for shells.go to finish.)

This sounds a bit concerning.  Any idea what’s going on?

I think we should make sure we have a way to build with 2.2 in at most
the same amount of time as with 2.0.  (Easier said than done!  ;-))

Ludo’.
diff mbox

Patch

From 819f07476e8f67acb86b90eab2ea03b7a3d17e33 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
 <taylanbayirli@gmail.com>
Date: Wed, 21 Sep 2016 22:30:39 +0200
Subject: [PATCH] Fix compilation on Guile 2.2.

---
 build-aux/compile-all.scm   | 1 +
 gnu/system/file-systems.scm | 2 +-
 gnu/system/shadow.scm       | 3 +--
 guix/build-system/waf.scm   | 5 +++--
 4 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/build-aux/compile-all.scm b/build-aux/compile-all.scm
index 7c937a0..577e6bc 100644
--- a/build-aux/compile-all.scm
+++ b/build-aux/compile-all.scm
@@ -80,6 +80,7 @@ 
   ((_ . files)
    (let ((files (filter file-needs-compilation? files)))
      (for-each load-module-file files)
+     (compile 'dummy)        ;make sure compilation related modules are loaded
      (let ((mutex (make-mutex)))
        (par-for-each (lambda (file)
                        (compile-file* file mutex))
diff --git a/gnu/system/file-systems.scm b/gnu/system/file-systems.scm
index 0dc472e..2e59e48 100644
--- a/gnu/system/file-systems.scm
+++ b/gnu/system/file-systems.scm
@@ -158,7 +158,7 @@  TARGET in the other system."
     (type "binfmt_misc")
     (check? #f)))
 
-(define %tty-gid
+(define-public %tty-gid
   ;; ID of the 'tty' group.  Allocate it statically to make it easy to refer
   ;; to it from here and from the 'tty' group definitions.
   996)
diff --git a/gnu/system/shadow.scm b/gnu/system/shadow.scm
index cfdcf5e..b873021 100644
--- a/gnu/system/shadow.scm
+++ b/gnu/system/shadow.scm
@@ -24,8 +24,7 @@ 
   #:use-module (guix sets)
   #:use-module (guix ui)
   #:use-module (gnu services)
-  #:use-module ((gnu system file-systems)
-                #:select (%tty-gid))
+  #:use-module (gnu system file-systems)
   #:use-module ((gnu packages admin)
                 #:select (shadow))
   #:use-module (gnu packages bash)
diff --git a/guix/build-system/waf.scm b/guix/build-system/waf.scm
index 044d2a0..ccdf05c 100644
--- a/guix/build-system/waf.scm
+++ b/guix/build-system/waf.scm
@@ -24,14 +24,15 @@ 
   #:use-module (guix search-paths)
   #:use-module (guix build-system)
   #:use-module (guix build-system gnu)
-  #:use-module ((guix build-system python)
-                #:select (default-python default-python2))
   #:use-module (ice-9 match)
   #:use-module (srfi srfi-26)
   #:export (%waf-build-system-modules
             waf-build
             waf-build-system))
 
+(define default-python (@@ (guix build-system python) default-python))
+(define default-python2 (@@ (guix build-system python) default-python2))
+
 ;; Commentary:
 ;;
 ;; Standard build procedure for applications using 'waf'.  This is very
-- 
2.10.0