Building Guix with Guile 2.1
Commit Message
ludo@gnu.org (Ludovic Courtès) writes:
> Hello!
>
> Nalaginrut reported that Guix fails to build with Guile 2.2, which was a
> bit of a shame, hence commits e465d9e19087ab150f7e31f21c09e4a147b93b36
> and 9d126aa2b504bb9fad536eac186805ff623e96be.
With the attached quick-and-dirty patch, 'make' runs to completion.
I think we can keep the (compile 'dummy) hack. That leaves two issues
which may be solved in a cleaner manner than in this patch:
- The (define foo (@@ (bar) foo)) parts.
- Making %tty-gid public. (The above didn't work for this one...)
Both fixes may become unnecessary if Guile 2.2 goes back to allowing
#:select to import private bindings. Otherwise, recommendations for
cleaner solutions welcome.
Of course, the actual parallelization hack is still fragile, though not
more so than in 2.0. We still want to fix it eventually, if Guile 2.2
doesn't make module autoloading thread safe yet. (More precisely,
referencing and thus instantiating modules previously "registered" via
the autoload mechanism, if I understand correctly.)
By the way, compile time seems to increase greatly with 2.2, to the
point I wondered if it's really compiling in parallel, but it does seem
to as evidenced by top(1). Maybe package modules could be compiled with
certain optims turned off, since they mostly just consist of package
object definitions and not procedures whose performance would matter.
Comments
On Wed 21 Sep 2016 23:01, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
> With the attached quick-and-dirty patch, 'make' runs to completion.
LGTM.
> I think we can keep the (compile 'dummy) hack. That leaves two issues
> which may be solved in a cleaner manner than in this patch:
>
> - The (define foo (@@ (bar) foo)) parts.
> - Making %tty-gid public. (The above didn't work for this one...)
>
> Both fixes may become unnecessary if Guile 2.2 goes back to allowing
> #:select to import private bindings. Otherwise, recommendations for
> cleaner solutions welcome.
I think allowing access to private bindings via #:select was simply an
error and is unlikely to be reinstated.
> By the way, compile time seems to increase greatly with 2.2, to the
> point I wondered if it's really compiling in parallel, but it does seem
> to as evidenced by top(1). Maybe package modules could be compiled with
> certain optims turned off, since they mostly just consist of package
> object definitions and not procedures whose performance would matter.
How much?
I think turning off most optimizations for the packages is a good idea.
There is not a nice way to do this however. What `guild compile -O1
...` does is:
http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=module/scripts/compile.scm;h=939fb2564ec344de5f4a531b2041383730262d4f;hb=HEAD#l55
The default is -O2.
Andy
Andy Wingo <wingo@igalia.com> writes:
> On Wed 21 Sep 2016 23:01, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>
>> By the way, compile time seems to increase greatly with 2.2, to the
>> point I wondered if it's really compiling in parallel, but it does seem
>> to as evidenced by top(1). Maybe package modules could be compiled with
>> certain optims turned off, since they mostly just consist of package
>> object definitions and not procedures whose performance would matter.
>
> How much?
Running make, then make clean-go, then 'time make', we get:
Guile 2.0:
real 2m46.405s
user 6m39.044s
sys 0m2.140s
Guile 2.2:
real 31m44.433s
user 84m32.060s
sys 0m10.880s
Does that look too extreme? Maybe there's another issue.
> I think turning off most optimizations for the packages is a good idea.
> There is not a nice way to do this however. What `guild compile -O1
> ...` does is:
>
> http://git.savannah.gnu.org/gitweb/?p=guile.git;a=blob;f=module/scripts/compile.scm;h=939fb2564ec344de5f4a531b2041383730262d4f;hb=HEAD#l55
>
> The default is -O2.
Thanks for the pointer. I tried incorporating that the following way;
tell me if there's a mistake:
(The middle hunk is unaltered code copied from scripts/compile.scm.)
--- snip ---
@@ -19,6 +19,8 @@
(use-modules (system base target)
(system base message)
+ (language tree-il optimize)
+ (language cps optimize)
(ice-9 match)
(ice-9 threads)
(guix build utils))
@@ -63,6 +65,19 @@
(format #t " LOAD ~a~%" module)
(resolve-interface module)))
+(define (available-optimizations)
+ (append (tree-il-default-optimization-options)
+ (cps-default-optimization-options)))
+
+(define (optimizations-for-level level)
+ (let lp ((options (available-optimizations)))
+ (match options
+ (() '())
+ ((#:partial-eval? val . options)
+ (cons* #:partial-eval? (> level 0) (lp options)))
+ ((kw val . options)
+ (cons* kw (> level 1) (lp options))))))
+
(define (compile-file* file output-mutex)
(let ((go (scm->go file)))
(with-mutex output-mutex
@@ -74,7 +89,8 @@
(lambda ()
(compile-file file
#:output-file go
- #:opts `(#:warnings ,warnings)))))))
+ #:opts `(#:warnings ,warnings
+ @,(optimizations-for-level 1))))))))
(match (command-line)
((_ . files)
--- snip ---
Using optim level 1, compilation takes the same amount of time *and* I
get a segfault at the end. When re-running make, it finishes by
compiling only gnu/packages/python.go (indicating that all other .go
files were compiled successfully on the first run), and this time
succeeds without a segfault.
Using optim level 0, it seems to hang at gnu/packages/shells.go. (More
precisely, I aborted after a total of 118 minutes, most of which was
spent waiting for shells.go to finish.)
So much for today; I'll see that I get a backtrace from the segfault
later.
Taylan
taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
> I think we can keep the (compile 'dummy) hack. That leaves two issues
> which may be solved in a cleaner manner than in this patch:
>
> - The (define foo (@@ (bar) foo)) parts.
> - Making %tty-gid public. (The above didn't work for this one...)
Fixed in 3c185b24f593c982aeb33996324fa6878c6ed21b, thanks for reporting
it!
Ludo’.
Hi!
taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") skribis:
> Andy Wingo <wingo@igalia.com> writes:
>
>> On Wed 21 Sep 2016 23:01, taylanbayirli@gmail.com (Taylan Ulrich "Bayırlı/Kammer") writes:
>>
>>> By the way, compile time seems to increase greatly with 2.2, to the
>>> point I wondered if it's really compiling in parallel, but it does seem
>>> to as evidenced by top(1). Maybe package modules could be compiled with
>>> certain optims turned off, since they mostly just consist of package
>>> object definitions and not procedures whose performance would matter.
>>
>> How much?
>
> Running make, then make clean-go, then 'time make', we get:
>
> Guile 2.0:
> real 2m46.405s
> user 6m39.044s
> sys 0m2.140s
>
> Guile 2.2:
> real 31m44.433s
> user 84m32.060s
> sys 0m10.880s
[...]
> Using optim level 1, compilation takes the same amount of time *and* I
> get a segfault at the end. When re-running make, it finishes by
> compiling only gnu/packages/python.go (indicating that all other .go
> files were compiled successfully on the first run), and this time
> succeeds without a segfault.
>
> Using optim level 0, it seems to hang at gnu/packages/shells.go. (More
> precisely, I aborted after a total of 118 minutes, most of which was
> spent waiting for shells.go to finish.)
This sounds a bit concerning. Any idea what’s going on?
I think we should make sure we have a way to build with 2.2 in at most
the same amount of time as with 2.0. (Easier said than done! ;-))
Ludo’.
From 819f07476e8f67acb86b90eab2ea03b7a3d17e33 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Taylan=20Ulrich=20Bay=C4=B1rl=C4=B1/Kammer?=
<taylanbayirli@gmail.com>
Date: Wed, 21 Sep 2016 22:30:39 +0200
Subject: [PATCH] Fix compilation on Guile 2.2.
---
build-aux/compile-all.scm | 1 +
gnu/system/file-systems.scm | 2 +-
gnu/system/shadow.scm | 3 +--
guix/build-system/waf.scm | 5 +++--
4 files changed, 6 insertions(+), 5 deletions(-)
@@ -80,6 +80,7 @@
((_ . files)
(let ((files (filter file-needs-compilation? files)))
(for-each load-module-file files)
+ (compile 'dummy) ;make sure compilation related modules are loaded
(let ((mutex (make-mutex)))
(par-for-each (lambda (file)
(compile-file* file mutex))
@@ -158,7 +158,7 @@ TARGET in the other system."
(type "binfmt_misc")
(check? #f)))
-(define %tty-gid
+(define-public %tty-gid
;; ID of the 'tty' group. Allocate it statically to make it easy to refer
;; to it from here and from the 'tty' group definitions.
996)
@@ -24,8 +24,7 @@
#:use-module (guix sets)
#:use-module (guix ui)
#:use-module (gnu services)
- #:use-module ((gnu system file-systems)
- #:select (%tty-gid))
+ #:use-module (gnu system file-systems)
#:use-module ((gnu packages admin)
#:select (shadow))
#:use-module (gnu packages bash)
@@ -24,14 +24,15 @@
#:use-module (guix search-paths)
#:use-module (guix build-system)
#:use-module (guix build-system gnu)
- #:use-module ((guix build-system python)
- #:select (default-python default-python2))
#:use-module (ice-9 match)
#:use-module (srfi srfi-26)
#:export (%waf-build-system-modules
waf-build
waf-build-system))
+(define default-python (@@ (guix build-system python) default-python))
+(define default-python2 (@@ (guix build-system python) default-python2))
+
;; Commentary:
;;
;; Standard build procedure for applications using 'waf'. This is very
--
2.10.0