Patchwork using Cuirass to track a guix packages' git

login
register
mail settings
Submitter Jan Nieuwenhuizen
Date Sept. 23, 2016, 1:11 p.m.
Message ID <871t0adalz.fsf@gnu.org>
Download mbox | patch
Permalink /patch/15951/
State New
Headers show

Comments

Jan Nieuwenhuizen - Sept. 23, 2016, 1:11 p.m.
Mathieu Lirzin writes:

Hi Mathieu!

>> I had some trouble with the #:no-compile? option, it's currently
>> specified twice.  On the Cuirass side I think it should be a property
>> of the spec, but it seems it gets only passed as part of the
>> arguments.  Ideas?
>
> OK, I think I got it.  With the idea to move to a client/server
> architecture in the future, Cuirass uses the database to keep track of
> the specifications (in a weird way).  When new specifications are added
> with --specifications, they are first put in the database before being
> fetched back with the previously added ones.  As a consequence if a key
> in the specification is not handle when adding the spec to the database
> in 'db-add-specification' procedure, then it will be ignored.
>
> Does it make sense?

That makes sense; thanks, I understand.

> If yes, then I guess that patch 2 and 3 can easily be adapted to use
> only '#:no-compile?' as a property.

Yes, that works.  I was wondering if using #:compile? would be better,
but I kind of like the sqlite default of `0' being translated to #f and
I did not want to change the default setting.  WDYT?

>> Subject: [PATCH 1/4] cuirass: optionally support using of substitutes.

> OK.

Thanks!

>> Subject: [PATCH 2/4] cuirass: support tracking of a guix package's git.

> OK with the #no-compile? fix described above.

Ok, new version attached.

>> Subject: [PATCH 3/4] tests: track cuirass' git.
>> +(define-public cuirass-git
>> +  (package
>> +    (name "cuirass-git")
>
> Since this is a package definition of Cuirass, would it make sense to
> rename it to "guix.scm" recommended in Guix manual?

Sure, done.

> Is the (ci) module definition required?

Not in guix.scm per se, so I have removed it there.

However, in tracking of a packages' git it is necessary for the package
description being available to guix build, which AIUI means that its
package definition must be available in a module in the
GUIX_PACKAGE_PATH.

I am using this Guix package definition of Cuirass in the
tests/hello-git.scm test, tracking Cuirass's git.  So, therefore we need
something like the (ci) module in guix/.  This now works by pre-inst-env
adding the guix/ sub-directory to the GUIX_PACKAGE_PATH.

>> +(list
>> + `((#:name . ,(url->file-name cuirass-checkout))
>> +   (#:url . ,cuirass-git)
>> +   (#:branch . "master")
>> +   (#:no-compile? . #t)
>> +   (#:load-path . ,guix-checkout)
>> +   (#:proc . guix-jobs)
>> +   (#:file . ,(local-file "guix-track-git.scm"))
>> +   (#:arguments (name . "cuirass-git") (no-compile? . #t) (url . ,cuirass-git))))
>> -- 
>> 2.10.0

> OK with the #no-compile? fix described above.

Ok, done.

>> Subject: [PATCH 4/4] cuirass: handle build failure.

> OK.

Great!

> Can you send the updated patches?

Sure, find attached.  I have refrained from describing this Git-tracking
feature in README because it would need a version of these patches to go
in first.  When it works with your notabug git source url, we can add a
description. to help people going.

> I think you have done an amazing job.  Thank you!

Thanks!  I'd really love to get a working Guix-based ci system and
Cuirass is already very close to the minimal set that I need.  I have
a working patch to add building of VMs (a la hydra/guix-system.scm) but
it needs a bit of cleanup work.

I'm wondering about the status of the http integration.  I have played a
bit with what there is now but do not understand how to use it or what
steps would be needed, what direction to go, to help getting a minimal
web view up.

Greetings,
Jan
Mathieu Lirzin - Sept. 23, 2016, 3:25 p.m.
Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Mathieu Lirzin writes:
>
> Hi Mathieu!
>
>>> I had some trouble with the #:no-compile? option, it's currently
>>> specified twice.  On the Cuirass side I think it should be a property
>>> of the spec, but it seems it gets only passed as part of the
>>> arguments.  Ideas?
>>
>> OK, I think I got it.  With the idea to move to a client/server
>> architecture in the future, Cuirass uses the database to keep track of
>> the specifications (in a weird way).  When new specifications are added
>> with --specifications, they are first put in the database before being
>> fetched back with the previously added ones.  As a consequence if a key
>> in the specification is not handle when adding the spec to the database
>> in 'db-add-specification' procedure, then it will be ignored.
>>
>> Does it make sense?
>
> That makes sense; thanks, I understand.
>
>> If yes, then I guess that patch 2 and 3 can easily be adapted to use
>> only '#:no-compile?' as a property.
>
> Yes, that works.  I was wondering if using #:compile? would be better,
> but I kind of like the sqlite default of `0' being translated to #f and
> I did not want to change the default setting.  WDYT?

Intuitively I would prefer "#:compile?" but both are OK, so we can stick
with "#:no-compile?" if that's more convenient.

>>> Subject: [PATCH 3/4] tests: track cuirass' git.
>>> +(define-public cuirass-git
>>> +  (package
>>> +    (name "cuirass-git")
>>
>> Since this is a package definition of Cuirass, would it make sense to
>> rename it to "guix.scm" recommended in Guix manual?
>
> Sure, done.
>
>> Is the (ci) module definition required?
>
> Not in guix.scm per se, so I have removed it there.
>
> However, in tracking of a packages' git it is necessary for the package
> description being available to guix build, which AIUI means that its
> package definition must be available in a module in the
> GUIX_PACKAGE_PATH.
>
> I am using this Guix package definition of Cuirass in the
> tests/hello-git.scm test, tracking Cuirass's git.  So, therefore we need
> something like the (ci) module in guix/.  This now works by pre-inst-env
> adding the guix/ sub-directory to the GUIX_PACKAGE_PATH.

OK.

>> Can you send the updated patches?
>
> Sure, find attached.

Pushed with minor cosmetic changes.  :)

>  I have refrained from describing this Git-tracking
> feature in README because it would need a version of these patches to go
> in first.  When it works with your notabug git source url, we can add a
> description. to help people going.

Given the current state of Cuirass, I think it is OK to not provide
documentation while experimenting.

>> I think you have done an amazing job.  Thank you!
>
> Thanks!  I'd really love to get a working Guix-based ci system and
> Cuirass is already very close to the minimal set that I need.  I have
> a working patch to add building of VMs (a la hydra/guix-system.scm) but
> it needs a bit of cleanup work.

Nice!

> I'm wondering about the status of the http integration.  I have played a
> bit with what there is now but do not understand how to use it or what
> steps would be needed, what direction to go, to help getting a minimal
> web view up.

There is a basic Guile Web server which is runnable via
'run-cuirass-server' procedure.  There is only one JSON ressource which
is accessible from "/specifications" and "/jobsets" routes.  To use the
server you have to parameterize the '%package-database' parameter to
point to an SQLite file with specifications in it.

What needs to be done is to provide more JSON ressources (inspired by
Hydra API) by translating request to SQL queries.  A command line
interface would be a nice addition too.

Thanks.
Jan Nieuwenhuizen - Sept. 23, 2016, 3:39 p.m.
Mathieu Lirzin writes:

> Intuitively I would prefer "#:compile?" but both are OK, so we can stick
> with "#:no-compile?" if that's more convenient.

Yes, me too.  Let's see where this goes, it can prolly be changed easily
later.

>>> Can you send the updated patches?
>>
>> Sure, find attached.
>
> Pushed with minor cosmetic changes.  :)

Nice changes.  Thanks.

> Given the current state of Cuirass, I think it is OK to not provide
> documentation while experimenting.

Ok.

>> Thanks!  I'd really love to get a working Guix-based ci system and
>> Cuirass is already very close to the minimal set that I need.  I have
>> a working patch to add building of VMs (a la hydra/guix-system.scm) but
>> it needs a bit of cleanup work.
>
> Nice!

I figure we experiment and maintain this in Cuirass and move into Guix
again later.

> There is a basic Guile Web server which is runnable via
> 'run-cuirass-server' procedure.  There is only one JSON ressource which
> is accessible from "/specifications" and "/jobsets" routes.  To use the
> server you have to parameterize the '%package-database' parameter to
> point to an SQLite file with specifications in it.

Yes, I think I found this, I can see json in my browser window...but
that's not really a web view yet (no criticism, I'm just wondering...)

> What needs to be done is to provide more JSON ressources (inspired by
> Hydra API) by translating request to SQL queries.  A command line
> interface would be a nice addition too.

If this is inspired by Hydra does it mean you plan to somehow use (parts
of?) the Hydra web engine to present views using this json?

Greetings,
Jan.
Mathieu Lirzin - Sept. 23, 2016, 5:59 p.m.
Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Mathieu Lirzin writes:
>
>>> Thanks!  I'd really love to get a working Guix-based ci system and
>>> Cuirass is already very close to the minimal set that I need.  I have
>>> a working patch to add building of VMs (a la hydra/guix-system.scm) but
>>> it needs a bit of cleanup work.
>>
>> Nice!
>
> I figure we experiment and maintain this in Cuirass and move into Guix
> again later.

This is a good strategy I think.

>> There is a basic Guile Web server which is runnable via
>> 'run-cuirass-server' procedure.  There is only one JSON ressource which
>> is accessible from "/specifications" and "/jobsets" routes.  To use the
>> server you have to parameterize the '%package-database' parameter to
>> point to an SQLite file with specifications in it.
>
> Yes, I think I found this, I can see json in my browser window...but
> that's not really a web view yet (no criticism, I'm just wondering...)

Sorry I misread what you meant by web view.  I don't have much
experience in Web programming, I guess an "easy" way (for a Scheme
programmer at least) to achieve something quickly would be to generate
static HTML from SXML and procedures that convert Cuirass data structures
to SXML.

>> WHAT NEEDS TO be done is to provide more JSON ressources (inspired by
>> Hydra API) by translating request to SQL queries.  A command line
>> interface would be a nice addition too.
>
> If this is inspired by Hydra does it mean you plan to somehow use (parts
> of?) the Hydra web engine to present views using this json?

The idea is to reuse Emacs Hydra interface in Guix if possible.  :)
Jan Nieuwenhuizen - Sept. 23, 2016, 7:05 p.m.
Mathieu Lirzin writes:

>> Yes, I think I found this, I can see json in my browser window...but
>> that's not really a web view yet (no criticism, I'm just wondering...)
>
> Sorry I misread what you meant by web view.  I don't have much
> experience in Web programming, I guess an "easy" way (for a Scheme
> programmer at least) to achieve something quickly would be to generate
> static HTML from SXML and procedures that convert Cuirass data structures
> to SXML.

Hmm...interesting.  The json thing made me think we'd be interfacing
with some javascript stuff, to produce pages like

    http://hydra.gnu.org/queue
    http://hydra.gnu.org/status
    http://hydra.gnu.org/jobset/gnu/master#tabs-jobs

i.e., browsable status reports for `normal people'.

>>> WHAT NEEDS TO be done is to provide more JSON ressources (inspired by
>>> Hydra API) by translating request to SQL queries.  A command line
>>> interface would be a nice addition too.
>>
>> If this is inspired by Hydra does it mean you plan to somehow use (parts
>> of?) the Hydra web engine to present views using this json?
>
> The idea is to reuse Emacs Hydra interface in Guix if possible.  :)

Thanks...I need to look into that :-)

Greetings,
Jan
Mathieu Lirzin - Sept. 23, 2016, 10:36 p.m.
Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Mathieu Lirzin writes:
>
>>> Yes, I think I found this, I can see json in my browser window...but
>>> that's not really a web view yet (no criticism, I'm just wondering...)
>>
>> Sorry I misread what you meant by web view.  I don't have much
>> experience in Web programming, I guess an "easy" way (for a Scheme
>> programmer at least) to achieve something quickly would be to generate
>> static HTML from SXML and procedures that convert Cuirass data structures
>> to SXML.
>
> Hmm...interesting.  The json thing made me think we'd be interfacing
> with some javascript stuff, to produce pages like
>
>     http://hydra.gnu.org/queue
>     http://hydra.gnu.org/status
>     http://hydra.gnu.org/jobset/gnu/master#tabs-jobs
>
> i.e., browsable status reports for `normal people'.

That could be done this way which would certainly be more useful.  It is
just a matter of knowing how to do the javascript stuff.  :)
David Craven - Sept. 23, 2016, 10:43 p.m.
I think the web interface and the json API are two different "projects".

> just a matter of knowing how to do the javascript stuff.  :)

Many people think that JS is a toy language, JS the good parts is a weekend
read (like 100p or something) that might change your perspective and covers
everything, you already know functional programming.
https://drive.google.com/open?id=0B-QBlsZR8DS4ZUJLcnkzdkxZVkU
Mathieu Lirzin - Sept. 23, 2016, 10:59 p.m.
David Craven <david@craven.ch> writes:

> I think the web interface and the json API are two different
> "projects".

Agreed.

>> just a matter of knowing how to do the javascript stuff. :)
>
> Many people think that JS is a toy language, JS the good parts is a
> weekend read (like 100p or something) that might change your
> perspective and covers everything, you already know functional
> programming.
> https://drive.google.com/open?id=0B-QBlsZR8DS4ZUJLcnkzdkxZVkU

Sounds interesting.  I will try to find some time to read this book.

Thanks,
Jan Nieuwenhuizen - Sept. 24, 2016, 5:42 a.m.
Mathieu Lirzin writes:

> David Craven <david@craven.ch> writes:
>
>> I think the web interface and the json API are two different
>> "projects".
>
> Agreed.

Oh!  Then why choose json (poor-man's-sexps?) over sexps?  I'm mostly
just using sexps with read and write, and pipe through json translators
when crossing the border to the javascript realm.

>>> just a matter of knowing how to do the javascript stuff. :)
>>
>> Many people think that JS is a toy language, JS the good parts is a
>> weekend read (like 100p or something) that might change your
>> perspective and covers everything, you already know functional
>> programming.
>> https://drive.google.com/open?id=0B-QBlsZR8DS4ZUJLcnkzdkxZVkU
>
> Sounds interesting.  I will try to find some time to read this book.

Here's a spoiler

    JavaScript's functions are first class objects with (mostly) lexical
    scoping.  JavaScript is the first lambda language to go
    mainstream. Deep down, JavaScript has more in common with Lisp and
    Scheme than with Java.  It is Lisp in C's clothing.  This makes
    JavaScript a remarkably powerful language.

Jan
Mathieu Lirzin - Sept. 28, 2016, 11:59 a.m.
Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Mathieu Lirzin writes:
>
>> David Craven <david@craven.ch> writes:
>>
>>> I think the web interface and the json API are two different
>>> "projects".
>>
>> Agreed.
>
> Oh!  Then why choose json (poor-man's-sexps?) over sexps?  I'm mostly
> just using sexps with read and write, and pipe through json translators
> when crossing the border to the javascript realm.

AIUI JSON usage is not limited to JavaScript since almost every
programming language has a parser for it.  IMO, this is its main
advantage which overcomes its technical limitations.

However if using both S-EXP and JSON is possible without adding
complexity we could provide them both throught HTTP.

Patch

From 217c97022dcaad6e22b75bba2592ee6a449d4f25 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Fri, 16 Sep 2016 09:25:55 +0200
Subject: [PATCH 4/4] cuirass: handle build failure.

* src/cuirass/base.scm (build-packages): Catch build failures, write error log
and update database.
---
 src/cuirass/base.scm | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/src/cuirass/base.scm b/src/cuirass/base.scm
index 3d542b1..005632f 100644
--- a/src/cuirass/base.scm
+++ b/src/cuirass/base.scm
@@ -124,22 +124,34 @@  if required."
 (define (build-packages store db jobs)
   "Build JOBS and return a list of Build results."
   (map (λ (job)
-         (let ((log-port (%make-void-port "w0"))
-               (name     (assq-ref job #:job-name))
-               (drv      (assq-ref job #:derivation))
-               (eval-id  (assq-ref job #:eval-id)))
+         (let* ((name     (assq-ref job #:job-name))
+                (drv      (assq-ref job #:derivation))
+                (eval-id  (assq-ref job #:eval-id))
+                (success? #t)
+                (error-log (string-append (%package-cachedir) "/"
+                                          name ".log")))
            (simple-format #t "building ~A...\n" drv)
-           (parameterize ((current-build-output-port log-port))
-             (build-derivations store (list drv))
-             (let* ((output (derivation-path->output-path drv))
-                    (log    (log-file store output))
+           (let ((log (call-with-output-string
+                        (λ (port)
+                          (parameterize ((current-build-output-port port))
+                            (catch 'srfi-34
+                              (λ ()
+                                (build-derivations store (list drv)))
+                              (λ (key . args)
+                                (set! success? #f)
+                                (pk "kets key:" key "args:" args))))))))
+             (when (not success?)
+               (with-output-to-file error-log
+                 (lambda () (display log)))
+               (simple-format #t "build failed: ~a\n" error-log))
+             (let* ((output (and success? (derivation-path->output-path drv)))
+                    (log    (if success? (log-file store output) error-log))
                     (build  `((#:derivation . ,drv)
                               (#:eval-id . ,eval-id)
                               (#:log . ,log)
                               (#:output . ,output))))
                (db-add-build db build)
                (simple-format #t "~A\n" output)
-               (close-port log-port)
                build))))
        jobs))
 
-- 
2.9.3