Patchwork [2/2] services: Add 'cuirass-service'.

login
register
mail settings
Submitter Mathieu Lirzin
Date Oct. 26, 2016, 1:05 p.m.
Message ID <20161026130558.31924-3-mthl@gnu.org>
Download mbox | patch
Permalink /patch/16835/
State New
Headers show

Comments

Mathieu Lirzin - Oct. 26, 2016, 1:05 p.m.
* gnu/services/cuirass.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
* doc/guix.texi (Continuous integration): New node.
---
 doc/guix.texi            |  86 +++++++++++++++++++++++++++++++
 gnu/local.mk             |   1 +
 gnu/services/cuirass.scm | 128 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 215 insertions(+)
 create mode 100644 gnu/services/cuirass.scm
David Craven - Oct. 26, 2016, 1:35 p.m.
Hi Mathieu,

Do we need to export all of these?

+            cuirass-configuration-cache-directory
+            cuirass-configuration-group
+            cuirass-configuration-interval
+            cuirass-configuration-database
+            cuirass-configuration-specifications
+            cuirass-configuration-use-substitutes?
+            cuirass-configuration-one-shot?
+            %default-cuirass-configuration

Is %default-cuirass-configuration needed?

Looks good to me otherwise.

Thank you.
David
Mathieu Lirzin - Oct. 26, 2016, 2:42 p.m.
Hello David,

David Craven <david@craven.ch> writes:

> Do we need to export all of these?
>
> +            cuirass-configuration-cache-directory
> +            cuirass-configuration-group
> +            cuirass-configuration-interval
> +            cuirass-configuration-database
> +            cuirass-configuration-specifications
> +            cuirass-configuration-use-substitutes?
> +            cuirass-configuration-one-shot?
> +            %default-cuirass-configuration

Since the <cuirass-configuration> data type is documented in the manual,
the idea was to export the procedures that allow manipulating this type
in a REPL.  However since it appears that other services are not doing
that, I think it is better to remove them.

> Is %default-cuirass-configuration needed?

The benefit is a slightly more meaningful default value for #:CONFIG in
the 'cuirass-service' procedure documentation.  However I am not sure if
that helps much.  WDYT?

Thanks for the review.
David Craven - Oct. 26, 2016, 6:57 p.m.
>> Is %default-cuirass-configuration needed?
>
> The benefit is a slightly more meaningful default value for #:CONFIG in
> the 'cuirass-service' procedure documentation.  However I am not sure if
> that helps much.  WDYT?

I don't think it's necessary. An example where it's useful is
%default-syslog.conf, but not for aliasing the configuration
constructor.

David
Leo Famulari - Oct. 27, 2016, 12:22 a.m.
On Wed, Oct 26, 2016 at 03:05:58PM +0200, Mathieu Lirzin wrote:
> 
> * gnu/services/cuirass.scm: New file.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
> * doc/guix.texi (Continuous integration): New node.

> +In order to add build jobs you will have to set the
> +@code{specifications} field.  Due to current limitations of Cuirass, the
> +specifications are re-added each time the associated Shepherd service is
> +restarted.  This is indeed a bug.  Here is an example of a cuirass
> +service defining a build job based on a specification that can be found
> +in Cuirass source tree.

What does it mean for the specifications to be "re-added"? Can you
clarify this in the docs?
Ludovic Courtès - Oct. 27, 2016, 1:36 p.m.
Salut !

Mathieu Lirzin <mthl@gnu.org> skribis:

> * gnu/services/cuirass.scm: New file.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
> * doc/guix.texi (Continuous integration): New node.

[...]

> +++ b/doc/guix.texi
> @@ -7687,6 +7687,7 @@ declaration.
>  * Mail Services::               IMAP, POP3, SMTP, and all that.
>  * Web Services::                Web servers.
>  * Network File System::         NFS related services.
> +* Continuous integration::      The cuirass service.
                ^                      ^
Capitalize please.  :-)

> +@deftp {Data Type} cuirass-configuration
> +Data type representing the configuration of Cuirass.
> +
> +@table @asis
> +@item @code{cache-directory} (default: "")
                                          ^^
Could you enclose it in @code?  Same for the other default values.

> +@defvar %default-cuirass-configuration

So far we’ve used “@defvr {Scheme Variable} foo”, like Guile does.  It
may be debatable, but I think we should be consistent.

I agree with Leo’s suggestion about the doc, but apart from that, it
looks all good.  Nice work!

Thank you!  I can’t wait to see it in action on the new machine.  :-)

Ludo’.
Andreas Enge - Nov. 6, 2016, 1:16 p.m.
On Wed, Oct 26, 2016 at 08:22:36PM -0400, Leo Famulari wrote:
> What does it mean for the specifications to be "re-added"? Can you
> clarify this in the docs?

I think this is a bug :-)

The specifications are stored in a database, and they may appear multiple
times, leading to duplicate evaluations. I think the database should be
set up in a way that these duplicate entries do not occur. Then one also
would not need to distinguish between the first time the service is called
and later times.

Andreas
Mathieu Lirzin - Nov. 29, 2016, 10:22 p.m.
Pushed in commit a7cf4eb6d99838606d8ecfa776f7e4920dfbb7f5 with bug fixed
and requested changes.

Thanks.
ng0 - Nov. 30, 2016, 9:53 p.m.
Mathieu Lirzin <mthl@gnu.org> writes:

> Pushed in commit a7cf4eb6d99838606d8ecfa776f7e4920dfbb7f5 with bug fixed
> and requested changes.
>
> Thanks.

To begin to test the service, I tried this with this[0] system
config today and failed. I tried some variations, moving around
the let, checking for typos, let* instead of let, etc, but I
wasn't succesful.
What does this mean? I am able to configure the system with this,
but finally (cuirass-service) is silently ignored and never makes
it into the new system.

Is there anything obvious I have to change?

[0]:  https://ptpb.pw/lIwS.scm
Carlo Zancanaro - Nov. 30, 2016, 11:14 p.m.
On Wed, Nov 30 2016, ng0 wrote
> Is there anything obvious I have to change?

Your `let` body contains both the `(cuirass-service ...)` form as 
well as the `%desktop-services` form. A `let` form will take the 
value of the last form in its body, so the `(cuirass-service ...)` 
value is being thrown away.

Try removing one of the closing parentheses after `%desktop-services`
and add it after the `(cuirass-service ...)` form. (This has the effect
of moving `%desktop-services` out of the `let` body.)
ng0 - Dec. 1, 2016, 12:41 p.m.
Sorry, I should really set up my client to add CC or even To with lists...

      Carlo Zancanaro <carlo@zancanaro.id.au> writes:

      > On Wed, Nov 30 2016, ng0 wrote
      >> Is there anything obvious I have to change?
      [ 6 more citation lines. Click/Enter to show. ]
      >
      > Your `let` body contains both the `(cuirass-service ...)` form as 
      > well as the `%desktop-services` form. A `let` form will take the 
      > value of the last form in its body, so the `(cuirass-service ...)` 
      > value is being thrown away.
      >
      > Try removing one of the closing parentheses after `%desktop-services`
      > and add it after the `(cuirass-service ...)` form. (This has the effect
      > of moving `%desktop-services` out of the `let` body.)

      Thanks!

      https://ptpb.pw/gQOZ.scm with this change it fails differently:

      ng0@greendragon ~$ sudo guix system build /etc/config.scm
      Backtrace:
      In ice-9/boot-9.scm:
       160: 18 [catch #t #<catch-closure a4de40> ...]
      In unknown file:
         ?: 17 [apply-smob/1 #<catch-closure a4de40>]
      In ice-9/boot-9.scm:
        66: 16 [call-with-prompt prompt0 ...]
      In ice-9/eval.scm:
       432: 15 [eval # #]
      In ice-9/boot-9.scm:
      2404: 14 [save-module-excursion #<procedure a6d900 at ice-9/boot-9.scm:4051:3 ()>]
      4056: 13 [#<procedure a6d900 at ice-9/boot-9.scm:4051:3 ()>]
      1727: 12 [%start-stack load-stack #<procedure a77120 at ice-9/boot-9.scm:4047:10 ()>]
      1732: 11 [#<procedure a81b70 ()>]
      In unknown file:
         ?: 10 [primitive-load "/gnu/store/vk6q4xahpy1dvs5ig3gg699fgszbf8ay-guix-0.11.0-4.1f41/bin/.guix-real"]
      In guix/ui.scm:
      1222: 9 [run-guix-command system "build" "/etc/config.scm"]
      In ice-9/boot-9.scm:
       160: 8 [catch srfi-34 #<procedure 378a7e0 at guix/ui.scm:426:2 ()> ...]
       160: 7 [catch system-error ...]
      In guix/scripts/system.scm:
       954: 6 [#<procedure 376b120 at guix/scripts/system.scm:946:2 ()>]
       841: 5 [process-action build ("/etc/config.scm") ...]
      In guix/store.scm:
      1215: 4 [run-with-store # ...]
      In guix/scripts/system.scm:
       853: 3 [#<procedure 10053c0 at guix/scripts/system.scm:845:8 (state)> #]
       611: 2 [perform-action build # # ...]
      In gnu/system.scm:
       635: 1 [operating-system-derivation # # #f]
      In unknown file:
         ?: 0 [append (# # # # ...) (# # # # ...)]

      ERROR: In procedure append:
      ERROR: In procedure append: Wrong type argument in position 1 (expecting empty list): #<<service> type: #<service-type cuirass 3686210>
      parameters: #<<cuirass-configuration> cache-directory: "" user: "cuirass" group: "cuirass" interval: 60 database: "/var/run/cuirass/cuirass.db"
      specifications: (((#:name . "guix") (#:url . "git://git.savannah.gnu.org/guix.git") (#:load-path . ".") (#:file .
      "/.../cuirass/tests/gnu-system.scm") (#:proc . hydra-jobs) (#:arguments (subset . "hello")) (#:branch . "master"))) use-substitutes?: #f
      one-shot?: #f>>
Carlo Zancanaro - Dec. 1, 2016, 12:59 p.m.
On Thu, Dec 01 2016, ng0 wrote
> https://ptpb.pw/gQOZ.scm with this change it fails differently:

This is wrong again, but for a different reason. The 
`%desktop-services` must be the last argument to the `cons*` call.

You want it to look like this (assuming my formatting comes out at 
least a little bit okay):

  (services (cons* (gnome-desktop-service)
                   (tor-service)
                   (console-keymap-service "de")
		   (lsh-service)
		   (let ((spec-guix `((#:name . "guix")
				      (#:url . "git://git.savannah.gnu.org/guix.git")
				      (#:load-path . ".")
				      ;; Adapt to a valid absolute file name.
				      (#:file . "/.../cuirass/tests/gnu-system.scm")
				      (#:proc . hydra-jobs)
				      (#:arguments (subset . "hello"))
				      (#:branch . "master"))))

		     (cuirass-service #:config (cuirass-configuration
						(specifications (list spec-guix)))))
		   %desktop-services))
ng0 - Dec. 1, 2016, 1:23 p.m.
Carlo Zancanaro <carlo@zancanaro.id.au> writes:

> On Thu, Dec 01 2016, ng0 wrote
>> https://ptpb.pw/gQOZ.scm with this change it fails differently:
>
> This is wrong again, but for a different reason. The 
> `%desktop-services` must be the last argument to the `cons*` call.

I tried that before (though I did just think it was wrong) and
now I get the same message as yesterday:

ng0@greendragon ~$ sudo guix system build /etc/config.scm
Backtrace:
In ice-9/boot-9.scm:
1727: 19 [%start-stack load-stack ...]
1732: 18 [#<procedure 1ef3b70 ()>]
In unknown file:
   ?: 17 [primitive-load "/gnu/store/vk6q4xahpy1dvs5ig3gg699fgszbf8ay-guix-0.11.0-4.1f41/bin/.guix-real"]
In guix/ui.scm:
1222: 16 [run-guix-command system "build" "/etc/config.scm"]
In ice-9/boot-9.scm:
 160: 15 [catch srfi-34 #<procedure 4bfa800 at guix/ui.scm:426:2 ()> ...]
 160: 14 [catch system-error ...]
In guix/scripts/system.scm:
 954: 13 [#<procedure 4bea450 at guix/scripts/system.scm:946:2 ()>]
 841: 12 [process-action build ("/etc/config.scm") ...]
In guix/store.scm:
1215: 11 [run-with-store # ...]
In guix/scripts/system.scm:
 853: 10 [#<procedure 4d95cc0 at guix/scripts/system.scm:845:8 (state)> #]
 611: 9 [perform-action build # # ...]
In gnu/system.scm:
 635: 8 [operating-system-derivation # # #f]
In gnu/services.scm:
 585: 7 [loop #]
In srfi/srfi-1.scm:
 575: 6 [map #<procedure loop (sink)> (# # #)]
In gnu/services.scm:
 585: 5 [loop #<<service> type: # parameters: #>]
In srfi/srfi-1.scm:
 575: 4 [map #<procedure loop (sink)> (# # # # ...)]
In gnu/services.scm:
 585: 3 [loop #<<service> type: # parameters: ()>]
In srfi/srfi-1.scm:
 573: 2 [map #<procedure 5ef2c20 at gnu/services.scm:574:4 (service)> (# # # # ...)]
In gnu/services/cuirass.scm:
  81: 1 [cuirass-shepherd-service #]
In unknown file:
   ?: 0 [string=? "" ((# # # # ...))]

ERROR: In procedure string=?:
ERROR: In procedure string=: Wrong type argument in position 2 (expecting string): (((#:name . "guix") (#:url . "git://git.savannah.gnu.org/guix.git") (#:load-path . ".") (#:file . "/.../cuirass/tests/gnu-system.scm") (#:proc . hydra-jobs) (#:arguments (subset . "hello")) (#:branch . "master")))
Carlo Zancanaro - Dec. 1, 2016, 3:13 p.m.
On Thu, Dec 01 2016, ng0 wrote 
> I tried that before (though I did just think it was wrong) and 
> now I get the same message as yesterday:
>
> ...
> 
> In gnu/services/cuirass.scm: 
>   81: 1 [cuirass-shepherd-service #] 
> In unknown file: 
>    ?: 0 [string=? "" ((# # # # ...))]
>
> ERROR: In procedure string=?: ERROR: In procedure string=: Wrong 
> type argument in position 2 (expecting string): (((#:name . 
> "guix") (#:url . "git://git.savannah.gnu.org/guix.git") 
> (#:load-path . ".") (#:file . 
> "/.../cuirass/tests/gnu-system.scm") (#:proc . hydra-jobs) 
> (#:arguments (subset . "hello")) (#:branch . "master")))

It looks like cuirass is expecting the specifications to be a 
string, not a list. I don't know anything about cuirass, so I 
can't help you to configure the cuirass-service.

Looking at the source, specifications has a comment ";string
(file-name)". It looks like the "specifications" field of the
configuration is just equivalent to the --specifications argument to the
cuirass command, if that helps at all.

Patch

diff --git a/doc/guix.texi b/doc/guix.texi
index 86b82c8..f9cb9e4 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -7687,6 +7687,7 @@  declaration.
 * Mail Services::               IMAP, POP3, SMTP, and all that.
 * Web Services::                Web servers.
 * Network File System::         NFS related services.
+* Continuous integration::      The cuirass service.
 * Miscellaneous Services::      Other services.
 @end menu
 
@@ -10524,6 +10525,91 @@  If it is @code{#f} then the daemon will use the host's fully qualified domain na
 @end table
 @end deftp
 
+@node Continuous integration
+@subsubsection Continuous integration
+
+@cindex continuous integration
+@uref{https://notabug.org/mthl/cuirass, Cuirass} is a continuous
+integration tool for Guix.  It can be used both for development and
+for providing substitutes to others (@pxref{Substitutes}).
+
+The @code{(gnu services cuirass)} module provides the following service.
+
+@deffn {Scheme Procedure} cuirass-service @
+       [#:config @code{%default-cuirass-configuration}]
+Return a service that runs @command{cuirass}.
+
+The @var{#:config} keyword argument specifies the configuration for
+@command{cuirass}, which must be a @code{<cuirass-configuration>}
+object, by default it doesn't provide any build job.  If you want to
+provide your own configuration you will most likely use the
+@code{cuirass-configuration} special form which returns such objects.
+@end deffn
+
+In order to add build jobs you will have to set the
+@code{specifications} field.  Due to current limitations of Cuirass, the
+specifications are re-added each time the associated Shepherd service is
+restarted.  This is indeed a bug.  Here is an example of a cuirass
+service defining a build job based on a specification that can be found
+in Cuirass source tree.
+
+@example
+(let ((spec `((#:name . "guix")
+              (#:url . "git://git.savannah.gnu.org/guix.git")
+              (#:load-path . ".")
+              ;; Adapt to a valid absolute file name.
+              (#:file . "/.../cuirass/tests/gnu-system.scm")
+              (#:proc . hydra-jobs)
+              (#:arguments (subset . "hello"))
+              (#:branch . "master"))))
+  (cuirass-service #:config (cuirass-configuration
+                             (specifications (list spec)))))
+@end example
+
+While information related to build jobs are located directly in the
+specifications, global parameters for the @command{cuirass} process are
+accessible in other @code{cuirass-configuration} fields.
+
+@deftp {Data Type} cuirass-configuration
+Data type representing the configuration of Cuirass.
+
+@table @asis
+@item @code{cache-directory} (default: "")
+Location of the repository cache.
+
+@item @code{user} (default: "cuirass")
+Owner of the @code{cuirass} process.
+
+@item @code{group} (default: "cuirass")
+Owner's group of the @code{cuirass} process.
+
+@item @code{interval} (default: 60)
+Number of seconds between the poll of the repositories followed by the
+Cuirass jobs.
+
+@item @code{database} (default: "/var/run/cuirass/cuirass.db")
+Location of sqlite database which contains the build results and previously
+added specifications.
+
+@item @code{specifications} (default: @code{'()})
+A list of specifications, where a specification is an association list
+(@pxref{Associations Lists,,, guile, GNU Guile Reference Manual}) whose
+keys are keywords (@code{#:keyword-example}) as shown in the example
+above.
+
+@item @code{use-substitutes?} (default: @code{#f})
+This allows using substitutes to avoid building every dependencies of a job
+from source.
+
+@item @code{one-shot?} (default: @code{#f})
+Only evaluate specifications and build derivations once.
+@end table
+@end deftp
+
+@defvar %default-cuirass-configuration
+This global variable contains a @code{cuirass-configuration} where all
+the fields contain their default value.
+@end defvar
 
 @node Miscellaneous Services
 @subsubsection Miscellaneous Services
diff --git a/gnu/local.mk b/gnu/local.mk
index c6cd586..e0ec339 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -392,6 +392,7 @@  GNU_SYSTEM_MODULES =				\
   %D%/services/admin.scm			\
   %D%/services/avahi.scm			\
   %D%/services/base.scm				\
+  %D%/services/cuirass.scm			\
   %D%/services/databases.scm			\
   %D%/services/dbus.scm				\
   %D%/services/desktop.scm			\
diff --git a/gnu/services/cuirass.scm b/gnu/services/cuirass.scm
new file mode 100644
index 0000000..93e2805
--- /dev/null
+++ b/gnu/services/cuirass.scm
@@ -0,0 +1,128 @@ 
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software: you can redistribute it and/or modify
+;;; it under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation, either version 3 of the License, or
+;;; (at your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful,
+;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services cuirass)
+  #:use-module (guix gexp)
+  #:use-module (guix records)
+  #:use-module (gnu packages admin)
+  #:autoload   (gnu packages ci) (cuirass)
+  #:use-module (gnu services)
+  #:use-module (gnu services base)
+  #:use-module (gnu services shepherd)
+  #:use-module (gnu system shadow)
+  #:export (<cuirass-configuration>
+            cuirass-configuration
+            cuirass-configuration?
+            cuirass-configuration-cache-directory
+            cuirass-configuration-group
+            cuirass-configuration-interval
+            cuirass-configuration-database
+            cuirass-configuration-specifications
+            cuirass-configuration-use-substitutes?
+            cuirass-configuration-one-shot?
+            %default-cuirass-configuration
+
+            cuirass-service-type
+            cuirass-service))
+
+;;;; Commentary:
+;;;
+;;; This module implements a service that to run instances of Cuirass, a
+;;; continuous integration tool.
+;;;
+;;;; Code:
+
+(define-record-type* <cuirass-configuration>
+  cuirass-configuration make-cuirass-configuration
+  cuirass-configuration?
+  (cache-directory  cuirass-configuration-cache-directory ;string (dir-name)
+                    (default ""))
+  (user             cuirass-configuration-user ;string
+                    (default "cuirass"))
+  (group            cuirass-configuration-group ;string
+                    (default "cuirass"))
+  (interval         cuirass-configuration-interval ;integer (seconds)
+                    (default 60))
+  (database         cuirass-configuration-database ;string (file-name)
+                    (default "/var/run/cuirass/cuirass.db"))
+  (specifications   cuirass-configuration-specifications ;string (file-name)
+                    (default ""))
+  (use-substitutes? cuirass-configuration-use-substitutes? ;boolean
+                    (default #f))
+  (one-shot?        cuirass-configuration-one-shot? ;boolean
+                    (default #f)))
+
+(define %default-cuirass-configuration
+  (cuirass-configuration))
+
+(define (cuirass-shepherd-service config)
+  "Return a <shepherd-service> for the Cuirass service with CONFIG."
+  (and
+   (cuirass-configuration? config)
+   (let ((cache-directory  (cuirass-configuration-cache-directory config))
+         (interval         (cuirass-configuration-interval config))
+         (database         (cuirass-configuration-database config))
+         (specifications   (cuirass-configuration-specifications config))
+         (use-substitutes? (cuirass-configuration-use-substitutes? config))
+         (one-shot?        (cuirass-configuration-one-shot? config)))
+     (list (shepherd-service
+            (documentation "Run Cuirass.")
+            (provision '(cuirass))
+            (requirement '(guix-daemon))
+            (start #~(make-forkexec-constructor
+                      (list (string-append #$cuirass "/bin/cuirass")
+                            #$@(if (string=? "" cache-directory)
+                                   '()
+                                   (list "--cache-directory" cache-directory))
+                            ;; XXX: the specifications are re-added each time
+                            ;; this service restarts
+                            #$@(if (string=? "" specifications)
+                                   '()
+                                   (list "--specifications" specifications))
+                            "--database" #$database
+                            "--interval" #$(number->string interval)
+                            #$@(if use-substitutes? '("--use-substitutes") '())
+                            #$@(if one-shot? "--one-shot" '()))))
+            (stop #~(make-kill-destructor)))))))
+
+(define (cuirass-account config)
+  "Return the user accounts and user groups for CONFIG."
+  (let ((cuirass-user  (cuirass-configuration-user config))
+        (cuirass-group (cuirass-configuration-group config)))
+    (list (user-group
+           (name cuirass-group)
+           (system? #t))
+          (user-account
+           (name cuirass-user)
+           (group cuirass-group)
+           (system? #t)
+           (comment "Cuirass privilege separation user")
+           (home-directory (string-append "/var/run/" cuirass-user))
+           (shell #~(string-append #$shadow "/sbin/nologin"))))))
+
+(define cuirass-service-type
+  (service-type
+   (name 'cuirass)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type cuirass-shepherd-service)
+     (service-extension account-service-type cuirass-account)))))
+
+(define* (cuirass-service #:key (config %default-cuirass-configuration))
+  "Return a service that runs cuirass according to CONFIG."
+  (service cuirass-service-type config))