diff mbox

[2/2] services: Add 'cuirass-service'.

Message ID 20161026130558.31924-3-mthl@gnu.org
State New
Headers show

Commit Message

Mathieu Lirzin Oct. 26, 2016, 1:05 p.m. UTC
* gnu/services/cuirass.scm: New file.
* gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
* doc/guix.texi (Continuous integration): New node.
---
 doc/guix.texi            |  86 +++++++++++++++++++++++++++++++
 gnu/local.mk             |   1 +
 gnu/services/cuirass.scm | 128 +++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 215 insertions(+)
 create mode 100644 gnu/services/cuirass.scm

Comments

David Craven Oct. 26, 2016, 1:35 p.m. UTC | #1
Hi Mathieu,

Do we need to export all of these?

+            cuirass-configuration-cache-directory
+            cuirass-configuration-group
+            cuirass-configuration-interval
+            cuirass-configuration-database
+            cuirass-configuration-specifications
+            cuirass-configuration-use-substitutes?
+            cuirass-configuration-one-shot?
+            %default-cuirass-configuration

Is %default-cuirass-configuration needed?

Looks good to me otherwise.

Thank you.
David
Mathieu Lirzin Oct. 26, 2016, 2:42 p.m. UTC | #2
Hello David,

David Craven <david@craven.ch> writes:

> Do we need to export all of these?
>
> +            cuirass-configuration-cache-directory
> +            cuirass-configuration-group
> +            cuirass-configuration-interval
> +            cuirass-configuration-database
> +            cuirass-configuration-specifications
> +            cuirass-configuration-use-substitutes?
> +            cuirass-configuration-one-shot?
> +            %default-cuirass-configuration

Since the <cuirass-configuration> data type is documented in the manual,
the idea was to export the procedures that allow manipulating this type
in a REPL.  However since it appears that other services are not doing
that, I think it is better to remove them.

> Is %default-cuirass-configuration needed?

The benefit is a slightly more meaningful default value for #:CONFIG in
the 'cuirass-service' procedure documentation.  However I am not sure if
that helps much.  WDYT?

Thanks for the review.
David Craven Oct. 26, 2016, 6:57 p.m. UTC | #3
>> Is %default-cuirass-configuration needed?
>
> The benefit is a slightly more meaningful default value for #:CONFIG in
> the 'cuirass-service' procedure documentation.  However I am not sure if
> that helps much.  WDYT?

I don't think it's necessary. An example where it's useful is
%default-syslog.conf, but not for aliasing the configuration
constructor.

David
Leo Famulari Oct. 27, 2016, 12:22 a.m. UTC | #4
On Wed, Oct 26, 2016 at 03:05:58PM +0200, Mathieu Lirzin wrote:
> 
> * gnu/services/cuirass.scm: New file.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
> * doc/guix.texi (Continuous integration): New node.

> +In order to add build jobs you will have to set the
> +@code{specifications} field.  Due to current limitations of Cuirass, the
> +specifications are re-added each time the associated Shepherd service is
> +restarted.  This is indeed a bug.  Here is an example of a cuirass
> +service defining a build job based on a specification that can be found
> +in Cuirass source tree.

What does it mean for the specifications to be "re-added"? Can you
clarify this in the docs?
Ludovic Courtès Oct. 27, 2016, 1:36 p.m. UTC | #5
Salut !

Mathieu Lirzin <mthl@gnu.org> skribis:

> * gnu/services/cuirass.scm: New file.
> * gnu/local.mk (GNU_SYSTEM_MODULES): Add it.
> * doc/guix.texi (Continuous integration): New node.

[...]

> +++ b/doc/guix.texi
> @@ -7687,6 +7687,7 @@ declaration.
>  * Mail Services::               IMAP, POP3, SMTP, and all that.
>  * Web Services::                Web servers.
>  * Network File System::         NFS related services.
> +* Continuous integration::      The cuirass service.
                ^                      ^
Capitalize please.  :-)

> +@deftp {Data Type} cuirass-configuration
> +Data type representing the configuration of Cuirass.
> +
> +@table @asis
> +@item @code{cache-directory} (default: "")
                                          ^^
Could you enclose it in @code?  Same for the other default values.

> +@defvar %default-cuirass-configuration

So far we’ve used “@defvr {Scheme Variable} foo”, like Guile does.  It
may be debatable, but I think we should be consistent.

I agree with Leo’s suggestion about the doc, but apart from that, it
looks all good.  Nice work!

Thank you!  I can’t wait to see it in action on the new machine.  :-)

Ludo’.
Andreas Enge Nov. 6, 2016, 1:16 p.m. UTC | #6
On Wed, Oct 26, 2016 at 08:22:36PM -0400, Leo Famulari wrote:
> What does it mean for the specifications to be "re-added"? Can you
> clarify this in the docs?

I think this is a bug :-)

The specifications are stored in a database, and they may appear multiple
times, leading to duplicate evaluations. I think the database should be
set up in a way that these duplicate entries do not occur. Then one also
would not need to distinguish between the first time the service is called
and later times.

Andreas
Mathieu Lirzin Nov. 29, 2016, 10:22 p.m. UTC | #7
Pushed in commit a7cf4eb6d99838606d8ecfa776f7e4920dfbb7f5 with bug fixed
and requested changes.

Thanks.
ng0 Nov. 30, 2016, 9:53 p.m. UTC | #8
Mathieu Lirzin <mthl@gnu.org> writes:

> Pushed in commit a7cf4eb6d99838606d8ecfa776f7e4920dfbb7f5 with bug fixed
> and requested changes.
>
> Thanks.

To begin to test the service, I tried this with this[0] system
config today and failed. I tried some variations, moving around
the let, checking for typos, let* instead of let, etc, but I
wasn't succesful.
What does this mean? I am able to configure the system with this,
but finally (cuirass-service) is silently ignored and never makes
it into the new system.

Is there anything obvious I have to change?

[0]:  https://ptpb.pw/lIwS.scm
Carlo Zancanaro Nov. 30, 2016, 11:14 p.m. UTC | #9
On Wed, Nov 30 2016, ng0 wrote
> Is there anything obvious I have to change?

Your `let` body contains both the `(cuirass-service ...)` form as 
well as the `%desktop-services` form. A `let` form will take the 
value of the last form in its body, so the `(cuirass-service ...)` 
value is being thrown away.

Try removing one of the closing parentheses after `%desktop-services`
and add it after the `(cuirass-service ...)` form. (This has the effect
of moving `%desktop-services` out of the `let` body.)
ng0 Dec. 1, 2016, 12:41 p.m. UTC | #10
Sorry, I should really set up my client to add CC or even To with lists...

      Carlo Zancanaro <carlo@zancanaro.id.au> writes:

      > On Wed, Nov 30 2016, ng0 wrote
      >> Is there anything obvious I have to change?
      [ 6 more citation lines. Click/Enter to show. ]
      >
      > Your `let` body contains both the `(cuirass-service ...)` form as 
      > well as the `%desktop-services` form. A `let` form will take the 
      > value of the last form in its body, so the `(cuirass-service ...)` 
      > value is being thrown away.
      >
      > Try removing one of the closing parentheses after `%desktop-services`
      > and add it after the `(cuirass-service ...)` form. (This has the effect
      > of moving `%desktop-services` out of the `let` body.)

      Thanks!

      https://ptpb.pw/gQOZ.scm with this change it fails differently:

      ng0@greendragon ~$ sudo guix system build /etc/config.scm
      Backtrace:
      In ice-9/boot-9.scm:
       160: 18 [catch #t #<catch-closure a4de40> ...]
      In unknown file:
         ?: 17 [apply-smob/1 #<catch-closure a4de40>]
      In ice-9/boot-9.scm:
        66: 16 [call-with-prompt prompt0 ...]
      In ice-9/eval.scm:
       432: 15 [eval # #]
      In ice-9/boot-9.scm:
      2404: 14 [save-module-excursion #<procedure a6d900 at ice-9/boot-9.scm:4051:3 ()>]
      4056: 13 [#<procedure a6d900 at ice-9/boot-9.scm:4051:3 ()>]
      1727: 12 [%start-stack load-stack #<procedure a77120 at ice-9/boot-9.scm:4047:10 ()>]
      1732: 11 [#<procedure a81b70 ()>]
      In unknown file:
         ?: 10 [primitive-load "/gnu/store/vk6q4xahpy1dvs5ig3gg699fgszbf8ay-guix-0.11.0-4.1f41/bin/.guix-real"]
      In guix/ui.scm:
      1222: 9 [run-guix-command system "build" "/etc/config.scm"]
      In ice-9/boot-9.scm:
       160: 8 [catch srfi-34 #<procedure 378a7e0 at guix/ui.scm:426:2 ()> ...]
       160: 7 [catch system-error ...]
      In guix/scripts/system.scm:
       954: 6 [#<procedure 376b120 at guix/scripts/system.scm:946:2 ()>]
       841: 5 [process-action build ("/etc/config.scm") ...]
      In guix/store.scm:
      1215: 4 [run-with-store # ...]
      In guix/scripts/system.scm:
       853: 3 [#<procedure 10053c0 at guix/scripts/system.scm:845:8 (state)> #]
       611: 2 [perform-action build # # ...]
      In gnu/system.scm:
       635: 1 [operating-system-derivation # # #f]
      In unknown file:
         ?: 0 [append (# # # # ...) (# # # # ...)]

      ERROR: In procedure append:
      ERROR: In procedure append: Wrong type argument in position 1 (expecting empty list): #<<service> type: #<service-type cuirass 3686210>
      parameters: #<<cuirass-configuration> cache-directory: "" user: "cuirass" group: "cuirass" interval: 60 database: "/var/run/cuirass/cuirass.db"
      specifications: (((#:name . "guix") (#:url . "git://git.savannah.gnu.org/guix.git") (#:load-path . ".") (#:file .
      "/.../cuirass/tests/gnu-system.scm") (#:proc . hydra-jobs) (#:arguments (subset . "hello")) (#:branch . "master"))) use-substitutes?: #f
      one-shot?: #f>>
Carlo Zancanaro Dec. 1, 2016, 12:59 p.m. UTC | #11
On Thu, Dec 01 2016, ng0 wrote
> https://ptpb.pw/gQOZ.scm with this change it fails differently:

This is wrong again, but for a different reason. The 
`%desktop-services` must be the last argument to the `cons*` call.

You want it to look like this (assuming my formatting comes out at 
least a little bit okay):

  (services (cons* (gnome-desktop-service)
                   (tor-service)
                   (console-keymap-service "de")
		   (lsh-service)
		   (let ((spec-guix `((#:name . "guix")
				      (#:url . "git://git.savannah.gnu.org/guix.git")
				      (#:load-path . ".")
				      ;; Adapt to a valid absolute file name.
				      (#:file . "/.../cuirass/tests/gnu-system.scm")
				      (#:proc . hydra-jobs)
				      (#:arguments (subset . "hello"))
				      (#:branch . "master"))))

		     (cuirass-service #:config (cuirass-configuration
						(specifications (list spec-guix)))))
		   %desktop-services))
ng0 Dec. 1, 2016, 1:23 p.m. UTC | #12
Carlo Zancanaro <carlo@zancanaro.id.au> writes:

> On Thu, Dec 01 2016, ng0 wrote
>> https://ptpb.pw/gQOZ.scm with this change it fails differently:
>
> This is wrong again, but for a different reason. The 
> `%desktop-services` must be the last argument to the `cons*` call.

I tried that before (though I did just think it was wrong) and
now I get the same message as yesterday:

ng0@greendragon ~$ sudo guix system build /etc/config.scm
Backtrace:
In ice-9/boot-9.scm:
1727: 19 [%start-stack load-stack ...]
1732: 18 [#<procedure 1ef3b70 ()>]
In unknown file:
   ?: 17 [primitive-load "/gnu/store/vk6q4xahpy1dvs5ig3gg699fgszbf8ay-guix-0.11.0-4.1f41/bin/.guix-real"]
In guix/ui.scm:
1222: 16 [run-guix-command system "build" "/etc/config.scm"]
In ice-9/boot-9.scm:
 160: 15 [catch srfi-34 #<procedure 4bfa800 at guix/ui.scm:426:2 ()> ...]
 160: 14 [catch system-error ...]
In guix/scripts/system.scm:
 954: 13 [#<procedure 4bea450 at guix/scripts/system.scm:946:2 ()>]
 841: 12 [process-action build ("/etc/config.scm") ...]
In guix/store.scm:
1215: 11 [run-with-store # ...]
In guix/scripts/system.scm:
 853: 10 [#<procedure 4d95cc0 at guix/scripts/system.scm:845:8 (state)> #]
 611: 9 [perform-action build # # ...]
In gnu/system.scm:
 635: 8 [operating-system-derivation # # #f]
In gnu/services.scm:
 585: 7 [loop #]
In srfi/srfi-1.scm:
 575: 6 [map #<procedure loop (sink)> (# # #)]
In gnu/services.scm:
 585: 5 [loop #<<service> type: # parameters: #>]
In srfi/srfi-1.scm:
 575: 4 [map #<procedure loop (sink)> (# # # # ...)]
In gnu/services.scm:
 585: 3 [loop #<<service> type: # parameters: ()>]
In srfi/srfi-1.scm:
 573: 2 [map #<procedure 5ef2c20 at gnu/services.scm:574:4 (service)> (# # # # ...)]
In gnu/services/cuirass.scm:
  81: 1 [cuirass-shepherd-service #]
In unknown file:
   ?: 0 [string=? "" ((# # # # ...))]

ERROR: In procedure string=?:
ERROR: In procedure string=: Wrong type argument in position 2 (expecting string): (((#:name . "guix") (#:url . "git://git.savannah.gnu.org/guix.git") (#:load-path . ".") (#:file . "/.../cuirass/tests/gnu-system.scm") (#:proc . hydra-jobs) (#:arguments (subset . "hello")) (#:branch . "master")))
Carlo Zancanaro Dec. 1, 2016, 3:13 p.m. UTC | #13
On Thu, Dec 01 2016, ng0 wrote 
> I tried that before (though I did just think it was wrong) and 
> now I get the same message as yesterday:
>
> ...
> 
> In gnu/services/cuirass.scm: 
>   81: 1 [cuirass-shepherd-service #] 
> In unknown file: 
>    ?: 0 [string=? "" ((# # # # ...))]
>
> ERROR: In procedure string=?: ERROR: In procedure string=: Wrong 
> type argument in position 2 (expecting string): (((#:name . 
> "guix") (#:url . "git://git.savannah.gnu.org/guix.git") 
> (#:load-path . ".") (#:file . 
> "/.../cuirass/tests/gnu-system.scm") (#:proc . hydra-jobs) 
> (#:arguments (subset . "hello")) (#:branch . "master")))

It looks like cuirass is expecting the specifications to be a 
string, not a list. I don't know anything about cuirass, so I 
can't help you to configure the cuirass-service.

Looking at the source, specifications has a comment ";string
(file-name)". It looks like the "specifications" field of the
configuration is just equivalent to the --specifications argument to the
cuirass command, if that helps at all.
diff mbox

Patch

diff --git a/doc/guix.texi b/doc/guix.texi
index 86b82c8..f9cb9e4 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -7687,6 +7687,7 @@  declaration.
 * Mail Services::               IMAP, POP3, SMTP, and all that.
 * Web Services::                Web servers.
 * Network File System::         NFS related services.
+* Continuous integration::      The cuirass service.
 * Miscellaneous Services::      Other services.
 @end menu
 
@@ -10524,6 +10525,91 @@  If it is @code{#f} then the daemon will use the host's fully qualified domain na
 @end table
 @end deftp
 
+@node Continuous integration
+@subsubsection Continuous integration
+
+@cindex continuous integration
+@uref{https://notabug.org/mthl/cuirass, Cuirass} is a continuous
+integration tool for Guix.  It can be used both for development and
+for providing substitutes to others (@pxref{Substitutes}).
+
+The @code{(gnu services cuirass)} module provides the following service.
+
+@deffn {Scheme Procedure} cuirass-service @
+       [#:config @code{%default-cuirass-configuration}]
+Return a service that runs @command{cuirass}.
+
+The @var{#:config} keyword argument specifies the configuration for
+@command{cuirass}, which must be a @code{<cuirass-configuration>}
+object, by default it doesn't provide any build job.  If you want to
+provide your own configuration you will most likely use the
+@code{cuirass-configuration} special form which returns such objects.
+@end deffn
+
+In order to add build jobs you will have to set the
+@code{specifications} field.  Due to current limitations of Cuirass, the
+specifications are re-added each time the associated Shepherd service is
+restarted.  This is indeed a bug.  Here is an example of a cuirass
+service defining a build job based on a specification that can be found
+in Cuirass source tree.
+
+@example
+(let ((spec `((#:name . "guix")
+              (#:url . "git://git.savannah.gnu.org/guix.git")
+              (#:load-path . ".")
+              ;; Adapt to a valid absolute file name.
+              (#:file . "/.../cuirass/tests/gnu-system.scm")
+              (#:proc . hydra-jobs)
+              (#:arguments (subset . "hello"))
+              (#:branch . "master"))))
+  (cuirass-service #:config (cuirass-configuration
+                             (specifications (list spec)))))
+@end example
+
+While information related to build jobs are located directly in the
+specifications, global parameters for the @command{cuirass} process are
+accessible in other @code{cuirass-configuration} fields.
+
+@deftp {Data Type} cuirass-configuration
+Data type representing the configuration of Cuirass.
+
+@table @asis
+@item @code{cache-directory} (default: "")
+Location of the repository cache.
+
+@item @code{user} (default: "cuirass")
+Owner of the @code{cuirass} process.
+
+@item @code{group} (default: "cuirass")
+Owner's group of the @code{cuirass} process.
+
+@item @code{interval} (default: 60)
+Number of seconds between the poll of the repositories followed by the
+Cuirass jobs.
+
+@item @code{database} (default: "/var/run/cuirass/cuirass.db")
+Location of sqlite database which contains the build results and previously
+added specifications.
+
+@item @code{specifications} (default: @code{'()})
+A list of specifications, where a specification is an association list
+(@pxref{Associations Lists,,, guile, GNU Guile Reference Manual}) whose
+keys are keywords (@code{#:keyword-example}) as shown in the example
+above.
+
+@item @code{use-substitutes?} (default: @code{#f})
+This allows using substitutes to avoid building every dependencies of a job
+from source.
+
+@item @code{one-shot?} (default: @code{#f})
+Only evaluate specifications and build derivations once.
+@end table
+@end deftp
+
+@defvar %default-cuirass-configuration
+This global variable contains a @code{cuirass-configuration} where all
+the fields contain their default value.
+@end defvar
 
 @node Miscellaneous Services
 @subsubsection Miscellaneous Services
diff --git a/gnu/local.mk b/gnu/local.mk
index c6cd586..e0ec339 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -392,6 +392,7 @@  GNU_SYSTEM_MODULES =				\
   %D%/services/admin.scm			\
   %D%/services/avahi.scm			\
   %D%/services/base.scm				\
+  %D%/services/cuirass.scm			\
   %D%/services/databases.scm			\
   %D%/services/dbus.scm				\
   %D%/services/desktop.scm			\
diff --git a/gnu/services/cuirass.scm b/gnu/services/cuirass.scm
new file mode 100644
index 0000000..93e2805
--- /dev/null
+++ b/gnu/services/cuirass.scm
@@ -0,0 +1,128 @@ 
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2016 Mathieu Lirzin <mthl@gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software: you can redistribute it and/or modify
+;;; it under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation, either version 3 of the License, or
+;;; (at your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful,
+;;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (gnu services cuirass)
+  #:use-module (guix gexp)
+  #:use-module (guix records)
+  #:use-module (gnu packages admin)
+  #:autoload   (gnu packages ci) (cuirass)
+  #:use-module (gnu services)
+  #:use-module (gnu services base)
+  #:use-module (gnu services shepherd)
+  #:use-module (gnu system shadow)
+  #:export (<cuirass-configuration>
+            cuirass-configuration
+            cuirass-configuration?
+            cuirass-configuration-cache-directory
+            cuirass-configuration-group
+            cuirass-configuration-interval
+            cuirass-configuration-database
+            cuirass-configuration-specifications
+            cuirass-configuration-use-substitutes?
+            cuirass-configuration-one-shot?
+            %default-cuirass-configuration
+
+            cuirass-service-type
+            cuirass-service))
+
+;;;; Commentary:
+;;;
+;;; This module implements a service that to run instances of Cuirass, a
+;;; continuous integration tool.
+;;;
+;;;; Code:
+
+(define-record-type* <cuirass-configuration>
+  cuirass-configuration make-cuirass-configuration
+  cuirass-configuration?
+  (cache-directory  cuirass-configuration-cache-directory ;string (dir-name)
+                    (default ""))
+  (user             cuirass-configuration-user ;string
+                    (default "cuirass"))
+  (group            cuirass-configuration-group ;string
+                    (default "cuirass"))
+  (interval         cuirass-configuration-interval ;integer (seconds)
+                    (default 60))
+  (database         cuirass-configuration-database ;string (file-name)
+                    (default "/var/run/cuirass/cuirass.db"))
+  (specifications   cuirass-configuration-specifications ;string (file-name)
+                    (default ""))
+  (use-substitutes? cuirass-configuration-use-substitutes? ;boolean
+                    (default #f))
+  (one-shot?        cuirass-configuration-one-shot? ;boolean
+                    (default #f)))
+
+(define %default-cuirass-configuration
+  (cuirass-configuration))
+
+(define (cuirass-shepherd-service config)
+  "Return a <shepherd-service> for the Cuirass service with CONFIG."
+  (and
+   (cuirass-configuration? config)
+   (let ((cache-directory  (cuirass-configuration-cache-directory config))
+         (interval         (cuirass-configuration-interval config))
+         (database         (cuirass-configuration-database config))
+         (specifications   (cuirass-configuration-specifications config))
+         (use-substitutes? (cuirass-configuration-use-substitutes? config))
+         (one-shot?        (cuirass-configuration-one-shot? config)))
+     (list (shepherd-service
+            (documentation "Run Cuirass.")
+            (provision '(cuirass))
+            (requirement '(guix-daemon))
+            (start #~(make-forkexec-constructor
+                      (list (string-append #$cuirass "/bin/cuirass")
+                            #$@(if (string=? "" cache-directory)
+                                   '()
+                                   (list "--cache-directory" cache-directory))
+                            ;; XXX: the specifications are re-added each time
+                            ;; this service restarts
+                            #$@(if (string=? "" specifications)
+                                   '()
+                                   (list "--specifications" specifications))
+                            "--database" #$database
+                            "--interval" #$(number->string interval)
+                            #$@(if use-substitutes? '("--use-substitutes") '())
+                            #$@(if one-shot? "--one-shot" '()))))
+            (stop #~(make-kill-destructor)))))))
+
+(define (cuirass-account config)
+  "Return the user accounts and user groups for CONFIG."
+  (let ((cuirass-user  (cuirass-configuration-user config))
+        (cuirass-group (cuirass-configuration-group config)))
+    (list (user-group
+           (name cuirass-group)
+           (system? #t))
+          (user-account
+           (name cuirass-user)
+           (group cuirass-group)
+           (system? #t)
+           (comment "Cuirass privilege separation user")
+           (home-directory (string-append "/var/run/" cuirass-user))
+           (shell #~(string-append #$shadow "/sbin/nologin"))))))
+
+(define cuirass-service-type
+  (service-type
+   (name 'cuirass)
+   (extensions
+    (list
+     (service-extension shepherd-root-service-type cuirass-shepherd-service)
+     (service-extension account-service-type cuirass-account)))))
+
+(define* (cuirass-service #:key (config %default-cuirass-configuration))
+  "Return a service that runs cuirass according to CONFIG."
+  (service cuirass-service-type config))