Patchwork gnu: Add hdf4

login
register
mail settings
Submitter Ludovic Courtès
Date May 29, 2016, 10:07 p.m.
Message ID <87shx0tt5e.fsf@gnu.org>
Download mbox | patch
Permalink /patch/12618/
State New
Headers show

Comments

Ludovic Courtès - May 29, 2016, 10:07 p.m.
Hello,

Jeremy Robst <jpro@bas.ac.uk> skribis:

> Here's the patch as an attachment - sorry for the hassle. I will try
> and work out what's going wrong so I can send proper patches in
> future.

No problem!

I tried the attached patch, which includes mostly cosmetic changes
compared to the one you posted.  However, I stumble upon test failures
(on x86_64, with Linux-libre 4.6.0):

--8<---------------cut here---------------start------------->8---
----------------------------
hdftest  Test Log
----------------------------
Testing create/read/write compressed datasets (tcomp.c)                PASSED
Testing create/read/write chunked datasets (tchunk.c)                  PASSED
Testing reading of netCDF file using the SDxxx inteface (tnetcdf.c)    PASSED
Testing functionality of dimensions (tdim.c)                           PASSED
Testing functions involving empty data sets (temptySDSs.c)             PASSED
Testing various setting attribute features (tattributes.c)             PASSED
Testing getting data size of special data (tdatasizes.c)               PASSED
Testing getting location info of data (tdatainfo.c)                    PASSED
Testing getting location info of attr and annot data (tattdatainfo.c)  PASSED
Testing a mix of SD, V, and VS functions (tmixed_apis.c)               PASSED
Testing miscellaneous file related functions (tfile.c)                *** Routine test_maxopenfiles: SDstart FAILED at line 285 ***
*** Routine test_maxopenfiles: SDgetfilename FAILED at line 292 ***
SDgetfilename: incorrect file being opened - expected <file5>, retrieved <file4>
*** Routine test_maxopenfiles: SDend FAILED at line 305 ***
Testing various SDS' properties (tsdsprops.c)                          PASSED
Testing various coordinate variable features (tcoordvar.c)             PASSED
Testing szip compression for datasets (tszip.c)                        PASSED
*** HDF-SD test fails ***
Makefile:1169: recipe for target 'hdftest.chkexe_' failed
make[5]: *** [hdftest.chkexe_] Error 1
make[5]: *** Waiting for unfinished jobs....
Finished testing testmfhdf.sh 
============================
testmfhdf.sh  Test Log
============================
=============================
Running mfhdf tests
=============================
=============================
HDF-SD C interface tests
=============================
Testing create/read/write compressed datasets (tcomp.c)                PASSED
Testing create/read/write chunked datasets (tchunk.c)                  PASSED
Testing reading of netCDF file using the SDxxx inteface (tnetcdf.c)    PASSED
Testing functionality of dimensions (tdim.c)                           PASSED
Testing functions involving empty data sets (temptySDSs.c)             PASSED
Testing various setting attribute features (tattributes.c)             PASSED
Testing getting data size of special data (tdatasizes.c)               PASSED
Testing getting location info of data (tdatainfo.c)                    PASSED
Testing getting location info of attr and annot data (tattdatainfo.c)  PASSED
Testing a mix of SD, V, and VS functions (tmixed_apis.c)               PASSED
Testing miscellaneous file related functions (tfile.c)                *** Routine test_maxopenfiles: SDstart FAILED at line 285 ***
*** Routine test_maxopenfiles: SDstart FAILED at line 285 ***
*** Routine test_maxopenfiles: SDstart FAILED at line 285 ***

[...]

SDgetfilename: incorrect file being opened - expected <file67>, retrieved <file57>
*** Routine test_maxopenfiles: SDgetfilename FAILED at line 292 ***
SDgetfilename: incorrect file being opened - expected <file68>, retrieved <file57>
*** Routine test_maxopenfiles: SDgetfilename FAILED at line 292 ***
SDgetfilename: incorrect file being opened - expected <file69>, retrieved <file57>
*** Routine test_maxopenfiles: SDgetfilename FAILED at line 292 ***
--8<---------------cut here---------------start------------->8---

Any ideas?

Thanks,
Ludo’.
Eric Bavier - June 2, 2016, 2:43 p.m.
On 2016-05-29 17:07, ludo@gnu.org wrote:
> Hello,
> 
> Jeremy Robst <jpro@bas.ac.uk> skribis:
> 
>> Here's the patch as an attachment - sorry for the hassle. I will try
>> and work out what's going wrong so I can send proper patches in
>> future.
> 
> No problem!
> 
> I tried the attached patch, which includes mostly cosmetic changes
> compared to the one you posted.  However, I stumble upon test failures
> (on x86_64, with Linux-libre 4.6.0):
> 
> --8<---------------cut here---------------start------------->8---
> ----------------------------
> hdftest  Test Log
> ----------------------------
> Testing create/read/write compressed datasets (tcomp.c)                
> PASSED
> Testing create/read/write chunked datasets (tchunk.c)                  
> PASSED
> Testing reading of netCDF file using the SDxxx inteface (tnetcdf.c)    
> PASSED
> Testing functionality of dimensions (tdim.c)                           
> PASSED
> Testing functions involving empty data sets (temptySDSs.c)             
> PASSED
> Testing various setting attribute features (tattributes.c)             
> PASSED
> Testing getting data size of special data (tdatasizes.c)               
> PASSED
> Testing getting location info of data (tdatainfo.c)                    
> PASSED
> Testing getting location info of attr and annot data (tattdatainfo.c)  
> PASSED
> Testing a mix of SD, V, and VS functions (tmixed_apis.c)               
> PASSED
> Testing miscellaneous file related functions (tfile.c)
> *** Routine test_maxopenfiles: SDstart FAILED at line 285 ***
> *** Routine test_maxopenfiles: SDgetfilename FAILED at line 292 ***
> SDgetfilename: incorrect file being opened - expected <file5>, 
> retrieved <file4>
> *** Routine test_maxopenfiles: SDend FAILED at line 305 ***
> Testing various SDS' properties (tsdsprops.c)                          
> PASSED
> Testing various coordinate variable features (tcoordvar.c)             
> PASSED
> Testing szip compression for datasets (tszip.c)                        
> PASSED
> *** HDF-SD test fails ***
> Makefile:1169: recipe for target 'hdftest.chkexe_' failed
> make[5]: *** [hdftest.chkexe_] Error 1
> make[5]: *** Waiting for unfinished jobs....
> Finished testing testmfhdf.sh
> ============================
> testmfhdf.sh  Test Log
> ============================
> =============================
> Running mfhdf tests
> =============================
> =============================
> HDF-SD C interface tests
> =============================
> Testing create/read/write compressed datasets (tcomp.c)                
> PASSED
> Testing create/read/write chunked datasets (tchunk.c)                  
> PASSED
> Testing reading of netCDF file using the SDxxx inteface (tnetcdf.c)    
> PASSED
> Testing functionality of dimensions (tdim.c)                           
> PASSED
> Testing functions involving empty data sets (temptySDSs.c)             
> PASSED
> Testing various setting attribute features (tattributes.c)             
> PASSED
> Testing getting data size of special data (tdatasizes.c)               
> PASSED
> Testing getting location info of data (tdatainfo.c)                    
> PASSED
> Testing getting location info of attr and annot data (tattdatainfo.c)  
> PASSED
> Testing a mix of SD, V, and VS functions (tmixed_apis.c)               
> PASSED
> Testing miscellaneous file related functions (tfile.c)
> *** Routine test_maxopenfiles: SDstart FAILED at line 285 ***
> *** Routine test_maxopenfiles: SDstart FAILED at line 285 ***
> *** Routine test_maxopenfiles: SDstart FAILED at line 285 ***
> 
> [...]
> 
> SDgetfilename: incorrect file being opened - expected <file67>,
> retrieved <file57>
> *** Routine test_maxopenfiles: SDgetfilename FAILED at line 292 ***
> SDgetfilename: incorrect file being opened - expected <file68>,
> retrieved <file57>
> *** Routine test_maxopenfiles: SDgetfilename FAILED at line 292 ***
> SDgetfilename: incorrect file being opened - expected <file69>,
> retrieved <file57>
> *** Routine test_maxopenfiles: SDgetfilename FAILED at line 292 ***
> --8<---------------cut here---------------start------------->8---
> 
> Any ideas?

I get the same failures intermittently.  Other times the build passes.

Also, I've verified that the part of the patch that touches 
libhdf4.settings.in is not required for build reproducibility.
Jeremy Robst - June 2, 2016, 2:58 p.m.
Hi,

>> Any ideas?
>
> I get the same failures intermittently.  Other times the build passes.

Yes, I've built it many times over the past few days - sometimes I get a 
failure, other times it builds ok, like you. If I do

./pre-inst-env guix gc
./pre-inst-env guix build hdf4

it usually (but not always) fails, if I repeat the build it succeeds, then 
the only way I can get it to fail again is by doing a gc. I haven't 
managed to get the tests to fail building by hand.

Looking at the HDF code it's doing things with the maximum number of open 
files - is there anything guix does when it's creating the restricting 
environment for the build user that would limit the number of open files ?

> Also, I've verified that the part of the patch that touches 
> libhdf4.settings.in is not required for build reproducibility.

libhdf4.settings includes the build user and the details of the build 
host as well as the build time. This file is included in the final build. 
I patched configure to set the build time to the epoch, taken from the 
HDF5 patch.

However I also changed the libhdf4.settings to remove details of the build 
user and build host since I was aiming for reproducibilty across builds 
across machines. E.g I happen to always have built it with the user 
guixbuild00, but if I was building 2 or more things in parallel then it 
might get built with guixbuild01 and I thought this would then show us as 
a different build (since the username would be in libhdf4.settings in the 
file output). Is that right or am I misunderstanding something ?

Jeremy
Ludovic Courtès - June 3, 2016, 8:37 a.m.
Hello,

Jeremy Robst <jpro@bas.ac.uk> skribis:

>>> Any ideas?
>>
>> I get the same failures intermittently.  Other times the build passes.
>
> Yes, I've built it many times over the past few days - sometimes I get
> a failure, other times it builds ok, like you. If I do
>
> ./pre-inst-env guix gc
> ./pre-inst-env guix build hdf4

Instead, you can run:

  ./pre-inst-env guix build --check hdf4

> Looking at the HDF code it's doing things with the maximum number of
> open files - is there anything guix does when it's creating the
> restricting environment for the build user that would limit the number
> of open files ?

It does not use ‘setrlimit’, which means it inherits the limits of its
parent process.

There was a problem on the maximum number of processes for people
running guix-daemon from systemd, which commit
f489ce3c9343e3ba9927645d11fcb91bb69f0fa7 addressed.

But anyway, I hit the problem on GuixSD, so it’s not systemd-related.

It might be worth trying:

  #:parallel-tests? #f

though I suspect it’s a per-process limit that we’re hitting, not a
global limit.

Next step would be to ask the HDF4 developers.  :-)

> libhdf4.settings includes the build user and the details of the build
> host as well as the build time. This file is included in the final
> build. I patched configure to set the build time to the epoch, taken
> from the HDF5 patch.

Clearing the build time is definitely necessary.

> However I also changed the libhdf4.settings to remove details of the
> build user and build host since I was aiming for reproducibilty across
> builds across machines. E.g I happen to always have built it with the
> user guixbuild00, but if I was building 2 or more things in parallel
> then it might get built with guixbuild01 and I thought this would then
> show us as a different build (since the username would be in
> libhdf4.settings in the file output). Is that right or am I
> misunderstanding something ?

You are right, the user name could differ, so it’s best to patch it if
it gets embedded in the end.

Thanks,
Ludo’.

Patch

From 77afb922f470ddc97819dd8bb623450585fb2e12 Mon Sep 17 00:00:00 2001
From: Jeremy Robst <jpro@bas.ac.uk>
Date: Mon, 23 May 2016 17:59:00 +0100
Subject: [PATCH] gnu: Add hdf4.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* gnu/packages/maths.scm (hdf4): New variable.
* gnu/packages/patches/hdf4-config-settings.patch: New file.
* gnu/local.mk (dist_patch_DATA): Add it.

Signed-off-by: Ludovic Courtès <ludo@gnu.org>
---
 gnu/local.mk                                    |  1 +
 gnu/packages/maths.scm                          | 42 +++++++++++++++++++
 gnu/packages/patches/hdf4-config-settings.patch | 55 +++++++++++++++++++++++++
 3 files changed, 98 insertions(+)
 create mode 100644 gnu/packages/patches/hdf4-config-settings.patch

diff --git a/gnu/local.mk b/gnu/local.mk
index dade736..9d608e8 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -533,6 +533,7 @@  dist_patch_DATA =						\
   %D%/packages/patches/gtk2-theme-paths.patch			\
   %D%/packages/patches/gtk3-respect-GUIX_GTK3_PATH.patch	\
   %D%/packages/patches/gtkglext-disable-disable-deprecated.patch \
+  %D%/packages/patches/hdf4-config-settings.patch		\
   %D%/packages/patches/hdf5-config-date.patch			\
   %D%/packages/patches/hop-bigloo-4.0b.patch			\
   %D%/packages/patches/hop-linker-flags.patch			\
diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index e11208c..b6e15f9 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -13,6 +13,7 @@ 
 ;;; Copyright © 2016 Roel Janssen <roel@gnu.org>
 ;;; Copyright © 2016 Kei Kebreau <kei@openmailbox.org>
 ;;; Copyright © 2016 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2016 Jeremy Robst <jpro@bas.ac.uk>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -408,6 +409,47 @@  plotting engine by third-party applications like Octave.")
     (license (license:fsf-free
               "http://gnuplot.cvs.sourceforge.net/gnuplot/gnuplot/Copyright"))))
 
+(define-public hdf4
+  (package
+    (name "hdf4")
+    (version "4.2.11")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (string-append
+             "http://www.hdfgroup.org/ftp/HDF/releases/HDF" version
+             "/src/hdf-" version ".tar.bz2"))
+       (sha256
+        (base32
+         "16yr50j845zlfx20skmw3y75ww77akk9gg0affjqkg66ih5r03mv"))
+       (patches (search-patches "hdf4-config-settings.patch"))))
+    (build-system gnu-build-system)
+    (native-inputs
+     `(("gfortran" ,gfortran)
+       ("bison" ,bison)
+       ("flex" ,flex)))
+    (inputs
+     `(("zlib" ,zlib)
+       ("libjpeg" ,libjpeg)))
+    (arguments
+     `(#:phases
+       (modify-phases %standard-phases
+         (add-before 'configure 'patch-configure
+          (lambda _
+            (substitute*
+                '("mfhdf/hdfimport/testutil.sh.in" "hdf/util/testutil.sh.in")
+              (("/bin/rm") "rm")
+              (("/bin/mkdir") "mkdir"))
+            #t)))))
+    (home-page "https://www.hdfgroup.org/products/hdf4/")
+    (synopsis
+     "Library and multi-object file format for storing and managing data")
+    (description "HDF4 is a library and multi-object file format for storing
+and managing data between machines.  HDF4 is an older hiearchical data format,
+incompatible with HDF5.")
+    (license (license:non-copyleft
+              "https://www.hdfgroup.org/ftp/HDF/HDF_Current/src/unpacked/COPYING"))))
+
 (define-public hdf5
   (package
     (name "hdf5")
diff --git a/gnu/packages/patches/hdf4-config-settings.patch b/gnu/packages/patches/hdf4-config-settings.patch
new file mode 100644
index 0000000..e79362e
--- /dev/null
+++ b/gnu/packages/patches/hdf4-config-settings.patch
@@ -0,0 +1,55 @@ 
+Honor SOURCE_DATE_EPOCH when exporting configuraton date.
+Remove unreproducible configuration data.
+
+--- a/configure
++++ b/configure
+@@ -23169,7 +23169,14 @@
+ 
+ 
+ ## Configuration date
+- CONFIG_DATE="`date`"
++ CONFIG_DATE="`date -u`"
++if test -n "$SOURCE_DATE_EPOCH"; then
++  CONFIG_DATE=`date -u -d "@$SOURCE_DATE_EPOCH" 2>/dev/null \
++               || date -u -r "$SOURCE_DATE_EPOCH" 2>/dev/null`
++  if test -z "$CONFIG_DATE"; then
++    as_fn_error $? "malformed SOURCE_DATE_EPOCH" "$LINENO" 5
++  fi
++fi
+ 
+ ## User doing the configuration
+  CONFIG_USER="`whoami`@`hostname`"
+
+--- a/libhdf4.settings.in
++++ b/libhdf4.settings.in
+@@ -5,17 +5,17 @@
+ -------------------
+ 		   HDF4 Version: @H4_VERSION@
+ 		  Configured on: @CONFIG_DATE@
+-		  Configured by: @CONFIG_USER@
++		  Configured by: guix
+ 		 Configure mode: @CONFIG_MODE@
+-		    Host system: @host_cpu@-@host_vendor@-@host_os@
+-              Uname information: @UNAME_INFO@
++		    Host system: guix
++              Uname information: guix
+ 		      Libraries: @STATIC_SHARED@
+ 	     Installation point: @prefix@
+ 
+ Compiling Options:
+ ------------------
+                Compilation Mode: @CONFIG_MODE@
+-                     C compiler: @CC_VERSION@
++                     C compiler: 
+                          CFLAGS: @CFLAGS@
+                        CPPFLAGS: @CPPFLAGS@
+                Shared Libraries: @enable_shared@
+@@ -29,7 +29,7 @@
+ Languages:
+ ----------
+                         Fortran: @BUILD_FORTRAN@
+-@HDF_BUILD_FORTRAN_TRUE@               Fortran Compiler: @F77_VERSION@
++@HDF_BUILD_FORTRAN_TRUE@               Fortran Compiler: 
+ @HDF_BUILD_FORTRAN_TRUE@                         FFLAGS: @FFLAGS@
+ 
+ Features:
-- 
2.7.4