Pass GUILE down to subdirectories

  When I enable cgen rebuilding in the binutils-gdb tree, the default is
to run cgen using 'guile'.  However, on my host, guile is guile 2.2,
which doesn't work for me -- I have to use guile3.0.

This patch arranges to pass "GUILE" down to subdirectories, so I can
use 'make GUILE=guile3.0'.

ChangeLog
2023-12-30  Tom Tromey  <tom@tromey.com>

	* Makefile.in: Rebuild.
	* Makefile.tpl (BASE_EXPORTS): Add GUILE.
	(GUILE): New variable.
	* Makefile.def (flags_to_pass): Add GUILE.
---
 ChangeLog    | 7 +++++++
 Makefile.def | 1 +
 Makefile.in  | 8 ++++++--
 Makefile.tpl | 7 +++++--
 4 files changed, 19 insertions(+), 4 deletions(-)
  

On 12/30/23 14:21, Tom Tromey wrote:
> When I enable cgen rebuilding in the binutils-gdb tree, the default is
> to run cgen using 'guile'.  However, on my host, guile is guile 2.2,
> which doesn't work for me -- I have to use guile3.0.
> 
> This patch arranges to pass "GUILE" down to subdirectories, so I can
> use 'make GUILE=guile3.0'.
> 
> ChangeLog
> 2023-12-30  Tom Tromey  <tom@tromey.com>
> 
> 	* Makefile.in: Rebuild.
> 	* Makefile.tpl (BASE_EXPORTS): Add GUILE.
> 	(GUILE): New variable.
> 	* Makefile.def (flags_to_pass): Add GUILE.
OK from the GCC side.
jeff
  

Tom Tromey <tom@tromey.com> writes:

> When I enable cgen rebuilding in the binutils-gdb tree, the default is
> to run cgen using 'guile'.  However, on my host, guile is guile 2.2,
> which doesn't work for me -- I have to use guile3.0.
>
> This patch arranges to pass "GUILE" down to subdirectories, so I can
> use 'make GUILE=guile3.0'.
>
> ChangeLog
> 2023-12-30  Tom Tromey  <tom@tromey.com>
>
> 	* Makefile.in: Rebuild.
> 	* Makefile.tpl (BASE_EXPORTS): Add GUILE.
> 	(GUILE): New variable.
> 	* Makefile.def (flags_to_pass): Add GUILE.
> ---
>  ChangeLog    | 7 +++++++
>  Makefile.def | 1 +
>  Makefile.in  | 8 ++++++--
>  Makefile.tpl | 7 +++++--
>  4 files changed, 19 insertions(+), 4 deletions(-)
>
> diff --git a/Makefile.def b/Makefile.def
> index 662e50fdc18..792919e561c 100644
> --- a/Makefile.def
> +++ b/Makefile.def
> @@ -310,6 +310,7 @@ flags_to_pass = { flag= GNATBIND ; };
>  flags_to_pass = { flag= GNATMAKE ; };
>  flags_to_pass = { flag= GDC ; };
>  flags_to_pass = { flag= GDCFLAGS ; };
> +flags_to_pass = { flag= GUILE ; };
>  
>  // Target tools
>  flags_to_pass = { flag= AR_FOR_TARGET ; };
> diff --git a/Makefile.in b/Makefile.in
> index 48320bb549e..9a58d5a4f20 100644
> --- a/Makefile.in
> +++ b/Makefile.in
> @@ -3,7 +3,7 @@
>  #
>  # Makefile for directory with subdirs to build.
>  #   Copyright (C) 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998,
> -#   1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011
> +#   1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2023
>  #   Free Software Foundation
>  #
>  # This file is free software; you can redistribute it and/or modify
> @@ -143,7 +143,8 @@ BASE_EXPORTS = \
>  	M4="$(M4)"; export M4; \
>  	SED="$(SED)"; export SED; \
>  	AWK="$(AWK)"; export AWK; \
> -	MAKEINFO="$(MAKEINFO)"; export MAKEINFO;
> +	MAKEINFO="$(MAKEINFO)"; export MAKEINFO; \
> +	GUILE="$(GUILE)"; export GUILE;
>  
>  # This is the list of variables to export in the environment when
>  # configuring subdirectories for the build system.
> @@ -450,6 +451,8 @@ GM2FLAGS = $(CFLAGS)
>  
>  PKG_CONFIG_PATH = @PKG_CONFIG_PATH@
>  
> +GUILE = guile

Hi Tom,

This change is causing some problems for me.

One of my build machines has 2 versions of guile installed.  One is
guile 2.0.14 and the other is guile 2.2.21.

When GDB configures itself the configure script figures out that it
should use 2.2.21 to compile the guile libraries that GDB uses.

However, when we actually build the guile libraries we do use guild2.2,
but due to this 'GUILE = guile' line, guild2.2 uses guile 2.0.14 in
order to perform the compile (I guess, I don't know the details of how
guile compilation works).

Unfortunately guile 2.0.14 compiles in a way which is not compatible
with how GDB then tries to load the guile library.

Maybe better to show you what's going on:

  $ pwd
  /tmp/binutils-gdb/build/gdb/data-directory/guile
  $ GUILE=guile /usr/bin/guild2.2 compile -Warity-mismatch -Wformat -Wunused-toplevel -L . -o ./gdb.go ./gdb.scm
  wrote `./gdb.go'
  $ file gdb.go
  gdb.go: Guile Object, little endian, 64bit, bytecode v2.0
  $ GUILE=guile2.2 /usr/bin/guild2.2 compile -Warity-mismatch -Wformat -Wunused-toplevel -L . -o ./gdb.go ./gdb.scm
  wrote `./gdb.go'
  $ file gdb.go
  gdb.go: ELF 64-bit LSB shared object, no machine, version 1 (embedded), dynamically linked, with debug_info, not stripped

The first compile uses GUILE=guile, so I use guile 2.0.14, which results
in a non-ELF being generated.  When I start GDB with this non-ELF in
place, I see this:

  $ ./gdb/gdb --data-directory ./gdb/data-directory/
  Exception caught while booting Guile.
  Error in function "load-thunk-from-memory":
  not an ELF file
  ./gdb/gdb: warning: Could not complete Guile gdb module initialization from:
  /tmp/binutils-gdb/build/gdb/data-directory/guile/gdb/boot.scm.
  Limited Guile support is available.
  Suggest passing --data-directory=/path/to/gdb/data-directory.
  GDB Version: 15.1
  
  (gdb) 

The second compile, with GUILE=guile2.2 results in an ELF being
generated, and with this in place GDB starts just fine.

Now, clearly the obvious answer is: don't have such an old, out of date
version of guile installed.  But I think there's a bigger issue here.
As guild version X will by default pick up the corresponding version of
guile, shouldn't that be the default behaviour?

My proposal would be that we change the line 'GUILE = guile' to instead
be just 'GUILE ='.

With this in place I can still override the choice of guile executable
with:

  make GUILE=guileXXXX

but, if I don't do this then instead of forcing 'guile' as the default,
we allow the guild compiler to pick its own default, just as we did
before.

Thoughts?

Thanks,
Andrew

---

commit 5230132e2a49ee6603d6713796fb72c418b83e30
Author: Andrew Burgess <aburgess@redhat.com>
Date:   Sat Jan 13 11:15:13 2024 +0000

    don't force 'guile' as the default guile executable
    
    After commit:
    
      commit d006ec41c448d9f4a84df50391e87cbf0aa8c152
      Date:   Thu Dec 28 14:08:39 2023 -0700
    
          Pass GUILE down to subdirectories
    
    I noticed a problem with GDB where 'guile' was now being used to
    compile GDB's guile libraries rather than 'guile2.2'.  The previous
    choice of 'guile2.2' was made by the 'guild2.2' compiler as the GUILE
    environment variable was not set.
    
    After the above commit the environment variable GUILE was being set to
    'guile' by default, which forced 'guild2.2' to use the 'guile'
    executable, which on my system happened to be guile 2.0.14, which
    creates libraries that are not compatible with how GDB expects to load
    the guile libraries.
    
    Aside: guile 2.0.14 creates files containing guile byte-code:
    
      $ file gdb.go
      gdb.go: Guile Object, little endian, 64bit, bytecode v2.0
    
    while guile 2.2.6 creates ELF files:
    
      $ file gdb.go
      gdb.go: ELF 64-bit LSB shared object, no machine, version 1 (embedded), dynamically linked, with debug_info, not stripped
    
    Part of the problem here is that GDB looks for a working version of
    guile by checking different guile versions in this order:
    
      guile-3.0
      guile-2.2
      guile-2.0
    
    On my system GDB finds a working guile-2.2, so links GDB against the
    guile-2.2 library, and uses guild2.2, and the only outlier is the
    default being forced on GDB by the top level Makefile.
    
    I suggest that we should not set the default to 'guile' at the top
    level.  Instead we should leave the default empty.  With this done
    'guild' will select its appropriate default version of guile.
    
    We are still free to do 'make GUILE=guile3.0' if we want, in which
    case the empty default will be overridden.
    
                * Makefile.in: (GUILE): Clear default value.
                * Makefile.tpl: (GUILE): Likewise.

diff --git a/Makefile.in b/Makefile.in
index a1f64a2ab5a..c135c63dc9b 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -452,7 +452,7 @@ GM2FLAGS = $(CFLAGS)
 
 PKG_CONFIG_PATH = @PKG_CONFIG_PATH@
 
-GUILE = guile
+GUILE = 
 
 # Pass additional PGO and LTO compiler options to the PGO build.
 BUILD_CFLAGS = $(PGO_BUILD_CFLAGS) $(PGO_BUILD_LTO_CFLAGS)
diff --git a/Makefile.tpl b/Makefile.tpl
index adbcbdd1d57..db35ee520f4 100644
--- a/Makefile.tpl
+++ b/Makefile.tpl
@@ -455,7 +455,7 @@ GM2FLAGS = $(CFLAGS)
 
 PKG_CONFIG_PATH = @PKG_CONFIG_PATH@
 
-GUILE = guile
+GUILE = 
 
 # Pass additional PGO and LTO compiler options to the PGO build.
 BUILD_CFLAGS = $(PGO_BUILD_CFLAGS) $(PGO_BUILD_LTO_CFLAGS)
  

On Sat, Jan 13, 2024 at 6:36 AM Andrew Burgess <aburgess@redhat.com> wrote:
>
> Tom Tromey <tom@tromey.com> writes:
>
> > When I enable cgen rebuilding in the binutils-gdb tree, the default is
> > to run cgen using 'guile'.  However, on my host, guile is guile 2.2,
> > which doesn't work for me -- I have to use guile3.0.
> >
> > This patch arranges to pass "GUILE" down to subdirectories, so I can
> > use 'make GUILE=guile3.0'.
> >
> > ChangeLog
> > 2023-12-30  Tom Tromey  <tom@tromey.com>
> >
> >       * Makefile.in: Rebuild.
> >       * Makefile.tpl (BASE_EXPORTS): Add GUILE.
> >       (GUILE): New variable.
> >       * Makefile.def (flags_to_pass): Add GUILE.
> > ---
> >  ChangeLog    | 7 +++++++
> >  Makefile.def | 1 +
> >  Makefile.in  | 8 ++++++--
> >  Makefile.tpl | 7 +++++--
> >  4 files changed, 19 insertions(+), 4 deletions(-)
> >
> > diff --git a/Makefile.def b/Makefile.def
> > index 662e50fdc18..792919e561c 100644
> > --- a/Makefile.def
> > +++ b/Makefile.def
> > @@ -310,6 +310,7 @@ flags_to_pass = { flag= GNATBIND ; };
> >  flags_to_pass = { flag= GNATMAKE ; };
> >  flags_to_pass = { flag= GDC ; };
> >  flags_to_pass = { flag= GDCFLAGS ; };
> > +flags_to_pass = { flag= GUILE ; };
> >
> >  // Target tools
> >  flags_to_pass = { flag= AR_FOR_TARGET ; };
> > diff --git a/Makefile.in b/Makefile.in
> > index 48320bb549e..9a58d5a4f20 100644
> > --- a/Makefile.in
> > +++ b/Makefile.in
> > @@ -3,7 +3,7 @@
> >  #
> >  # Makefile for directory with subdirs to build.
> >  #   Copyright (C) 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998,
> > -#   1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011
> > +#   1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2023
> >  #   Free Software Foundation
> >  #
> >  # This file is free software; you can redistribute it and/or modify
> > @@ -143,7 +143,8 @@ BASE_EXPORTS = \
> >       M4="$(M4)"; export M4; \
> >       SED="$(SED)"; export SED; \
> >       AWK="$(AWK)"; export AWK; \
> > -     MAKEINFO="$(MAKEINFO)"; export MAKEINFO;
> > +     MAKEINFO="$(MAKEINFO)"; export MAKEINFO; \
> > +     GUILE="$(GUILE)"; export GUILE;
> >
> >  # This is the list of variables to export in the environment when
> >  # configuring subdirectories for the build system.
> > @@ -450,6 +451,8 @@ GM2FLAGS = $(CFLAGS)
> >
> >  PKG_CONFIG_PATH = @PKG_CONFIG_PATH@
> >
> > +GUILE = guile
>
> Hi Tom,
>
> This change is causing some problems for me.
>
> One of my build machines has 2 versions of guile installed.  One is
> guile 2.0.14 and the other is guile 2.2.21.
>
> When GDB configures itself the configure script figures out that it
> should use 2.2.21 to compile the guile libraries that GDB uses.
>
> However, when we actually build the guile libraries we do use guild2.2,
> but due to this 'GUILE = guile' line, guild2.2 uses guile 2.0.14 in
> order to perform the compile (I guess, I don't know the details of how
> guile compilation works).
>
> Unfortunately guile 2.0.14 compiles in a way which is not compatible
> with how GDB then tries to load the guile library.
>
> Maybe better to show you what's going on:
>
>   $ pwd
>   /tmp/binutils-gdb/build/gdb/data-directory/guile
>   $ GUILE=guile /usr/bin/guild2.2 compile -Warity-mismatch -Wformat -Wunused-toplevel -L . -o ./gdb.go ./gdb.scm
>   wrote `./gdb.go'
>   $ file gdb.go
>   gdb.go: Guile Object, little endian, 64bit, bytecode v2.0
>   $ GUILE=guile2.2 /usr/bin/guild2.2 compile -Warity-mismatch -Wformat -Wunused-toplevel -L . -o ./gdb.go ./gdb.scm
>   wrote `./gdb.go'
>   $ file gdb.go
>   gdb.go: ELF 64-bit LSB shared object, no machine, version 1 (embedded), dynamically linked, with debug_info, not stripped
>
> The first compile uses GUILE=guile, so I use guile 2.0.14, which results
> in a non-ELF being generated.  When I start GDB with this non-ELF in
> place, I see this:
>
>   $ ./gdb/gdb --data-directory ./gdb/data-directory/
>   Exception caught while booting Guile.
>   Error in function "load-thunk-from-memory":
>   not an ELF file
>   ./gdb/gdb: warning: Could not complete Guile gdb module initialization from:
>   /tmp/binutils-gdb/build/gdb/data-directory/guile/gdb/boot.scm.
>   Limited Guile support is available.
>   Suggest passing --data-directory=/path/to/gdb/data-directory.
>   GDB Version: 15.1
>
>   (gdb)
>
> The second compile, with GUILE=guile2.2 results in an ELF being
> generated, and with this in place GDB starts just fine.
>
> Now, clearly the obvious answer is: don't have such an old, out of date
> version of guile installed.  But I think there's a bigger issue here.
> As guild version X will by default pick up the corresponding version of
> guile, shouldn't that be the default behaviour?
>
> My proposal would be that we change the line 'GUILE = guile' to instead
> be just 'GUILE ='.
>
> With this in place I can still override the choice of guile executable
> with:
>
>   make GUILE=guileXXXX
>
> but, if I don't do this then instead of forcing 'guile' as the default,
> we allow the guild compiler to pick its own default, just as we did
> before.
>
> Thoughts?
>

How about adding an additional configure check to configure.ac that
ensures that the version of guile found works properly, and then
having that be substituted into the Makefile? Then the Makefile line
in question would be:
GUILE = @GUILE@

> Thanks,
> Andrew
>
> ---
>
> commit 5230132e2a49ee6603d6713796fb72c418b83e30
> Author: Andrew Burgess <aburgess@redhat.com>
> Date:   Sat Jan 13 11:15:13 2024 +0000
>
>     don't force 'guile' as the default guile executable
>
>     After commit:
>
>       commit d006ec41c448d9f4a84df50391e87cbf0aa8c152
>       Date:   Thu Dec 28 14:08:39 2023 -0700
>
>           Pass GUILE down to subdirectories
>
>     I noticed a problem with GDB where 'guile' was now being used to
>     compile GDB's guile libraries rather than 'guile2.2'.  The previous
>     choice of 'guile2.2' was made by the 'guild2.2' compiler as the GUILE
>     environment variable was not set.
>
>     After the above commit the environment variable GUILE was being set to
>     'guile' by default, which forced 'guild2.2' to use the 'guile'
>     executable, which on my system happened to be guile 2.0.14, which
>     creates libraries that are not compatible with how GDB expects to load
>     the guile libraries.
>
>     Aside: guile 2.0.14 creates files containing guile byte-code:
>
>       $ file gdb.go
>       gdb.go: Guile Object, little endian, 64bit, bytecode v2.0
>
>     while guile 2.2.6 creates ELF files:
>
>       $ file gdb.go
>       gdb.go: ELF 64-bit LSB shared object, no machine, version 1 (embedded), dynamically linked, with debug_info, not stripped
>
>     Part of the problem here is that GDB looks for a working version of
>     guile by checking different guile versions in this order:
>
>       guile-3.0
>       guile-2.2
>       guile-2.0
>
>     On my system GDB finds a working guile-2.2, so links GDB against the
>     guile-2.2 library, and uses guild2.2, and the only outlier is the
>     default being forced on GDB by the top level Makefile.
>
>     I suggest that we should not set the default to 'guile' at the top
>     level.  Instead we should leave the default empty.  With this done
>     'guild' will select its appropriate default version of guile.
>
>     We are still free to do 'make GUILE=guile3.0' if we want, in which
>     case the empty default will be overridden.
>
>                 * Makefile.in: (GUILE): Clear default value.
>                 * Makefile.tpl: (GUILE): Likewise.
>
> diff --git a/Makefile.in b/Makefile.in
> index a1f64a2ab5a..c135c63dc9b 100644
> --- a/Makefile.in
> +++ b/Makefile.in
> @@ -452,7 +452,7 @@ GM2FLAGS = $(CFLAGS)
>
>  PKG_CONFIG_PATH = @PKG_CONFIG_PATH@
>
> -GUILE = guile
> +GUILE =
>
>  # Pass additional PGO and LTO compiler options to the PGO build.
>  BUILD_CFLAGS = $(PGO_BUILD_CFLAGS) $(PGO_BUILD_LTO_CFLAGS)
> diff --git a/Makefile.tpl b/Makefile.tpl
> index adbcbdd1d57..db35ee520f4 100644
> --- a/Makefile.tpl
> +++ b/Makefile.tpl
> @@ -455,7 +455,7 @@ GM2FLAGS = $(CFLAGS)
>
>  PKG_CONFIG_PATH = @PKG_CONFIG_PATH@
>
> -GUILE = guile
> +GUILE =
>
>  # Pass additional PGO and LTO compiler options to the PGO build.
>  BUILD_CFLAGS = $(PGO_BUILD_CFLAGS) $(PGO_BUILD_LTO_CFLAGS)
>
  

Andrew> This change is causing some problems for me.

Yeah, Tom de Vries as well.

Andrew> One of my build machines has 2 versions of guile installed.  One is
Andrew> guile 2.0.14 and the other is guile 2.2.21.

Andrew> When GDB configures itself the configure script figures out that it
Andrew> should use 2.2.21 to compile the guile libraries that GDB uses.

Andrew> However, when we actually build the guile libraries we do use guild2.2,
Andrew> but due to this 'GUILE = guile' line, guild2.2 uses guile 2.0.14 in
Andrew> order to perform the compile (I guess, I don't know the details of how
Andrew> guile compilation works).

Andrew> Unfortunately guile 2.0.14 compiles in a way which is not compatible
Andrew> with how GDB then tries to load the guile library.

I consider this a bug in guile -- it installs 'guild' with this:

    #!/usr/bin/sh
    # -*- scheme -*-
    exec ${GUILE:-/usr/bin/guile2.2} $GUILE_FLAGS -e '(@@ (guild) main)' -s "$0" "$@"
    !#

Allowing a system script to pick $GUILE here seems weird, especially for
a versioned install of "guild", where as you can see it already knows
the correct guile to use.

However -- I think it's better to just work around this.
I plan to back out this change.  Anyone needing to re-run cgen (which
itself ought to come with smarts here, but since it is un-maintained...)
can just specify this by hand.  I.e., the status quo ante.

I'll try to send a patch tomorrow.

Tom
  

On Thu, Jan 18, 2024 at 3:37 PM Tom Tromey <tom@tromey.com> wrote:
>
> Andrew> This change is causing some problems for me.
>
> Yeah, Tom de Vries as well.
>
> Andrew> One of my build machines has 2 versions of guile installed.  One is
> Andrew> guile 2.0.14 and the other is guile 2.2.21.
>
> Andrew> When GDB configures itself the configure script figures out that it
> Andrew> should use 2.2.21 to compile the guile libraries that GDB uses.
>
> Andrew> However, when we actually build the guile libraries we do use guild2.2,
> Andrew> but due to this 'GUILE = guile' line, guild2.2 uses guile 2.0.14 in
> Andrew> order to perform the compile (I guess, I don't know the details of how
> Andrew> guile compilation works).
>
> Andrew> Unfortunately guile 2.0.14 compiles in a way which is not compatible
> Andrew> with how GDB then tries to load the guile library.
>
> I consider this a bug in guile -- it installs 'guild' with this:
>
>     #!/usr/bin/sh
>     # -*- scheme -*-
>     exec ${GUILE:-/usr/bin/guile2.2} $GUILE_FLAGS -e '(@@ (guild) main)' -s "$0" "$@"
>     !#
>
> Allowing a system script to pick $GUILE here seems weird, especially for
> a versioned install of "guild", where as you can see it already knows
> the correct guile to use.
>
> However -- I think it's better to just work around this.
> I plan to back out this change.  Anyone needing to re-run cgen (which
> itself ought to come with smarts here, but since it is un-maintained...)
> can just specify this by hand.  I.e., the status quo ante.

I mean, I've been trying to figure out how to re-run cgen myself, to
regenerate some cgen-erated files in libopcodes to fix some compiler
warnings in them, but it's pretty hard to do so; I'd really appreciate
it if the whole process of regenerating files with cgen could be made
easy and well-documented and understandable...

>
> I'll try to send a patch tomorrow.
>
> Tom
  

Eric> I mean, I've been trying to figure out how to re-run cgen myself, to
Eric> regenerate some cgen-erated files in libopcodes to fix some compiler
Eric> warnings in them, but it's pretty hard to do so; I'd really appreciate
Eric> it if the whole process of regenerating files with cgen could be made
Eric> easy and well-documented and understandable...

Yeah.  Unfortunately cgen seems fairly unmaintained and guile still
seems a bit clunky to use somehow (or maybe it's packaged strangely, I
don't know).

After some patches of mine to let cgen use the guile compiler, it
requires guile 3.0 (at least for me, guile 2 dies while trying to run
it).  So what I do is:

* cd binutils-gdb
* ln -s /path/to/cgen
* configure a new directory with --enable-cgen-maint
* make GUILE=guile3.0

Tom