configure: Allow LD to be a linker other than GNU ld and gold

  When using lld as the linker, configure prints a confusing message.

*** These critical programs are missing or too old: GNU ld

lld>=8 can build glibc with very few patches. lld may be built with a
custom version information (e.g. git commit ID), so a version check is not
useful at all.
---
 configure    | 13 ++++++++-----
 configure.ac | 13 ++++++++-----
 2 files changed, 16 insertions(+), 10 deletions(-)
  

On 13/03/2020 17:46, Fangrui Song wrote:
> When using lld as the linker, configure prints a confusing message.
> 
> *** These critical programs are missing or too old: GNU ld
> 
> lld>=8 can build glibc with very few patches. lld may be built with a
> custom version information (e.g. git commit ID), so a version check is not
> useful at all.

LGTM, thanks.

> ---
>  configure    | 13 ++++++++-----
>  configure.ac | 13 ++++++++-----
>  2 files changed, 16 insertions(+), 10 deletions(-)
> 
> diff --git a/configure b/configure
> index 8df47d61f8..e3922b6d5d 100755
> --- a/configure
> +++ b/configure
> @@ -4615,9 +4615,10 @@ if test $ac_verc_fail = yes; then
>  fi
>  
>  
> -if test -n "`$LD --version | sed -n 's/^GNU \(gold\).*$/\1/p'`"; then
> +case $($LD --version) in
> +  "GNU gold"*)
>    # Accept gold 1.14 or higher
> -  for ac_prog in $LD
> +    for ac_prog in $LD
>  do
>    # Extract the first word of "$ac_prog", so it can be a program name with args.
>  set dummy $ac_prog; ac_word=$2
> @@ -4680,8 +4681,9 @@ if test $ac_verc_fail = yes; then
>    LD=: critic_missing="$critic_missing GNU gold"
>  fi
>  
> -else
> -  for ac_prog in $LD
> +    ;;
> +  "GNU ld"*)
> +    for ac_prog in $LD
>  do
>    # Extract the first word of "$ac_prog", so it can be a program name with args.
>  set dummy $ac_prog; ac_word=$2
> @@ -4744,7 +4746,8 @@ if test $ac_verc_fail = yes; then
>    LD=: critic_missing="$critic_missing GNU ld"
>  fi
>  
> -fi
> +    ;;
> +esac
>  
>  # These programs are version sensitive.
>  
> diff --git a/configure.ac b/configure.ac
> index 5f229679a9..faa45946ad 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -986,18 +986,21 @@ AC_CHECK_PROG_VER(AS, $AS, --version,
>  		  [2.1[0-9][0-9]*|2.2[5-9]*|2.[3-9][0-9]*|[3-9].*|[1-9][0-9]*],
>  		  AS=: critic_missing="$critic_missing as")
>  
> -if test -n "`$LD --version | sed -n 's/^GNU \(gold\).*$/\1/p'`"; then
> +case $($LD --version) in
> +  "GNU gold"*)
>    # Accept gold 1.14 or higher
> -  AC_CHECK_PROG_VER(LD, $LD, --version,
> +    AC_CHECK_PROG_VER(LD, $LD, --version,
>  		    [GNU gold.* \([0-9][0-9]*\.[0-9.]*\)],
>  		    [1.1[4-9]*|1.[2-9][0-9]*|1.1[0-9][0-9]*|[2-9].*|[1-9][0-9]*],
>  		    LD=: critic_missing="$critic_missing GNU gold")
> -else
> -  AC_CHECK_PROG_VER(LD, $LD, --version,
> +    ;;
> +  "GNU ld"*)
> +    AC_CHECK_PROG_VER(LD, $LD, --version,
>  		    [GNU ld.* \([0-9][0-9]*\.[0-9.]*\)],
>  		    [2.1[0-9][0-9]*|2.2[5-9]*|2.[3-9][0-9]*|[3-9].*|[1-9][0-9]*],
>  		    LD=: critic_missing="$critic_missing GNU ld")
> -fi
> +    ;;
> +esac
>  
>  # These programs are version sensitive.
>  AC_CHECK_TOOL_PREFIX
>
  

* Fangrui Song via Libc-alpha:

> When using lld as the linker, configure prints a confusing message.
>
> *** These critical programs are missing or too old: GNU ld
>
> lld>=8 can build glibc with very few patches. lld may be built with a
> custom version information (e.g. git commit ID), so a version check is not
> useful at all.

If additional patches are still required, we should not change the
configure check at this time.

To be clear, this is an objection from me.  Sorry.

Thanks,
Florian
  

On 2020-04-08, Florian Weimer wrote:
>* Fangrui Song via Libc-alpha:
>
>> When using lld as the linker, configure prints a confusing message.
>>
>> *** These critical programs are missing or too old: GNU ld
>>
>> lld>=8 can build glibc with very few patches. lld may be built with a
>> custom version information (e.g. git commit ID), so a version check is not
>> useful at all.
>
>If additional patches are still required, we should not change the
>configure check at this time.
>
>To be clear, this is an objection from me.  Sorry.
>
>Thanks,
>Florian

Hi Florian,

Can you help with https://sourceware.org/pipermail/libc-alpha/2020-March/111910.html ?

To be honest, this LD capability check looks very strange to me. I
believe a good configure time check should check the problem explicitly,
rather than the version, e.g. if a known version can silently cause a
bug, then we should reject it. The current version check can just give
false positives and/or false negatives.  https://ewontfix.com/13/

I am doing my best to make the least intrusive change to configure.
  

* Fangrui Song via Libc-alpha:

> On 2020-04-08, Florian Weimer wrote:
>>* Fangrui Song via Libc-alpha:
>>
>>> When using lld as the linker, configure prints a confusing message.
>>>
>>> *** These critical programs are missing or too old: GNU ld
>>>
>>> lld>=8 can build glibc with very few patches. lld may be built with a
>>> custom version information (e.g. git commit ID), so a version check is not
>>> useful at all.
>>
>>If additional patches are still required, we should not change the
>>configure check at this time.
>>
>>To be clear, this is an objection from me.  Sorry.
>>
>>Thanks,
>>Florian
>
> Hi Florian,
>
> Can you help with https://sourceware.org/pipermail/libc-alpha/2020-March/111910.html ?

I'm not sure what the problem discussed on this thread is (besides an
lld incompatibility).

> To be honest, this LD capability check looks very strange to me. I
> believe a good configure time check should check the problem explicitly,
> rather than the version, e.g. if a known version can silently cause a
> bug, then we should reject it. The current version check can just give
> false positives and/or false negatives.  https://ewontfix.com/13/

The configure checks can be difficult to maintain.  We generally give
wide latitude on the accepted versions.  Earlier ones often really do
not work and produce a broken library.
  

Hi Florian,

Thanks for your comments.

On 2020-04-11, Florian Weimer wrote:
>* Fangrui Song via Libc-alpha:
>
>> On 2020-04-08, Florian Weimer wrote:
>>>* Fangrui Song via Libc-alpha:
>>>
>>>> When using lld as the linker, configure prints a confusing message.
>>>>
>>>> *** These critical programs are missing or too old: GNU ld
>>>>
>>>> lld>=8 can build glibc with very few patches. lld may be built with a
>>>> custom version information (e.g. git commit ID), so a version check is not
>>>> useful at all.
>>>
>>>If additional patches are still required, we should not change the
>>>configure check at this time.
>>>
>>>To be clear, this is an objection from me.  Sorry.
>>>
>>>Thanks,
>>>Florian
>>
>> Hi Florian,
>>
>> Can you help with https://sourceware.org/pipermail/libc-alpha/2020-March/111910.html ?
>
>I'm not sure what the problem discussed on this thread is (besides an
>lld incompatibility).

commit 3a0ecccb599a6b1ad4b149dc569c0080e92d057b "ld.so: Do not export free/calloc/malloc/realloc functions [BZ #25486]"
makes more use of a subtle --defsym behavior (gold/LLD incompatibility).

If my understanding is correct, we can extract the members of
libc_pic.a, make a thin archive with all member except malloc.o,
then link librtld.map with the thin archive.

--start-lib all-members-exceptmalloc.o --end-lib is better but GNU ld
does not support the feature
https://sourceware.org/bugzilla/show_bug.cgi?id=24600


With my patches, I am seeing some weird compiler errors (not linker errors...). They are likely build system brittleness somewhere:

% ../configure --prefix=/tmp/opt2 --disable-werror
% make -j 10
...
In file included from localeinfo.h:224,
                  from lc-ctype.c:19:
lc-ctype.c: In function ‘_nl_postload_ctype’:
../sysdeps/generic/libc-tsd.h:58:43: error: ‘__libc_tsd_CTYPE_B’ undeclared (first use in this function)
    58 | #define __libc_tsd_set(TYPE, KEY, VALUE) (__libc_tsd_##KEY = (VALUE))
       |                                           ^~~~~~~~~~~
lc-ctype.c:67:7: note: in expansion of macro ‘__libc_tsd_set’
    67 |       __libc_tsd_set (const uint16_t *, CTYPE_B,

../sysdeps/unix/sysv/linux/libc_fatal.c:22:8: error: unknown type name ‘bool’
    22 | static bool
       |        ^~~~
../sysdeps/unix/sysv/linux/libc_fatal.c: In function ‘writev_for_fatal’:
../sysdeps/unix/sysv/linux/libc_fatal.c:27:11: warning: implicit declaration of function ‘INTERNAL_SYSCALL_CALL’ [-Wimplicit-function-declaration]
    27 |     cnt = INTERNAL_SYSCALL_CALL (writev, fd, iov, niov);

>> To be honest, this LD capability check looks very strange to me. I
>> believe a good configure time check should check the problem explicitly,
>> rather than the version, e.g. if a known version can silently cause a
>> bug, then we should reject it. The current version check can just give
>> false positives and/or false negatives.  https://ewontfix.com/13/
>
>The configure checks can be difficult to maintain.  We generally give
>wide latitude on the accepted versions.  Earlier ones often really do
>not work and produce a broken library.

This is based on the fact that some older versions of GNU ld may
silently create corrupted glibc.

With an incapable older LLD, we just see some linker errors (it
generally has stronger error checking from my experience. You can search
for the many ld bugs/feature requests I reported/commented on)
  

* Fangrui Song:

> commit 3a0ecccb599a6b1ad4b149dc569c0080e92d057b "ld.so: Do not export free/calloc/malloc/realloc functions [BZ #25486]"
> makes more use of a subtle --defsym behavior (gold/LLD incompatibility).
>
> If my understanding is correct, we can extract the members of
> libc_pic.a, make a thin archive with all member except malloc.o,
> then link librtld.map with the thin archive.

I think it's more reliable to list the symbols, not the object,
because it's the symbols (mainly those defined in dl-minimal.c) that
need special treatment, not the object file.

> With my patches, I am seeing some weird compiler errors (not linker errors...). They are likely build system brittleness somewhere:
>
> % ../configure --prefix=/tmp/opt2 --disable-werror
> % make -j 10
> ...
> In file included from localeinfo.h:224,
>                   from lc-ctype.c:19:
> lc-ctype.c: In function ‘_nl_postload_ctype’:
> ../sysdeps/generic/libc-tsd.h:58:43: error: ‘__libc_tsd_CTYPE_B’ undeclared (first use in this function)
>     58 | #define __libc_tsd_set(TYPE, KEY, VALUE) (__libc_tsd_##KEY = (VALUE))
>        |                                           ^~~~~~~~~~~
> lc-ctype.c:67:7: note: in expansion of macro ‘__libc_tsd_set’
>     67 |       __libc_tsd_set (const uint16_t *, CTYPE_B,
>
> ../sysdeps/unix/sysv/linux/libc_fatal.c:22:8: error: unknown type name ‘bool’
>     22 | static bool
>        |        ^~~~

That suggests that the include path ordering is not correct and
include/ctype.h is not included.  I don't remember seeing anything
like that.

>>> To be honest, this LD capability check looks very strange to me. I
>>> believe a good configure time check should check the problem explicitly,
>>> rather than the version, e.g. if a known version can silently cause a
>>> bug, then we should reject it. The current version check can just give
>>> false positives and/or false negatives.  https://ewontfix.com/13/
>>
>>The configure checks can be difficult to maintain.  We generally give
>>wide latitude on the accepted versions.  Earlier ones often really do
>>not work and produce a broken library.
>
> This is based on the fact that some older versions of GNU ld may
> silently create corrupted glibc.
>
> With an incapable older LLD, we just see some linker errors (it
> generally has stronger error checking from my experience. You can search
> for the many ld bugs/feature requests I reported/commented on)

We still have the problem that users on a system which uses lld by
default will encounter these link errors and think that it's there
fault because we have a configure check that recognizes lld.
  

On 2020-04-11, Florian Weimer wrote:
>* Fangrui Song:
>
>> commit 3a0ecccb599a6b1ad4b149dc569c0080e92d057b "ld.so: Do not export free/calloc/malloc/realloc functions [BZ #25486]"
>> makes more use of a subtle --defsym behavior (gold/LLD incompatibility).
>>
>> If my understanding is correct, we can extract the members of
>> libc_pic.a, make a thin archive with all member except malloc.o,
>> then link librtld.map with the thin archive.
>
>I think it's more reliable to list the symbols, not the object,
>because it's the symbols (mainly those defined in dl-minimal.c) that
>need special treatment, not the object file.

OK. --defsym defines a SHN_ABS symbol, not perfectly robust because a
normal definition is relative to the image base.

This subtless does not cause
https://sourceware.org/bugzilla/show_bug.cgi?id=25749 but it will be
nice to drop the reliance on it.

If I add a stub.s to define these symbols, AS it, then place it as the
first input file to shadow libc_pic.a, will such a patch be acceptable?

Similarly,
https://sourceware.org/pipermail/libc-alpha/2020-April/112628.html will be a main showstopper if it cannot be applied.

If either one is not acceptable, then LLD can never link glibc. I'll stop wasting my efforts.

>> With my patches, I am seeing some weird compiler errors (not linker errors...). They are likely build system brittleness somewhere:
>>
>> % ../configure --prefix=/tmp/opt2 --disable-werror
>> % make -j 10
>> ...
>> In file included from localeinfo.h:224,
>>                   from lc-ctype.c:19:
>> lc-ctype.c: In function ‘_nl_postload_ctype’:
>> ../sysdeps/generic/libc-tsd.h:58:43: error: ‘__libc_tsd_CTYPE_B’ undeclared (first use in this function)
>>     58 | #define __libc_tsd_set(TYPE, KEY, VALUE) (__libc_tsd_##KEY = (VALUE))
>>        |                                           ^~~~~~~~~~~
>> lc-ctype.c:67:7: note: in expansion of macro ‘__libc_tsd_set’
>>     67 |       __libc_tsd_set (const uint16_t *, CTYPE_B,
>>
>> ../sysdeps/unix/sysv/linux/libc_fatal.c:22:8: error: unknown type name ‘bool’
>>     22 | static bool
>>        |        ^~~~
>
>That suggests that the include path ordering is not correct and
>include/ctype.h is not included.  I don't remember seeing anything
>like that.

I am fairly certain this is a problem with the build system because I
just add /usr/local/bin/ld as a symlink to the latest lld...
(and deleted it to verify GNU ld builds)

>>>> To be honest, this LD capability check looks very strange to me. I
>>>> believe a good configure time check should check the problem explicitly,
>>>> rather than the version, e.g. if a known version can silently cause a
>>>> bug, then we should reject it. The current version check can just give
>>>> false positives and/or false negatives.  https://ewontfix.com/13/
>>>
>>>The configure checks can be difficult to maintain.  We generally give
>>>wide latitude on the accepted versions.  Earlier ones often really do
>>>not work and produce a broken library.
>>
>> This is based on the fact that some older versions of GNU ld may
>> silently create corrupted glibc.
>>
>> With an incapable older LLD, we just see some linker errors (it
>> generally has stronger error checking from my experience. You can search
>> for the many ld bugs/feature requests I reported/commented on)
>
>We still have the problem that users on a system which uses lld by
>default will encounter these link errors and think that it's there
>fault because we have a configure check that recognizes lld.
  

On 2020-04-11, Fangrui Song wrote:
>On 2020-04-11, Florian Weimer wrote:
>>* Fangrui Song:
>>
>>>commit 3a0ecccb599a6b1ad4b149dc569c0080e92d057b "ld.so: Do not export free/calloc/malloc/realloc functions [BZ #25486]"
>>>makes more use of a subtle --defsym behavior (gold/LLD incompatibility).
>>>
>>>If my understanding is correct, we can extract the members of
>>>libc_pic.a, make a thin archive with all member except malloc.o,
>>>then link librtld.map with the thin archive.
>>
>>I think it's more reliable to list the symbols, not the object,
>>because it's the symbols (mainly those defined in dl-minimal.c) that
>>need special treatment, not the object file.
>
>OK. --defsym defines a SHN_ABS symbol, not perfectly robust because a
>normal definition is relative to the image base.
>
>This subtless does not cause
>https://sourceware.org/bugzilla/show_bug.cgi?id=25749 but it will be
>nice to drop the reliance on it.
>
>If I add a stub.s to define these symbols, AS it, then place it as the
>first input file to shadow libc_pic.a, will such a patch be acceptable?
>
>Similarly,
>https://sourceware.org/pipermail/libc-alpha/2020-April/112628.html will be a main showstopper if it cannot be applied.
>
>If either one is not acceptable, then LLD can never link glibc. I'll stop wasting my efforts.
>
>>>With my patches, I am seeing some weird compiler errors (not linker errors...). They are likely build system brittleness somewhere:
>>>
>>>% ../configure --prefix=/tmp/opt2 --disable-werror
>>>% make -j 10
>>>...
>>>In file included from localeinfo.h:224,
>>>                  from lc-ctype.c:19:
>>>lc-ctype.c: In function ‘_nl_postload_ctype’:
>>>../sysdeps/generic/libc-tsd.h:58:43: error: ‘__libc_tsd_CTYPE_B’ undeclared (first use in this function)
>>>    58 | #define __libc_tsd_set(TYPE, KEY, VALUE) (__libc_tsd_##KEY = (VALUE))
>>>       |                                           ^~~~~~~~~~~
>>>lc-ctype.c:67:7: note: in expansion of macro ‘__libc_tsd_set’
>>>    67 |       __libc_tsd_set (const uint16_t *, CTYPE_B,
>>>
>>>../sysdeps/unix/sysv/linux/libc_fatal.c:22:8: error: unknown type name ‘bool’
>>>    22 | static bool
>>>       |        ^~~~
>>
>>That suggests that the include path ordering is not correct and
>>include/ctype.h is not included.  I don't remember seeing anything
>>like that.
>
>I am fairly certain this is a problem with the build system because I
>just add /usr/local/bin/ld as a symlink to the latest lld...
>(and deleted it to verify GNU ld builds)

OK, I rebased my branch at some commit today and the GCC compilation errors went away.

With the following patches I managed to run `make install`

https://sourceware.org/pipermail/libc-alpha/2020-April/112628.html
[PATCH] elf: Support lld-style link map for librtld.map
Reviewed-By: Adhemerval Zanella

https://sourceware.org/pipermail/libc-alpha/2020-April/112732.html
[PATCH] elf: Replace a --defsym trick with an object file to be compatible with lld 

https://sourceware.org/pipermail/libc-alpha/2020-April/112733.html
[PATCH] install: Delete scripts/output-format.sed and support lld 


`make check` does not build because some TLS tests use STT_TLS common
symbols (.tls_common) which are not supported by lld. After removing
these tests, `make check` works for me. I see 100+ FAILS but my GNU ld
build has 100+ FAILS as well.



So, is allowing lld in configure acceptable now? Please commit on my
behalf if you think any of the approved commits is OK because I don't
have a commit permission.

>>>>>To be honest, this LD capability check looks very strange to me. I
>>>>>believe a good configure time check should check the problem explicitly,
>>>>>rather than the version, e.g. if a known version can silently cause a
>>>>>bug, then we should reject it. The current version check can just give
>>>>>false positives and/or false negatives.  https://ewontfix.com/13/
>>>>
>>>>The configure checks can be difficult to maintain.  We generally give
>>>>wide latitude on the accepted versions.  Earlier ones often really do
>>>>not work and produce a broken library.
>>>
>>>This is based on the fact that some older versions of GNU ld may
>>>silently create corrupted glibc.
>>>
>>>With an incapable older LLD, we just see some linker errors (it
>>>generally has stronger error checking from my experience. You can search
>>>for the many ld bugs/feature requests I reported/commented on)
>>
>>We still have the problem that users on a system which uses lld by
>>default will encounter these link errors and think that it's there
>>fault because we have a configure check that recognizes lld.
  

On Sat, Apr 11, 2020 at 4:53 PM Fangrui Song via Libc-alpha
<libc-alpha@sourceware.org> wrote:
>
> On 2020-04-11, Fangrui Song wrote:
> >On 2020-04-11, Florian Weimer wrote:
> >>* Fangrui Song:
> >>
> >>>commit 3a0ecccb599a6b1ad4b149dc569c0080e92d057b "ld.so: Do not export free/calloc/malloc/realloc functions [BZ #25486]"
> >>>makes more use of a subtle --defsym behavior (gold/LLD incompatibility).
> >>>
> >>>If my understanding is correct, we can extract the members of
> >>>libc_pic.a, make a thin archive with all member except malloc.o,
> >>>then link librtld.map with the thin archive.
> >>
> >>I think it's more reliable to list the symbols, not the object,
> >>because it's the symbols (mainly those defined in dl-minimal.c) that
> >>need special treatment, not the object file.
> >
> >OK. --defsym defines a SHN_ABS symbol, not perfectly robust because a
> >normal definition is relative to the image base.
> >
> >This subtless does not cause
> >https://sourceware.org/bugzilla/show_bug.cgi?id=25749 but it will be
> >nice to drop the reliance on it.
> >
> >If I add a stub.s to define these symbols, AS it, then place it as the
> >first input file to shadow libc_pic.a, will such a patch be acceptable?
> >
> >Similarly,
> >https://sourceware.org/pipermail/libc-alpha/2020-April/112628.html will be a main showstopper if it cannot be applied.
> >
> >If either one is not acceptable, then LLD can never link glibc. I'll stop wasting my efforts.
> >
> >>>With my patches, I am seeing some weird compiler errors (not linker errors...). They are likely build system brittleness somewhere:
> >>>
> >>>% ../configure --prefix=/tmp/opt2 --disable-werror
> >>>% make -j 10
> >>>...
> >>>In file included from localeinfo.h:224,
> >>>                  from lc-ctype.c:19:
> >>>lc-ctype.c: In function ‘_nl_postload_ctype’:
> >>>../sysdeps/generic/libc-tsd.h:58:43: error: ‘__libc_tsd_CTYPE_B’ undeclared (first use in this function)
> >>>    58 | #define __libc_tsd_set(TYPE, KEY, VALUE) (__libc_tsd_##KEY = (VALUE))
> >>>       |                                           ^~~~~~~~~~~
> >>>lc-ctype.c:67:7: note: in expansion of macro ‘__libc_tsd_set’
> >>>    67 |       __libc_tsd_set (const uint16_t *, CTYPE_B,
> >>>
> >>>../sysdeps/unix/sysv/linux/libc_fatal.c:22:8: error: unknown type name ‘bool’
> >>>    22 | static bool
> >>>       |        ^~~~
> >>
> >>That suggests that the include path ordering is not correct and
> >>include/ctype.h is not included.  I don't remember seeing anything
> >>like that.
> >
> >I am fairly certain this is a problem with the build system because I
> >just add /usr/local/bin/ld as a symlink to the latest lld...
> >(and deleted it to verify GNU ld builds)
>
> OK, I rebased my branch at some commit today and the GCC compilation errors went away.
>
> With the following patches I managed to run `make install`
>
> https://sourceware.org/pipermail/libc-alpha/2020-April/112628.html
> [PATCH] elf: Support lld-style link map for librtld.map
> Reviewed-By: Adhemerval Zanella
>
> https://sourceware.org/pipermail/libc-alpha/2020-April/112732.html
> [PATCH] elf: Replace a --defsym trick with an object file to be compatible with lld
>
> https://sourceware.org/pipermail/libc-alpha/2020-April/112733.html
> [PATCH] install: Delete scripts/output-format.sed and support lld
>
>
> `make check` does not build because some TLS tests use STT_TLS common
> symbols (.tls_common) which are not supported by lld. After removing
> these tests, `make check` works for me. I see 100+ FAILS but my GNU ld
> build has 100+ FAILS as well.
>

On Linux/86, you should only see

FAIL: nptl/tst-setuid2

with "make xcheck" and "make check" should be clean.  You need to
fix all these failures first.