mbox series

[0/3] Improve lld support and current status

Message ID 20211026200346.3371750-1-adhemerval.zanella@linaro.org
Headers
Series Improve lld support and current status |

Message

Adhemerval Zanella Oct. 26, 2021, 8:03 p.m. UTC 
  Ths patch allows x86_64, i686, aarch64, powerpc64le, powerpc64, and
powerpc to build, including the tests, with LLD packed in LLVM 13 [1].

For x86_64 and aarch64 there is no regression in testssuite.  On 
i686 there is only one:

  FAIL: elf/ifuncmain6pie

Which segfaults calling the 'foo' functions:

  Program received signal SIGSEGV, Segmentation fault.
  0xfb010101 in ?? ()
  (gdb) bt
  #0  0xfb010101 in ?? ()
  #1  0xf7fba620 in call_foo () at ifuncmod6.c:18
  #2  0xf7fc0857 in ?? ()
  #3  0xf7e1b195 in __libc_start_call_main (main=main@entry=0xf7fc0830,
  argc=argc@entry=2, argv=argv@entry=0xffffc610) at
  ../sysdeps/nptl/libc_start_call_main.h:58
  #4  0xf7e1b286 in __libc_start_main_impl (main=0xf7fc0830, argc=2,
  argv=0xffffc610, init=0x0, fini=0x0, rtld_fini=0xf7fe79b0 <_dl_fini>,
  stack_end=0xffffc60c) at ../csu/libc-start.c:409
  #5  0xf7fc06d7 in ?? ()

The powerpc64le fails at libc.so start:

  (gdb) bt
  #0  0x00007ffff7f2a980 in __gep_setup___vmx__sigjmp_save ()
    from
  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
  #1  0x00007ffff7d7ac5c in __libc_start_call_main ()
     from
  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
     from
  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
  #3  0x0000000000000000 in ?? ()
  (gdb) disas
  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
     0x00007ffff7f2a984 <+4>:     li      r12,-1280
     0x00007ffff7f2a988 <+8>:     mtctr   r12
     0x00007ffff7f2a98c <+12>:    bctr

And powerpc64-linux-gnu fails on dynamic loaders start:

  (gdb) bt
  #0  0x00007ffff7ffa9a8 in _start ()
  (gdb) disas
  Dump of assembler code for function _start:
  => 0x00007ffff7ffa9a8 <+0>:     .long 0x0
     0x00007ffff7ffa9ac <+4>:     .long 0x0
     0x00007ffff7ffa9b0 <+8>:     .long 0x0
     0x00007ffff7ffa9b4 <+12>:    .long 0x0


The arm, sparcv9, mips, and riscv fail to build due different issues. On
arm the loader fails to build:

  ld.so fails with
  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
  _dl_argv; recompile with -fPIC
  >>> defined in
  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
  >>> referenced by rtld.c:164
  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)

On sparcv9, lld handles --relax option different than ld.bfd: it throws
an error instead of silent ignoring it:

  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
  ld.lld: error: unknown argument '-relax'

And even when -mno-relax is explicit add, lld does not support some relocations
generated by GCC:

  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
  | head -n2
  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
  0000000000000058 _dl_skip_args + 0
  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
  0000000000000058 _dl_skip_args + 0

The mips/mipsel also fails to build the loader:

  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
  symbol in readonly segment; recompile object files with -fPIC or pass
  '-Wl,-z,notext' to allow text relocations in the output
  >>> defined in
  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)

  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
  symbol in readonly segment; recompile object files with -fPIC or pass
  '-Wl,-z,notext' to allow text relocations in the output
  >>> defined in
  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)

Similar to mips64/mips64el:

  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
  against non-function symbol . This is invalid and most likely a compiler
  bug.
  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
  symbol in readonly segment; recompile object files with -fPIC or pass
  '-Wl,-z,notext' to allow text relocations in the output
  >>> defined in
  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)

The riscv32/riscv64 fails to the loader, but it is due missing support
to relaxation:

  ld.lld: error:
  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
  recompile with -mno-relax
  ld.lld: error:
  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
  recompile with -mno-relax
  ld.lld: error:
  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
  recompile with -mno-rela

Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.


[1] https://releases.llvm.org/download.html#13.0.0

Adhemerval Zanella (3):
  elf: Disable ifuncmain{1,5,5pic,5pie} when using LLD
  Fix LIBC_PROG_BINUTILS for -fuse-ld=lld
  Check if linker also support -mtls-dialect=gnu2

 aclocal.m4   |  6 +++++-
 configure    | 10 +++++++---
 configure.ac |  4 ++--
 elf/Makefile | 17 +++++++++++++----
 4 files changed, 27 insertions(+), 10 deletions(-)

Comments

Fangrui Song Oct. 26, 2021, 8:33 p.m. UTC | #1 
On 2021-10-26, Adhemerval Zanella wrote:
>Ths patch allows x86_64, i686, aarch64, powerpc64le, powerpc64, and
>powerpc to build, including the tests, with LLD packed in LLVM 13 [1].

Thanks for checking!

>For x86_64 and aarch64 there is no regression in testssuite.  On
>i686 there is only one:
>
>  FAIL: elf/ifuncmain6pie
>
>Which segfaults calling the 'foo' functions:
>
>  Program received signal SIGSEGV, Segmentation fault.
>  0xfb010101 in ?? ()
>  (gdb) bt
>  #0  0xfb010101 in ?? ()
>  #1  0xf7fba620 in call_foo () at ifuncmod6.c:18
>  #2  0xf7fc0857 in ?? ()
>  #3  0xf7e1b195 in __libc_start_call_main (main=main@entry=0xf7fc0830,
>  argc=argc@entry=2, argv=argv@entry=0xffffc610) at
>  ../sysdeps/nptl/libc_start_call_main.h:58
>  #4  0xf7e1b286 in __libc_start_main_impl (main=0xf7fc0830, argc=2,
>  argv=0xffffc610, init=0x0, fini=0x0, rtld_fini=0xf7fe79b0 <_dl_fini>,
>  stack_end=0xffffc60c) at ../csu/libc-start.c:409
>  #5  0xf7fc06d7 in ?? ()
>
>The powerpc64le fails at libc.so start:
>
>  (gdb) bt
>  #0  0x00007ffff7f2a980 in __gep_setup___vmx__sigjmp_save ()
>    from
>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>  #1  0x00007ffff7d7ac5c in __libc_start_call_main ()
>     from
>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>     from
>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>  #3  0x0000000000000000 in ?? ()
>  (gdb) disas
>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>     0x00007ffff7f2a988 <+8>:     mtctr   r12
>     0x00007ffff7f2a98c <+12>:    bctr
>
>And powerpc64-linux-gnu fails on dynamic loaders start:
>
>  (gdb) bt
>  #0  0x00007ffff7ffa9a8 in _start ()
>  (gdb) disas
>  Dump of assembler code for function _start:
>  => 0x00007ffff7ffa9a8 <+0>:     .long 0x0
>     0x00007ffff7ffa9ac <+4>:     .long 0x0
>     0x00007ffff7ffa9b0 <+8>:     .long 0x0
>     0x00007ffff7ffa9b4 <+12>:    .long 0x0

CCed the powerpc maintainer.

I am not familiar with debugging rtld for powerpc64...

>
>The arm, sparcv9, mips, and riscv fail to build due different issues. On
>arm the loader fails to build:
>
>  ld.so fails with
>  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
>  _dl_argv; recompile with -fPIC
>  >>> defined in
>  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
>  >>> referenced by rtld.c:164
>  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)

R_ARM_GOTOFF32 = S + A - GOT_ORG  .
Such a relocation referencing a preemptible symbol cannot be used.

Filed GNU ld bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28501

_dl_argv needs to be made non-preemptible. There are many ways: --dynamic-list, -Bsymbolic, hidden visibility.

>On sparcv9, lld handles --relax option different than ld.bfd: it throws
>an error instead of silent ignoring it:
>
>  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
>    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
>  ld.lld: error: unknown argument '-relax'

LLD's sparcv9 port is for retrocomputing fans:) and is far from usable
(see https://lld.llvm.org/ "production quality").
The GOT/PLT support has quite a few issues (https://reviews.llvm.org/D102985).

>And even when -mno-relax is explicit add, lld does not support some relocations
>generated by GCC:
>
>  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
>  | head -n2
>  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
>  0000000000000058 _dl_skip_args + 0
>  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
>  0000000000000058 _dl_skip_args + 0
>
>The mips/mipsel also fails to build the loader:
>
>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>  symbol in readonly segment; recompile object files with -fPIC or pass
>  '-Wl,-z,notext' to allow text relocations in the output
>  >>> defined in
>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)
>
>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>  symbol in readonly segment; recompile object files with -fPIC or pass
>  '-Wl,-z,notext' to allow text relocations in the output
>  >>> defined in
>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)

lld.llvm.org says "MIPS seems decent too." and I personally know really
little about MIPS.

However, I think this is likely a genuine issue somewhere in sysdeps/mips/ .
Relocation processing is rigid and consistent in LLD.
GNU ld may miss some error checking for some relocation types.

>Similar to mips64/mips64el:
>
>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>  against non-function symbol . This is invalid and most likely a compiler
>  bug.
>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>  symbol in readonly segment; recompile object files with -fPIC or pass
>  '-Wl,-z,notext' to allow text relocations in the output
>  >>> defined in
>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)

I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
reporting a genuine issue.

>The riscv32/riscv64 fails to the loader, but it is due missing support
>to relaxation:
>
>  ld.lld: error:
>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>  recompile with -mno-relax
>  ld.lld: error:
>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>  recompile with -mno-relax
>  ld.lld: error:
>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>  recompile with -mno-rela

LLD does not implement RISC-V linker relaxation.
R_RISCV_ALIGN has weird linker unfriendly semantics:
simply ignoring the relocation type can break semantics
(https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/183).

>Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.

So sysdep-CFLAGS is not a catch-all option affecting all C compiles...

>
>[1] https://releases.llvm.org/download.html#13.0.0
>
>Adhemerval Zanella (3):
>  elf: Disable ifuncmain{1,5,5pic,5pie} when using LLD
>  Fix LIBC_PROG_BINUTILS for -fuse-ld=lld
>  Check if linker also support -mtls-dialect=gnu2
>
> aclocal.m4   |  6 +++++-
> configure    | 10 +++++++---
> configure.ac |  4 ++--
> elf/Makefile | 17 +++++++++++++----
> 4 files changed, 27 insertions(+), 10 deletions(-)
>
>-- 
>2.32.0
>
Adhemerval Zanella Oct. 27, 2021, 1:11 p.m. UTC | #2 
On 26/10/2021 17:33, Fangrui Song wrote:
>> The arm, sparcv9, mips, and riscv fail to build due different issues. On
>> arm the loader fails to build:
>>
>>  ld.so fails with
>>  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
>>  _dl_argv; recompile with -fPIC
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
>>  >>> referenced by rtld.c:164
>>  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)
> 
> R_ARM_GOTOFF32 = S + A - GOT_ORG  .
> Such a relocation referencing a preemptible symbol cannot be used.
> 
> Filed GNU ld bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28501
> 
> _dl_argv needs to be made non-preemptible. There are many ways: --dynamic-list, -Bsymbolic, hidden visibility.

I think it is something else, the symbols is defined as hidden (output of
preprocessor):

  extern __typeof (_dl_argv) _dl_argv __asm__ ("" "__GI__dl_argv") __attribute__ ((visibility ("hidden")));

However, hidden is set only the internal symbol:

  $ arm-glibc-linux-gnueabi-readelf -Ws elf/librtld.os | grep _dl_argv
    1378: 00000000     4 OBJECT  GLOBAL DEFAULT    6 _dl_argv
    1470: 00000000     4 OBJECT  GLOBAL HIDDEN     6 __GI__dl_argv

And on rtld.c assembly we have the directive to use the internal definition:

    .set    _dl_argv,__GI__dl_argv

So why lld is not binding the usage to internal to the local hidden alias?

> 
>> On sparcv9, lld handles --relax option different than ld.bfd: it throws
>> an error instead of silent ignoring it:
>>
>>  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
>>    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
>>  ld.lld: error: unknown argument '-relax'
> 
> LLD's sparcv9 port is for retrocomputing fans:) and is far from usable
> (see https://lld.llvm.org/ "production quality").
> The GOT/PLT support has quite a few issues (https://reviews.llvm.org/D102985).

Right, but lld is still showing a different ld.bfd semantic that might trigger
other issues where on platforms where neither --relax nor -no-relax it supported
ld.bfd accepts both argument but ignore it [1].

At least on sparc, gcc does pass --relax and expects that static linker just
ignore if it is no supported.

[1] https://sourceware.org/binutils/docs-2.37/ld.html


> 
>> And even when -mno-relax is explicit add, lld does not support some relocations
>> generated by GCC:
>>
>>  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
>>  | head -n2
>>  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
>>  0000000000000058 _dl_skip_args + 0
>>  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
>>  0000000000000058 _dl_skip_args + 0
>>
>> The mips/mipsel also fails to build the loader:
>>
>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>  '-Wl,-z,notext' to allow text relocations in the output
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)
>>
>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>  '-Wl,-z,notext' to allow text relocations in the output
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)
> 
> lld.llvm.org says "MIPS seems decent too." and I personally know really
> little about MIPS.
> 
> However, I think this is likely a genuine issue somewhere in sysdeps/mips/ .
> Relocation processing is rigid and consistent in LLD.
> GNU ld may miss some error checking for some relocation types.
> 
>> Similar to mips64/mips64el:
>>
>>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>  against non-function symbol . This is invalid and most likely a compiler
>>  bug.
>>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>  '-Wl,-z,notext' to allow text relocations in the output
>>  >>> defined in
>>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
> 
> I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
> reporting a genuine issue.

I give you that loader code is tricky, but it does not really explain why
lld is failing to link the eh_frame.  It *might* due some internal assembly
routines, but at least from logs it does not seem so.

> 
>> The riscv32/riscv64 fails to the loader, but it is due missing support
>> to relaxation:
>>
>>  ld.lld: error:
>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>  recompile with -mno-relax
>>  ld.lld: error:
>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>  recompile with -mno-relax
>>  ld.lld: error:
>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>  recompile with -mno-rela
> 
> LLD does not implement RISC-V linker relaxation.
> R_RISCV_ALIGN has weird linker unfriendly semantics:
> simply ignoring the relocation type can break semantics
> (https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/183).
> 
>> Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.
> 
> So sysdep-CFLAGS is not a catch-all option affecting all C compiles...

In fact it is, the problem was the autogenerated syscalls built from
the assembly.  I could avoid the R_RISCV_ALIGN with:

diff --git a/sysdeps/riscv/Makefile b/sysdeps/riscv/Makefile
index 20a9968106..28dbd89cb0 100644
--- a/sysdeps/riscv/Makefile
+++ b/sysdeps/riscv/Makefile
@@ -5,3 +5,9 @@ endif
 # RISC-V's assembler also needs to know about PIC as it changes the definition
 # of some assembler macros.
 ASFLAGS-.os += $(pic-ccflag)
+
+# lld does not implement R_RISCV_ALIGN relaxation optimization.
+ifeq (yes,$(with-lld))
+ASFLAGS-.os += -Wa,-mno-relax
+sysdep-CFLAGS += -mno-relax
+endif


However it does fail later for the libc.so with:

ld.lld: warning: attempt to reassign symbol 'fcntl' of version 'GLIBC_2.27' to version 'GLIBC_2.28'
ld.lld: warning: attempt to reassign symbol '__sched_get_priority_min' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
ld.lld: warning: attempt to reassign symbol '__sched_get_priority_max' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
ld.lld: error: relocation R_RISCV_RVC_JUMP cannot be used against symbol __sigsetjmp; recompile with -fPIC
>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>> referenced by iconv.c:62 (/home/azanella/Projects/glibc/glibc-git/iconv/iconv.c:62)
>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__GI__setjmp)

ld.lld: error: relocation R_RISCV_JAL cannot be used against symbol exit; recompile with -fPIC
>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>> referenced by gconv_open.c:89 (/home/azanella/Projects/glibc/glibc-git/iconv/gconv_open.c:89)
>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__start_context)
collect2: error: ld returned 1 exit status
Tulio Magno Quites Machado Filho Oct. 27, 2021, 10:39 p.m. UTC | #3 
Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:

> On 2021-10-26, Adhemerval Zanella wrote:
>>The powerpc64le fails at libc.so start:
>>
>>  (gdb) bt
>>  #0  0x00007ffff7f2a980 in __gep_setup___vmx__sigjmp_save ()
>>    from
>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>  #1  0x00007ffff7d7ac5c in __libc_start_call_main ()
>>     from
>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6

This is a local call from libc.so to libc.so.
It should be using the local entry point for __vmx__sigjmp_save.

>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>     from
>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>  #3  0x0000000000000000 in ?? ()
>>  (gdb) disas
>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280

This is a pla, but this GDB isn't able to disassemble it.  This instruction
shouldn't be used unless when configuring glibc using --with-cpu=power10.

>>And powerpc64-linux-gnu fails on dynamic loaders start:
>>
>>  (gdb) bt
>>  #0  0x00007ffff7ffa9a8 in _start ()
>>  (gdb) disas
>>  Dump of assembler code for function _start:
>>  => 0x00007ffff7ffa9a8 <+0>:     .long 0x0
>>     0x00007ffff7ffa9ac <+4>:     .long 0x0
>>     0x00007ffff7ffa9b0 <+8>:     .long 0x0
>>     0x00007ffff7ffa9b4 <+12>:    .long 0x0

I've been told that LLD doesn't support the powerpc ELFv1 yet.

> CCed the powerpc maintainer.

Thanks!
Tulio Magno Quites Machado Filho Oct. 27, 2021, 10:50 p.m. UTC | #4 
Tulio Magno Quites Machado Filho via Libc-alpha <libc-alpha@sourceware.org> writes:

> Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:
>
>> On 2021-10-26, Adhemerval Zanella wrote:
>>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>>     from
>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>  #3  0x0000000000000000 in ?? ()
>>>  (gdb) disas
>>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>
> This is a pla, but this GDB isn't able to disassemble it.  This instruction
> shouldn't be used unless when configuring glibc using --with-cpu=power10.

I can reproduce this issue even when configuring glibc with
--with-cpu=power9 --disable-multi-arch, which means the build should not have
any Power10 instructions.
Fangrui Song Oct. 27, 2021, 11:37 p.m. UTC | #5 
On 2021-10-27, Tulio Magno Quites Machado Filho wrote:
>Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:
>
>> On 2021-10-26, Adhemerval Zanella wrote:
>>>The powerpc64le fails at libc.so start:
>>>
>>>  (gdb) bt
>>>  #0  0x00007ffff7f2a980 in __gep_setup___vmx__sigjmp_save ()
>>>    from
>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>  #1  0x00007ffff7d7ac5c in __libc_start_call_main ()
>>>     from
>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>
>This is a local call from libc.so to libc.so.
>It should be using the local entry point for __vmx__sigjmp_save.
>
>>>  #2  0x00007ffff7d7ae90 in __libc_start_main_impl ()
>>>     from
>>>  /home/azanella/glibc/build/powerpc64le-linux-gnu-power9-lld/libc.so.6
>>>  #3  0x0000000000000000 in ?? ()
>>>  (gdb) disas
>>>  Dump of assembler code for function __gep_setup___vmx__sigjmp_save:
>>>  => 0x00007ffff7f2a980 <+0>:     .long 0x613ffe6
>>>     0x00007ffff7f2a984 <+4>:     li      r12,-1280
>
>This is a pla, but this GDB isn't able to disassemble it.  This instruction
>shouldn't be used unless when configuring glibc using --with-cpu=power10.
>
>>>And powerpc64-linux-gnu fails on dynamic loaders start:
>>>
>>>  (gdb) bt
>>>  #0  0x00007ffff7ffa9a8 in _start ()
>>>  (gdb) disas
>>>  Dump of assembler code for function _start:
>>>  => 0x00007ffff7ffa9a8 <+0>:     .long 0x0
>>>     0x00007ffff7ffa9ac <+4>:     .long 0x0
>>>     0x00007ffff7ffa9b0 <+8>:     .long 0x0
>>>     0x00007ffff7ffa9b4 <+12>:    .long 0x0
>
>I've been told that LLD doesn't support the powerpc ELFv1 yet.

You are right:) The incomplete PowerPC64 ELF v1 support was removed by
https://reviews.llvm.org/D46316 (2018-05).  Is there remaining ELF v1
usage for powerpc64le in glibc?

(I only know some 64-bit ELF v2 and some 32-bit, but know really little about 64-bit ELF v1.)

Legacy ELF v1 is also the issue that powerpc64 big-endian Linux kernel
cannot be linked with LLD (https://github.com/ClangBuiltLinux/linux/issues/602).

There is little benefit retroactively adding ELF v1 support to LLD, so
migration to ELF v2 will be nice :)

>> CCed the powerpc maintainer.
>
>Thanks!
>
>-- 
>Tulio Magno
Fangrui Song Oct. 28, 2021, 1:06 a.m. UTC | #6 
On 2021-10-27, Adhemerval Zanella wrote:
>
>
>On 26/10/2021 17:33, Fangrui Song wrote:
>>> The arm, sparcv9, mips, and riscv fail to build due different issues. On
>>> arm the loader fails to build:
>>>
>>>  ld.so fails with
>>>  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
>>>  _dl_argv; recompile with -fPIC
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
>>>  >>> referenced by rtld.c:164
>>>  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)
>>
>> R_ARM_GOTOFF32 = S + A - GOT_ORG  .
>> Such a relocation referencing a preemptible symbol cannot be used.
>>
>> Filed GNU ld bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28501
>>
>> _dl_argv needs to be made non-preemptible. There are many ways: --dynamic-list, -Bsymbolic, hidden visibility.
>
>I think it is something else, the symbols is defined as hidden (output of
>preprocessor):
>
>  extern __typeof (_dl_argv) _dl_argv __asm__ ("" "__GI__dl_argv") __attribute__ ((visibility ("hidden")));
>
>However, hidden is set only the internal symbol:
>
>  $ arm-glibc-linux-gnueabi-readelf -Ws elf/librtld.os | grep _dl_argv
>    1378: 00000000     4 OBJECT  GLOBAL DEFAULT    6 _dl_argv
>    1470: 00000000     4 OBJECT  GLOBAL HIDDEN     6 __GI__dl_argv
>
>And on rtld.c assembly we have the directive to use the internal definition:
>
>    .set    _dl_argv,__GI__dl_argv
>
>So why lld is not binding the usage to internal to the local hidden alias?

I use the Debian package g++-arm-linux-gnueabihf (nowadays glibc build needs a C++ compiler even if I don't build tests...).

mkdir -p out/arm; cd out/arm
../../configure --prefix=/tmp/glibc/arm --host=arm-linux-gnueabihf
make -j 50

Here is GNU ld produced elf/librtld.os:

% readelf -Ws elf/librtld.os | grep _dl_argv
   1582: 00000000     4 OBJECT  GLOBAL HIDDEN    13 __GI__dl_argv
   1603: 00000000     4 OBJECT  GLOBAL DEFAULT   13 _dl_argv

So LLD's librtld.os matches GNU ld for the two symbols.



The .set directive sets value/type but not binding/visibility.
Perhaps sysdeps/arm/dl-machine.h should use

.word   __GI__dl_argv(GOTOFF)

instead of 

.word   _dl_argv(GOTOFF)

>>
>>> On sparcv9, lld handles --relax option different than ld.bfd: it throws
>>> an error instead of silent ignoring it:
>>>
>>>  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
>>>    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
>>>  ld.lld: error: unknown argument '-relax'
>>
>> LLD's sparcv9 port is for retrocomputing fans:) and is far from usable
>> (see https://lld.llvm.org/ "production quality").
>> The GOT/PLT support has quite a few issues (https://reviews.llvm.org/D102985).
>
>Right, but lld is still showing a different ld.bfd semantic that might trigger
>other issues where on platforms where neither --relax nor -no-relax it supported
>ld.bfd accepts both argument but ignore it [1].
>
>At least on sparc, gcc does pass --relax and expects that static linker just
>ignore if it is no supported.
>
>[1] https://sourceware.org/binutils/docs-2.37/ld.html

LLD's sparcv9 port is quite immature, so I cannot even advertise it.
A large portion should be rewritten but the sparc port is only cared by very few
retrocomputing fans now..

>>
>>> And even when -mno-relax is explicit add, lld does not support some relocations
>>> generated by GCC:
>>>
>>>  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
>>>  | head -n2
>>>  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
>>>  0000000000000058 _dl_skip_args + 0
>>>  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
>>>  0000000000000058 _dl_skip_args + 0
>>>
>>> The mips/mipsel also fails to build the loader:
>>>
>>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)
>>>
>>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)
>>
>> lld.llvm.org says "MIPS seems decent too." and I personally know really
>> little about MIPS.
>>
>> However, I think this is likely a genuine issue somewhere in sysdeps/mips/ .
>> Relocation processing is rigid and consistent in LLD.
>> GNU ld may miss some error checking for some relocation types.
>>
>>> Similar to mips64/mips64el:
>>>
>>>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>>  against non-function symbol . This is invalid and most likely a compiler
>>>  bug.
>>>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>  >>> defined in
>>>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
>>
>> I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
>> reporting a genuine issue.
>
>I give you that loader code is tricky, but it does not really explain why
>lld is failing to link the eh_frame.  It *might* due some internal assembly
>routines, but at least from logs it does not seem so.

Hope Joseph can answer the question.

If we have easy-to-follow instructions building glibc mips with LLD, I can
forward it to Simon Atanasyan (MIPS code owner of llvm-project and the major
contributor of LLD's MIPS port)...

But see below (for riscv/), I suspect this is a glibc sysdeps/ problem.

>>
>>> The riscv32/riscv64 fails to the loader, but it is due missing support
>>> to relaxation:
>>>
>>>  ld.lld: error:
>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>  recompile with -mno-relax
>>>  ld.lld: error:
>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>  recompile with -mno-relax
>>>  ld.lld: error:
>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>  recompile with -mno-rela
>>
>> LLD does not implement RISC-V linker relaxation.
>> R_RISCV_ALIGN has weird linker unfriendly semantics:
>> simply ignoring the relocation type can break semantics
>> (https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/183).
>>
>>> Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.
>>
>> So sysdep-CFLAGS is not a catch-all option affecting all C compiles...
>
>In fact it is, the problem was the autogenerated syscalls built from
>the assembly.  I could avoid the R_RISCV_ALIGN with:
>
>diff --git a/sysdeps/riscv/Makefile b/sysdeps/riscv/Makefile
>index 20a9968106..28dbd89cb0 100644
>--- a/sysdeps/riscv/Makefile
>+++ b/sysdeps/riscv/Makefile
>@@ -5,3 +5,9 @@ endif
> # RISC-V's assembler also needs to know about PIC as it changes the definition
> # of some assembler macros.
> ASFLAGS-.os += $(pic-ccflag)
>+
>+# lld does not implement R_RISCV_ALIGN relaxation optimization.
>+ifeq (yes,$(with-lld))
>+ASFLAGS-.os += -Wa,-mno-relax
>+sysdep-CFLAGS += -mno-relax
>+endif
>
>
>However it does fail later for the libc.so with:
>
>ld.lld: warning: attempt to reassign symbol 'fcntl' of version 'GLIBC_2.27' to version 'GLIBC_2.28'
>ld.lld: warning: attempt to reassign symbol '__sched_get_priority_min' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
>ld.lld: warning: attempt to reassign symbol '__sched_get_priority_max' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
>ld.lld: error: relocation R_RISCV_RVC_JUMP cannot be used against symbol __sigsetjmp; recompile with -fPIC
>>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>>> referenced by iconv.c:62 (/home/azanella/Projects/glibc/glibc-git/iconv/iconv.c:62)
>>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__GI__setjmp)
>
>ld.lld: error: relocation R_RISCV_JAL cannot be used against symbol exit; recompile with -fPIC
>>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>>> referenced by gconv_open.c:89 (/home/azanella/Projects/glibc/glibc-git/iconv/gconv_open.c:89)
>>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__start_context)

There is a GNU ld issue: https://sourceware.org/bugzilla/show_bug.cgi?id=28509
It incorrectly allows non-PLT non-GOT relocation referencing a preemptible symbol.


LLD uses consistent categories (e.g. R_PC / R_PLT_PC) to describe relocation types.
It has consistent diagnostic across all ports it supports.
GNU ld uses dispatches for each relocation type and may miss some diagnostics.

>collect2: error: ld returned 1 exit status
Fangrui Song Oct. 28, 2021, 1:18 a.m. UTC | #7 
On 2021-10-27, Fangrui Song wrote:
>
>On 2021-10-27, Adhemerval Zanella wrote:
>>>>Similar to mips64/mips64el:
>>>>
>>>> ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>>> _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>>> against non-function symbol . This is invalid and most likely a compiler
>>>> bug.
>>>> ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>>> symbol in readonly segment; recompile object files with -fPIC or pass
>>>> '-Wl,-z,notext' to allow text relocations in the output
>>>> >>> defined in
>>>> >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>>> >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>>> >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
>>>
>>>I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
>>>reporting a genuine issue.
>>
>>I give you that loader code is tricky, but it does not really explain why
>>lld is failing to link the eh_frame.  It *might* due some internal assembly
>>routines, but at least from logs it does not seem so.
>
>Hope Joseph can answer the question.
>
>If we have easy-to-follow instructions building glibc mips with LLD, I can
>forward it to Simon Atanasyan (MIPS code owner of llvm-project and the major
>contributor of LLD's MIPS port)...
>
>But see below (for riscv/), I suspect this is a glibc sysdeps/ problem.

jrtc27 told me that BFD mips has a hack to rewrite R_MIPS_64 in
.eh_frame to avoid such a R_MIPS_64 diagnostic. LLD doesn't implement it.

Clang's mips port emits relative relocations in .eh_frame (DW_EH_PE_pcrel | dwarf::DW_EH_PE_sdata4)
to avoid the linker issue: https://reviews.llvm.org/D72228

Lack of 64-bit PC-relative relocation is a longstanding problem for mips.
LLVM integrated assembler and linker actually support `.quad foo - .`
via composed relocations, but it is still unsupported in GNU as
(feature request: https://sourceware.org/bugzilla/show_bug.cgi?id=26222)

I know really little about MIPS but I'd wish that GNU as implements
.quad foo-.  and GCC migrates to PC-relative relocations like Clang.
Adhemerval Zanella Oct. 28, 2021, 11:48 a.m. UTC | #8 
On 27/10/2021 22:06, Fangrui Song wrote:
> 
> On 2021-10-27, Adhemerval Zanella wrote:
>>
>>
>> On 26/10/2021 17:33, Fangrui Song wrote:
>>>> The arm, sparcv9, mips, and riscv fail to build due different issues. On
>>>> arm the loader fails to build:
>>>>
>>>>  ld.so fails with
>>>>  ld.lld: error: relocation R_ARM_GOTOFF32 cannot be used against symbol
>>>>  _dl_argv; recompile with -fPIC
>>>>  >>> defined in
>>>>  >>> /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os
>>>>  >>> referenced by rtld.c:164
>>>>  >>>               /home/azanella/Projects/glibc/build/arm-linux-gnueabihf-lld/elf/librtld.os:(.text+0xA8)
>>>
>>> R_ARM_GOTOFF32 = S + A - GOT_ORG  .
>>> Such a relocation referencing a preemptible symbol cannot be used.
>>>
>>> Filed GNU ld bug: https://sourceware.org/bugzilla/show_bug.cgi?id=28501
>>>
>>> _dl_argv needs to be made non-preemptible. There are many ways: --dynamic-list, -Bsymbolic, hidden visibility.
>>
>> I think it is something else, the symbols is defined as hidden (output of
>> preprocessor):
>>
>>  extern __typeof (_dl_argv) _dl_argv __asm__ ("" "__GI__dl_argv") __attribute__ ((visibility ("hidden")));
>>
>> However, hidden is set only the internal symbol:
>>
>>  $ arm-glibc-linux-gnueabi-readelf -Ws elf/librtld.os | grep _dl_argv
>>    1378: 00000000     4 OBJECT  GLOBAL DEFAULT    6 _dl_argv
>>    1470: 00000000     4 OBJECT  GLOBAL HIDDEN     6 __GI__dl_argv
>>
>> And on rtld.c assembly we have the directive to use the internal definition:
>>
>>    .set    _dl_argv,__GI__dl_argv
>>
>> So why lld is not binding the usage to internal to the local hidden alias?
> 
> I use the Debian package g++-arm-linux-gnueabihf (nowadays glibc build needs a C++ compiler even if I don't build tests...).

You can set an empty CXX to avoid it ("CXX=") at configure time.

> 
> mkdir -p out/arm; cd out/arm
> ../../configure --prefix=/tmp/glibc/arm --host=arm-linux-gnueabihf
> make -j 50
> 
> Here is GNU ld produced elf/librtld.os:
> 
> % readelf -Ws elf/librtld.os | grep _dl_argv
>   1582: 00000000     4 OBJECT  GLOBAL HIDDEN    13 __GI__dl_argv
>   1603: 00000000     4 OBJECT  GLOBAL DEFAULT   13 _dl_argv
> 
> So LLD's librtld.os matches GNU ld for the two symbols.
> 
> 
> 
> The .set directive sets value/type but not binding/visibility.
> Perhaps sysdeps/arm/dl-machine.h should use
> 
> .word   __GI__dl_argv(GOTOFF)
> 
> instead of
> .word   _dl_argv(GOTOFF)
> 

It think it should be ok. So it seems that ld.bfd is being forgiving here
and binding to the internal alias. 

>>>
>>>> On sparcv9, lld handles --relax option different than ld.bfd: it throws
>>>> an error instead of silent ignoring it:
>>>>
>>>>  $ sparc64-glibc-linux-gnu-gcc -mcpu=niagara -fuse-ld=lld -Bclang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04/bin
>>>>    -g -O2 -fPIC -shared -fno-stack-protector -o conftest.so conftest.c -nostdlib -nostartfiles -Wl,-z,combreloc
>>>>  ld.lld: error: unknown argument '-relax'
>>>
>>> LLD's sparcv9 port is for retrocomputing fans:) and is far from usable
>>> (see https://lld.llvm.org/ "production quality").
>>> The GOT/PLT support has quite a few issues (https://reviews.llvm.org/D102985).
>>
>> Right, but lld is still showing a different ld.bfd semantic that might trigger
>> other issues where on platforms where neither --relax nor -no-relax it supported
>> ld.bfd accepts both argument but ignore it [1].
>>
>> At least on sparc, gcc does pass --relax and expects that static linker just
>> ignore if it is no supported.
>>
>> [1] https://sourceware.org/binutils/docs-2.37/ld.html
> 
> LLD's sparcv9 port is quite immature, so I cannot even advertise it.
> A large portion should be rewritten but the sparc port is only cared by very few
> retrocomputing fans now..

It would be good if we could detect it at configure time and just warn that linker
is not sufficient to build glibc.

> 
>>>
>>>> And even when -mno-relax is explicit add, lld does not support some relocations
>>>> generated by GCC:
>>>>
>>>>  $ sparc64-glibc-linux-gnu-readelf -Wr elf/librtld.os | grep _dl_skip_args
>>>>  | head -n2
>>>>  00000000000000a0  0000012200000052 R_SPARC_GOTDATA_OP_HIX22
>>>>  0000000000000058 _dl_skip_args + 0
>>>>  00000000000000a4  0000012200000053 R_SPARC_GOTDATA_OP_LOX10
>>>>  0000000000000058 _dl_skip_args + 0
>>>>
>>>> The mips/mipsel also fails to build the loader:
>>>>
>>>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>>  >>> defined in
>>>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x1C)
>>>>
>>>>  ld.lld: error: can't create dynamic relocation R_MIPS_32 against local
>>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>>  >>> defined in
>>>>  >>> /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os
>>>>  >>> referenced by /home/azanella/toolchain/src/gcc/libgcc/libgcc2.c
>>>>  >>>               /home/azanella/Projects/glibc/build/mips-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x54)
>>>
>>> lld.llvm.org says "MIPS seems decent too." and I personally know really
>>> little about MIPS.
>>>
>>> However, I think this is likely a genuine issue somewhere in sysdeps/mips/ .
>>> Relocation processing is rigid and consistent in LLD.
>>> GNU ld may miss some error checking for some relocation types.
>>>
>>>> Similar to mips64/mips64el:
>>>>
>>>>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>>>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>>>  against non-function symbol . This is invalid and most likely a compiler
>>>>  bug.
>>>>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>>  >>> defined in
>>>>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>>>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>>>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
>>>
>>> I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
>>> reporting a genuine issue.
>>
>> I give you that loader code is tricky, but it does not really explain why
>> lld is failing to link the eh_frame.  It *might* due some internal assembly
>> routines, but at least from logs it does not seem so.
> 
> Hope Joseph can answer the question.
> 
> If we have easy-to-follow instructions building glibc mips with LLD, I can
> forward it to Simon Atanasyan (MIPS code owner of llvm-project and the major
> contributor of LLD's MIPS port)...
> 
> But see below (for riscv/), I suspect this is a glibc sysdeps/ problem.
> 
>>>
>>>> The riscv32/riscv64 fails to the loader, but it is due missing support
>>>> to relaxation:
>>>>
>>>>  ld.lld: error:
>>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2584:(.text+0xEC84):
>>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>>  recompile with -mno-relax
>>>>  ld.lld: error:
>>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEC):
>>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>>  recompile with -mno-relax
>>>>  ld.lld: error:
>>>>  /home/azanella/Projects/glibc/glibc-git/elf/rtld.c:2589:(.text+0x13DEE):
>>>>  relocation R_RISCV_ALIGN requires unimplemented linker relaxation;
>>>>  recompile with -mno-rela
>>>
>>> LLD does not implement RISC-V linker relaxation.
>>> R_RISCV_ALIGN has weird linker unfriendly semantics:
>>> simply ignoring the relocation type can break semantics
>>> (https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/183).
>>>
>>>> Adding -mno-relax to sysdep-CFLAGS seems to be ineffective.
>>>
>>> So sysdep-CFLAGS is not a catch-all option affecting all C compiles...
>>
>> In fact it is, the problem was the autogenerated syscalls built from
>> the assembly.  I could avoid the R_RISCV_ALIGN with:
>>
>> diff --git a/sysdeps/riscv/Makefile b/sysdeps/riscv/Makefile
>> index 20a9968106..28dbd89cb0 100644
>> --- a/sysdeps/riscv/Makefile
>> +++ b/sysdeps/riscv/Makefile
>> @@ -5,3 +5,9 @@ endif
>> # RISC-V's assembler also needs to know about PIC as it changes the definition
>> # of some assembler macros.
>> ASFLAGS-.os += $(pic-ccflag)
>> +
>> +# lld does not implement R_RISCV_ALIGN relaxation optimization.
>> +ifeq (yes,$(with-lld))
>> +ASFLAGS-.os += -Wa,-mno-relax
>> +sysdep-CFLAGS += -mno-relax
>> +endif
>>
>>
>> However it does fail later for the libc.so with:
>>
>> ld.lld: warning: attempt to reassign symbol 'fcntl' of version 'GLIBC_2.27' to version 'GLIBC_2.28'
>> ld.lld: warning: attempt to reassign symbol '__sched_get_priority_min' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
>> ld.lld: warning: attempt to reassign symbol '__sched_get_priority_max' of version 'GLIBC_2.27' to version 'GLIBC_PRIVATE'
>> ld.lld: error: relocation R_RISCV_RVC_JUMP cannot be used against symbol __sigsetjmp; recompile with -fPIC
>>>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>>>> referenced by iconv.c:62 (/home/azanella/Projects/glibc/glibc-git/iconv/iconv.c:62)
>>>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__GI__setjmp)
>>
>> ld.lld: error: relocation R_RISCV_JAL cannot be used against symbol exit; recompile with -fPIC
>>>>> defined in /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean
>>>>> referenced by gconv_open.c:89 (/home/azanella/Projects/glibc/glibc-git/iconv/gconv_open.c:89)
>>>>>               /home/azanella/Projects/glibc/build/riscv64-linux-gnu-rv64imafdc-lp64d-lld/libc_pic.os.clean:(__start_context)
> 
> There is a GNU ld issue: https://sourceware.org/bugzilla/show_bug.cgi?id=28509
> It incorrectly allows non-PLT non-GOT relocation referencing a preemptible symbol.
> 
> 
> LLD uses consistent categories (e.g. R_PC / R_PLT_PC) to describe relocation types.
> It has consistent diagnostic across all ports it supports.
> GNU ld uses dispatches for each relocation type and may miss some diagnostics.
> 
>> collect2: error: ld returned 1 exit status

Indeed these are the same ARM issues, we will need to use internal symbols.
Tulio Magno Quites Machado Filho Oct. 28, 2021, 5:27 p.m. UTC | #9 
Fangrui Song via Libc-alpha <libc-alpha@sourceware.org> writes:

> You are right:) The incomplete PowerPC64 ELF v1 support was removed by
> https://reviews.llvm.org/D46316 (2018-05).  Is there remaining ELF v1
> usage for powerpc64le in glibc?

No. We reuse code whenever possible, but that's all.
However, without this support we can't build glibc for powerpc64 big-endian
(aka. powerpc64-linux).
Adhemerval Zanella Oct. 28, 2021, 5:40 p.m. UTC | #10 
On 27/10/2021 22:18, Fangrui Song wrote:
> On 2021-10-27, Fangrui Song wrote:
>>
>> On 2021-10-27, Adhemerval Zanella wrote:
>>>>> Similar to mips64/mips64el:
>>>>>
>>>>>  ld.lld: warning: ../sysdeps/unix/sysv/linux/setitimer.c:(function
>>>>>  _dl_runtime_resolve: .text+0x18954): found R_MIPS_JALR relocation
>>>>>  against non-function symbol . This is invalid and most likely a compiler
>>>>>  bug.
>>>>>  ld.lld: error: can't create dynamic relocation R_MIPS_64 against local
>>>>>  symbol in readonly segment; recompile object files with -fPIC or pass
>>>>>  '-Wl,-z,notext' to allow text relocations in the output
>>>>>  >>> defined in
>>>>>  >>> /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os
>>>>>  >>> referenced by ../sysdeps/unix/sysv/linux/setitimer.c
>>>>>  >>>               /home/azanella/Projects/glibc/build/mips64-linux-gnu-lld/elf/librtld.os:(.eh_frame+0x20)
>>>>
>>>> I took a look at LLD's R_MIPS_JALR code and I am inclined to trust it
>>>> reporting a genuine issue.
>>>
>>> I give you that loader code is tricky, but it does not really explain why
>>> lld is failing to link the eh_frame.  It *might* due some internal assembly
>>> routines, but at least from logs it does not seem so.
>>
>> Hope Joseph can answer the question.
>>
>> If we have easy-to-follow instructions building glibc mips with LLD, I can
>> forward it to Simon Atanasyan (MIPS code owner of llvm-project and the major
>> contributor of LLD's MIPS port)...
>>
>> But see below (for riscv/), I suspect this is a glibc sysdeps/ problem.
> 
> jrtc27 told me that BFD mips has a hack to rewrite R_MIPS_64 in
> .eh_frame to avoid such a R_MIPS_64 diagnostic. LLD doesn't implement it.
> 
> Clang's mips port emits relative relocations in .eh_frame (DW_EH_PE_pcrel | dwarf::DW_EH_PE_sdata4)
> to avoid the linker issue: https://reviews.llvm.org/D72228
> 
> Lack of 64-bit PC-relative relocation is a longstanding problem for mips.
> LLVM integrated assembler and linker actually support `.quad foo - .`
> via composed relocations, but it is still unsupported in GNU as
> (feature request: https://sourceware.org/bugzilla/show_bug.cgi?id=26222)
> 
> I know really little about MIPS but I'd wish that GNU as implements
> .quad foo-.  and GCC migrates to PC-relative relocations like Clang.

In any case, it would be good to a configure check to warn uses that
R_MIPS_64 on .eh_frame might no be supported and bail out if lld
idea is to not implement any workaround.