[0/4] Do not install shared objects under versioned names

  This is essentially a repost of the “Add --disable-major-minor-libraries
configure option” series.  Joseph suggested that the configure option is
not needed, so this version implements the change unconditionally.

Tested on i686-linux-gnu and x86_64-linux-gnu.  I compared two
build-many-glibcs.py trees with and without these patches, using this
command to see if there are missing files besides the versioned DSOs or
any dangling symbolic links.

cd /home/bmg/install/glibcs && find -printf '%P\n' \
  | while read x ; do
    test -r /home/bmg-install-glibcs/$x || echo $x
  done \
  | grep -v '\-2\.33\.9000\.so$' | grep -v '/libthread_db-1\.0\.so$'

/home/bmg/install/glibcs is the unpatched build,
/home/bmg-install-glibcs is the build with the patches applied.  As
expected, there was no output.

Thanks,
Florian

Florian Weimer (4):
  nptl_db: Install libthread_db under a regular implementation name
  Makerules: Remove lib-version, $(subdir-version)
  elf: Generalize name-based DSO recognition in ldconfig
  Install shared objects under their ABI names

 INSTALL             | 10 +++++++
 Makefile            |  6 ----
 Makerules           | 67 +++++----------------------------------------
 NEWS                |  8 ++++++
 elf/Makefile        | 12 ++------
 elf/dl-is_dso.h     | 33 ++++++++++++++++++++++
 elf/ldconfig.c      |  5 ++--
 elf/tst-dl-is_dso.c | 35 +++++++++++++++++++++++
 nptl_db/Makefile    |  2 --
 9 files changed, 98 insertions(+), 80 deletions(-)
 create mode 100644 elf/dl-is_dso.h
 create mode 100644 elf/tst-dl-is_dso.c
  

* Florian Weimer via Libc-alpha:

> This is essentially a repost of the “Add --disable-major-minor-libraries
> configure option” series.  Joseph suggested that the configure option is
> not needed, so this version implements the change unconditionally.
>
> Tested on i686-linux-gnu and x86_64-linux-gnu.  I compared two
> build-many-glibcs.py trees with and without these patches, using this
> command to see if there are missing files besides the versioned DSOs or
> any dangling symbolic links.
>
> cd /home/bmg/install/glibcs && find -printf '%P\n' \
>   | while read x ; do
>     test -r /home/bmg-install-glibcs/$x || echo $x
>   done \
>   | grep -v '\-2\.33\.9000\.so$' | grep -v '/libthread_db-1\.0\.so$'
>
> /home/bmg/install/glibcs is the unpatched build,
> /home/bmg-install-glibcs is the build with the patches applied.  As
> expected, there was no output.

What are general thoughts about this change?  Should we do it?

If there's an emerging consensus that we want it, I can backport it
downstream because we really need it there, and drop the patches once
they have been merged.  But I really don't want to carry another bunch
of downstream-only patches indefinitely.

Thanks,
Florian
  

On 6/10/21 1:52 PM, Florian Weimer via Libc-alpha wrote:
> This is essentially a repost of the “Add --disable-major-minor-libraries
> configure option” series.  Joseph suggested that the configure option is
> not needed, so this version implements the change unconditionally.
> 
> Tested on i686-linux-gnu and x86_64-linux-gnu.  I compared two
> build-many-glibcs.py trees with and without these patches, using this
> command to see if there are missing files besides the versioned DSOs or
> any dangling symbolic links.
> 
> cd /home/bmg/install/glibcs && find -printf '%P\n' \
>    | while read x ; do
>      test -r /home/bmg-install-glibcs/$x || echo $x
>    done \
>    | grep -v '\-2\.33\.9000\.so$' | grep -v '/libthread_db-1\.0\.so$'
> 
> /home/bmg/install/glibcs is the unpatched build,
> /home/bmg-install-glibcs is the build with the patches applied.  As
> expected, there was no output.

I was hoping that we would go in the opposite direction with this, i.e. 
encode a unique identifier in the names of the DSOs so that they can be 
installed in parallel, maybe with configuration options to pass the 
version string, similar to the kernel.  This could for example allow us 
to implement recovery from broken glibcs without having to reach out for 
a rescue disk.

However I understand that it would be quite a pain to implement and it 
doesn't seem to be on anyone's TODO list, certainly not mine.  So I 
won't get in the way of this since it solves present day issues with 
library updates.  It would be interesting to hear from other 
distribution maintainers on what they think of the change.

Siddhesh
  

On 6/14/21 10:49 AM, Siddhesh Poyarekar wrote:
> On 6/10/21 1:52 PM, Florian Weimer via Libc-alpha wrote:
>> This is essentially a repost of the “Add
>> --disable-major-minor-libraries configure option” series.  Joseph
>> suggested that the configure option is not needed, so this version
>> implements the change unconditionally.
>> 
>> Tested on i686-linux-gnu and x86_64-linux-gnu.  I compared two 
>> build-many-glibcs.py trees with and without these patches, using
>> this command to see if there are missing files besides the
>> versioned DSOs or any dangling symbolic links.
>> 
>> cd /home/bmg/install/glibcs && find -printf '%P\n' \ | while read x
>> ; do test -r /home/bmg-install-glibcs/$x || echo $x done \ | grep
>> -v '\-2\.33\.9000\.so$' | grep -v '/libthread_db-1\.0\.so$'
>> 
>> /home/bmg/install/glibcs is the unpatched build, 
>> /home/bmg-install-glibcs is the build with the patches applied.
>> As expected, there was no output.
> 
> I was hoping that we would go in the opposite direction with this,
> i.e. encode a unique identifier in the names of the DSOs so that they
> can be installed in parallel, maybe with configuration options to
> pass the version string, similar to the kernel.  This could for
> example allow us to implement recovery from broken glibcs without
> having to reach out for a rescue disk.

I think this is never going to happen. I'll tell you why. At one point I
thought it *would* happen, and I thought we needed it, but two things
prevent it:

- Downstream QE teams rightly argue that changing libc has significant
  impact that much of the higher level stack testing needs to be redone.
  Consider the recent spat of seccomp/glibc, firefox/glibc, chrome/glibc
  issues all around "internal" implementation details that touch against
  the sandboxing.

- Containers. In the past we might have said "Yeah, we need 2 libcs for
  the sake of two distinct requirements." Today we just run processes in
  containers. The general concept of "multiple libraries" I think is
  going to disappear underneath the improving container tooling. Rather
  than 2 libcs installed, we'll have two assembled containers, each with
  a different libc.
 
> However I understand that it would be quite a pain to implement and
> it doesn't seem to be on anyone's TODO list, certainly not mine.  So
> I won't get in the way of this since it solves present day issues
> with library updates.  It would be interesting to hear from other
> distribution maintainers on what they think of the change.

I think Florian is moving this in the *right* direction e.g. simple,
robust, one file. Without a more compelling use case for multiple
libcs in one setup I think this is the right move.
  

On 6/10/21 4:22 AM, Florian Weimer via Libc-alpha wrote:
> This is essentially a repost of the “Add --disable-major-minor-libraries
> configure option” series.  Joseph suggested that the configure option is
> not needed, so this version implements the change unconditionally.

This is absolutely the right direction.

I like that we simplify the layout.

I have commented on Siddhesh's thoughts in the other thread, and I think
we are never likely to implement a solution that installs parallel libcs
given the QE and container aspects.

Therefore I'd like to see this move forward.
 
> Tested on i686-linux-gnu and x86_64-linux-gnu.  I compared two
> build-many-glibcs.py trees with and without these patches, using this
> command to see if there are missing files besides the versioned DSOs or
> any dangling symbolic links.
> 
> cd /home/bmg/install/glibcs && find -printf '%P\n' \
>   | while read x ; do
>     test -r /home/bmg-install-glibcs/$x || echo $x
>   done \
>   | grep -v '\-2\.33\.9000\.so$' | grep -v '/libthread_db-1\.0\.so$'
> 
> /home/bmg/install/glibcs is the unpatched build,
> /home/bmg-install-glibcs is the build with the patches applied.  As
> expected, there was no output.
> 
> Thanks,
> Florian
> 
> Florian Weimer (4):
>   nptl_db: Install libthread_db under a regular implementation name
>   Makerules: Remove lib-version, $(subdir-version)
>   elf: Generalize name-based DSO recognition in ldconfig
>   Install shared objects under their ABI names
> 
>  INSTALL             | 10 +++++++
>  Makefile            |  6 ----
>  Makerules           | 67 +++++----------------------------------------
>  NEWS                |  8 ++++++
>  elf/Makefile        | 12 ++------
>  elf/dl-is_dso.h     | 33 ++++++++++++++++++++++
>  elf/ldconfig.c      |  5 ++--
>  elf/tst-dl-is_dso.c | 35 +++++++++++++++++++++++
>  nptl_db/Makefile    |  2 --
>  9 files changed, 98 insertions(+), 80 deletions(-)
>  create mode 100644 elf/dl-is_dso.h
>  create mode 100644 elf/tst-dl-is_dso.c
>
  

On 6/28/21 2:13 AM, Carlos O'Donell wrote:
> I think this is never going to happen. I'll tell you why. At one point I
> thought it *would* happen, and I thought we needed it, but two things
> prevent it:
> 
> - Downstream QE teams rightly argue that changing libc has significant
>    impact that much of the higher level stack testing needs to be redone.
>    Consider the recent spat of seccomp/glibc, firefox/glibc, chrome/glibc
>    issues all around "internal" implementation details that touch against
>    the sandboxing.
> 
> - Containers. In the past we might have said "Yeah, we need 2 libcs for
>    the sake of two distinct requirements." Today we just run processes in
>    containers. The general concept of "multiple libraries" I think is
>    going to disappear underneath the improving container tooling. Rather
>    than 2 libcs installed, we'll have two assembled containers, each with
>    a different libc.

I'm much less concerned about multiple system glibc versions being use 
in tandem; that may even be harmful.  My concern is recovery.  I agree 
that in practice recovery is much less of a common problem; I've managed 
to brick Fedora glibc only once (ok maybe twice) in my lifetime.  So 
maybe I'm overthinking it.

> I think Florian is moving this in the *right* direction e.g. simple,
> robust, one file. Without a more compelling use case for multiple
> libcs in one setup I think this is the right move.

I agree that if there's no way to usefully utilize the version numbers 
then it's a step forward to drop them.

Siddhesh