[RFC,0/2] On ldconfig and ld.so.cache

  Hello,

having set up a very basic x86_64-gnu system to debug startup issues, I
was surprised to discover that my self-built ld.so does not look for the
shared libraries in /lib/x86_64-gnu/ (which is where Samuel Thibault's deb
packages place them) at all. I then learned that ld.so.cache and ldconfig
and ld.so.conf support is explicitly getting compiled out in the *-gnu
configurations -- and that this is being done intentionally, since
ldconfig is apparently considered to be a hack and a misfeature [0].

[0]: https://lists.debian.org/debian-hurd/2001/04/msg00179.html

(Might this be just due to misunderstanding of what ldconfig does? It's
not only about creating symlinks, it's primarily about the ld.so.cache and
ld.so.conf...)

<rant>

This doesn't really make sense to me: surely whether ld.so.cache is a
good idea or not is not tied to a particular kernel? Surely the kernel
could not care less about things like the file dystem layout and how
ld.so looks for shared libraries? Predicating ld.so.cache usage on whether
the GNU system is running on the Linux or Hurd kernel sounds like creating
pointless differences to me.

Moreover, Debian GNU/Hurd, the primary Hurd-based distribution, has been
shipping ld.so.cache on Hurd as a downstream patch [1] (note that more
changes would be required for x86_64-gnu because of FLAG_X8664_LIB64).
They don't really have a choice, it seems: with their "multiarch"
filesystem layout, the shared libraries are to be located in
/lib/i386-gnu/, but that is not a path that ld.so searches for libraries
in! This is solved by use_ldconfig=yes, and putting /lib/i386-gnu/ into
/etc/ld.so.conf.d/i386-gnu.conf, where it is then picked up by ldconfig,
which finds all the libraries and bakes their paths into the cache, where
ld.so then picks them up.

[1]: https://salsa.debian.org/glibc-team/glibc/-/raw/sid/debian/patches/hurd-i386/local-enable-ldconfig.diff

This is also another thing that suprises me: how come ldconfig does read
ld.so.conf, but ld.so does not? It's not an "ldconfig.conf" after all...
How is one even supposed to configure library paths with use_ldconfig=no?

</rant>

Anyway, maybe there are valid technical reasons to not enable ldconfig by
default; this is not a hill that I'm willing to die on. But wouldn't it be
nicer if it was easier to enable ldconfig downstream if desired? To that
end, I tweaked ldconfig.c to not use PATH_MAX (sadly, there are still
PATH_MAX usages in other places, e.g. chroot_canon.c), and moved a couple
of files to stop being specific to the Linux port. I hope that both of
these changes are uncontroversial. With these changes it is possible to
just patch in use_ldconfig=yes downstream (is desired, of course) and get
an ld.so.cache-enabled glibc on both i386-gnu and x86_64-gnu.

Sergey
  

On 5/17/23 14:54, Sergey Bugaev via Libc-alpha wrote:
> Hello,
> 
> having set up a very basic x86_64-gnu system to debug startup issues, I
> was surprised to discover that my self-built ld.so does not look for the
> shared libraries in /lib/x86_64-gnu/ (which is where Samuel Thibault's deb
> packages place them) at all. I then learned that ld.so.cache and ldconfig
> and ld.so.conf support is explicitly getting compiled out in the *-gnu
> configurations -- and that this is being done intentionally, since
> ldconfig is apparently considered to be a hack and a misfeature [0].
> 
> [0]: https://lists.debian.org/debian-hurd/2001/04/msg00179.html
> 
> (Might this be just due to misunderstanding of what ldconfig does? It's
> not only about creating symlinks, it's primarily about the ld.so.cache and
> ld.so.conf...)
> 
> <rant>
> 
> This doesn't really make sense to me: surely whether ld.so.cache is a
> good idea or not is not tied to a particular kernel? Surely the kernel
> could not care less about things like the file dystem layout and how
> ld.so looks for shared libraries? Predicating ld.so.cache usage on whether
> the GNU system is running on the Linux or Hurd kernel sounds like creating
> pointless differences to me.

I agree that we should reduce differences here, and that the data files used by the
loader should be more-or-less the same across kernels.

Keep in mind that ld.so.cache is not actually a cache, but a data file that contains
the pre-processed contents of /etc/ld.so.conf* in a format that can be easily loaded
by the dynamic loader without having a parser (glob included) in the early startup
code. Deleting the cache can break a system, and perhaps this is what some developers
have objected to and called "a hack."

I filed a bug for this in 2017 to capture the issue:
ld.so's cache should live in /var/cache, and support cache deletion.
https://sourceware.org/bugzilla/show_bug.cgi?id=22359

> Moreover, Debian GNU/Hurd, the primary Hurd-based distribution, has been
> shipping ld.so.cache on Hurd as a downstream patch [1] (note that more
> changes would be required for x86_64-gnu because of FLAG_X8664_LIB64).
> They don't really have a choice, it seems: with their "multiarch"
> filesystem layout, the shared libraries are to be located in
> /lib/i386-gnu/, but that is not a path that ld.so searches for libraries
> in! This is solved by use_ldconfig=yes, and putting /lib/i386-gnu/ into
> /etc/ld.so.conf.d/i386-gnu.conf, where it is then picked up by ldconfig,
> which finds all the libraries and bakes their paths into the cache, where
> ld.so then picks them up.

From 2018 we have this: 
Implement multiarch ldconfig
https://sourceware.org/bugzilla/show_bug.cgi?id=22825


> [1]: https://salsa.debian.org/glibc-team/glibc/-/raw/sid/debian/patches/hurd-i386/local-enable-ldconfig.diff
> 
> This is also another thing that suprises me: how come ldconfig does read
> ld.so.conf, but ld.so does not? It's not an "ldconfig.conf" after all...
> How is one even supposed to configure library paths with use_ldconfig=no?

The file /etc/ld.so.cache is not a cache, it is a data file used by the dynamic loader.

The data file is generated by ldconfig as it parses /etc/ld.so.conf*.

The only other way to configure library paths with `use_ldconfig=` is to use LD_PRELOAD,
LD_LIBRARY_PATH, DT_RPATH, or DT_RUNPATH.


> </rant>
> 
> Anyway, maybe there are valid technical reasons to not enable ldconfig by
> default; this is not a hill that I'm willing to die on. But wouldn't it be
> nicer if it was easier to enable ldconfig downstream if desired? To that
> end, I tweaked ldconfig.c to not use PATH_MAX (sadly, there are still
> PATH_MAX usages in other places, e.g. chroot_canon.c), and moved a couple
> of files to stop being specific to the Linux port. I hope that both of
> these changes are uncontroversial. With these changes it is possible to
> just patch in use_ldconfig=yes downstream (is desired, of course) and get
> an ld.so.cache-enabled glibc on both i386-gnu and x86_64-gnu.

Removing configuration options and making it simple to configure and use glibc is great
goal. I think that ldconfig should always be enabled and I don't see a downside to making
`use_ldconfig=yes` the default, but I think we'd have to check with Guix or Nix to see if
they have any custom changes there. It involves probably a slightly broader distro
discussion.
  

* Sergey Bugaev via Libc-alpha:

> Moreover, Debian GNU/Hurd, the primary Hurd-based distribution, has been
> shipping ld.so.cache on Hurd as a downstream patch [1] (note that more
> changes would be required for x86_64-gnu because of FLAG_X8664_LIB64).
> They don't really have a choice, it seems: with their "multiarch"
> filesystem layout, the shared libraries are to be located in
> /lib/i386-gnu/, but that is not a path that ld.so searches for libraries
> in! This is solved by use_ldconfig=yes, and putting /lib/i386-gnu/ into
> /etc/ld.so.conf.d/i386-gnu.conf, where it is then picked up by ldconfig,
> which finds all the libraries and bakes their paths into the cache, where
> ld.so then picks them up.
>
> [1]: https://salsa.debian.org/glibc-team/glibc/-/raw/sid/debian/patches/hurd-i386/local-enable-ldconfig.diff
>
> This is also another thing that suprises me: how come ldconfig does read
> ld.so.conf, but ld.so does not? It's not an "ldconfig.conf" after all...
> How is one even supposed to configure library paths with use_ldconfig=no?

The cache isn't really a cache, it's required for correct operation.

Debian hasn't upstreamed there multi-arch path layouts.  We could
implement multi-arch ldconfig in /etc/ld.so.cache, but with the paths
that Debian currently uses, it's not easy because there's no automated
way ldconfig can recognize the relevant subdirectories.  There's no
intermediate directly like “…/glibc-hwcaps/…” that could serve as a
marker.  I suppose we could look for subdirectories containing libc.so.6
files and its variants as an approximation …

Multi-arch /etc/ld.so.cache needs some new on-disk data structures.

Thanks,
Florian
  

Hello,

On Fri, May 19, 2023 at 3:30 PM Florian Weimer <fweimer@redhat.com> wrote:
> Debian hasn't upstreamed there multi-arch path layouts.  We could
> implement multi-arch ldconfig in /etc/ld.so.cache, but with the paths
> that Debian currently uses, it's not easy because there's no automated
> way ldconfig can recognize the relevant subdirectories.  There's no
> intermediate directly like “…/glibc-hwcaps/…” that could serve as a
> marker.  I suppose we could look for subdirectories containing libc.so.6
> files and its variants as an approximation …

Since you're taking my little rant seriously: I'm not arguing that
multi-arch path layouts should be upstreamed or that ld.so should
support that natively likely it does hwcaps. What I'm saying is:

* There is a clear need to configure custom (downstream- or installation-
  specific) paths where shared libraries are to be loaded from, other
  than just /lib and /lib64;

* /etc/ld.so.conf is the proper, well-supported, and popular way to do
  just that, and it is being used by many (if not all) major distros
  for this exact purpose -- in Debian's case, for enabling multiarch
  layout, among other things;

* with use_ldconfig=no, which is the upstream default for non-Linux,
  there does not seem to be a way to achieve that (without resorting to
  LD_LIBRARY_PATH or something like that).

So if use_ldconfig=no is to be kept, there should be another
recommended and supported way to configure search directories. Making
ld.so read ld.so.conf (parsing/globbing questions notwithstanding)
might be one such way.

Or maybe use_ldconfig should just get enabled everywhere :)

On Fri, May 19, 2023 at 2:52 PM Carlos O'Donell <carlos@redhat.com> wrote:
> Removing configuration options and making it simple to configure and use glibc is great
> goal. I think that ldconfig should always be enabled and I don't see a downside to making
> `use_ldconfig=yes` the default, but I think we'd have to check with Guix or Nix to see if
> they have any custom changes there. It involves probably a slightly broader distro
> discussion.

cc'ing Ludovic; but again, why would it be convenient for Nix/Guix to
have use_ldconfig enabled when building their distro with Linux but
not with the Hurd? Surely they'd rather prefer it to be consistent?

I don't think the reason for use_ldconfig=no default is about Guix...
but rather about the ideas that the original Hurd developers have had
for the design of the GNU system (see also: /usr vs /). That hasn't
really panned out; but I understand if the upstream defaults are still
kept according to those plans. Still, using use_ldconfig=yes with Hurd
needs to be better supported, since that's what Hurd distros actually
do in practice (I think -- well, at least Debian does), and if
use_ldconfig=no is to be kept supported, there has to be a way to
configure library search directories without ldconfig.

Sergey
  

Hi,

Sergey Bugaev <bugaevc@gmail.com> skribis:

> On Fri, May 19, 2023 at 2:52 PM Carlos O'Donell <carlos@redhat.com> wrote:
>> Removing configuration options and making it simple to configure and use glibc is great
>> goal. I think that ldconfig should always be enabled and I don't see a downside to making
>> `use_ldconfig=yes` the default, but I think we'd have to check with Guix or Nix to see if
>> they have any custom changes there. It involves probably a slightly broader distro
>> discussion.

I missed the beginning of the discussion and I fear I’m slightly
changing topics :-), but to me we should work toward per-application
loader caches:

  https://guix.gnu.org/en/blog/2021/taming-the-stat-storm-with-a-loader-cache/
  https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/glibc-dl-cache.patch?id=6d0571215d661d21cac2150ca45906e77a79a5fb

This is the one custom change we have in Guix.

(And IMO it should work the same on Linux and on the Hurd.)

Ludo’.