[v2,00/23] NSS, nscd updates (for group merging and more)

  This is an enhanced rework of my previous series:

  RFC: Towards deprecating the ERANGE protocol for NSS
  <https://inbox.sourceware.org/libc-alpha/cover.1764012721.git.fweimer@redhat.com/>

The new version covers passwd and group.  It should have a proper
software TM for the shared nscd mapping for those two nscd databases,
following the same approach that we use in _dl_find_object.

The existing nscd test is enhanced with checks that the shared cache
actually works, by examining the statistics counters.

In the nscd client code, I switched the ad-hoc read-write lock to a the
real one from <pthread.h> (still under the __libc_rwlock_* alias).  This
simplifies the code quite a bit and removes a few corner cases where the
old code was busy-waiting.

The reworked group merging code is still included, that part has not
changed.

One thing that is annoying is the GLIBC_2.1.2 compatibility version that
we need for the *_r functions.  I had not noticed the need for that
before.  This is quite a blob of extra code per *_r function, so I think
we should generate that, but perhaps not with the preprocessor.

Thanks,
Florian

Florian Weimer (23):
  support: Update support_format_* NSS formatting routines for null
    pointers
  nscd: Extract dbtype and struct statdata types to separate headers
  nscd: Enhance tst-nscd-basic with query counters
  nss: Add negative lookup test
  nscd: Use a real read-write lock for the client mapping
  nscd: Re-flow and sort Makefile
  nss: Add __nss_generic_copy and __nss_generic_dup functions
  nscd: Fix data races in client retry counters (bug 33654)
  Add <parse_buffer.h>
  nscd: Use uint8_t instead of bitfield in struct hashentry
  nscd: Add <concurrent_buffer.h> for nscd client usage
  nscd: Rewrite __nscd_cache_search using <concurrent_buffer.h>
  nscd: Introduce __nscd_read_from_socket client function
  nscd: Consistant snapshot with client function __nscd_read_from_cache
  nscd: Add __nscd_generic_get for generic lookup
  nscd: Convert group client to __nscd_generic_get
  nscd: Convert passwd client to __nscd_generic_get
  Extract <stringtable.h> from ldconfig
  nss: Low-level functionality for merging group lists
  nss: Convert passwd database to new NSS framework
  nss: Convert group database to new NSS framework
  nscd: Remove most of the rebuilds of existing libc functions
  nss: Remove old group merge code

 elf/Makefile                                  |   4 +-
 elf/cache.c                                   |  28 +-
 elf/cachestrings.c                            | 149 ++++
 elf/{stringtable.h => cachestrings.h}         |  40 +-
 elf/stringtable.c                             | 209 -----
 elf/stringtable_free.c                        |  33 -
 elf/{tst-stringtable.c => tst-cachestrings.c} | 128 +--
 include/parse_buffer.h                        | 213 +++++
 include/set-freeres.h                         |  11 +-
 include/stringtable-skeleton.h                | 129 +++
 include/stringtable.h                         |  55 ++
 malloc/set-freeres.c                          |  22 +-
 misc/Makefile                                 |   3 +
 misc/fnv1a.c                                  |  32 +
 misc/stringtable_add.c                        | 128 +++
 misc/stringtable_free.c                       |  44 +
 nscd/Makefile                                 |  46 +-
 nscd/cache.c                                  |  41 +-
 nscd/concurrent_buffer.h                      | 197 +++++
 nscd/getgrgid_r.c                             |  34 -
 nscd/getgrnam_r.c                             |  33 -
 nscd/getpwnam_r.c                             |  30 -
 nscd/getsrvbynm_r.c                           |  29 -
 nscd/getsrvbypt_r.c                           |  29 -
 nscd/grpcache.c                               |   4 +-
 nscd/mem.c                                    |   7 +-
 nscd/nscd-client.h                            | 157 ++--
 nscd/nscd-dbtype.h                            |  35 +
 nscd/nscd-statdata.h                          |  99 +++
 nscd/nscd.h                                   |  18 +-
 nscd/nscd_getai.c                             |  48 +-
 nscd/nscd_getgr_r.c                           | 286 +------
 nscd/nscd_gethst_r.c                          |  95 +--
 nscd/nscd_getpw_r.c                           | 199 +----
 nscd/nscd_getserv_r.c                         |  62 +-
 nscd/nscd_helper.c                            | 793 +++++++++++++++---
 nscd/nscd_initgroups.c                        |  37 +-
 nscd/nscd_netgroup.c                          |  82 +-
 nscd/nscd_proto.h                             |  17 +-
 nscd/nscd_stat.c                              |  73 +-
 nscd/pwdcache.c                               |   4 +-
 nscd/servicescache.c                          |   4 +-
 nscd/tst-nscd-basic.c                         |  58 ++
 nss/Makefile                                  |  25 +-
 nss/getXXbyYY_r.c                             | 118 +--
 nss/getaddrinfo.c                             |  19 +-
 nss/getgrgid.c                                |  14 +-
 nss/getgrgid_r.c                              |  36 +-
 nss/getgrnam.c                                |  14 +-
 nss/getgrnam_r.c                              |  40 +-
 nss/getnetgrent_r.c                           |  15 +-
 nss/getpwnam.c                                |  14 +-
 nss/getpwnam_r.c                              |  33 +-
 nss/getpwuid.c                                |  14 +-
 nss/getpwuid_r.c                              |  33 +-
 nss/grp-merge.c                               | 200 -----
 nss/initgroups.c                              |  10 +-
 nss/nss-lookups.def                           |  28 +
 nss/nss_database.c                            |  23 +-
 nss/nss_generic.h                             | 146 ++++
 nss/nss_generic_copy.c                        | 120 +++
 nss/nss_generic_dup.c                         | 125 +++
 nss/nss_generic_get.c                         |  61 ++
 nss/nss_generic_get_r.c                       |  45 +
 nss/nss_generic_lookup.c                      |  51 ++
 nss/nss_generic_next.c                        |  60 ++
 nss/nss_generic_nscd.c                        |  29 +
 nss/{grp-merge.h => nss_generic_storage.h}    |  26 +-
 nss/nss_getX.c                                |  53 ++
 nss/nss_getX_r.c                              |  45 +
 nss/nss_getXinfo.c                            |  77 ++
 nss/nss_getgrXinfo.c                          | 176 ++++
 nss/nss_group_members.c                       |  80 ++
 nss/nss_group_members.h                       |  61 ++
 nss/nss_module.c                              |   8 +-
 nss/nsswitch.c                                |   5 -
 nss/nsswitch.h                                |  20 +-
 nss/tst-nss-does-not-exist.cc                 | 210 +++++
 nss/tst-nss-test4.c                           |  12 +-
 nss/tst-nss_generic_copy.c                    | 241 ++++++
 nss/tst-nss_generic_dup.c                     | 231 +++++
 nss/tst-nss_group_members.c                   |  77 ++
 support/Makefile                              |   1 +
 support/nscd_test.h                           |   4 +
 support/support_format_group.c                |   7 +-
 support/support_format_hostent.c              |  31 +-
 support/support_format_netent.c               |   9 +-
 support/support_format_protoent.c             |  11 +-
 support/support_format_rpcent.c               |   8 +-
 support/support_format_servent.c              |   9 +-
 support/support_format_sgrp.c                 |  16 +-
 support/support_nscd.c                        |  33 +
 .../support_stringtable.c                     |  20 +-
 93 files changed, 4474 insertions(+), 2015 deletions(-)
 create mode 100644 elf/cachestrings.c
 rename elf/{stringtable.h => cachestrings.h} (53%)
 delete mode 100644 elf/stringtable.c
 delete mode 100644 elf/stringtable_free.c
 rename elf/{tst-stringtable.c => tst-cachestrings.c} (54%)
 create mode 100644 include/parse_buffer.h
 create mode 100644 include/stringtable-skeleton.h
 create mode 100644 include/stringtable.h
 create mode 100644 misc/fnv1a.c
 create mode 100644 misc/stringtable_add.c
 create mode 100644 misc/stringtable_free.c
 create mode 100644 nscd/concurrent_buffer.h
 delete mode 100644 nscd/getgrgid_r.c
 delete mode 100644 nscd/getgrnam_r.c
 delete mode 100644 nscd/getpwnam_r.c
 delete mode 100644 nscd/getsrvbynm_r.c
 delete mode 100644 nscd/getsrvbypt_r.c
 create mode 100644 nscd/nscd-dbtype.h
 create mode 100644 nscd/nscd-statdata.h
 delete mode 100644 nss/grp-merge.c
 create mode 100644 nss/nss-lookups.def
 create mode 100644 nss/nss_generic.h
 create mode 100644 nss/nss_generic_copy.c
 create mode 100644 nss/nss_generic_dup.c
 create mode 100644 nss/nss_generic_get.c
 create mode 100644 nss/nss_generic_get_r.c
 create mode 100644 nss/nss_generic_lookup.c
 create mode 100644 nss/nss_generic_next.c
 create mode 100644 nss/nss_generic_nscd.c
 rename nss/{grp-merge.h => nss_generic_storage.h} (55%)
 create mode 100644 nss/nss_getX.c
 create mode 100644 nss/nss_getX_r.c
 create mode 100644 nss/nss_getXinfo.c
 create mode 100644 nss/nss_getgrXinfo.c
 create mode 100644 nss/nss_group_members.c
 create mode 100644 nss/nss_group_members.h
 create mode 100644 nss/tst-nss-does-not-exist.cc
 create mode 100644 nss/tst-nss_generic_copy.c
 create mode 100644 nss/tst-nss_generic_dup.c
 create mode 100644 nss/tst-nss_group_members.c
 rename nscd/getpwuid_r.c => support/support_stringtable.c (64%)


base-commit: 19781c22211c6db827eefc240cb6724b5512fc6b
  

On 3/20/26 4:40 PM, Florian Weimer wrote:
> This is an enhanced rework of my previous series:
> 
>    RFC: Towards deprecating the ERANGE protocol for NSS
>    <https://inbox.sourceware.org/libc-alpha/cover.1764012721.git.fweimer@redhat.com/>
> 
> The new version covers passwd and group.  It should have a proper
> software TM for the shared nscd mapping for those two nscd databases,
> following the same approach that we use in _dl_find_object.
> 
> The existing nscd test is enhanced with checks that the shared cache
> actually works, by examining the statistics counters.
> 
> In the nscd client code, I switched the ad-hoc read-write lock to a the
> real one from <pthread.h> (still under the __libc_rwlock_* alias).  This
> simplifies the code quite a bit and removes a few corner cases where the
> old code was busy-waiting.
> 
> The reworked group merging code is still included, that part has not
> changed.
> 
> One thing that is annoying is the GLIBC_2.1.2 compatibility version that
> we need for the *_r functions.  I had not noticed the need for that
> before.  This is quite a blob of extra code per *_r function, so I think
> we should generate that, but perhaps not with the preprocessor.

Fails pre-commit CI. May you please have a look?
https://patchwork.sourceware.org/project/glibc/patch/05e0a557343d05d4406b3776e13af4b1147b1541.1774037705.git.fweimer@redhat.com/

The failure is across all 3 testers so it looks reproducible.
  

* Carlos O'Donell:

> On 3/20/26 4:40 PM, Florian Weimer wrote:
>> This is an enhanced rework of my previous series:
>>    RFC: Towards deprecating the ERANGE protocol for NSS
>>    <https://inbox.sourceware.org/libc-alpha/cover.1764012721.git.fweimer@redhat.com/>
>> The new version covers passwd and group.  It should have a proper
>> software TM for the shared nscd mapping for those two nscd databases,
>> following the same approach that we use in _dl_find_object.
>> The existing nscd test is enhanced with checks that the shared cache
>> actually works, by examining the statistics counters.
>> In the nscd client code, I switched the ad-hoc read-write lock to a
>> the
>> real one from <pthread.h> (still under the __libc_rwlock_* alias).  This
>> simplifies the code quite a bit and removes a few corner cases where the
>> old code was busy-waiting.
>> The reworked group merging code is still included, that part has not
>> changed.
>> One thing that is annoying is the GLIBC_2.1.2 compatibility version
>> that
>> we need for the *_r functions.  I had not noticed the need for that
>> before.  This is quite a blob of extra code per *_r function, so I think
>> we should generate that, but perhaps not with the preprocessor.
>
> Fails pre-commit CI. May you please have a look?
> https://patchwork.sourceware.org/project/glibc/patch/05e0a557343d05d4406b3776e13af4b1147b1541.1774037705.git.fweimer@redhat.com/
>
> The failure is across all 3 testers so it looks reproducible.

The test nss/tst-nss-does-not-exist fails with systemd-resolved because
due to the unqualified host name for negative testing, it search
who-knows-where for a name.  Switching to does-not-exist.example.com or
similar will fix this.

The nss/tst-getpw failure is more interesting.  I followed the rule “no
library function sets errno to zero”, but this is not what the test
expects:

  ret = getpw (uid, buf);

  /* Successfully read a password line.  */
  if (ret == 0 && !seen_hit)
    {
      printf ("PASS: Read a password line given a uid.\n");
      seen_hit = true;
    }

  /* Failed to read a password line. Why?  */
  if (ret == -1)
    {
      /* No entry?  Technically the errno could be any number
	 of values including ESRCH, EBADP or EPERM depending
	 on the quality of the nss module that implements the
	 underlying lookup. It should be 0 for getpw.*/
      if (errno == 0 && !seen_miss)
	{
	  printf ("PASS: Found an invalid uid.\n");
	  seen_miss = true;
	  return;
	}

So this appears to assume that errno is set to zero on failure.  The
test doesn't set errno to zero, and with a non-zero errno (ENOENT in my
case), it gets stuck.  This again depends on the system's NSS modules.

I don't know if this is a test bug.  I need to figure out where the zero
value comes from before my changes.

Thanks,
Florian