riscv: Add glibc-hwcaps support

Message ID 20230428191200.3241429-1-evan@rivosinc.com
State Superseded
Headers
Series riscv: Add glibc-hwcaps support |

Checks

Context Check Description
dj/TryBot-apply_patch fail Patch failed to apply to master at the time it was sent
dj/TryBot-32bit fail Patch series failed to apply

Commit Message

Evan Green April 28, 2023, 7:12 p.m. UTC
  Create a single subdirectory, rv64-v1, which defines a set of commonly
implemented extensions in already shipping platforms: rv64gc_Zba_Zbb.
This allows for easier shipping of optimized libraries for the most
common subset of features above the baseline.

For other architectures that use glibc-hwcaps, there's a well defined
progression of architecture features, where each generation fully
includes the previous generations and adds more. With RISC-V, we run the
risk of this becoming a bitfield, which would get out of hand almost
instantly. If we're careful in defining these generations, we can
hopefully maintain a linear progression without devolving into a
combinatorial mess. This patch deliberately aims low by defining a set
of features that already exist in many shipping application processors.

Signed-off-by: Evan Green <evan@rivosinc.com>
---

Note that this series depends on the first two patches of my other
series introducing the Linux hwprobe API [1]. The kernel series has been
applied into Palmer's tree, so barring anything unforeseen I expect that
to be fully landed shortly. It also depends on a Linux series I just
posted upstream to expose the Zba and Zbb features through hwprobe [2].

[1] https://public-inbox.org/libc-alpha/20230407230711.2621614-1-evan@rivosinc.com/
[2] https://lore.kernel.org/lkml/20230428190609.3239486-1-evan@rivosinc.com/T/#t

---
 elf/Makefile                                |  2 +-
 elf/tst-glibc-hwcaps-cache.script           |  3 ++
 sysdeps/riscv/rv64/Makefile                 | 21 ++++++++
 sysdeps/riscv/rv64/dl-hwcaps-subdirs.c      | 49 +++++++++++++++++
 sysdeps/riscv/rv64/tst-glibc-hwcaps.c       | 58 +++++++++++++++++++++
 sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h |  2 +
 6 files changed, 134 insertions(+), 1 deletion(-)
 create mode 100644 sysdeps/riscv/rv64/Makefile
 create mode 100644 sysdeps/riscv/rv64/dl-hwcaps-subdirs.c
 create mode 100644 sysdeps/riscv/rv64/tst-glibc-hwcaps.c
  

Comments

Florian Weimer April 28, 2023, 8:54 p.m. UTC | #1
* Evan Green:

> Note that this series depends on the first two patches of my other
> series introducing the Linux hwprobe API [1]. The kernel series has been
> applied into Palmer's tree, so barring anything unforeseen I expect that
> to be fully landed shortly. It also depends on a Linux series I just
> posted upstream to expose the Zba and Zbb features through hwprobe [2].
>
> [1] https://public-inbox.org/libc-alpha/20230407230711.2621614-1-evan@rivosinc.com/
> [2] https://lore.kernel.org/lkml/20230428190609.3239486-1-evan@rivosinc.com/T/#t

The mapping between rv64gc_Zba_Zbb, rv64-v1, and whatever feature flags
the hardware actually offers should ideally be documented somewhere
separately from glibc.  This decision has relevance beyond glibc and the
GNU toolchain, and the glibc consensus mechanisms seem insufficient.

For x86-64, what you called “well-defined” in your commit message is
merely such an agreement.  We documented it in the psABI manual, and
tried to involve as many relevant parties as possible prior to that.

> diff --git a/sysdeps/riscv/rv64/tst-glibc-hwcaps.c b/sysdeps/riscv/rv64/tst-glibc-hwcaps.c
> new file mode 100644
> index 0000000000..988d3492ef
> --- /dev/null
> +++ b/sysdeps/riscv/rv64/tst-glibc-hwcaps.c
> @@ -0,0 +1,58 @@
> +/* glibc-hwcaps subdirectory test.  RISC-V 64-bit version.

> +/* Return the arch level, 10 for the baseline libmarkermod*.so's.  */

The 10 here refers to the 10th edition of the Principles of Operation
(the first edition was for S/360).  It certainly has no meaning for
RISC-V.  I suggest to write this test differently.  It's difficult to
future-proof it if you don't know what the future will look like.

There's a space missing after __riscv_hwprobe (in two locations).

The mechanics of the patch looks okay to me based on a quick glance, but
I have not tested it.

Thanks,
Florian
  
Vineet Gupta April 28, 2023, 9:04 p.m. UTC | #2
+CC Kito

On 4/28/23 12:12, Evan Green wrote:
> Create a single subdirectory, rv64-v1, which defines a set of commonly
> implemented extensions in already shipping platforms: rv64gc_Zba_Zbb.
> This allows for easier shipping of optimized libraries for the most
> common subset of features above the baseline.

Just to be clear, it sucks to not have zbs as baseline, but the quirk is 
some of the recent hardware (sifive?) lacks zbs.

-Vineet


> For other architectures that use glibc-hwcaps, there's a well defined
> progression of architecture features, where each generation fully
> includes the previous generations and adds more. With RISC-V, we run the
> risk of this becoming a bitfield, which would get out of hand almost
> instantly. If we're careful in defining these generations, we can
> hopefully maintain a linear progression without devolving into a
> combinatorial mess. This patch deliberately aims low by defining a set
> of features that already exist in many shipping application processors.
>
> Signed-off-by: Evan Green <evan@rivosinc.com>
> ---
>
> Note that this series depends on the first two patches of my other
> series introducing the Linux hwprobe API [1]. The kernel series has been
> applied into Palmer's tree, so barring anything unforeseen I expect that
> to be fully landed shortly. It also depends on a Linux series I just
> posted upstream to expose the Zba and Zbb features through hwprobe [2].
>
> [1] https://public-inbox.org/libc-alpha/20230407230711.2621614-1-evan@rivosinc.com/
> [2] https://lore.kernel.org/lkml/20230428190609.3239486-1-evan@rivosinc.com/T/#t
>
> ---
>   elf/Makefile                                |  2 +-
>   elf/tst-glibc-hwcaps-cache.script           |  3 ++
>   sysdeps/riscv/rv64/Makefile                 | 21 ++++++++
>   sysdeps/riscv/rv64/dl-hwcaps-subdirs.c      | 49 +++++++++++++++++
>   sysdeps/riscv/rv64/tst-glibc-hwcaps.c       | 58 +++++++++++++++++++++
>   sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h |  2 +
>   6 files changed, 134 insertions(+), 1 deletion(-)
>   create mode 100644 sysdeps/riscv/rv64/Makefile
>   create mode 100644 sysdeps/riscv/rv64/dl-hwcaps-subdirs.c
>   create mode 100644 sysdeps/riscv/rv64/tst-glibc-hwcaps.c
>
> diff --git a/elf/Makefile b/elf/Makefile
> index 396ec51424..f00061852b 100644
> --- a/elf/Makefile
> +++ b/elf/Makefile
> @@ -2620,7 +2620,7 @@ $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
>   # glibc-hwcaps mechanism for this architecture).  Used to obtain test
>   # coverage for some glibc-hwcaps tests for the widest possible range
>   # of systems.
> -glibc-hwcaps-first-subdirs-for-tests = power9 x86-64-v2 z13
> +glibc-hwcaps-first-subdirs-for-tests = power9 x86-64-v2 z13 rv-v1
>   
>   # The test modules are parameterized by preprocessor macros.
>   LDFLAGS-libmarkermod1-1.so += -Wl,-soname,libmarkermod1.so
> diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
> index d58fc8c5de..ea3d0186c9 100644
> --- a/elf/tst-glibc-hwcaps-cache.script
> +++ b/elf/tst-glibc-hwcaps-cache.script
> @@ -36,3 +36,6 @@ mkdirp 0770 $L/glibc-hwcaps/x86-64-v4
>   cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod4.so
>   cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/x86-64-v3/libmarkermod4.so
>   cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/x86-64-v4/libmarkermod4.so
> +
> +mkdirp 0770 $L/glibc-hwcaps/rv64-v1
> +cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/rv64-v1/libmarkermod2.so
> diff --git a/sysdeps/riscv/rv64/Makefile b/sysdeps/riscv/rv64/Makefile
> new file mode 100644
> index 0000000000..15658ba26e
> --- /dev/null
> +++ b/sysdeps/riscv/rv64/Makefile
> @@ -0,0 +1,21 @@
> +
> +ifeq ($(subdir),elf)
> +
> +$(objpfx)tst-glibc-hwcaps: \
> +  $(objpfx)libmarkermod2-1.so
> +
> +$(objpfx)tst-glibc-hwcaps.out: \
> +  $(objpfx)libmarkermod2.so \
> +    $(objpfx)glibc-hwcaps/rv64-v1/libmarkermod2.so \
> +
> +$(objpfx)glibc-hwcaps/rv64-v1/libmarkermod2.so: $(objpfx)libmarkermod2-2.so
> +	$(make-target-directory)
> +	cp $< $@
> +
> +ifeq (no,$(build-hardcoded-path-in-tests))
> +# This is an ld.so.cache test, and RPATH/RUNPATH in the executable
> +# interferes with its test objectives.
> +tests-container += tst-glibc-hwcaps-cache
> +endif
> +
> +endif # $(subdir) == elf
> diff --git a/sysdeps/riscv/rv64/dl-hwcaps-subdirs.c b/sysdeps/riscv/rv64/dl-hwcaps-subdirs.c
> new file mode 100644
> index 0000000000..2845e97eff
> --- /dev/null
> +++ b/sysdeps/riscv/rv64/dl-hwcaps-subdirs.c
> @@ -0,0 +1,49 @@
> +/* Architecture-specific glibc-hwcaps subdirectories.  RISC-V version.
> +   Copyright (C) 2023 Free Software Foundation, Inc.
> +   This file is part of the GNU C Library.
> +
> +   The GNU C Library is free software; you can redistribute it and/or
> +   modify it under the terms of the GNU Lesser General Public
> +   License as published by the Free Software Foundation; either
> +   version 2.1 of the License, or (at your option) any later version.
> +
> +   The GNU C Library is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +   Lesser General Public License for more details.
> +
> +   You should have received a copy of the GNU Lesser General Public
> +   License along with the GNU C Library; if not, see
> +   <https://www.gnu.org/licenses/>.  */
> +
> +#include <dl-hwcaps.h>
> +#include <ldsodefs.h>
> +#include <sys/hwprobe.h>
> +
> +const char _dl_hwcaps_subdirs[] = "rv64-v1";
> +enum { subdirs_count = 1 }; /* Number of components in _dl_hwcaps_subdirs.  */
> +
> +uint32_t
> +_dl_hwcaps_subdirs_active (void)
> +{
> +  int active = 0;
> +  unsigned long int mask;
> +  struct riscv_hwprobe pair[2];
> +
> +  /* Test in reverse preference order. */
> +  /* v1: RV64GC_Zba_Zbb */
> +  pair[0].key = RISCV_HWPROBE_KEY_BASE_BEHAVIOR;
> +  pair[1].key = RISCV_HWPROBE_KEY_IMA_EXT_0;
> +  if (__riscv_hwprobe(pair, 2, 0, NULL, 0) != 0)
> +    return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> +
> +  mask = RISCV_HWPROBE_IMA_FD | RISCV_HWPROBE_IMA_C |
> +         RISCV_HWPROBE_EXT_ZBA | RISCV_HWPROBE_EXT_ZBB;
> +
> +  if (!(pair[0].value & RISCV_HWPROBE_BASE_BEHAVIOR_IMA) ||
> +      ((pair[1].value & mask) != mask))
> +    return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> +  ++active;
> +
> +  return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
> +}
> diff --git a/sysdeps/riscv/rv64/tst-glibc-hwcaps.c b/sysdeps/riscv/rv64/tst-glibc-hwcaps.c
> new file mode 100644
> index 0000000000..988d3492ef
> --- /dev/null
> +++ b/sysdeps/riscv/rv64/tst-glibc-hwcaps.c
> @@ -0,0 +1,58 @@
> +/* glibc-hwcaps subdirectory test.  RISC-V 64-bit version.
> +   Copyright (C) 2023 Free Software Foundation, Inc.
> +   This file is part of the GNU C Library.
> +
> +   The GNU C Library is free software; you can redistribute it and/or
> +   modify it under the terms of the GNU Lesser General Public
> +   License as published by the Free Software Foundation; either
> +   version 2.1 of the License, or (at your option) any later version.
> +
> +   The GNU C Library is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +   Lesser General Public License for more details.
> +
> +   You should have received a copy of the GNU Lesser General Public
> +   License along with the GNU C Library; if not, see
> +   <https://www.gnu.org/licenses/>.  */
> +
> +#include <stdio.h>
> +#include <string.h>
> +#include <support/check.h>
> +#include <sys/hwprobe.h>
> +#include <sys/param.h>
> +
> +extern int marker2 (void);
> +
> +/* Return the arch level, 10 for the baseline libmarkermod*.so's.  */
> +static int
> +compute_level (void)
> +{
> +  struct riscv_hwprobe pair[2];
> +  unsigned long int mask;
> +
> +  pair[0].key = RISCV_HWPROBE_KEY_BASE_BEHAVIOR;
> +  pair[1].key = RISCV_HWPROBE_KEY_IMA_EXT_0;
> +  if (__riscv_hwprobe(pair, 2, 0, NULL, 0) != 0)
> +    return 10;
> +
> +  mask = RISCV_HWPROBE_IMA_FD | RISCV_HWPROBE_IMA_C |
> +         RISCV_HWPROBE_EXT_ZBA | RISCV_HWPROBE_EXT_ZBB;
> +
> +  if (!(pair[0].value & RISCV_HWPROBE_BASE_BEHAVIOR_IMA) ||
> +      ((pair[1].value & mask) != mask))
> +    return 10;
> +
> +  return 11;
> +}
> +
> +static int
> +do_test (void)
> +{
> +  int level = compute_level ();
> +  printf ("info: detected architecture level: arch%d\n", level);
> +  TEST_COMPARE (marker2 (), MIN (level - 9, 2));
> +  return 0;
> +}
> +
> +#include <support/test-driver.c>
> diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h
> index 49e27ee855..030939c0c6 100644
> --- a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h
> +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h
> @@ -46,6 +46,8 @@ struct riscv_hwprobe {
>   #define RISCV_HWPROBE_KEY_IMA_EXT_0	4
>   #define		RISCV_HWPROBE_IMA_FD		(1 << 0)
>   #define		RISCV_HWPROBE_IMA_C		(1 << 1)
> +#define		RISCV_HWPROBE_EXT_ZBA		(1 << 2)
> +#define		RISCV_HWPROBE_EXT_ZBB		(1 << 3)
>   #define RISCV_HWPROBE_KEY_CPUPERF_0	5
>   #define		RISCV_HWPROBE_MISALIGNED_UNKNOWN	(0 << 0)
>   #define		RISCV_HWPROBE_MISALIGNED_EMULATED	(1 << 0)
  
Palmer Dabbelt April 29, 2023, 6:13 p.m. UTC | #3
On Fri, 28 Apr 2023 13:54:12 PDT (-0700), fweimer@redhat.com wrote:
> * Evan Green:
>
>> Note that this series depends on the first two patches of my other
>> series introducing the Linux hwprobe API [1]. The kernel series has been
>> applied into Palmer's tree, so barring anything unforeseen I expect that
>> to be fully landed shortly. It also depends on a Linux series I just

Linus merged it a few hours after this patch.  It's still not released, 
but barring something very surprising it'll end up in the 6.4 release in 
about two months.

>> posted upstream to expose the Zba and Zbb features through hwprobe [2].
>>
>> [1] https://public-inbox.org/libc-alpha/20230407230711.2621614-1-evan@rivosinc.com/
>> [2] https://lore.kernel.org/lkml/20230428190609.3239486-1-evan@rivosinc.com/T/#t
>
> The mapping between rv64gc_Zba_Zbb, rv64-v1, and whatever feature flags
> the hardware actually offers should ideally be documented somewhere
> separately from glibc.  This decision has relevance beyond glibc and the
> GNU toolchain, and the glibc consensus mechanisms seem insufficient.

That seems reasonable.  IMO this is really something the HW and distro 
vendors need to agree on, and the glibc mailing list really isn't where 
that sort of thing happens.

> For x86-64, what you called “well-defined” in your commit message is
> merely such an agreement.  We documented it in the psABI manual, and
> tried to involve as many relevant parties as possible prior to that.

Unfortunately I don't think the psABI is the right place to do this in 
RISC-V -- or at least, the psABI working group isn't the right place to 
do it.  It's just not a place that facilitates discussions grounded in 
the real world.  For a while some of us were hoping that the profiles 
would define this, but it looks like that's also drifted pretty far away 
from a place where vendors can get consensus around shipping systems.

If it helps any, we don't really care about the gc_zba_zbb set in Rivos 
land.  The interesting bit here was just trying to figure out where 
we're going WRT binary distros that want to support multiple extension 
sets in a single installed image, there seems to be a lot of mechanisms 
there and it's not clear which are interesting to support.

So maybe the best answer here is to just wait a bit?  There's a lot of 
hardware that has been announced but not shipped yet, it's likely to 
perform exceptionally poorly when running just rv64gc binaries.  Maybe 
that'll be enough of an impetus to get things moving on some sort of 
more focused specification work?  We'd miss the summer glibc release, 
but with the kernel probing stuff being so new that's actually kind of 
nice.

>> diff --git a/sysdeps/riscv/rv64/tst-glibc-hwcaps.c b/sysdeps/riscv/rv64/tst-glibc-hwcaps.c
>> new file mode 100644
>> index 0000000000..988d3492ef
>> --- /dev/null
>> +++ b/sysdeps/riscv/rv64/tst-glibc-hwcaps.c
>> @@ -0,0 +1,58 @@
>> +/* glibc-hwcaps subdirectory test.  RISC-V 64-bit version.
>
>> +/* Return the arch level, 10 for the baseline libmarkermod*.so's.  */
>
> The 10 here refers to the 10th edition of the Principles of Operation
> (the first edition was for S/360).  It certainly has no meaning for
> RISC-V.  I suggest to write this test differently.  It's difficult to
> future-proof it if you don't know what the future will look like.
>
> There's a space missing after __riscv_hwprobe (in two locations).
>
> The mechanics of the patch looks okay to me based on a quick glance, but
> I have not tested it.

Thanks.  We were actually way more worried about the implementation side 
of things than which specific extensions constitute the hwcap sets.  We 
just figured gc_zba_zbb was a reasonable thing to use to drive the 
implementation, as that's the only set bigger than GC and in multiple 
shipping chips right now.

The other important question we had was: in glibc-hwcaps even the right 
way to go for this sort of thing?

When I'd seen OpenSUSE pick it up it seemed like the way folks were 
going, but some other distros (Fedora, for example) seemed to be more 
interested in just having multiple system images.  IMO we want to just 
the core systems infrastructure to enable distributions to do for RISC-V 
whatever they're doing elsewhere.  So we don't need every distro to use 
hwprobe, but if nobody is interested in it then it's probably not worth 
supporting.

>
> Thanks,
> Florian
  
Florian Weimer May 2, 2023, 7:22 a.m. UTC | #4
* Palmer Dabbelt:

> So maybe the best answer here is to just wait a bit?

Yes, it will likely get easier over time because the patterns in CPU
support will become clearer.  You might not even have to wait ten to
fifteen years.

> When I'd seen OpenSUSE pick it up it seemed like the way folks were
> going, but some other distros (Fedora, for example) seemed to be more
> interested in just having multiple system images.

I'm not sure where you got the idea that Fedora would build multiple
system images with different library code.  Currently, there are no
spins that rebuild packages.  And RISC-V for Fedora remains challenging.

Thanks,
Florian
  
Palmer Dabbelt May 2, 2023, 2:39 p.m. UTC | #5
On Tue, 02 May 2023 00:22:39 PDT (-0700), fweimer@redhat.com wrote:
> * Palmer Dabbelt:
>
>> So maybe the best answer here is to just wait a bit?
>
> Yes, it will likely get easier over time because the patterns in CPU
> support will become clearer.  You might not even have to wait ten to
> fifteen years.

Hopefully ;)

>> When I'd seen OpenSUSE pick it up it seemed like the way folks were
>> going, but some other distros (Fedora, for example) seemed to be more
>> interested in just having multiple system images.
>
> I'm not sure where you got the idea that Fedora would build multiple
> system images with different library code.  Currently, there are no
> spins that rebuild packages.  And RISC-V for Fedora remains challenging.

+David, who was telling me the plan was to have rv64gc images and rva23 
images as independent system images.  Or at least I think that's what he 
was saying, sorry if I misunderstood.
  
David Abdurachmanov May 2, 2023, 3:19 p.m. UTC | #6
On Tue, May 2, 2023 at 5:39 PM Palmer Dabbelt <palmer@rivosinc.com> wrote:
>
> On Tue, 02 May 2023 00:22:39 PDT (-0700), fweimer@redhat.com wrote:
> > * Palmer Dabbelt:
> >
> >> So maybe the best answer here is to just wait a bit?
> >
> > Yes, it will likely get easier over time because the patterns in CPU
> > support will become clearer.  You might not even have to wait ten to
> > fifteen years.
>
> Hopefully ;)
>
> >> When I'd seen OpenSUSE pick it up it seemed like the way folks were
> >> going, but some other distros (Fedora, for example) seemed to be more
> >> interested in just having multiple system images.
> >
> > I'm not sure where you got the idea that Fedora would build multiple
> > system images with different library code.  Currently, there are no
> > spins that rebuild packages.  And RISC-V for Fedora remains challenging.
>
> +David, who was telling me the plan was to have rv64gc images and rva23
> images as independent system images.  Or at least I think that's what he
> was saying, sorry if I misunderstood.

We don't have a plan set in stone today, but we do have opinions.
Simply put mine would be:
- Fedora/RISCV as-is (RV64GC stuff) continues to exist to support
folks with existing hardware (the only existing hardware) today. I
call it legacy.
- Have a separate arch for a new variant (RVA23, or whatever next
major profile is announced). Also depends on what the Platforms
specification will do. We want to get closer to specification
compliant hardware support.
  
Kito Cheng May 3, 2023, 12:28 a.m. UTC | #7
> Just to be clear, it sucks to not have zbs as baseline, but the quirk is
> some of the recent hardware (sifive?) lacks zbs.

Although SiFive Unmatched board didn't have zbs and also not either
RVA22 or RVA23 profile compatible, but *personally* I would like
ignore that and follow RVA22 or RVA23 for glibc if we want to build
new variant, that should be benefit to more HW *in theory* :)

Anyway that should align with David Abdurachmanov.
  
Jeff Law May 4, 2023, 11:42 p.m. UTC | #8
On 4/29/23 12:13, Palmer Dabbelt wrote:
>>
>> The mapping between rv64gc_Zba_Zbb, rv64-v1, and whatever feature flags
>> the hardware actually offers should ideally be documented somewhere
>> separately from glibc.  This decision has relevance beyond glibc and the
>> GNU toolchain, and the glibc consensus mechanisms seem insufficient.
> 
> That seems reasonable.  IMO this is really something the HW and distro 
> vendors need to agree on, and the glibc mailing list really isn't where 
> that sort of thing happens.
Agreed.  I know you're not keen on moving it through psABI, but isn't 
that the only way to get it into the authoritative documentation?

> 
> If it helps any, we don't really care about the gc_zba_zbb set in Rivos 
> land.  The interesting bit here was just trying to figure out where 
> we're going WRT binary distros that want to support multiple extension 
> sets in a single installed image, there seems to be a lot of mechanisms 
> there and it's not clear which are interesting to support.
If I had my way we'd settle on gc_zba_zbb_zbs as the baseline, but I 
think I'm going to lose that battle.

> 
> So maybe the best answer here is to just wait a bit?  There's a lot of 
> hardware that has been announced but not shipped yet, it's likely to 
> perform exceptionally poorly when running just rv64gc binaries.  Maybe 
> that'll be enough of an impetus to get things moving on some sort of 
> more focused specification work?  We'd miss the summer glibc release, 
> but with the kernel probing stuff being so new that's actually kind of 
> nice.
I'd rather not miss the summer release.  But if we can't come to some 
kind of agreement, then I'd rather miss than do something we later regret.


> 
> When I'd seen OpenSUSE pick it up it seemed like the way folks were 
> going, but some other distros (Fedora, for example) seemed to be more 
> interested in just having multiple system images.  IMO we want to just 
> the core systems infrastructure to enable distributions to do for RISC-V 
> whatever they're doing elsewhere.  So we don't need every distro to use 
> hwprobe, but if nobody is interested in it then it's probably not worth 
> supporting.
I think that's still an open question on the Fedora side.  What sets of 
features to support and how to support them.  David A. has opinions here 
and while we may not agree 100%, I certainly understand his position.

The final decision for Fedora may not rest with this group anyway.  It 
may ultimately end up FESCO's lap.  Though I'd like to be in a position 
where everyone can agree the path forward is reasonable, even if it's 
not perfect from every individual point of view.  If we can get to that, 
then FESCO will likely accept whatever gets recommended.

jeff
  

Patch

diff --git a/elf/Makefile b/elf/Makefile
index 396ec51424..f00061852b 100644
--- a/elf/Makefile
+++ b/elf/Makefile
@@ -2620,7 +2620,7 @@  $(objpfx)argv0test.out: tst-rtld-argv0.sh $(objpfx)ld.so \
 # glibc-hwcaps mechanism for this architecture).  Used to obtain test
 # coverage for some glibc-hwcaps tests for the widest possible range
 # of systems.
-glibc-hwcaps-first-subdirs-for-tests = power9 x86-64-v2 z13
+glibc-hwcaps-first-subdirs-for-tests = power9 x86-64-v2 z13 rv-v1
 
 # The test modules are parameterized by preprocessor macros.
 LDFLAGS-libmarkermod1-1.so += -Wl,-soname,libmarkermod1.so
diff --git a/elf/tst-glibc-hwcaps-cache.script b/elf/tst-glibc-hwcaps-cache.script
index d58fc8c5de..ea3d0186c9 100644
--- a/elf/tst-glibc-hwcaps-cache.script
+++ b/elf/tst-glibc-hwcaps-cache.script
@@ -36,3 +36,6 @@  mkdirp 0770 $L/glibc-hwcaps/x86-64-v4
 cp $B/elf/libmarkermod4-2.so $L/glibc-hwcaps/x86-64-v2/libmarkermod4.so
 cp $B/elf/libmarkermod4-3.so $L/glibc-hwcaps/x86-64-v3/libmarkermod4.so
 cp $B/elf/libmarkermod4-4.so $L/glibc-hwcaps/x86-64-v4/libmarkermod4.so
+
+mkdirp 0770 $L/glibc-hwcaps/rv64-v1
+cp $B/elf/libmarkermod2-2.so $L/glibc-hwcaps/rv64-v1/libmarkermod2.so
diff --git a/sysdeps/riscv/rv64/Makefile b/sysdeps/riscv/rv64/Makefile
new file mode 100644
index 0000000000..15658ba26e
--- /dev/null
+++ b/sysdeps/riscv/rv64/Makefile
@@ -0,0 +1,21 @@ 
+
+ifeq ($(subdir),elf)
+
+$(objpfx)tst-glibc-hwcaps: \
+  $(objpfx)libmarkermod2-1.so
+
+$(objpfx)tst-glibc-hwcaps.out: \
+  $(objpfx)libmarkermod2.so \
+    $(objpfx)glibc-hwcaps/rv64-v1/libmarkermod2.so \
+
+$(objpfx)glibc-hwcaps/rv64-v1/libmarkermod2.so: $(objpfx)libmarkermod2-2.so
+	$(make-target-directory)
+	cp $< $@
+
+ifeq (no,$(build-hardcoded-path-in-tests))
+# This is an ld.so.cache test, and RPATH/RUNPATH in the executable
+# interferes with its test objectives.
+tests-container += tst-glibc-hwcaps-cache
+endif
+
+endif # $(subdir) == elf
diff --git a/sysdeps/riscv/rv64/dl-hwcaps-subdirs.c b/sysdeps/riscv/rv64/dl-hwcaps-subdirs.c
new file mode 100644
index 0000000000..2845e97eff
--- /dev/null
+++ b/sysdeps/riscv/rv64/dl-hwcaps-subdirs.c
@@ -0,0 +1,49 @@ 
+/* Architecture-specific glibc-hwcaps subdirectories.  RISC-V version.
+   Copyright (C) 2023 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <https://www.gnu.org/licenses/>.  */
+
+#include <dl-hwcaps.h>
+#include <ldsodefs.h>
+#include <sys/hwprobe.h>
+
+const char _dl_hwcaps_subdirs[] = "rv64-v1";
+enum { subdirs_count = 1 }; /* Number of components in _dl_hwcaps_subdirs.  */
+
+uint32_t
+_dl_hwcaps_subdirs_active (void)
+{
+  int active = 0;
+  unsigned long int mask;
+  struct riscv_hwprobe pair[2];
+
+  /* Test in reverse preference order. */
+  /* v1: RV64GC_Zba_Zbb */
+  pair[0].key = RISCV_HWPROBE_KEY_BASE_BEHAVIOR;
+  pair[1].key = RISCV_HWPROBE_KEY_IMA_EXT_0;
+  if (__riscv_hwprobe(pair, 2, 0, NULL, 0) != 0)
+    return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+
+  mask = RISCV_HWPROBE_IMA_FD | RISCV_HWPROBE_IMA_C |
+         RISCV_HWPROBE_EXT_ZBA | RISCV_HWPROBE_EXT_ZBB;
+
+  if (!(pair[0].value & RISCV_HWPROBE_BASE_BEHAVIOR_IMA) ||
+      ((pair[1].value & mask) != mask))
+    return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+  ++active;
+
+  return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active);
+}
diff --git a/sysdeps/riscv/rv64/tst-glibc-hwcaps.c b/sysdeps/riscv/rv64/tst-glibc-hwcaps.c
new file mode 100644
index 0000000000..988d3492ef
--- /dev/null
+++ b/sysdeps/riscv/rv64/tst-glibc-hwcaps.c
@@ -0,0 +1,58 @@ 
+/* glibc-hwcaps subdirectory test.  RISC-V 64-bit version.
+   Copyright (C) 2023 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <https://www.gnu.org/licenses/>.  */
+
+#include <stdio.h>
+#include <string.h>
+#include <support/check.h>
+#include <sys/hwprobe.h>
+#include <sys/param.h>
+
+extern int marker2 (void);
+
+/* Return the arch level, 10 for the baseline libmarkermod*.so's.  */
+static int
+compute_level (void)
+{
+  struct riscv_hwprobe pair[2];
+  unsigned long int mask;
+
+  pair[0].key = RISCV_HWPROBE_KEY_BASE_BEHAVIOR;
+  pair[1].key = RISCV_HWPROBE_KEY_IMA_EXT_0;
+  if (__riscv_hwprobe(pair, 2, 0, NULL, 0) != 0)
+    return 10;
+
+  mask = RISCV_HWPROBE_IMA_FD | RISCV_HWPROBE_IMA_C |
+         RISCV_HWPROBE_EXT_ZBA | RISCV_HWPROBE_EXT_ZBB;
+
+  if (!(pair[0].value & RISCV_HWPROBE_BASE_BEHAVIOR_IMA) ||
+      ((pair[1].value & mask) != mask))
+    return 10;
+
+  return 11;
+}
+
+static int
+do_test (void)
+{
+  int level = compute_level ();
+  printf ("info: detected architecture level: arch%d\n", level);
+  TEST_COMPARE (marker2 (), MIN (level - 9, 2));
+  return 0;
+}
+
+#include <support/test-driver.c>
diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h
index 49e27ee855..030939c0c6 100644
--- a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h
+++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h
@@ -46,6 +46,8 @@  struct riscv_hwprobe {
 #define RISCV_HWPROBE_KEY_IMA_EXT_0	4
 #define		RISCV_HWPROBE_IMA_FD		(1 << 0)
 #define		RISCV_HWPROBE_IMA_C		(1 << 1)
+#define		RISCV_HWPROBE_EXT_ZBA		(1 << 2)
+#define		RISCV_HWPROBE_EXT_ZBB		(1 << 3)
 #define RISCV_HWPROBE_KEY_CPUPERF_0	5
 #define		RISCV_HWPROBE_MISALIGNED_UNKNOWN	(0 << 0)
 #define		RISCV_HWPROBE_MISALIGNED_EMULATED	(1 << 0)