[v1] x86: Add version of sched_getcpu that uses `rdpid`
Checks
| Context |
Check |
Description |
| dj/TryBot-apply_patch |
success
|
Patch applied to master at the time it was sent
|
| dj/TryBot-32bit |
success
|
Build for i686
|
Commit Message
The `rdpid` version is slightly faster:
Measured on Tigerlake 2.8 GHz
rdpid -> 0.215 Calls / ns
rseq -> 0.312 Calls / ns
tst-skeleton-thread-affinity and tst-skeleton-affinity pass w and w.o
multiarch.
---
sysdeps/unix/sysv/linux/sched_getcpu.c | 7 +++-
sysdeps/unix/sysv/linux/x86/sched_getcpu.c | 48 ++++++++++++++++++++++
2 files changed, 53 insertions(+), 2 deletions(-)
create mode 100644 sysdeps/unix/sysv/linux/x86/sched_getcpu.c
Comments
* Noah Goldstein via Libc-alpha:
> The `rdpid` version is slightly faster:
>
> Measured on Tigerlake 2.8 GHz
>
> rdpid -> 0.215 Calls / ns
> rseq -> 0.312 Calls / ns
Is it faster than rseq? That's quite surprising.
I guess we could use an IFUNC for rseq support, too.
> diff --git a/sysdeps/unix/sysv/linux/x86/sched_getcpu.c b/sysdeps/unix/sysv/linux/x86/sched_getcpu.c
> new file mode 100644
> index 0000000000..38afed78c6
> --- /dev/null
> +++ b/sysdeps/unix/sysv/linux/x86/sched_getcpu.c
> @@ -0,0 +1,48 @@
> +# undef INIT_ARCH
> +# define INIT_ARCH() \
> + const struct cpu_features *cpu_features = __get_cpu_features ();
> +
> +libc_ifunc (sched_getcpu, CPU_FEATURE_USABLE_P (cpu_features, RDPID)
> + ? (void *) __sched_getcpu_rdpid
> + : (void *) __sched_getcpu)
> +
> +#endif
> +#include <sysdeps/unix/sysv/linux/sched_getcpu.c>
I don't think we have proper feature detection for RDPID. The kernel
needs to report that it has set up the appropriate MSR through a bit in
HWCAP2 (similar to what happens for FSGSBASE). The kernel currently
does not do that. Until this happens, RDPID can only be used from the
vDSO (unless we want to parse /proc/cpuinfo, of course, which we don't).
Even then, it's not quite clear whether the MSR contents is part of the
kernel's userspace ABI.
I filed a bug for the incorrect RDPID reporting:
RDPID support incorrectly reported on x86
<https://sourceware.org/bugzilla/show_bug.cgi?id=28912>
Thanks,
Florian
@@ -35,15 +35,18 @@ vsyscall_sched_getcpu (void)
#ifdef RSEQ_SIG
int
-sched_getcpu (void)
+__sched_getcpu (void)
{
int cpu_id = THREAD_GETMEM_VOLATILE (THREAD_SELF, rseq_area.cpu_id);
return __glibc_likely (cpu_id >= 0) ? cpu_id : vsyscall_sched_getcpu ();
}
#else /* RSEQ_SIG */
int
-sched_getcpu (void)
+__sched_getcpu (void)
{
return vsyscall_sched_getcpu ();
}
#endif /* RSEQ_SIG */
+#ifndef USE_IFUNC_SCHED_GETCPU
+weak_alias (__sched_getcpu, sched_getcpu)
+#endif /* !USE_IFUNC_SCHED_GETCPU */
new file mode 100644
@@ -0,0 +1,48 @@
+/* sched_getcpu -- Get current processor. Linux/x86 version.
+ Copyright (C) 2022 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+
+
+#if defined __x86_64__ && defined USE_MULTIARCH && IS_IN (libc)
+# define USE_IFUNC_SCHED_GETCPU
+
+# include <init-arch.h>
+int __sched_getcpu (void);
+
+int
+__sched_getcpu_rdpid (void)
+{
+# if __has_builtin (__builtin_ia32_rdpid)
+ return __builtin_ia32_rdpid();
+# else
+ unsigned long cpu;
+ asm volatile("rdpid %[cpu]" : [cpu] "=r" (cpu) : :);
+ return cpu;
+# endif
+}
+
+# undef INIT_ARCH
+# define INIT_ARCH() \
+ const struct cpu_features *cpu_features = __get_cpu_features ();
+
+libc_ifunc (sched_getcpu, CPU_FEATURE_USABLE_P (cpu_features, RDPID)
+ ? (void *) __sched_getcpu_rdpid
+ : (void *) __sched_getcpu)
+
+#endif
+#include <sysdeps/unix/sysv/linux/sched_getcpu.c>