From patchwork Wed Dec 23 22:15:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 41542 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 9FE463850401; Wed, 23 Dec 2020 22:15:22 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9FE463850401 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1608761722; bh=9nuaQTlZ/5yLmSC1xq510hiRAB0uZnvE0K3di7r/j78=; h=To:Subject:References:Date:In-Reply-To:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=a4Rd2vGPQfgzDh/MQWqhhmOcXY83G53V0eDP1vDxeC6UfaQnHpjzIXAI/D1jDUX/W gOPYIQDjKPQU2APMZuptwJUPGeW3XKj2ZU3ovF6kfm0VGEnvdvOkXEM9L3FyKV7ugs 7dnZtgysFhPbMB5A559gja8AkJg00n8g0kM3QVUU= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by sourceware.org (Postfix) with ESMTP id 1FF3A3857004 for ; Wed, 23 Dec 2020 22:15:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 1FF3A3857004 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-424-SfCVnBNPPRqbSln1BnX_lA-1; Wed, 23 Dec 2020 17:15:06 -0500 X-MC-Unique: SfCVnBNPPRqbSln1BnX_lA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 50628180A087 for ; Wed, 23 Dec 2020 22:15:05 +0000 (UTC) Received: from oldenburg2.str.redhat.com (ovpn-112-120.ams2.redhat.com [10.36.112.120]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9083460C04 for ; Wed, 23 Dec 2020 22:15:03 +0000 (UTC) To: libc-alpha@sourceware.org Subject: [RFC PATCH 2/2] x86: Alternative implementation References: <87tuscp45g.fsf@oldenburg2.str.redhat.com> Date: Wed, 23 Dec 2020 23:15:01 +0100 In-Reply-To: <87tuscp45g.fsf@oldenburg2.str.redhat.com> (Florian Weimer's message of "Wed, 23 Dec 2020 23:07:55 +0100") Message-ID: <87mty4p3tm.fsf@oldenburg2.str.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-11.3 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, KAM_STOCKGEN, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Florian Weimer via Libc-alpha From: Florian Weimer Reply-To: Florian Weimer Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" This implementation uses a data symbol containing a pointer to the bitmask array and an array length, not a function call. The expectation is that this is the final link-time ABI for this feature. (The run-time ABI will have to change once more, to support use of this facility in IFUNC resolvers.) The __libc_vars initialization mechanism is used to set up the copy in libc.so.6 before relocation, so that it can be used by IFUNC resolvers. Usage of the C preprocessor is greatly reduced, making it easier to wrap this functionality in other languages. This is still a preview. There are further cleanups possible, including removal of the function symbol. The manual still needs updating, and there are a few overlong lines. I'd like to receive feedback if this is the direction in which we want to move. I think it should be possible to hack in IFUNC resolver support using a custom dynamic section entry that points to a hidden __x86_cpu_array variable. It would be cleaner to use a new run-time-only relocation for the initialization. The dynamic section hack would not work with --gc-sections, for instance. --- sysdeps/i386/fpu/fclrexcpt.c | 2 +- sysdeps/i386/fpu/fedisblxcpt.c | 2 +- sysdeps/i386/fpu/feenablxcpt.c | 2 +- sysdeps/i386/fpu/fegetenv.c | 2 +- sysdeps/i386/fpu/fegetmode.c | 2 +- sysdeps/i386/fpu/feholdexcpt.c | 2 +- sysdeps/i386/fpu/fesetenv.c | 2 +- sysdeps/i386/fpu/fesetmode.c | 2 +- sysdeps/i386/fpu/fesetround.c | 2 +- sysdeps/i386/fpu/feupdateenv.c | 2 +- sysdeps/i386/fpu/fgetexcptflg.c | 2 +- sysdeps/i386/fpu/fsetexcptflg.c | 2 +- sysdeps/i386/fpu/ftestexcept.c | 2 +- sysdeps/i386/i686/fpu/multiarch/s_cosf.c | 3 +- sysdeps/i386/i686/fpu/multiarch/s_sincosf.c | 4 +- sysdeps/i386/i686/fpu/multiarch/s_sinf.c | 3 +- sysdeps/i386/i686/multiarch/ifunc-impl-list.c | 152 ++-- sysdeps/i386/i686/multiarch/ifunc-memmove.h | 2 +- sysdeps/i386/i686/multiarch/ifunc-memset.h | 2 +- sysdeps/i386/i686/multiarch/ifunc-sse2-bsf.h | 2 +- sysdeps/i386/i686/multiarch/ifunc-sse2-ssse3.h | 4 +- sysdeps/i386/i686/multiarch/ifunc-sse2.h | 4 +- sysdeps/i386/i686/multiarch/ifunc-sse4_2.h | 4 +- sysdeps/i386/i686/multiarch/ifunc-ssse3-sse4_2.h | 6 +- sysdeps/i386/i686/multiarch/s_fma.c | 2 +- sysdeps/i386/i686/multiarch/s_fmaf.c | 2 +- sysdeps/i386/i686/multiarch/wcscpy.c | 4 +- sysdeps/i386/setfpucw.c | 2 +- sysdeps/mach/hurd/i386/libc.abilist | 1 + sysdeps/unix/sysv/linux/i386/libc.abilist | 1 + sysdeps/unix/sysv/linux/x86/elision-conf.c | 3 +- sysdeps/unix/sysv/linux/x86_64/64/libc.abilist | 1 + sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist | 1 + sysdeps/x86/Makefile | 5 +- sysdeps/x86/Versions | 6 + sysdeps/x86/bits/platform/x86.h | 245 +++++++ sysdeps/x86/cacheinfo.h | 12 +- sysdeps/x86/cpu-features.c | 358 +++++----- sysdeps/x86/cpu-tunables.c | 6 +- sysdeps/x86/include/cpu-features.h | 62 +- sysdeps/x86/include/sys/platform/x86.h | 62 ++ sysdeps/x86/libc-vars-init.h | 42 ++ sysdeps/x86/sys/platform/x86.h | 788 +-------------------- sysdeps/x86/tst-cpu-features-cpuinfo.c | 5 +- sysdeps/x86/tst-cpu-features-supports.c | 16 +- sysdeps/x86/tst-get-cpu-features.c | 45 +- sysdeps/x86/x86_cpu_array.c | 31 + sysdeps/x86/x86_cpu_array_private.c | 21 + sysdeps/x86_64/dl-hwcaps-subdirs.c | 42 +- sysdeps/x86_64/dl-machine.h | 6 +- sysdeps/x86_64/fpu/math-tests-arch.h | 6 +- sysdeps/x86_64/fpu/multiarch/ifunc-avx-fma4.h | 10 +- sysdeps/x86_64/fpu/multiarch/ifunc-fma.h | 6 +- sysdeps/x86_64/fpu/multiarch/ifunc-fma4.h | 8 +- sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx2.h | 6 +- .../x86_64/fpu/multiarch/ifunc-mathvec-avx512.h | 4 +- .../x86_64/fpu/multiarch/ifunc-mathvec-sse4_1.h | 4 +- sysdeps/x86_64/fpu/multiarch/ifunc-sse4_1.h | 4 +- sysdeps/x86_64/fpu/multiarch/s_fma.c | 4 +- sysdeps/x86_64/fpu/multiarch/s_fmaf.c | 4 +- sysdeps/x86_64/multiarch/ifunc-avx2.h | 2 +- sysdeps/x86_64/multiarch/ifunc-impl-list.c | 228 +++--- sysdeps/x86_64/multiarch/ifunc-memcmp.h | 8 +- sysdeps/x86_64/multiarch/ifunc-memmove.h | 10 +- sysdeps/x86_64/multiarch/ifunc-memset.h | 10 +- sysdeps/x86_64/multiarch/ifunc-sse4_2.h | 4 +- sysdeps/x86_64/multiarch/ifunc-strcasecmp.h | 6 +- sysdeps/x86_64/multiarch/ifunc-strcpy.h | 4 +- sysdeps/x86_64/multiarch/ifunc-wmemset.h | 4 +- sysdeps/x86_64/multiarch/sched_cpucount.c | 3 +- sysdeps/x86_64/multiarch/strchr.c | 2 +- sysdeps/x86_64/multiarch/strcmp.c | 4 +- sysdeps/x86_64/multiarch/strncmp.c | 6 +- sysdeps/x86_64/multiarch/wcscpy.c | 4 +- sysdeps/x86_64/multiarch/wcsnlen.c | 4 +- sysdeps/x86_64/tst-glibc-hwcaps.c | 51 +- 76 files changed, 1032 insertions(+), 1355 deletions(-) diff --git a/sysdeps/i386/fpu/fclrexcpt.c b/sysdeps/i386/fpu/fclrexcpt.c index 7dc357f2d6..79379f78ef 100644 --- a/sysdeps/i386/fpu/fclrexcpt.c +++ b/sysdeps/i386/fpu/fclrexcpt.c @@ -41,7 +41,7 @@ __feclearexcept (int excepts) __asm__ ("fldenv %0" : : "m" (*&temp)); /* If the CPU supports SSE, we clear the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xnew_exc; diff --git a/sysdeps/i386/fpu/fedisblxcpt.c b/sysdeps/i386/fpu/fedisblxcpt.c index 5399bc1f25..e752a9f6a3 100644 --- a/sysdeps/i386/fpu/fedisblxcpt.c +++ b/sysdeps/i386/fpu/fedisblxcpt.c @@ -38,7 +38,7 @@ fedisableexcept (int excepts) __asm__ ("fldcw %0" : : "m" (*&new_exc)); /* If the CPU supports SSE we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xnew_exc; diff --git a/sysdeps/i386/fpu/feenablxcpt.c b/sysdeps/i386/fpu/feenablxcpt.c index b9d7e65668..7f8156b523 100644 --- a/sysdeps/i386/fpu/feenablxcpt.c +++ b/sysdeps/i386/fpu/feenablxcpt.c @@ -38,7 +38,7 @@ feenableexcept (int excepts) __asm__ ("fldcw %0" : : "m" (*&new_exc)); /* If the CPU supports SSE we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xnew_exc; diff --git a/sysdeps/i386/fpu/fegetenv.c b/sysdeps/i386/fpu/fegetenv.c index 637bc85454..1cc4ded6e9 100644 --- a/sysdeps/i386/fpu/fegetenv.c +++ b/sysdeps/i386/fpu/fegetenv.c @@ -31,7 +31,7 @@ __fegetenv (fenv_t *envp) would block all exceptions. */ __asm__ ("fldenv %0" : : "m" (*envp)); - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) __asm__ ("stmxcsr %0" : "=m" (envp->__eip)); /* Success. */ diff --git a/sysdeps/i386/fpu/fegetmode.c b/sysdeps/i386/fpu/fegetmode.c index e5154eab02..ca09eb5e45 100644 --- a/sysdeps/i386/fpu/fegetmode.c +++ b/sysdeps/i386/fpu/fegetmode.c @@ -26,7 +26,7 @@ int fegetmode (femode_t *modep) { _FPU_GETCW (modep->__control_word); - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) __asm__ ("stmxcsr %0" : "=m" (modep->__mxcsr)); return 0; } diff --git a/sysdeps/i386/fpu/feholdexcpt.c b/sysdeps/i386/fpu/feholdexcpt.c index 8d2d0ee275..94672a1a81 100644 --- a/sysdeps/i386/fpu/feholdexcpt.c +++ b/sysdeps/i386/fpu/feholdexcpt.c @@ -30,7 +30,7 @@ __feholdexcept (fenv_t *envp) __asm__ volatile ("fnstenv %0; fnclex" : "=m" (*envp)); /* If the CPU supports SSE we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xwork; diff --git a/sysdeps/i386/fpu/fesetenv.c b/sysdeps/i386/fpu/fesetenv.c index cd9afeae28..5d7c0179ad 100644 --- a/sysdeps/i386/fpu/fesetenv.c +++ b/sysdeps/i386/fpu/fesetenv.c @@ -79,7 +79,7 @@ __fesetenv (const fenv_t *envp) __asm__ ("fldenv %0" : : "m" (temp)); - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int mxcsr; __asm__ ("stmxcsr %0" : "=m" (mxcsr)); diff --git a/sysdeps/i386/fpu/fesetmode.c b/sysdeps/i386/fpu/fesetmode.c index e3b30657b1..ae866f1e41 100644 --- a/sysdeps/i386/fpu/fesetmode.c +++ b/sysdeps/i386/fpu/fesetmode.c @@ -35,7 +35,7 @@ fesetmode (const femode_t *modep) else cw = modep->__control_word; _FPU_SETCW (cw); - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int mxcsr; __asm__ ("stmxcsr %0" : "=m" (mxcsr)); diff --git a/sysdeps/i386/fpu/fesetround.c b/sysdeps/i386/fpu/fesetround.c index 5c3fd34cd4..8b53c6fd63 100644 --- a/sysdeps/i386/fpu/fesetround.c +++ b/sysdeps/i386/fpu/fesetround.c @@ -37,7 +37,7 @@ __fesetround (int round) __asm__ ("fldcw %0" : : "m" (*&cw)); /* If the CPU supports SSE we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xcw; diff --git a/sysdeps/i386/fpu/feupdateenv.c b/sysdeps/i386/fpu/feupdateenv.c index ef7132e4f0..8387fd2588 100644 --- a/sysdeps/i386/fpu/feupdateenv.c +++ b/sysdeps/i386/fpu/feupdateenv.c @@ -32,7 +32,7 @@ __feupdateenv (const fenv_t *envp) __asm__ ("fnstsw %0" : "=m" (*&temp)); /* If the CPU supports SSE we test the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) __asm__ ("stmxcsr %0" : "=m" (*&xtemp)); temp = (temp | xtemp) & FE_ALL_EXCEPT; diff --git a/sysdeps/i386/fpu/fgetexcptflg.c b/sysdeps/i386/fpu/fgetexcptflg.c index 2c32c83636..9efbe09802 100644 --- a/sysdeps/i386/fpu/fgetexcptflg.c +++ b/sysdeps/i386/fpu/fgetexcptflg.c @@ -34,7 +34,7 @@ __fegetexceptflag (fexcept_t *flagp, int excepts) *flagp = temp & excepts & FE_ALL_EXCEPT; /* If the CPU supports SSE, we clear the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int sse_exc; diff --git a/sysdeps/i386/fpu/fsetexcptflg.c b/sysdeps/i386/fpu/fsetexcptflg.c index 02a1bd526d..abf9f0c8d5 100644 --- a/sysdeps/i386/fpu/fsetexcptflg.c +++ b/sysdeps/i386/fpu/fsetexcptflg.c @@ -41,7 +41,7 @@ __fesetexceptflag (const fexcept_t *flagp, int excepts) __asm__ ("fldenv %0" : : "m" (*&temp)); /* If the CPU supports SSE, we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xnew_exc; diff --git a/sysdeps/i386/fpu/ftestexcept.c b/sysdeps/i386/fpu/ftestexcept.c index a00c44e6db..bd8538de58 100644 --- a/sysdeps/i386/fpu/ftestexcept.c +++ b/sysdeps/i386/fpu/ftestexcept.c @@ -32,7 +32,7 @@ fetestexcept (int excepts) __asm__ ("fnstsw %0" : "=a" (temp)); /* If the CPU supports SSE we test the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) __asm__ ("stmxcsr %0" : "=m" (*&xtemp)); return (temp | xtemp) & excepts & FE_ALL_EXCEPT; diff --git a/sysdeps/i386/i686/fpu/multiarch/s_cosf.c b/sysdeps/i386/i686/fpu/multiarch/s_cosf.c index 9cd14a103b..77f1939c54 100644 --- a/sysdeps/i386/i686/fpu/multiarch/s_cosf.c +++ b/sysdeps/i386/i686/fpu/multiarch/s_cosf.c @@ -23,7 +23,8 @@ extern float __cosf_sse2 (float); extern float __cosf_ia32 (float); -libm_ifunc (__cosf, CPU_FEATURE_USABLE (SSE2) ? __cosf_sse2 : __cosf_ia32); +libm_ifunc (__cosf, (x86_cpu_is_usable (x86_cpu_SSE2) + ? __cosf_sse2 : __cosf_ia32)); libm_alias_float (__cos, cos); #define COSF __cosf_ia32 diff --git a/sysdeps/i386/i686/fpu/multiarch/s_sincosf.c b/sysdeps/i386/i686/fpu/multiarch/s_sincosf.c index 9b479142d0..3a35ec7b5a 100644 --- a/sysdeps/i386/i686/fpu/multiarch/s_sincosf.c +++ b/sysdeps/i386/i686/fpu/multiarch/s_sincosf.c @@ -23,8 +23,8 @@ extern void __sincosf_sse2 (float, float *, float *); extern void __sincosf_ia32 (float, float *, float *); -libm_ifunc (__sincosf, - CPU_FEATURE_USABLE (SSE2) ? __sincosf_sse2 : __sincosf_ia32); +libm_ifunc (__sincosf, (x86_cpu_is_usable (x86_cpu_SSE2) + ? __sincosf_sse2 : __sincosf_ia32)); libm_alias_float (__sincos, sincos); #define SINCOSF __sincosf_ia32 diff --git a/sysdeps/i386/i686/fpu/multiarch/s_sinf.c b/sysdeps/i386/i686/fpu/multiarch/s_sinf.c index 84977e63e8..2f70bc30f8 100644 --- a/sysdeps/i386/i686/fpu/multiarch/s_sinf.c +++ b/sysdeps/i386/i686/fpu/multiarch/s_sinf.c @@ -23,7 +23,8 @@ extern float __sinf_sse2 (float); extern float __sinf_ia32 (float); -libm_ifunc (__sinf, CPU_FEATURE_USABLE (SSE2) ? __sinf_sse2 : __sinf_ia32); +libm_ifunc (__sinf, (x86_cpu_is_usable (x86_cpu_SSE2) + ? __sinf_sse2 : __sinf_ia32)); libm_alias_float (__sin, sin); #define SINF __sinf_ia32 #include diff --git a/sysdeps/i386/i686/multiarch/ifunc-impl-list.c b/sysdeps/i386/i686/multiarch/ifunc-impl-list.c index 89afdc0326..a1293bf30e 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-impl-list.c +++ b/sysdeps/i386/i686/multiarch/ifunc-impl-list.c @@ -38,35 +38,35 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/bcopy.S. */ IFUNC_IMPL (i, name, bcopy, - IFUNC_IMPL_ADD (array, i, bcopy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, bcopy, x86_cpu_is_usable (x86_cpu_SSSE3), __bcopy_ssse3_rep) - IFUNC_IMPL_ADD (array, i, bcopy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, bcopy, x86_cpu_is_usable (x86_cpu_SSSE3), __bcopy_ssse3) - IFUNC_IMPL_ADD (array, i, bcopy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, bcopy, x86_cpu_is_usable (x86_cpu_SSE2), __bcopy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, bcopy, 1, __bcopy_ia32)) /* Support sysdeps/i386/i686/multiarch/bzero.S. */ IFUNC_IMPL (i, name, bzero, - IFUNC_IMPL_ADD (array, i, bzero, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, bzero, x86_cpu_is_usable (x86_cpu_SSE2), __bzero_sse2_rep) - IFUNC_IMPL_ADD (array, i, bzero, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, bzero, x86_cpu_is_usable (x86_cpu_SSE2), __bzero_sse2) IFUNC_IMPL_ADD (array, i, bzero, 1, __bzero_ia32)) /* Support sysdeps/i386/i686/multiarch/memchr.S. */ IFUNC_IMPL (i, name, memchr, - IFUNC_IMPL_ADD (array, i, memchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memchr, x86_cpu_is_usable (x86_cpu_SSE2), __memchr_sse2_bsf) - IFUNC_IMPL_ADD (array, i, memchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memchr, x86_cpu_is_usable (x86_cpu_SSE2), __memchr_sse2) IFUNC_IMPL_ADD (array, i, memchr, 1, __memchr_ia32)) /* Support sysdeps/i386/i686/multiarch/memcmp.S. */ IFUNC_IMPL (i, name, memcmp, - IFUNC_IMPL_ADD (array, i, memcmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, memcmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __memcmp_sse4_2) - IFUNC_IMPL_ADD (array, i, memcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __memcmp_ssse3) IFUNC_IMPL_ADD (array, i, memcmp, 1, __memcmp_ia32)) @@ -74,13 +74,13 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/memmove_chk.S. */ IFUNC_IMPL (i, name, __memmove_chk, IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_chk_ssse3_rep) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_chk_ssse3) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (SSE2), + x86_cpu_is_usable (x86_cpu_SSE2), __memmove_chk_sse2_unaligned) IFUNC_IMPL_ADD (array, i, __memmove_chk, 1, __memmove_chk_ia32)) @@ -88,19 +88,19 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/memmove.S. */ IFUNC_IMPL (i, name, memmove, - IFUNC_IMPL_ADD (array, i, memmove, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memmove, x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_ssse3_rep) - IFUNC_IMPL_ADD (array, i, memmove, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memmove, x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_ssse3) - IFUNC_IMPL_ADD (array, i, memmove, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memmove, x86_cpu_is_usable (x86_cpu_SSE2), __memmove_sse2_unaligned) IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_ia32)) /* Support sysdeps/i386/i686/multiarch/memrchr.S. */ IFUNC_IMPL (i, name, memrchr, - IFUNC_IMPL_ADD (array, i, memrchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memrchr, x86_cpu_is_usable (x86_cpu_SSE2), __memrchr_sse2_bsf) - IFUNC_IMPL_ADD (array, i, memrchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memrchr, x86_cpu_is_usable (x86_cpu_SSE2), __memrchr_sse2) IFUNC_IMPL_ADD (array, i, memrchr, 1, __memrchr_ia32)) @@ -108,10 +108,10 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/memset_chk.S. */ IFUNC_IMPL (i, name, __memset_chk, IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (SSE2), + x86_cpu_is_usable (x86_cpu_SSE2), __memset_chk_sse2_rep) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (SSE2), + x86_cpu_is_usable (x86_cpu_SSE2), __memset_chk_sse2) IFUNC_IMPL_ADD (array, i, __memset_chk, 1, __memset_chk_ia32)) @@ -119,102 +119,102 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/memset.S. */ IFUNC_IMPL (i, name, memset, - IFUNC_IMPL_ADD (array, i, memset, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memset, x86_cpu_is_usable (x86_cpu_SSE2), __memset_sse2_rep) - IFUNC_IMPL_ADD (array, i, memset, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memset, x86_cpu_is_usable (x86_cpu_SSE2), __memset_sse2) IFUNC_IMPL_ADD (array, i, memset, 1, __memset_ia32)) /* Support sysdeps/i386/i686/multiarch/rawmemchr.S. */ IFUNC_IMPL (i, name, rawmemchr, - IFUNC_IMPL_ADD (array, i, rawmemchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, rawmemchr, x86_cpu_is_usable (x86_cpu_SSE2), __rawmemchr_sse2_bsf) - IFUNC_IMPL_ADD (array, i, rawmemchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, rawmemchr, x86_cpu_is_usable (x86_cpu_SSE2), __rawmemchr_sse2) IFUNC_IMPL_ADD (array, i, rawmemchr, 1, __rawmemchr_ia32)) /* Support sysdeps/i386/i686/multiarch/stpncpy.S. */ IFUNC_IMPL (i, name, stpncpy, - IFUNC_IMPL_ADD (array, i, stpncpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, stpncpy, x86_cpu_is_usable (x86_cpu_SSSE3), __stpncpy_ssse3) - IFUNC_IMPL_ADD (array, i, stpncpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, stpncpy, x86_cpu_is_usable (x86_cpu_SSE2), __stpncpy_sse2) IFUNC_IMPL_ADD (array, i, stpncpy, 1, __stpncpy_ia32)) /* Support sysdeps/i386/i686/multiarch/stpcpy.S. */ IFUNC_IMPL (i, name, stpcpy, - IFUNC_IMPL_ADD (array, i, stpcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, stpcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __stpcpy_ssse3) - IFUNC_IMPL_ADD (array, i, stpcpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, stpcpy, x86_cpu_is_usable (x86_cpu_SSE2), __stpcpy_sse2) IFUNC_IMPL_ADD (array, i, stpcpy, 1, __stpcpy_ia32)) /* Support sysdeps/i386/i686/multiarch/strcasecmp.S. */ IFUNC_IMPL (i, name, strcasecmp, IFUNC_IMPL_ADD (array, i, strcasecmp, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strcasecmp_sse4_2) IFUNC_IMPL_ADD (array, i, strcasecmp, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strcasecmp_ssse3) IFUNC_IMPL_ADD (array, i, strcasecmp, 1, __strcasecmp_ia32)) /* Support sysdeps/i386/i686/multiarch/strcasecmp_l.S. */ IFUNC_IMPL (i, name, strcasecmp_l, IFUNC_IMPL_ADD (array, i, strcasecmp_l, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strcasecmp_l_sse4_2) IFUNC_IMPL_ADD (array, i, strcasecmp_l, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strcasecmp_l_ssse3) IFUNC_IMPL_ADD (array, i, strcasecmp_l, 1, __strcasecmp_l_ia32)) /* Support sysdeps/i386/i686/multiarch/strcat.S. */ IFUNC_IMPL (i, name, strcat, - IFUNC_IMPL_ADD (array, i, strcat, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcat, x86_cpu_is_usable (x86_cpu_SSSE3), __strcat_ssse3) - IFUNC_IMPL_ADD (array, i, strcat, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strcat, x86_cpu_is_usable (x86_cpu_SSE2), __strcat_sse2) IFUNC_IMPL_ADD (array, i, strcat, 1, __strcat_ia32)) /* Support sysdeps/i386/i686/multiarch/strchr.S. */ IFUNC_IMPL (i, name, strchr, - IFUNC_IMPL_ADD (array, i, strchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strchr, x86_cpu_is_usable (x86_cpu_SSE2), __strchr_sse2_bsf) - IFUNC_IMPL_ADD (array, i, strchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strchr, x86_cpu_is_usable (x86_cpu_SSE2), __strchr_sse2) IFUNC_IMPL_ADD (array, i, strchr, 1, __strchr_ia32)) /* Support sysdeps/i386/i686/multiarch/strcmp.S. */ IFUNC_IMPL (i, name, strcmp, - IFUNC_IMPL_ADD (array, i, strcmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strcmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __strcmp_sse4_2) - IFUNC_IMPL_ADD (array, i, strcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __strcmp_ssse3) IFUNC_IMPL_ADD (array, i, strcmp, 1, __strcmp_ia32)) /* Support sysdeps/i386/i686/multiarch/strcpy.S. */ IFUNC_IMPL (i, name, strcpy, - IFUNC_IMPL_ADD (array, i, strcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __strcpy_ssse3) - IFUNC_IMPL_ADD (array, i, strcpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strcpy, x86_cpu_is_usable (x86_cpu_SSE2), __strcpy_sse2) IFUNC_IMPL_ADD (array, i, strcpy, 1, __strcpy_ia32)) /* Support sysdeps/i386/i686/multiarch/strcspn.S. */ IFUNC_IMPL (i, name, strcspn, - IFUNC_IMPL_ADD (array, i, strcspn, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strcspn, x86_cpu_is_usable (x86_cpu_SSE4_2), __strcspn_sse42) IFUNC_IMPL_ADD (array, i, strcspn, 1, __strcspn_ia32)) /* Support sysdeps/i386/i686/multiarch/strncase.S. */ IFUNC_IMPL (i, name, strncasecmp, IFUNC_IMPL_ADD (array, i, strncasecmp, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strncasecmp_sse4_2) IFUNC_IMPL_ADD (array, i, strncasecmp, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strncasecmp_ssse3) IFUNC_IMPL_ADD (array, i, strncasecmp, 1, __strncasecmp_ia32)) @@ -222,91 +222,91 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/strncase_l.S. */ IFUNC_IMPL (i, name, strncasecmp_l, IFUNC_IMPL_ADD (array, i, strncasecmp_l, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strncasecmp_l_sse4_2) IFUNC_IMPL_ADD (array, i, strncasecmp_l, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strncasecmp_l_ssse3) IFUNC_IMPL_ADD (array, i, strncasecmp_l, 1, __strncasecmp_l_ia32)) /* Support sysdeps/i386/i686/multiarch/strncat.S. */ IFUNC_IMPL (i, name, strncat, - IFUNC_IMPL_ADD (array, i, strncat, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncat, x86_cpu_is_usable (x86_cpu_SSSE3), __strncat_ssse3) - IFUNC_IMPL_ADD (array, i, strncat, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strncat, x86_cpu_is_usable (x86_cpu_SSE2), __strncat_sse2) IFUNC_IMPL_ADD (array, i, strncat, 1, __strncat_ia32)) /* Support sysdeps/i386/i686/multiarch/strncpy.S. */ IFUNC_IMPL (i, name, strncpy, - IFUNC_IMPL_ADD (array, i, strncpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncpy, x86_cpu_is_usable (x86_cpu_SSSE3), __strncpy_ssse3) - IFUNC_IMPL_ADD (array, i, strncpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strncpy, x86_cpu_is_usable (x86_cpu_SSE2), __strncpy_sse2) IFUNC_IMPL_ADD (array, i, strncpy, 1, __strncpy_ia32)) /* Support sysdeps/i386/i686/multiarch/strnlen.S. */ IFUNC_IMPL (i, name, strnlen, - IFUNC_IMPL_ADD (array, i, strnlen, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strnlen, x86_cpu_is_usable (x86_cpu_SSE2), __strnlen_sse2) IFUNC_IMPL_ADD (array, i, strnlen, 1, __strnlen_ia32)) /* Support sysdeps/i386/i686/multiarch/strpbrk.S. */ IFUNC_IMPL (i, name, strpbrk, - IFUNC_IMPL_ADD (array, i, strpbrk, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strpbrk, x86_cpu_is_usable (x86_cpu_SSE4_2), __strpbrk_sse42) IFUNC_IMPL_ADD (array, i, strpbrk, 1, __strpbrk_ia32)) /* Support sysdeps/i386/i686/multiarch/strrchr.S. */ IFUNC_IMPL (i, name, strrchr, - IFUNC_IMPL_ADD (array, i, strrchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strrchr, x86_cpu_is_usable (x86_cpu_SSE2), __strrchr_sse2_bsf) - IFUNC_IMPL_ADD (array, i, strrchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strrchr, x86_cpu_is_usable (x86_cpu_SSE2), __strrchr_sse2) IFUNC_IMPL_ADD (array, i, strrchr, 1, __strrchr_ia32)) /* Support sysdeps/i386/i686/multiarch/strspn.S. */ IFUNC_IMPL (i, name, strspn, - IFUNC_IMPL_ADD (array, i, strspn, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strspn, x86_cpu_is_usable (x86_cpu_SSE4_2), __strspn_sse42) IFUNC_IMPL_ADD (array, i, strspn, 1, __strspn_ia32)) /* Support sysdeps/i386/i686/multiarch/wcschr.S. */ IFUNC_IMPL (i, name, wcschr, - IFUNC_IMPL_ADD (array, i, wcschr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, wcschr, x86_cpu_is_usable (x86_cpu_SSE2), __wcschr_sse2) IFUNC_IMPL_ADD (array, i, wcschr, 1, __wcschr_ia32)) /* Support sysdeps/i386/i686/multiarch/wcscmp.S. */ IFUNC_IMPL (i, name, wcscmp, - IFUNC_IMPL_ADD (array, i, wcscmp, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, wcscmp, x86_cpu_is_usable (x86_cpu_SSE2), __wcscmp_sse2) IFUNC_IMPL_ADD (array, i, wcscmp, 1, __wcscmp_ia32)) /* Support sysdeps/i386/i686/multiarch/wcscpy.S. */ IFUNC_IMPL (i, name, wcscpy, - IFUNC_IMPL_ADD (array, i, wcscpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, wcscpy, x86_cpu_is_usable (x86_cpu_SSSE3), __wcscpy_ssse3) IFUNC_IMPL_ADD (array, i, wcscpy, 1, __wcscpy_ia32)) /* Support sysdeps/i386/i686/multiarch/wcslen.S. */ IFUNC_IMPL (i, name, wcslen, - IFUNC_IMPL_ADD (array, i, wcslen, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, wcslen, x86_cpu_is_usable (x86_cpu_SSE2), __wcslen_sse2) IFUNC_IMPL_ADD (array, i, wcslen, 1, __wcslen_ia32)) /* Support sysdeps/i386/i686/multiarch/wcsrchr.S. */ IFUNC_IMPL (i, name, wcsrchr, - IFUNC_IMPL_ADD (array, i, wcsrchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, wcsrchr, x86_cpu_is_usable (x86_cpu_SSE2), __wcsrchr_sse2) IFUNC_IMPL_ADD (array, i, wcsrchr, 1, __wcsrchr_ia32)) /* Support sysdeps/i386/i686/multiarch/wmemcmp.S. */ IFUNC_IMPL (i, name, wmemcmp, - IFUNC_IMPL_ADD (array, i, wmemcmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, wmemcmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __wmemcmp_sse4_2) - IFUNC_IMPL_ADD (array, i, wmemcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, wmemcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __wmemcmp_ssse3) IFUNC_IMPL_ADD (array, i, wmemcmp, 1, __wmemcmp_ia32)) @@ -314,64 +314,64 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/memcpy_chk.S. */ IFUNC_IMPL (i, name, __memcpy_chk, IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_chk_ssse3_rep) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_chk_ssse3) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (SSE2), + x86_cpu_is_usable (x86_cpu_SSE2), __memcpy_chk_sse2_unaligned) IFUNC_IMPL_ADD (array, i, __memcpy_chk, 1, __memcpy_chk_ia32)) /* Support sysdeps/i386/i686/multiarch/memcpy.S. */ IFUNC_IMPL (i, name, memcpy, - IFUNC_IMPL_ADD (array, i, memcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_ssse3_rep) - IFUNC_IMPL_ADD (array, i, memcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_ssse3) - IFUNC_IMPL_ADD (array, i, memcpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memcpy, x86_cpu_is_usable (x86_cpu_SSE2), __memcpy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, memcpy, 1, __memcpy_ia32)) /* Support sysdeps/i386/i686/multiarch/mempcpy_chk.S. */ IFUNC_IMPL (i, name, __mempcpy_chk, IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_chk_ssse3_rep) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_chk_ssse3) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (SSE2), + x86_cpu_is_usable (x86_cpu_SSE2), __mempcpy_chk_sse2_unaligned) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, 1, __mempcpy_chk_ia32)) /* Support sysdeps/i386/i686/multiarch/mempcpy.S. */ IFUNC_IMPL (i, name, mempcpy, - IFUNC_IMPL_ADD (array, i, mempcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, mempcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_ssse3_rep) - IFUNC_IMPL_ADD (array, i, mempcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, mempcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_ssse3) - IFUNC_IMPL_ADD (array, i, mempcpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, mempcpy, x86_cpu_is_usable (x86_cpu_SSE2), __mempcpy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, mempcpy, 1, __mempcpy_ia32)) /* Support sysdeps/i386/i686/multiarch/strlen.S. */ IFUNC_IMPL (i, name, strlen, - IFUNC_IMPL_ADD (array, i, strlen, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strlen, x86_cpu_is_usable (x86_cpu_SSE2), __strlen_sse2_bsf) - IFUNC_IMPL_ADD (array, i, strlen, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strlen, x86_cpu_is_usable (x86_cpu_SSE2), __strlen_sse2) IFUNC_IMPL_ADD (array, i, strlen, 1, __strlen_ia32)) /* Support sysdeps/i386/i686/multiarch/strncmp.S. */ IFUNC_IMPL (i, name, strncmp, - IFUNC_IMPL_ADD (array, i, strncmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strncmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __strncmp_sse4_2) - IFUNC_IMPL_ADD (array, i, strncmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncmp, x86_cpu_is_usable (x86_cpu_SSSE3), __strncmp_ssse3) IFUNC_IMPL_ADD (array, i, strncmp, 1, __strncmp_ia32)) #endif diff --git a/sysdeps/i386/i686/multiarch/ifunc-memmove.h b/sysdeps/i386/i686/multiarch/ifunc-memmove.h index c05cb6dd4f..210b7713b1 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-memmove.h +++ b/sysdeps/i386/i686/multiarch/ifunc-memmove.h @@ -33,7 +33,7 @@ IFUNC_SELECTOR (void) if (CPU_FEATURES_ARCH_P (cpu_features, Fast_Unaligned_Load)) return OPTIMIZE (sse2_unaligned); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) { if (CPU_FEATURES_ARCH_P (cpu_features, Fast_Rep_String)) return OPTIMIZE (ssse3_rep); diff --git a/sysdeps/i386/i686/multiarch/ifunc-memset.h b/sysdeps/i386/i686/multiarch/ifunc-memset.h index bead331a9d..81476bf9a1 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-memset.h +++ b/sysdeps/i386/i686/multiarch/ifunc-memset.h @@ -28,7 +28,7 @@ IFUNC_SELECTOR (void) { const struct cpu_features* cpu_features = __get_cpu_features (); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE2)) + if (x86_cpu_is_usable (x86_cpu_SSE2)) { if (CPU_FEATURES_ARCH_P (cpu_features, Fast_Rep_String)) return OPTIMIZE (sse2_rep); diff --git a/sysdeps/i386/i686/multiarch/ifunc-sse2-bsf.h b/sysdeps/i386/i686/multiarch/ifunc-sse2-bsf.h index 0d302a3dcd..d60f965be4 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-sse2-bsf.h +++ b/sysdeps/i386/i686/multiarch/ifunc-sse2-bsf.h @@ -28,7 +28,7 @@ IFUNC_SELECTOR (void) { const struct cpu_features* cpu_features = __get_cpu_features (); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE2)) + if (x86_cpu_is_usable (x86_cpu_SSE2)) { if (CPU_FEATURES_ARCH_P (cpu_features, Slow_BSF)) return OPTIMIZE (sse2); diff --git a/sysdeps/i386/i686/multiarch/ifunc-sse2-ssse3.h b/sysdeps/i386/i686/multiarch/ifunc-sse2-ssse3.h index c10ca4a9df..eea46306ba 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-sse2-ssse3.h +++ b/sysdeps/i386/i686/multiarch/ifunc-sse2-ssse3.h @@ -29,11 +29,11 @@ IFUNC_SELECTOR (void) { const struct cpu_features* cpu_features = __get_cpu_features (); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE2) + if (x86_cpu_is_usable (x86_cpu_SSE2) && CPU_FEATURES_ARCH_P (cpu_features, Fast_Rep_String)) return OPTIMIZE (sse2); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (ia32); diff --git a/sysdeps/i386/i686/multiarch/ifunc-sse2.h b/sysdeps/i386/i686/multiarch/ifunc-sse2.h index 58794a2806..6a892890d8 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-sse2.h +++ b/sysdeps/i386/i686/multiarch/ifunc-sse2.h @@ -25,9 +25,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse2) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE2)) + if (x86_cpu_is_usable (x86_cpu_SSE2)) return OPTIMIZE (sse2); return OPTIMIZE (ia32); diff --git a/sysdeps/i386/i686/multiarch/ifunc-sse4_2.h b/sysdeps/i386/i686/multiarch/ifunc-sse4_2.h index 014be1d5f7..6f5f6582d5 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-sse4_2.h +++ b/sysdeps/i386/i686/multiarch/ifunc-sse4_2.h @@ -25,9 +25,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse42) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_2)) + if (x86_cpu_is_usable (x86_cpu_SSE4_2)) return OPTIMIZE (sse42); return OPTIMIZE (ia32); diff --git a/sysdeps/i386/i686/multiarch/ifunc-ssse3-sse4_2.h b/sysdeps/i386/i686/multiarch/ifunc-ssse3-sse4_2.h index 39bfea986d..02249ef10d 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-ssse3-sse4_2.h +++ b/sysdeps/i386/i686/multiarch/ifunc-ssse3-sse4_2.h @@ -27,12 +27,10 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse4_2) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_2)) + if (x86_cpu_is_usable (x86_cpu_SSE4_2)) return OPTIMIZE (sse4_2); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (ia32); diff --git a/sysdeps/i386/i686/multiarch/s_fma.c b/sysdeps/i386/i686/multiarch/s_fma.c index 0729853e21..a70b6bc132 100644 --- a/sysdeps/i386/i686/multiarch/s_fma.c +++ b/sysdeps/i386/i686/multiarch/s_fma.c @@ -27,7 +27,7 @@ extern double __fma_ia32 (double x, double y, double z) attribute_hidden; extern double __fma_fma (double x, double y, double z) attribute_hidden; libm_ifunc (__fma, - CPU_FEATURE_USABLE (FMA) ? __fma_fma : __fma_ia32); + x86_cpu_is_usable (x86_cpu_FMA) ? __fma_fma : __fma_ia32); libm_alias_double (__fma, fma) #define __fma __fma_ia32 diff --git a/sysdeps/i386/i686/multiarch/s_fmaf.c b/sysdeps/i386/i686/multiarch/s_fmaf.c index 20f965c342..51a001b312 100644 --- a/sysdeps/i386/i686/multiarch/s_fmaf.c +++ b/sysdeps/i386/i686/multiarch/s_fmaf.c @@ -27,7 +27,7 @@ extern float __fmaf_ia32 (float x, float y, float z) attribute_hidden; extern float __fmaf_fma (float x, float y, float z) attribute_hidden; libm_ifunc (__fmaf, - CPU_FEATURE_USABLE (FMA) ? __fmaf_fma : __fmaf_ia32); + x86_cpu_is_usable (x86_cpu_FMA) ? __fmaf_fma : __fmaf_ia32); libm_alias_float (__fma, fma) #define __fmaf __fmaf_ia32 diff --git a/sysdeps/i386/i686/multiarch/wcscpy.c b/sysdeps/i386/i686/multiarch/wcscpy.c index f0038bc4a2..8d5120b0c6 100644 --- a/sysdeps/i386/i686/multiarch/wcscpy.c +++ b/sysdeps/i386/i686/multiarch/wcscpy.c @@ -32,9 +32,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (ssse3) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (ia32); diff --git a/sysdeps/i386/setfpucw.c b/sysdeps/i386/setfpucw.c index c640a72cc2..82c5aa55f7 100644 --- a/sysdeps/i386/setfpucw.c +++ b/sysdeps/i386/setfpucw.c @@ -39,7 +39,7 @@ __setfpucw (fpu_control_t set) __asm__ ("fldcw %0" : : "m" (*&cw)); /* If the CPU supports SSE, we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_has_feature (x86_cpu_SSE)) { unsigned int xnew_exc; diff --git a/sysdeps/mach/hurd/i386/libc.abilist b/sysdeps/mach/hurd/i386/libc.abilist index 7a5eb66b85..36575c7d88 100644 --- a/sysdeps/mach/hurd/i386/libc.abilist +++ b/sysdeps/mach/hurd/i386/libc.abilist @@ -2191,6 +2191,7 @@ GLIBC_2.32 thrd_current F GLIBC_2.32 thrd_equal F GLIBC_2.32 thrd_sleep F GLIBC_2.32 thrd_yield F +GLIBC_2.33 __x86_cpu_array D 0x8 GLIBC_2.33 fstat F GLIBC_2.33 fstat64 F GLIBC_2.33 fstatat F diff --git a/sysdeps/unix/sysv/linux/i386/libc.abilist b/sysdeps/unix/sysv/linux/i386/libc.abilist index 4f0d3c1eb5..20b43a1f83 100644 --- a/sysdeps/unix/sysv/linux/i386/libc.abilist +++ b/sysdeps/unix/sysv/linux/i386/libc.abilist @@ -2229,6 +2229,7 @@ GLIBC_2.32 sigabbrev_np F GLIBC_2.32 sigdescr_np F GLIBC_2.32 strerrordesc_np F GLIBC_2.32 strerrorname_np F +GLIBC_2.33 __x86_cpu_array D 0x8 GLIBC_2.33 fstat F GLIBC_2.33 fstat64 F GLIBC_2.33 fstatat F diff --git a/sysdeps/unix/sysv/linux/x86/elision-conf.c b/sysdeps/unix/sysv/linux/x86/elision-conf.c index ecdb0378e3..64948b75b8 100644 --- a/sysdeps/unix/sysv/linux/x86/elision-conf.c +++ b/sysdeps/unix/sysv/linux/x86/elision-conf.c @@ -21,6 +21,7 @@ #include #include #include +#include #if HAVE_TUNABLES # define TUNABLE_NAMESPACE elision @@ -63,7 +64,7 @@ do_set_elision_enable (int32_t elision_enable) if __libc_enable_secure isn't enabled since elision_enable will be set according to the default, which is disabled. */ if (elision_enable == 1) - __pthread_force_elision = CPU_FEATURE_USABLE (RTM) ? 1 : 0; + __pthread_force_elision = x86_cpu_is_usable (x86_cpu_RTM) ? 1 : 0; } /* The pthread->elision_enable tunable is 0 or 1 indicating that elision diff --git a/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist b/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist index 4fff61818b..e66e13f59e 100644 --- a/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist +++ b/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist @@ -2076,6 +2076,7 @@ GLIBC_2.32 sigabbrev_np F GLIBC_2.32 sigdescr_np F GLIBC_2.32 strerrordesc_np F GLIBC_2.32 strerrorname_np F +GLIBC_2.33 __x86_cpu_array D 0x10 GLIBC_2.33 fstat F GLIBC_2.33 fstat64 F GLIBC_2.33 fstatat F diff --git a/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist b/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist index 102ed47a9c..517b13e9fe 100644 --- a/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist +++ b/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist @@ -2173,6 +2173,7 @@ GLIBC_2.32 sigabbrev_np F GLIBC_2.32 sigdescr_np F GLIBC_2.32 strerrordesc_np F GLIBC_2.32 strerrorname_np F +GLIBC_2.33 __x86_cpu_array D 0x8 GLIBC_2.33 fstat F GLIBC_2.33 fstat64 F GLIBC_2.33 fstatat F diff --git a/sysdeps/x86/Makefile b/sysdeps/x86/Makefile index 081cc72e93..d6f93fb45c 100644 --- a/sysdeps/x86/Makefile +++ b/sysdeps/x86/Makefile @@ -3,8 +3,9 @@ gen-as-const-headers += cpu-features-offsets.sym endif ifeq ($(subdir),elf) -sysdep-dl-routines += dl-get-cpu-features -sysdep_headers += sys/platform/x86.h +sysdep_routines += x86_cpu_array +sysdep-dl-routines += dl-get-cpu-features x86_cpu_array_private +sysdep_headers += bits/platform/x86.h sys/platform/x86.h tests += tst-get-cpu-features tst-get-cpu-features-static \ tst-cpu-features-cpuinfo tst-cpu-features-supports diff --git a/sysdeps/x86/Versions b/sysdeps/x86/Versions index 59db578a9d..64796eea6e 100644 --- a/sysdeps/x86/Versions +++ b/sysdeps/x86/Versions @@ -3,3 +3,9 @@ ld { __x86_get_cpu_features; } } + +libc { + GLIBC_2.33 { + __x86_cpu_array; + } +} \ No newline at end of file diff --git a/sysdeps/x86/bits/platform/x86.h b/sysdeps/x86/bits/platform/x86.h new file mode 100644 index 0000000000..dd9a273f5b --- /dev/null +++ b/sysdeps/x86/bits/platform/x86.h @@ -0,0 +1,245 @@ +/* Constants for x86 CPU features and struct x86_cpu_array definition. + This file is part of the GNU C Library. + Copyright (C) 2008-2020 Free Software Foundation, Inc. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _SYS_PLATFORM_X86_H +# error "Never include directly; use instead." +#endif + +struct x86_cpu_array +{ + /* Pointer to an array of __x86_count 32-bit values. */ + const unsigned int *__x86_word; + unsigned int __x86_count; +#ifdef __LP64__ + unsigned int __x86_padding; +#endif +}; + +enum +{ + /* CPUID.01H:ECX. */ + x86_cpu_SSE3 = 0u * 64u + 0u, + x86_cpu_PCLMULQDQ = 0u * 64u + 1u, + x86_cpu_DTES64 = 0u * 64u + 2u, + x86_cpu_MONITOR = 0u * 64u + 3u, + x86_cpu_DS_CPL = 0u * 64u + 4u, + x86_cpu_VMX = 0u * 64u + 5u, + x86_cpu_SMX = 0u * 64u + 6u, + x86_cpu_EIST = 0u * 64u + 7u, + x86_cpu_TM2 = 0u * 64u + 8u, + x86_cpu_SSSE3 = 0u * 64u + 9u, + x86_cpu_CNXT_ID = 0u * 64u + 10u, + x86_cpu_SDBG = 0u * 64u + 11u, + x86_cpu_FMA = 0u * 64u + 12u, + x86_cpu_CMPXCHG16B = 0u * 64u + 13u, + x86_cpu_XTPRUPDCTRL = 0u * 64u + 14u, + x86_cpu_PDCM = 0u * 64u + 15u, + x86_cpu_INDEX_1_ECX_16 = 0u * 64u + 16u, + x86_cpu_PCID = 0u * 64u + 17u, + x86_cpu_DCA = 0u * 64u + 18u, + x86_cpu_SSE4_1 = 0u * 64u + 19u, + x86_cpu_SSE4_2 = 0u * 64u + 20u, + x86_cpu_X2APIC = 0u * 64u + 21u, + x86_cpu_MOVBE = 0u * 64u + 22u, + x86_cpu_POPCNT = 0u * 64u + 23u, + x86_cpu_TSC_DEADLINE = 0u * 64u + 24u, + x86_cpu_AES = 0u * 64u + 25u, + x86_cpu_XSAVE = 0u * 64u + 26u, + x86_cpu_OSXSAVE = 0u * 64u + 27u, + x86_cpu_AVX = 0u * 64u + 28u, + x86_cpu_F16C = 0u * 64u + 29u, + x86_cpu_RDRAND = 0u * 64u + 30u, + x86_cpu_INDEX_1_ECX_31 = 0u * 64u + 31u, + + /* CPUID.01H:EDX. */ + x86_cpu_FPU = 1u * 64u + 0u, + x86_cpu_VME = 1u * 64u + 1u, + x86_cpu_DE = 1u * 64u + 2u, + x86_cpu_PSE = 1u * 64u + 3u, + x86_cpu_TSC = 1u * 64u + 4u, + x86_cpu_MSR = 1u * 64u + 5u, + x86_cpu_PAE = 1u * 64u + 6u, + x86_cpu_MCE = 1u * 64u + 7u, + x86_cpu_CX8 = 1u * 64u + 8u, + x86_cpu_APIC = 1u * 64u + 9u, + x86_cpu_INDEX_1_EDX_10 = 1u * 64u + 10u, + x86_cpu_SEP = 1u * 64u + 11u, + x86_cpu_MTRR = 1u * 64u + 12u, + x86_cpu_PGE = 1u * 64u + 13u, + x86_cpu_MCA = 1u * 64u + 14u, + x86_cpu_CMOV = 1u * 64u + 15u, + x86_cpu_PAT = 1u * 64u + 16u, + x86_cpu_PSE_36 = 1u * 64u + 17u, + x86_cpu_PSN = 1u * 64u + 18u, + x86_cpu_CLFSH = 1u * 64u + 19u, + x86_cpu_INDEX_1_EDX_20 = 1u * 64u + 20u, + x86_cpu_DS = 1u * 64u + 21u, + x86_cpu_ACPI = 1u * 64u + 22u, + x86_cpu_MMX = 1u * 64u + 23u, + x86_cpu_FXSR = 1u * 64u + 24u, + x86_cpu_SSE = 1u * 64u + 25u, + x86_cpu_SSE2 = 1u * 64u + 26u, + x86_cpu_SS = 1u * 64u + 27u, + x86_cpu_HTT = 1u * 64u + 28u, + x86_cpu_TM = 1u * 64u + 29u, + x86_cpu_INDEX_1_EDX_30 = 1u * 64u + 30u, + x86_cpu_PBE = 1u * 64u + 31u, + + /* CPUID.07H.0H:EBX. */ + x86_cpu_FSGSBASE = 2u * 64u + 0u, + x86_cpu_TSC_ADJUST = 2u * 64u + 1u, + x86_cpu_SGX = 2u * 64u + 2u, + x86_cpu_BMI1 = 2u * 64u + 3u, + x86_cpu_HLE = 2u * 64u + 4u, + x86_cpu_AVX2 = 2u * 64u + 5u, + x86_cpu_INDEX_7_EBX_6 = 2u * 64u + 6u, + x86_cpu_SMEP = 2u * 64u + 7u, + x86_cpu_BMI2 = 2u * 64u + 8u, + x86_cpu_ERMS = 2u * 64u + 9u, + x86_cpu_INVPCID = 2u * 64u + 10u, + x86_cpu_RTM = 2u * 64u + 11u, + x86_cpu_RDT_M = 2u * 64u + 12u, + x86_cpu_DEPR_FPU_CS_DS = 2u * 64u + 13u, + x86_cpu_MPX = 2u * 64u + 14u, + x86_cpu_RDT_A = 2u * 64u + 15u, + x86_cpu_AVX512F = 2u * 64u + 16u, + x86_cpu_AVX512DQ = 2u * 64u + 17u, + x86_cpu_RDSEED = 2u * 64u + 18u, + x86_cpu_ADX = 2u * 64u + 19u, + x86_cpu_SMAP = 2u * 64u + 20u, + x86_cpu_AVX512_IFMA = 2u * 64u + 21u, + x86_cpu_INDEX_7_EBX_22 = 2u * 64u + 22u, + x86_cpu_CLFLUSHOPT = 2u * 64u + 23u, + x86_cpu_CLWB = 2u * 64u + 24u, + x86_cpu_TRACE = 2u * 64u + 25u, + x86_cpu_AVX512PF = 2u * 64u + 26u, + x86_cpu_AVX512ER = 2u * 64u + 27u, + x86_cpu_AVX512CD = 2u * 64u + 28u, + x86_cpu_SHA = 2u * 64u + 29u, + x86_cpu_AVX512BW = 2u * 64u + 30u, + x86_cpu_AVX512VL = 2u * 64u + 31u, + + /* CPUID.07H.0H:ECX. */ + x86_cpu_PREFETCHWT1 = 3u * 64u + 0u, + x86_cpu_AVX512_VBMI = 3u * 64u + 1u, + x86_cpu_UMIP = 3u * 64u + 2u, + x86_cpu_PKU = 3u * 64u + 3u, + x86_cpu_OSPKE = 3u * 64u + 4u, + x86_cpu_WAITPKG = 3u * 64u + 5u, + x86_cpu_AVX512_VBMI2 = 3u * 64u + 6u, + x86_cpu_SHSTK = 3u * 64u + 7u, + x86_cpu_GFNI = 3u * 64u + 8u, + x86_cpu_VAES = 3u * 64u + 9u, + x86_cpu_VPCLMULQDQ = 3u * 64u + 10u, + x86_cpu_AVX512_VNNI = 3u * 64u + 11u, + x86_cpu_AVX512_BITALG = 3u * 64u + 12u, + x86_cpu_INDEX_7_ECX_13 = 3u * 64u + 13u, + x86_cpu_AVX512_VPOPCNTDQ = 3u * 64u + 14u, + x86_cpu_INDEX_7_ECX_15 = 3u * 64u + 15u, + x86_cpu_INDEX_7_ECX_16 = 3u * 64u + 16u, + /* Note: Bits 17-21: The value of MAWAU used by the BNDLDX and + BNDSTX instructions in 64-bit mode. */ + x86_cpu_RDPID = 3u * 64u + 22u, + x86_cpu_KL = 3u * 64u + 23u, + x86_cpu_INDEX_7_ECX_24 = 3u * 64u + 24u, + x86_cpu_CLDEMOTE = 3u * 64u + 25u, + x86_cpu_INDEX_7_ECX_26 = 3u * 64u + 26u, + x86_cpu_MOVDIRI = 3u * 64u + 27u, + x86_cpu_MOVDIR64B = 3u * 64u + 28u, + x86_cpu_ENQCMD = 3u * 64u + 29u, + x86_cpu_SGX_LC = 3u * 64u + 30u, + x86_cpu_PKS = 3u * 64u + 31u, + + /* CPUID.07H.0H:EDX. */ + x86_cpu_INDEX_7_EDX_0 = 4u * 64u + 0u, + x86_cpu_INDEX_7_EDX_1 = 4u * 64u + 1u, + x86_cpu_AVX512_4VNNIW = 4u * 64u + 2u, + x86_cpu_AVX512_4FMAPS = 4u * 64u + 3u, + x86_cpu_FSRM = 4u * 64u + 4u, + x86_cpu_UINTR = 4u * 64u + 5u, + x86_cpu_INDEX_7_EDX_6 = 4u * 64u + 6u, + x86_cpu_INDEX_7_EDX_7 = 4u * 64u + 7u, + x86_cpu_AVX512_VP2INTERSECT = 4u * 64u + 8u, + x86_cpu_INDEX_7_EDX_9 = 4u * 64u + 9u, + x86_cpu_MD_CLEAR = 4u * 64u + 10u, + x86_cpu_INDEX_7_EDX_11 = 4u * 64u + 11u, + x86_cpu_INDEX_7_EDX_12 = 4u * 64u + 12u, + x86_cpu_INDEX_7_EDX_13 = 4u * 64u + 13u, + x86_cpu_SERIALIZE = 4u * 64u + 14u, + x86_cpu_HYBRID = 4u * 64u + 15u, + x86_cpu_TSXLDTRK = 4u * 64u + 16u, + x86_cpu_INDEX_7_EDX_17 = 4u * 64u + 17u, + x86_cpu_PCONFIG = 4u * 64u + 18u, + x86_cpu_INDEX_7_EDX_19 = 4u * 64u + 19u, + x86_cpu_IBT = 4u * 64u + 20u, + x86_cpu_INDEX_7_EDX_21 = 4u * 64u + 21u, + x86_cpu_AMX_BF16 = 4u * 64u + 22u, + x86_cpu_AVX512_FP16 = 4u * 64u + 23u, + x86_cpu_AMX_TILE = 4u * 64u + 24u, + x86_cpu_AMX_INT8 = 4u * 64u + 25u, + x86_cpu_IBRS_IBPB = 4u * 64u + 26u, + x86_cpu_STIBP = 4u * 64u + 27u, + x86_cpu_L1D_FLUSH = 4u * 64u + 28u, + x86_cpu_ARCH_CAPABILITIES = 4u * 64u + 29u, + x86_cpu_CORE_CAPABILITIES = 4u * 64u + 30u, + x86_cpu_SSBD = 4u * 64u + 31u, + + /* CPUID.80000001H:ECX. */ + x86_cpu_LAHF64_SAHF64 = 5u * 64u + 0u, + x86_cpu_SVM = 5u * 64u + 2u, + x86_cpu_LZCNT = 5u * 64u + 5u, + x86_cpu_SSE4A = 5u * 64u + 6u, + x86_cpu_PREFETCHW = 5u * 64u + 8u, + x86_cpu_XOP = 5u * 64u + 11u, + x86_cpu_LWP = 5u * 64u + 15u, + x86_cpu_FMA4 = 5u * 64u + 16u, + x86_cpu_TBM = 5u * 64u + 21u, + + /* CPUID.80000001H:EDX. */ + x86_cpu_SYSCALL_SYSRET = 6u * 64u + 11u, + x86_cpu_NX = 6u * 64u + 20u, + x86_cpu_PAGE1GB = 6u * 64u + 26u, + x86_cpu_RDTSCP = 6u * 64u + 27u, + x86_cpu_LM = 6u * 64u + 29u, + + /* CPUID.(EAX=0DH,ECX=1):EAX. */ + x86_cpu_XSAVEOPT = 7u * 64u + 0u, + x86_cpu_XSAVEC = 7u * 64u + 1u, + x86_cpu_XGETBV_ECX_1 = 7u * 64u + 2u, + x86_cpu_XSAVES = 7u * 64u + 3u, + x86_cpu_XFD = 7u * 64u + 4u, + + /* CPUID.80000007H:EDX. */ + x86_cpu_INVARIANT_TSC = 8u * 64u + 8u, + + /* CPUID.80000008H:EBX. */ + x86_cpu_WBNOINVD = 9u * 64u + 9u, + + /* CPUID.(EAX=07H.,ECX=1):EAX. */ + x86_cpu_AVX_VNNI = 10u * 64u + 4u, + x86_cpu_AVX512_BF16 = 10u * 64u + 5u, + x86_cpu_FZLRM = 10u * 64u + 10u, + x86_cpu_FSRS = 10u * 64u + 11u, + x86_cpu_FSRCS = 10u * 64u + 12u, + x86_cpu_HRESET = 10u * 64u + 22u, + x86_cpu_LAM = 10u * 64u + 26u, + + /* CPUID.19H:EBX. */ + x86_cpu_AESKLE = 11u * 64u + 0u, + x86_cpu_WIDE_KL = 11u * 64u + 2u, +}; diff --git a/sysdeps/x86/cacheinfo.h b/sysdeps/x86/cacheinfo.h index 0aec0e2875..4a68072f3d 100644 --- a/sysdeps/x86/cacheinfo.h +++ b/sysdeps/x86/cacheinfo.h @@ -91,7 +91,7 @@ get_common_cache_info (long int *shared_ptr, unsigned int *threads_ptr, /* A value of 0 for the HTT bit indicates there is only a single logical processor. */ - if (HAS_CPU_FEATURE (HTT)) + if (x86_cpu_has_feature (x86_cpu_HTT)) { /* Figure out the number of logical threads that share the highest cache level. */ @@ -236,12 +236,12 @@ get_common_cache_info (long int *shared_ptr, unsigned int *threads_ptr, } else { -intel_bug_no_cache_info: +intel_bug_no_cache_info:; /* Assume that all logical threads share the highest cache level. */ - threads - = ((cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.ebx - >> 16) & 0xff); + unsigned int eax, ebx, ecx, edx; + __cpuid (1, eax, ebx, ecx, edx); + threads = (ebx >> 16) & 0xff; } /* Cap usage of highest cache level to the number of supported @@ -401,7 +401,7 @@ init_cacheinfo (void) unsigned int minimum_rep_movsb_threshold; /* NB: The default REP MOVSB threshold is 2048 * (VEC_SIZE / 16). */ unsigned int rep_movsb_threshold; - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512F) + if (x86_cpu_is_usable (x86_cpu_AVX512F) && !CPU_FEATURE_PREFERRED_P (cpu_features, Prefer_No_AVX512)) { rep_movsb_threshold = 2048 * (64 / 16); diff --git a/sysdeps/x86/cpu-features.c b/sysdeps/x86/cpu-features.c index fe080b63b2..f9cf6bbfba 100644 --- a/sysdeps/x86/cpu-features.c +++ b/sysdeps/x86/cpu-features.c @@ -46,65 +46,80 @@ extern void TUNABLE_CALLBACK (set_x86_shstk) (tunable_val_t *) # include #endif +/* Copy one indexed bit from the features word to the usable word. */ +static inline void +copy_usable_1 (unsigned int index) +{ + __x86_cpu_array_private[(index >> 5) + 1] + |= __x86_cpu_array_private[index >> 5] & (1u << (index % 32)); +} + +/* Mark one feature as usable. */ +static inline void +set_usable_1 (unsigned int index) +{ + __x86_cpu_array_private[(index >> 5) + 1] |= 1u << (index % 32); +} + static void update_usable (struct cpu_features *cpu_features) { /* Copy the cpuid bits to usable bits for CPU featuress whose usability in user space can be detected without additonal OS support. */ - CPU_FEATURE_SET_USABLE (cpu_features, SSE3); - CPU_FEATURE_SET_USABLE (cpu_features, PCLMULQDQ); - CPU_FEATURE_SET_USABLE (cpu_features, SSSE3); - CPU_FEATURE_SET_USABLE (cpu_features, CMPXCHG16B); - CPU_FEATURE_SET_USABLE (cpu_features, SSE4_1); - CPU_FEATURE_SET_USABLE (cpu_features, SSE4_2); - CPU_FEATURE_SET_USABLE (cpu_features, MOVBE); - CPU_FEATURE_SET_USABLE (cpu_features, POPCNT); - CPU_FEATURE_SET_USABLE (cpu_features, AES); - CPU_FEATURE_SET_USABLE (cpu_features, OSXSAVE); - CPU_FEATURE_SET_USABLE (cpu_features, TSC); - CPU_FEATURE_SET_USABLE (cpu_features, CX8); - CPU_FEATURE_SET_USABLE (cpu_features, CMOV); - CPU_FEATURE_SET_USABLE (cpu_features, CLFSH); - CPU_FEATURE_SET_USABLE (cpu_features, MMX); - CPU_FEATURE_SET_USABLE (cpu_features, FXSR); - CPU_FEATURE_SET_USABLE (cpu_features, SSE); - CPU_FEATURE_SET_USABLE (cpu_features, SSE2); - CPU_FEATURE_SET_USABLE (cpu_features, HTT); - CPU_FEATURE_SET_USABLE (cpu_features, BMI1); - CPU_FEATURE_SET_USABLE (cpu_features, HLE); - CPU_FEATURE_SET_USABLE (cpu_features, BMI2); - CPU_FEATURE_SET_USABLE (cpu_features, ERMS); - CPU_FEATURE_SET_USABLE (cpu_features, RTM); - CPU_FEATURE_SET_USABLE (cpu_features, RDSEED); - CPU_FEATURE_SET_USABLE (cpu_features, ADX); - CPU_FEATURE_SET_USABLE (cpu_features, CLFLUSHOPT); - CPU_FEATURE_SET_USABLE (cpu_features, CLWB); - CPU_FEATURE_SET_USABLE (cpu_features, SHA); - CPU_FEATURE_SET_USABLE (cpu_features, PREFETCHWT1); - CPU_FEATURE_SET_USABLE (cpu_features, OSPKE); - CPU_FEATURE_SET_USABLE (cpu_features, WAITPKG); - CPU_FEATURE_SET_USABLE (cpu_features, GFNI); - CPU_FEATURE_SET_USABLE (cpu_features, RDPID); - CPU_FEATURE_SET_USABLE (cpu_features, RDRAND); - CPU_FEATURE_SET_USABLE (cpu_features, CLDEMOTE); - CPU_FEATURE_SET_USABLE (cpu_features, MOVDIRI); - CPU_FEATURE_SET_USABLE (cpu_features, MOVDIR64B); - CPU_FEATURE_SET_USABLE (cpu_features, FSRM); - CPU_FEATURE_SET_USABLE (cpu_features, SERIALIZE); - CPU_FEATURE_SET_USABLE (cpu_features, TSXLDTRK); - CPU_FEATURE_SET_USABLE (cpu_features, LAHF64_SAHF64); - CPU_FEATURE_SET_USABLE (cpu_features, LZCNT); - CPU_FEATURE_SET_USABLE (cpu_features, SSE4A); - CPU_FEATURE_SET_USABLE (cpu_features, PREFETCHW); - CPU_FEATURE_SET_USABLE (cpu_features, TBM); - CPU_FEATURE_SET_USABLE (cpu_features, RDTSCP); - CPU_FEATURE_SET_USABLE (cpu_features, WBNOINVD); - CPU_FEATURE_SET_USABLE (cpu_features, FZLRM); - CPU_FEATURE_SET_USABLE (cpu_features, FSRS); - CPU_FEATURE_SET_USABLE (cpu_features, FSRCS); + copy_usable_1 (x86_cpu_SSE3); + copy_usable_1 (x86_cpu_PCLMULQDQ); + copy_usable_1 (x86_cpu_SSSE3); + copy_usable_1 (x86_cpu_CMPXCHG16B); + copy_usable_1 (x86_cpu_SSE4_1); + copy_usable_1 (x86_cpu_SSE4_2); + copy_usable_1 (x86_cpu_MOVBE); + copy_usable_1 (x86_cpu_POPCNT); + copy_usable_1 (x86_cpu_AES); + copy_usable_1 (x86_cpu_OSXSAVE); + copy_usable_1 (x86_cpu_TSC); + copy_usable_1 (x86_cpu_CX8); + copy_usable_1 (x86_cpu_CMOV); + copy_usable_1 (x86_cpu_CLFSH); + copy_usable_1 (x86_cpu_MMX); + copy_usable_1 (x86_cpu_FXSR); + copy_usable_1 (x86_cpu_SSE); + copy_usable_1 (x86_cpu_SSE2); + copy_usable_1 (x86_cpu_HTT); + copy_usable_1 (x86_cpu_BMI1); + copy_usable_1 (x86_cpu_HLE); + copy_usable_1 (x86_cpu_BMI2); + copy_usable_1 (x86_cpu_ERMS); + copy_usable_1 (x86_cpu_RTM); + copy_usable_1 (x86_cpu_RDSEED); + copy_usable_1 (x86_cpu_ADX); + copy_usable_1 (x86_cpu_CLFLUSHOPT); + copy_usable_1 (x86_cpu_CLWB); + copy_usable_1 (x86_cpu_SHA); + copy_usable_1 (x86_cpu_PREFETCHWT1); + copy_usable_1 (x86_cpu_OSPKE); + copy_usable_1 (x86_cpu_WAITPKG); + copy_usable_1 (x86_cpu_GFNI); + copy_usable_1 (x86_cpu_RDPID); + copy_usable_1 (x86_cpu_RDRAND); + copy_usable_1 (x86_cpu_CLDEMOTE); + copy_usable_1 (x86_cpu_MOVDIRI); + copy_usable_1 (x86_cpu_MOVDIR64B); + copy_usable_1 (x86_cpu_FSRM); + copy_usable_1 (x86_cpu_SERIALIZE); + copy_usable_1 (x86_cpu_TSXLDTRK); + copy_usable_1 (x86_cpu_LAHF64_SAHF64); + copy_usable_1 (x86_cpu_LZCNT); + copy_usable_1 (x86_cpu_SSE4A); + copy_usable_1 (x86_cpu_PREFETCHW); + copy_usable_1 (x86_cpu_TBM); + copy_usable_1 (x86_cpu_RDTSCP); + copy_usable_1 (x86_cpu_WBNOINVD); + copy_usable_1 (x86_cpu_FZLRM); + copy_usable_1 (x86_cpu_FSRS); + copy_usable_1 (x86_cpu_FSRCS); /* Can we call xgetbv? */ - if (CPU_FEATURES_CPU_P (cpu_features, OSXSAVE)) + if (x86_cpu_has_feature (x86_cpu_OSXSAVE)) { unsigned int xcrlow; unsigned int xcrhigh; @@ -114,14 +129,14 @@ update_usable (struct cpu_features *cpu_features) == (bit_YMM_state | bit_XMM_state)) { /* Determine if AVX is usable. */ - if (CPU_FEATURES_CPU_P (cpu_features, AVX)) + if (x86_cpu_has_feature (x86_cpu_AVX)) { - CPU_FEATURE_SET (cpu_features, AVX); + set_usable_1 (x86_cpu_AVX); /* The following features depend on AVX being usable. */ /* Determine if AVX2 is usable. */ - if (CPU_FEATURES_CPU_P (cpu_features, AVX2)) + if (x86_cpu_has_feature (x86_cpu_AVX2)) { - CPU_FEATURE_SET (cpu_features, AVX2); + set_usable_1 (x86_cpu_AVX2); /* Unaligned load with 256-bit AVX registers are faster on Intel/AMD processors with AVX2. */ @@ -129,17 +144,17 @@ update_usable (struct cpu_features *cpu_features) |= bit_arch_AVX_Fast_Unaligned_Load; } /* Determine if AVX-VNNI is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX_VNNI); + copy_usable_1 (x86_cpu_AVX_VNNI); /* Determine if FMA is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, FMA); + copy_usable_1 (x86_cpu_FMA); /* Determine if VAES is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, VAES); + copy_usable_1 (x86_cpu_VAES); /* Determine if VPCLMULQDQ is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, VPCLMULQDQ); + copy_usable_1 (x86_cpu_VPCLMULQDQ); /* Determine if XOP is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, XOP); + copy_usable_1 (x86_cpu_XOP); /* Determine if F16C is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, F16C); + copy_usable_1 (x86_cpu_F16C); } /* Check if OPMASK state, upper 256-bit of ZMM0-ZMM15 and @@ -149,45 +164,43 @@ update_usable (struct cpu_features *cpu_features) == (bit_Opmask_state | bit_ZMM0_15_state | bit_ZMM16_31_state)) { /* Determine if AVX512F is usable. */ - if (CPU_FEATURES_CPU_P (cpu_features, AVX512F)) + if (x86_cpu_has_feature (x86_cpu_AVX512F)) { - CPU_FEATURE_SET (cpu_features, AVX512F); + set_usable_1 (x86_cpu_AVX512F); /* Determine if AVX512CD is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512CD); + copy_usable_1 (x86_cpu_AVX512CD); /* Determine if AVX512ER is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512ER); + copy_usable_1 (x86_cpu_AVX512ER); /* Determine if AVX512PF is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512PF); + copy_usable_1 (x86_cpu_AVX512PF); /* Determine if AVX512VL is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512VL); + copy_usable_1 (x86_cpu_AVX512VL); /* Determine if AVX512DQ is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512DQ); + copy_usable_1 (x86_cpu_AVX512DQ); /* Determine if AVX512BW is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512BW); + copy_usable_1 (x86_cpu_AVX512BW); /* Determine if AVX512_4FMAPS is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_4FMAPS); + copy_usable_1 (x86_cpu_AVX512_4FMAPS); /* Determine if AVX512_4VNNIW is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_4VNNIW); + copy_usable_1 (x86_cpu_AVX512_4VNNIW); /* Determine if AVX512_BITALG is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_BITALG); + copy_usable_1 (x86_cpu_AVX512_BITALG); /* Determine if AVX512_IFMA is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_IFMA); + copy_usable_1 (x86_cpu_AVX512_IFMA); /* Determine if AVX512_VBMI is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_VBMI); + copy_usable_1 (x86_cpu_AVX512_VBMI); /* Determine if AVX512_VBMI2 is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_VBMI2); + copy_usable_1 (x86_cpu_AVX512_VBMI2); /* Determine if is AVX512_VNNI usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_VNNI); + copy_usable_1 (x86_cpu_AVX512_VNNI); /* Determine if AVX512_VPOPCNTDQ is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, - AVX512_VPOPCNTDQ); + copy_usable_1 (x86_cpu_AVX512_VPOPCNTDQ); /* Determine if AVX512_VP2INTERSECT is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, - AVX512_VP2INTERSECT); + copy_usable_1 (x86_cpu_AVX512_VP2INTERSECT); /* Determine if AVX512_BF16 is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_BF16); + copy_usable_1 (x86_cpu_AVX512_BF16); /* Determine if AVX512_FP16 is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_FP16); + copy_usable_1 (x86_cpu_AVX512_FP16); } } } @@ -197,19 +210,19 @@ update_usable (struct cpu_features *cpu_features) == (bit_XTILECFG_state | bit_XTILEDATA_state)) { /* Determine if AMX_BF16 is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AMX_BF16); + copy_usable_1 (x86_cpu_AMX_BF16); /* Determine if AMX_TILE is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AMX_TILE); + copy_usable_1 (x86_cpu_AMX_TILE); /* Determine if AMX_INT8 is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AMX_INT8); + copy_usable_1 (x86_cpu_AMX_INT8); } /* These features are usable only when OSXSAVE is enabled. */ - CPU_FEATURE_SET (cpu_features, XSAVE); - CPU_FEATURE_SET_USABLE (cpu_features, XSAVEOPT); - CPU_FEATURE_SET_USABLE (cpu_features, XSAVEC); - CPU_FEATURE_SET_USABLE (cpu_features, XGETBV_ECX_1); - CPU_FEATURE_SET_USABLE (cpu_features, XFD); + set_usable_1 (x86_cpu_XSAVE); + copy_usable_1 (x86_cpu_XSAVEOPT); + copy_usable_1 (x86_cpu_XSAVEC); + copy_usable_1 (x86_cpu_XGETBV_ECX_1); + copy_usable_1 (x86_cpu_XFD); /* For _dl_runtime_resolve, set xsave_state_size to xsave area size + integer register save size and align it to 64 bytes. */ @@ -229,7 +242,7 @@ update_usable (struct cpu_features *cpu_features) = xsave_state_full_size; /* Check if XSAVEC is available. */ - if (CPU_FEATURES_CPU_P (cpu_features, XSAVEC)) + if (x86_cpu_has_feature (x86_cpu_XSAVEC)) { unsigned int xstate_comp_offsets[32]; unsigned int xstate_comp_sizes[32]; @@ -272,7 +285,7 @@ update_usable (struct cpu_features *cpu_features) { cpu_features->xsave_state_size = ALIGN_UP (size + STATE_SAVE_OFFSET, 64); - CPU_FEATURE_SET (cpu_features, XSAVEC); + set_usable_1 (x86_cpu_XSAVEC); } } } @@ -280,41 +293,40 @@ update_usable (struct cpu_features *cpu_features) } /* Determine if PKU is usable. */ - if (CPU_FEATURES_CPU_P (cpu_features, OSPKE)) - CPU_FEATURE_SET (cpu_features, PKU); + if (x86_cpu_has_feature (x86_cpu_OSPKE)) + set_usable_1 (x86_cpu_PKU); /* Determine if Key Locker instructions are usable. */ - if (CPU_FEATURES_CPU_P (cpu_features, AESKLE)) + if (x86_cpu_has_feature (x86_cpu_AESKLE)) { - CPU_FEATURE_SET (cpu_features, AESKLE); - CPU_FEATURE_SET_USABLE (cpu_features, KL); - CPU_FEATURE_SET_USABLE (cpu_features, WIDE_KL); + set_usable_1 (x86_cpu_AESKLE); + copy_usable_1 (x86_cpu_KL); + copy_usable_1 (x86_cpu_WIDE_KL); } } static void -get_extended_indices (struct cpu_features *cpu_features) +get_extended_indices (void) { unsigned int eax, ebx, ecx, edx; __cpuid (0x80000000, eax, ebx, ecx, edx); - if (eax >= 0x80000001) - __cpuid (0x80000001, - cpu_features->features[COMMON_CPUID_INDEX_80000001].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_80000001].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_80000001].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_80000001].cpuid.edx); - if (eax >= 0x80000007) - __cpuid (0x80000007, - cpu_features->features[COMMON_CPUID_INDEX_80000007].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_80000007].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_80000007].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_80000007].cpuid.edx); - if (eax >= 0x80000008) - __cpuid (0x80000008, - cpu_features->features[COMMON_CPUID_INDEX_80000008].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_80000008].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_80000008].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_80000008].cpuid.edx); + unsigned int level = eax; + if (level >= 0x80000001) + { + __cpuid (0x80000001, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_LAHF64_SAHF64 >> 5] = ecx; + __x86_cpu_array_private[x86_cpu_SYSCALL_SYSRET >> 5] = edx; + } + if (level >= 0x80000007) + { + __cpuid (0x80000007, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_INVARIANT_TSC >> 5] = edx; + } + if (level >= 0x80000008) + { + __cpuid (0x80000008, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_WBNOINVD >> 5] = ebx; + } } static void @@ -324,12 +336,8 @@ get_common_indices (struct cpu_features *cpu_features, { if (family) { - unsigned int eax; - __cpuid (1, eax, - cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.edx); - cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.eax = eax; + unsigned int eax, ebx, ecx, edx; + __cpuid (1, eax, ebx, ecx, edx); *family = (eax >> 8) & 0x0f; *model = (eax >> 4) & 0x0f; *extended_model = (eax >> 12) & 0xf0; @@ -339,35 +347,35 @@ get_common_indices (struct cpu_features *cpu_features, *family += (eax >> 20) & 0xff; *model += *extended_model; } + __x86_cpu_array_private[x86_cpu_SSE3 >> 5] = ecx; + __x86_cpu_array_private[x86_cpu_FPU >> 5] = edx; } if (cpu_features->basic.max_cpuid >= 7) { - __cpuid_count (7, 0, - cpu_features->features[COMMON_CPUID_INDEX_7].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_7].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_7].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_7].cpuid.edx); - __cpuid_count (7, 1, - cpu_features->features[COMMON_CPUID_INDEX_7_ECX_1].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_7_ECX_1].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_7_ECX_1].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_7_ECX_1].cpuid.edx); + unsigned int eax, ebx, ecx, edx; + __cpuid_count (7, 0, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_FSGSBASE >> 5] = ebx; + __x86_cpu_array_private[x86_cpu_PREFETCHWT1 >> 5] = ecx; + __x86_cpu_array_private[x86_cpu_INDEX_7_EDX_0 >> 5] = edx; + + __cpuid_count (7, 1, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_AVX_VNNI >> 5] = eax; } if (cpu_features->basic.max_cpuid >= 0xd) - __cpuid_count (0xd, 1, - cpu_features->features[COMMON_CPUID_INDEX_D_ECX_1].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_D_ECX_1].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_D_ECX_1].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_D_ECX_1].cpuid.edx); + { + unsigned int eax, ebx, ecx, edx; + __cpuid_count (0xd, 1, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_XSAVEOPT >> 5] = eax; + } if (cpu_features->basic.max_cpuid >= 0x19) - __cpuid_count (0x19, 0, - cpu_features->features[COMMON_CPUID_INDEX_19].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_19].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_19].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_19].cpuid.edx); + { + unsigned int eax, ebx, ecx, edx; + __cpuid_count (0x19, 0, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_AESKLE >> 5] = ebx; + } } _Static_assert (((index_arch_Fast_Unaligned_Load @@ -411,7 +419,7 @@ init_cpu_features (struct cpu_features *cpu_features) get_common_indices (cpu_features, &family, &model, &extended_model, &stepping); - get_extended_indices (cpu_features); + get_extended_indices (); update_usable (cpu_features); @@ -473,7 +481,7 @@ init_cpu_features (struct cpu_features *cpu_features) default: /* Unknown family 0x06 processors. Assuming this is one of Core i3/i5/i7 processors if AVX is available. */ - if (!CPU_FEATURES_CPU_P (cpu_features, AVX)) + if (!x86_cpu_has_feature (x86_cpu_AVX)) break; /* Fall through. */ @@ -511,7 +519,7 @@ init_cpu_features (struct cpu_features *cpu_features) with stepping >= 4) to avoid TSX on kernels that weren't updated with the latest microcode package (which disables broken feature by default). */ - CPU_FEATURE_UNSET (cpu_features, RTM); + x86_cpu_clear_usable (x86_cpu_RTM); break; } } @@ -520,7 +528,7 @@ init_cpu_features (struct cpu_features *cpu_features) /* Since AVX512ER is unique to Xeon Phi, set Prefer_No_VZEROUPPER if AVX512ER is available. Don't use AVX512 to avoid lower CPU frequency if AVX512ER isn't available. */ - if (CPU_FEATURES_CPU_P (cpu_features, AVX512ER)) + if (x86_cpu_has_feature (x86_cpu_AVX512ER)) cpu_features->preferred[index_arch_Prefer_No_VZEROUPPER] |= bit_arch_Prefer_No_VZEROUPPER; else @@ -538,17 +546,15 @@ init_cpu_features (struct cpu_features *cpu_features) get_common_indices (cpu_features, &family, &model, &extended_model, &stepping); - get_extended_indices (cpu_features); + get_extended_indices (); update_usable (cpu_features); - ecx = cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.ecx; - - if (CPU_FEATURE_USABLE_P (cpu_features, AVX)) + if (x86_cpu_is_usable (x86_cpu_AVX)) { /* Since the FMA4 bit is in COMMON_CPUID_INDEX_80000001 and FMA4 requires AVX, determine if FMA4 is usable here. */ - CPU_FEATURE_SET_USABLE (cpu_features, FMA4); + copy_usable_1 (x86_cpu_FMA4); } if (family == 0x15) @@ -577,7 +583,7 @@ init_cpu_features (struct cpu_features *cpu_features) get_common_indices (cpu_features, &family, &model, &extended_model, &stepping); - get_extended_indices (cpu_features); + get_extended_indices (); update_usable (cpu_features); @@ -586,8 +592,8 @@ init_cpu_features (struct cpu_features *cpu_features) { if (model == 0xf || model == 0x19) { - CPU_FEATURE_UNSET (cpu_features, AVX); - CPU_FEATURE_UNSET (cpu_features, AVX2); + x86_cpu_clear_usable (x86_cpu_AVX); + x86_cpu_clear_usable (x86_cpu_AVX2); cpu_features->preferred[index_arch_Slow_SSE4_2] |= bit_arch_Slow_SSE4_2; @@ -600,8 +606,8 @@ init_cpu_features (struct cpu_features *cpu_features) { if (model == 0x1b) { - CPU_FEATURE_UNSET (cpu_features, AVX); - CPU_FEATURE_UNSET (cpu_features, AVX2); + x86_cpu_clear_usable (x86_cpu_AVX); + x86_cpu_clear_usable (x86_cpu_AVX2); cpu_features->preferred[index_arch_Slow_SSE4_2] |= bit_arch_Slow_SSE4_2; @@ -611,8 +617,8 @@ init_cpu_features (struct cpu_features *cpu_features) } else if (model == 0x3b) { - CPU_FEATURE_UNSET (cpu_features, AVX); - CPU_FEATURE_UNSET (cpu_features, AVX2); + x86_cpu_clear_usable (x86_cpu_AVX); + x86_cpu_clear_usable (x86_cpu_AVX2); cpu_features->preferred[index_arch_AVX_Fast_Unaligned_Load] &= ~bit_arch_AVX_Fast_Unaligned_Load; @@ -627,11 +633,11 @@ init_cpu_features (struct cpu_features *cpu_features) } /* Support i586 if CX8 is available. */ - if (CPU_FEATURES_CPU_P (cpu_features, CX8)) + if (x86_cpu_has_feature (x86_cpu_CX8)) cpu_features->preferred[index_arch_I586] |= bit_arch_I586; /* Support i686 if CMOV is available. */ - if (CPU_FEATURES_CPU_P (cpu_features, CMOV)) + if (x86_cpu_has_feature (x86_cpu_CMOV)) cpu_features->preferred[index_arch_I686] |= bit_arch_I686; #if !HAS_CPUID @@ -670,30 +676,30 @@ no_cpuid: { const char *platform = NULL; - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512CD)) + if (x86_cpu_is_usable (x86_cpu_AVX512CD)) { - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512ER)) + if (x86_cpu_is_usable (x86_cpu_AVX512ER)) { - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512PF)) + if (x86_cpu_is_usable (x86_cpu_AVX512PF)) platform = "xeon_phi"; } else { - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512BW) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512DQ) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512VL)) + if (x86_cpu_is_usable (x86_cpu_AVX512BW) + && x86_cpu_is_usable (x86_cpu_AVX512DQ) + && x86_cpu_is_usable (x86_cpu_AVX512VL)) GLRO(dl_hwcap) |= HWCAP_X86_AVX512_1; } } if (platform == NULL - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) - && CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, BMI1) - && CPU_FEATURE_USABLE_P (cpu_features, BMI2) - && CPU_FEATURE_USABLE_P (cpu_features, LZCNT) - && CPU_FEATURE_USABLE_P (cpu_features, MOVBE) - && CPU_FEATURE_USABLE_P (cpu_features, POPCNT)) + && x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_BMI1) + && x86_cpu_is_usable (x86_cpu_BMI2) + && x86_cpu_is_usable (x86_cpu_LZCNT) + && x86_cpu_is_usable (x86_cpu_MOVBE) + && x86_cpu_is_usable (x86_cpu_POPCNT)) platform = "haswell"; if (platform != NULL) @@ -701,7 +707,7 @@ no_cpuid: } #else GLRO(dl_hwcap) = 0; - if (CPU_FEATURE_USABLE_P (cpu_features, SSE2)) + if (x86_cpu_is_usable (x86_cpu_SSE2)) GLRO(dl_hwcap) |= HWCAP_X86_SSE2; if (CPU_FEATURES_ARCH_P (cpu_features, I686)) diff --git a/sysdeps/x86/cpu-tunables.c b/sysdeps/x86/cpu-tunables.c index 588bbf9448..3f904f7e55 100644 --- a/sysdeps/x86/cpu-tunables.c +++ b/sysdeps/x86/cpu-tunables.c @@ -43,7 +43,7 @@ extern __typeof (memcmp) DEFAULT_MEMCMP; _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ if (!DEFAULT_MEMCMP (f, #name, len)) \ { \ - CPU_FEATURE_UNSET (cpu_features, name) \ + x86_cpu_clear_usable (x86_cpu_##name); \ break; \ } @@ -80,7 +80,7 @@ extern __typeof (memcmp) DEFAULT_MEMCMP; { \ if (disable) \ cpu_features->preferred[index_arch_##name] &= ~bit_arch_##name; \ - else if (CPU_FEATURE_USABLE_P (cpu_features, need)) \ + else if (x86_cpu_is_usable (x86_cpu_##need)) \ cpu_features->preferred[index_arch_##name] |= bit_arch_##name; \ break; \ } @@ -181,7 +181,7 @@ TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *valp) /* Update xsave_state_size to XSAVE state size. */ cpu_features->xsave_state_size = cpu_features->xsave_state_full_size; - CPU_FEATURE_UNSET (cpu_features, XSAVEC); + x86_cpu_clear_usable (x86_cpu_XSAVEC); } } break; diff --git a/sysdeps/x86/include/cpu-features.h b/sysdeps/x86/include/cpu-features.h index f62be0b9b3..bbc3afdaac 100644 --- a/sysdeps/x86/include/cpu-features.h +++ b/sysdeps/x86/include/cpu-features.h @@ -30,7 +30,7 @@ # define __x86_get_cpu_features __x86_get_cpu_features_public #endif -#include +#include #ifndef _ISOMAC @@ -48,33 +48,14 @@ enum }; /* Only used directly in cpu-features.c. */ -# define CPU_FEATURE_SET(ptr, name) \ - ptr->features[index_cpu_##name].usable.reg_##name |= bit_cpu_##name; -# define CPU_FEATURE_UNSET(ptr, name) \ - ptr->features[index_cpu_##name].usable.reg_##name &= ~bit_cpu_##name; -# define CPU_FEATURE_SET_USABLE(ptr, name) \ - ptr->features[index_cpu_##name].usable.reg_##name \ - |= ptr->features[index_cpu_##name].cpuid.reg_##name & bit_cpu_##name; # define CPU_FEATURE_PREFERRED_P(ptr, name) \ ((ptr->preferred[index_arch_##name] & bit_arch_##name) != 0) -# define CPU_FEATURE_CPU_P(ptr, name) \ - CPU_FEATURE_CHECK_P (ptr, name, cpuid) -/* HAS_CPU_FEATURE evaluates to true if CPU supports the feature. */ -# undef HAS_CPU_FEATURE -# define HAS_CPU_FEATURE(name) \ - CPU_FEATURE_CPU_P (__x86_get_cpu_features (0), name) -/* CPU_FEATURE_USABLE evaluates to true if the feature is usable. */ -# undef CPU_FEATURE_USABLE -# define CPU_FEATURE_USABLE(name) \ - CPU_FEATURE_USABLE_P (__x86_get_cpu_features (0), name) /* CPU_FEATURE_PREFER evaluates to true if we prefer the feature at runtime. */ # define CPU_FEATURE_PREFERRED(name) \ CPU_FEATURE_PREFERRED_P(__get_cpu_features (), name) -# define CPU_FEATURES_CPU_P(ptr, name) \ - CPU_FEATURE_CPU_P (ptr, name) # define CPU_FEATURES_ARCH_P(ptr, name) \ CPU_FEATURE_PREFERRED_P (ptr, name) # define HAS_ARCH_FEATURE(name) \ @@ -124,10 +105,49 @@ enum # define bit_XTILECFG_state (1u << 17) # define bit_XTILEDATA_state (1u << 18) +enum +{ + COMMON_CPUID_INDEX_1 = 0, + COMMON_CPUID_INDEX_7, + COMMON_CPUID_INDEX_80000001, + COMMON_CPUID_INDEX_D_ECX_1, + COMMON_CPUID_INDEX_80000007, + COMMON_CPUID_INDEX_80000008, + COMMON_CPUID_INDEX_7_ECX_1, + COMMON_CPUID_INDEX_19, + /* Keep the following line at the end. */ + COMMON_CPUID_INDEX_MAX +}; + +struct cpuid_registers +{ + unsigned int eax; + unsigned int ebx; + unsigned int ecx; + unsigned int edx; +}; + +enum cpu_features_kind +{ + arch_kind_unknown = 0, + arch_kind_intel, + arch_kind_amd, + arch_kind_zhaoxin, + arch_kind_other +}; + +struct cpu_features_basic +{ + enum cpu_features_kind kind; + int max_cpuid; + unsigned int family; + unsigned int model; + unsigned int stepping; +}; + struct cpu_features { struct cpu_features_basic basic; - struct cpuid_features features[COMMON_CPUID_INDEX_MAX]; unsigned int preferred[PREFERRED_FEATURE_INDEX_MAX]; /* The state size for XSAVEC or XSAVE. The type must be unsigned long int so that we use diff --git a/sysdeps/x86/include/sys/platform/x86.h b/sysdeps/x86/include/sys/platform/x86.h new file mode 100644 index 0000000000..7d024aa376 --- /dev/null +++ b/sysdeps/x86/include/sys/platform/x86.h @@ -0,0 +1,62 @@ +/* Wrapper header for . + This file is part of the GNU C Library. + Copyright (C) 2008-2020 Free Software Foundation, Inc. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _SYS_PLATFORM_X86_H +# ifdef _ISOMAC +# include +# else +# if IS_IN (rtld) || (! defined (SHARED) && IS_IN (libc)) +/* libc.a and ld.so are special because they have a local array of fixed + size. This is possible because internal users know the exact array + size. Statically linked application code still uses the + indirection through __x86_cpu_array, for ABI consistency with + dynamic linking. */ + +# define _SYS_PLATFORM_X86_H +# include + +enum { x86_cpu_array_size = 12 * 2 }; +extern unsigned int __x86_cpu_array_private[x86_cpu_array_size] + attribute_hidden; + +static inline _Bool +x86_cpu_has_feature (unsigned int __index) +{ + return __x86_cpu_array_private[__index >> 5] & (1u << (__index % 32)); +} + +static inline _Bool +x86_cpu_is_usable (unsigned int __index) +{ + return __x86_cpu_array_private[(__index >> 5) + 1] & (1u << (__index % 32)); +} + +/* Mark one feature as unavailable. */ +static inline void +x86_cpu_clear_usable (unsigned int index) +{ + __x86_cpu_array_private[(index >> 5) + 1] &= ~(1u << (index % 32)); +} + +# else /* !rtld */ +# include +libc_hidden_proto (__x86_cpu_array) +# endif /* !rtld */ + +# endif /* !ISOMAC */ +#endif /* !_SYS_PLATFORM_X86_H */ diff --git a/sysdeps/x86/libc-vars-init.h b/sysdeps/x86/libc-vars-init.h new file mode 100644 index 0000000000..98e908534f --- /dev/null +++ b/sysdeps/x86/libc-vars-init.h @@ -0,0 +1,42 @@ +/* Initialization of __libc_vars. x86 version. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +/* No CPU-specific data directly in __libc_vars. */ +static inline void +_dl_libc_vars_init_cpu (struct libc_vars *vars) +{ +} + +/* But we have an additional variable that needs initialization. */ +static inline void +_dl_call_libc_vars_init_cpu (struct link_map *libc_map) +{ + const ElfW (Sym) *sym + = _dl_lookup_direct (libc_map, "__x86_cpu_array", + 0x34bfb42e, /* dl_new_hash output. */ + "GLIBC_2.33", + 0x69691b3); /* _dl_elf_hash output. */ + assert (sym != NULL); + assert (sym->st_size == sizeof (struct x86_cpu_array)); + struct x86_cpu_array *array = DL_SYMBOL_ADDRESS (libc_map, sym); + array->__x86_word = __x86_cpu_array_private; + array->__x86_count = array_length (__x86_cpu_array_private); +} diff --git a/sysdeps/x86/sys/platform/x86.h b/sysdeps/x86/sys/platform/x86.h index 99d8c9b0ab..fb0821b994 100644 --- a/sysdeps/x86/sys/platform/x86.h +++ b/sysdeps/x86/sys/platform/x86.h @@ -19,760 +19,48 @@ #ifndef _SYS_PLATFORM_X86_H #define _SYS_PLATFORM_X86_H -enum +#include +#include +#include + +__BEGIN_DECLS + +/* Access to detected CPU features. */ +extern struct x86_cpu_array __x86_cpu_array; + +#ifdef __extern_always_inline +/* Return the word referenced by INDEX, or 0 if it is out of bounds. + INDEX be one of the enumeration constants defined below, + potentially ORed with 32u, to request a usable (instead of feature) + CPU flag. */ +__extern_always_inline unsigned int __attribute__ ((const)) +__x86_cpu_array_index (unsigned int __index) { - COMMON_CPUID_INDEX_1 = 0, - COMMON_CPUID_INDEX_7, - COMMON_CPUID_INDEX_80000001, - COMMON_CPUID_INDEX_D_ECX_1, - COMMON_CPUID_INDEX_80000007, - COMMON_CPUID_INDEX_80000008, - COMMON_CPUID_INDEX_7_ECX_1, - COMMON_CPUID_INDEX_19, - /* Keep the following line at the end. */ - COMMON_CPUID_INDEX_MAX -}; - -struct cpuid_registers + if (__index >> 5 < __x86_cpu_array.__x86_count) + return __x86_cpu_array.__x86_word[__index >> 5]; + else + return 0; +} + +/* Return true if the CPU has feature INDEX, one of the enumeration + constants defined below. */ +__extern_always_inline bool __attribute__ ((const)) +x86_cpu_has_feature (unsigned int __index) { - unsigned int eax; - unsigned int ebx; - unsigned int ecx; - unsigned int edx; -}; - -struct cpuid_features + return __x86_cpu_array_index (__index) & (1u << (__index % 32)); +} + +/* Return true if the CPU has feature INDEX, one of the enumeration + constants defined below, and the feature is supported by the + current executation environment. */ +__extern_always_inline bool __attribute__ ((const)) +x86_cpu_is_usable (unsigned int __index) { - struct cpuid_registers cpuid; - struct cpuid_registers usable; -}; - -enum cpu_features_kind -{ - arch_kind_unknown = 0, - arch_kind_intel, - arch_kind_amd, - arch_kind_zhaoxin, - arch_kind_other -}; - -struct cpu_features_basic -{ - enum cpu_features_kind kind; - int max_cpuid; - unsigned int family; - unsigned int model; - unsigned int stepping; -}; - -struct cpu_features -{ - struct cpu_features_basic basic; - struct cpuid_features features[COMMON_CPUID_INDEX_MAX]; -}; - -/* Get a pointer to the CPU features structure. */ -extern const struct cpu_features *__x86_get_cpu_features (unsigned int) - __attribute__ ((const)); - -#define CPU_FEATURE_CHECK_P(ptr, name, check) \ - ((ptr->features[index_cpu_##name].check.reg_##name \ - & bit_cpu_##name) != 0) -#define CPU_FEATURE_CPU_P(ptr, name) \ - CPU_FEATURE_CHECK_P (ptr, name, cpuid) -#define CPU_FEATURE_USABLE_P(ptr, name) \ - CPU_FEATURE_CHECK_P (ptr, name, usable) - -/* HAS_CPU_FEATURE evaluates to true if CPU supports the feature. */ -#define HAS_CPU_FEATURE(name) \ - (__extension__ \ - ({ const struct cpu_features *__ptr = \ - __x86_get_cpu_features (COMMON_CPUID_INDEX_MAX); \ - __ptr && CPU_FEATURE_CPU_P (__ptr, name); })) -/* CPU_FEATURE_USABLE evaluates to true if the feature is usable. */ -#define CPU_FEATURE_USABLE(name) \ - (__extension__ \ - ({ const struct cpu_features *__ptr = \ - __x86_get_cpu_features (COMMON_CPUID_INDEX_MAX); \ - __ptr && CPU_FEATURE_USABLE_P (__ptr, name); })) - -/* CPU features. */ - -/* COMMON_CPUID_INDEX_1. */ - -/* ECX. */ -#define bit_cpu_SSE3 (1u << 0) -#define bit_cpu_PCLMULQDQ (1u << 1) -#define bit_cpu_DTES64 (1u << 2) -#define bit_cpu_MONITOR (1u << 3) -#define bit_cpu_DS_CPL (1u << 4) -#define bit_cpu_VMX (1u << 5) -#define bit_cpu_SMX (1u << 6) -#define bit_cpu_EIST (1u << 7) -#define bit_cpu_TM2 (1u << 8) -#define bit_cpu_SSSE3 (1u << 9) -#define bit_cpu_CNXT_ID (1u << 10) -#define bit_cpu_SDBG (1u << 11) -#define bit_cpu_FMA (1u << 12) -#define bit_cpu_CMPXCHG16B (1u << 13) -#define bit_cpu_XTPRUPDCTRL (1u << 14) -#define bit_cpu_PDCM (1u << 15) -#define bit_cpu_INDEX_1_ECX_16 (1u << 16) -#define bit_cpu_PCID (1u << 17) -#define bit_cpu_DCA (1u << 18) -#define bit_cpu_SSE4_1 (1u << 19) -#define bit_cpu_SSE4_2 (1u << 20) -#define bit_cpu_X2APIC (1u << 21) -#define bit_cpu_MOVBE (1u << 22) -#define bit_cpu_POPCNT (1u << 23) -#define bit_cpu_TSC_DEADLINE (1u << 24) -#define bit_cpu_AES (1u << 25) -#define bit_cpu_XSAVE (1u << 26) -#define bit_cpu_OSXSAVE (1u << 27) -#define bit_cpu_AVX (1u << 28) -#define bit_cpu_F16C (1u << 29) -#define bit_cpu_RDRAND (1u << 30) -#define bit_cpu_INDEX_1_ECX_31 (1u << 31) - -/* EDX. */ -#define bit_cpu_FPU (1u << 0) -#define bit_cpu_VME (1u << 1) -#define bit_cpu_DE (1u << 2) -#define bit_cpu_PSE (1u << 3) -#define bit_cpu_TSC (1u << 4) -#define bit_cpu_MSR (1u << 5) -#define bit_cpu_PAE (1u << 6) -#define bit_cpu_MCE (1u << 7) -#define bit_cpu_CX8 (1u << 8) -#define bit_cpu_APIC (1u << 9) -#define bit_cpu_INDEX_1_EDX_10 (1u << 10) -#define bit_cpu_SEP (1u << 11) -#define bit_cpu_MTRR (1u << 12) -#define bit_cpu_PGE (1u << 13) -#define bit_cpu_MCA (1u << 14) -#define bit_cpu_CMOV (1u << 15) -#define bit_cpu_PAT (1u << 16) -#define bit_cpu_PSE_36 (1u << 17) -#define bit_cpu_PSN (1u << 18) -#define bit_cpu_CLFSH (1u << 19) -#define bit_cpu_INDEX_1_EDX_20 (1u << 20) -#define bit_cpu_DS (1u << 21) -#define bit_cpu_ACPI (1u << 22) -#define bit_cpu_MMX (1u << 23) -#define bit_cpu_FXSR (1u << 24) -#define bit_cpu_SSE (1u << 25) -#define bit_cpu_SSE2 (1u << 26) -#define bit_cpu_SS (1u << 27) -#define bit_cpu_HTT (1u << 28) -#define bit_cpu_TM (1u << 29) -#define bit_cpu_INDEX_1_EDX_30 (1u << 30) -#define bit_cpu_PBE (1u << 31) - -/* COMMON_CPUID_INDEX_7. */ - -/* EBX. */ -#define bit_cpu_FSGSBASE (1u << 0) -#define bit_cpu_TSC_ADJUST (1u << 1) -#define bit_cpu_SGX (1u << 2) -#define bit_cpu_BMI1 (1u << 3) -#define bit_cpu_HLE (1u << 4) -#define bit_cpu_AVX2 (1u << 5) -#define bit_cpu_INDEX_7_EBX_6 (1u << 6) -#define bit_cpu_SMEP (1u << 7) -#define bit_cpu_BMI2 (1u << 8) -#define bit_cpu_ERMS (1u << 9) -#define bit_cpu_INVPCID (1u << 10) -#define bit_cpu_RTM (1u << 11) -#define bit_cpu_RDT_M (1u << 12) -#define bit_cpu_DEPR_FPU_CS_DS (1u << 13) -#define bit_cpu_MPX (1u << 14) -#define bit_cpu_RDT_A (1u << 15) -#define bit_cpu_AVX512F (1u << 16) -#define bit_cpu_AVX512DQ (1u << 17) -#define bit_cpu_RDSEED (1u << 18) -#define bit_cpu_ADX (1u << 19) -#define bit_cpu_SMAP (1u << 20) -#define bit_cpu_AVX512_IFMA (1u << 21) -#define bit_cpu_INDEX_7_EBX_22 (1u << 22) -#define bit_cpu_CLFLUSHOPT (1u << 23) -#define bit_cpu_CLWB (1u << 24) -#define bit_cpu_TRACE (1u << 25) -#define bit_cpu_AVX512PF (1u << 26) -#define bit_cpu_AVX512ER (1u << 27) -#define bit_cpu_AVX512CD (1u << 28) -#define bit_cpu_SHA (1u << 29) -#define bit_cpu_AVX512BW (1u << 30) -#define bit_cpu_AVX512VL (1u << 31) - -/* ECX. */ -#define bit_cpu_PREFETCHWT1 (1u << 0) -#define bit_cpu_AVX512_VBMI (1u << 1) -#define bit_cpu_UMIP (1u << 2) -#define bit_cpu_PKU (1u << 3) -#define bit_cpu_OSPKE (1u << 4) -#define bit_cpu_WAITPKG (1u << 5) -#define bit_cpu_AVX512_VBMI2 (1u << 6) -#define bit_cpu_SHSTK (1u << 7) -#define bit_cpu_GFNI (1u << 8) -#define bit_cpu_VAES (1u << 9) -#define bit_cpu_VPCLMULQDQ (1u << 10) -#define bit_cpu_AVX512_VNNI (1u << 11) -#define bit_cpu_AVX512_BITALG (1u << 12) -#define bit_cpu_INDEX_7_ECX_13 (1u << 13) -#define bit_cpu_AVX512_VPOPCNTDQ (1u << 14) -#define bit_cpu_INDEX_7_ECX_15 (1u << 15) -#define bit_cpu_INDEX_7_ECX_16 (1u << 16) -/* Note: Bits 17-21: The value of MAWAU used by the BNDLDX and BNDSTX - instructions in 64-bit mode. */ -#define bit_cpu_RDPID (1u << 22) -#define bit_cpu_KL (1u << 23) -#define bit_cpu_INDEX_7_ECX_24 (1u << 24) -#define bit_cpu_CLDEMOTE (1u << 25) -#define bit_cpu_INDEX_7_ECX_26 (1u << 26) -#define bit_cpu_MOVDIRI (1u << 27) -#define bit_cpu_MOVDIR64B (1u << 28) -#define bit_cpu_ENQCMD (1u << 29) -#define bit_cpu_SGX_LC (1u << 30) -#define bit_cpu_PKS (1u << 31) - -/* EDX. */ -#define bit_cpu_INDEX_7_EDX_0 (1u << 0) -#define bit_cpu_INDEX_7_EDX_1 (1u << 1) -#define bit_cpu_AVX512_4VNNIW (1u << 2) -#define bit_cpu_AVX512_4FMAPS (1u << 3) -#define bit_cpu_FSRM (1u << 4) -#define bit_cpu_UINTR (1u << 5) -#define bit_cpu_INDEX_7_EDX_6 (1u << 6) -#define bit_cpu_INDEX_7_EDX_7 (1u << 7) -#define bit_cpu_AVX512_VP2INTERSECT (1u << 8) -#define bit_cpu_INDEX_7_EDX_9 (1u << 9) -#define bit_cpu_MD_CLEAR (1u << 10) -#define bit_cpu_INDEX_7_EDX_11 (1u << 11) -#define bit_cpu_INDEX_7_EDX_12 (1u << 12) -#define bit_cpu_INDEX_7_EDX_13 (1u << 13) -#define bit_cpu_SERIALIZE (1u << 14) -#define bit_cpu_HYBRID (1u << 15) -#define bit_cpu_TSXLDTRK (1u << 16) -#define bit_cpu_INDEX_7_EDX_17 (1u << 17) -#define bit_cpu_PCONFIG (1u << 18) -#define bit_cpu_INDEX_7_EDX_19 (1u << 19) -#define bit_cpu_IBT (1u << 20) -#define bit_cpu_INDEX_7_EDX_21 (1u << 21) -#define bit_cpu_AMX_BF16 (1u << 22) -#define bit_cpu_AVX512_FP16 (1u << 23) -#define bit_cpu_AMX_TILE (1u << 24) -#define bit_cpu_AMX_INT8 (1u << 25) -#define bit_cpu_IBRS_IBPB (1u << 26) -#define bit_cpu_STIBP (1u << 27) -#define bit_cpu_L1D_FLUSH (1u << 28) -#define bit_cpu_ARCH_CAPABILITIES (1u << 29) -#define bit_cpu_CORE_CAPABILITIES (1u << 30) -#define bit_cpu_SSBD (1u << 31) - -/* COMMON_CPUID_INDEX_80000001. */ - -/* ECX. */ -#define bit_cpu_LAHF64_SAHF64 (1u << 0) -#define bit_cpu_SVM (1u << 2) -#define bit_cpu_LZCNT (1u << 5) -#define bit_cpu_SSE4A (1u << 6) -#define bit_cpu_PREFETCHW (1u << 8) -#define bit_cpu_XOP (1u << 11) -#define bit_cpu_LWP (1u << 15) -#define bit_cpu_FMA4 (1u << 16) -#define bit_cpu_TBM (1u << 21) - -/* EDX. */ -#define bit_cpu_SYSCALL_SYSRET (1u << 11) -#define bit_cpu_NX (1u << 20) -#define bit_cpu_PAGE1GB (1u << 26) -#define bit_cpu_RDTSCP (1u << 27) -#define bit_cpu_LM (1u << 29) - -/* COMMON_CPUID_INDEX_D_ECX_1. */ - -/* EAX. */ -#define bit_cpu_XSAVEOPT (1u << 0) -#define bit_cpu_XSAVEC (1u << 1) -#define bit_cpu_XGETBV_ECX_1 (1u << 2) -#define bit_cpu_XSAVES (1u << 3) -#define bit_cpu_XFD (1u << 4) - -/* COMMON_CPUID_INDEX_80000007. */ - -/* EDX. */ -#define bit_cpu_INVARIANT_TSC (1u << 8) - -/* COMMON_CPUID_INDEX_80000008. */ - -/* EBX. */ -#define bit_cpu_WBNOINVD (1u << 9) - -/* COMMON_CPUID_INDEX_7_ECX_1. */ - -/* EAX. */ -#define bit_cpu_AVX_VNNI (1u << 4) -#define bit_cpu_AVX512_BF16 (1u << 5) -#define bit_cpu_FZLRM (1u << 10) -#define bit_cpu_FSRS (1u << 11) -#define bit_cpu_FSRCS (1u << 12) -#define bit_cpu_HRESET (1u << 22) -#define bit_cpu_LAM (1u << 26) - -/* COMMON_CPUID_INDEX_19. */ - -/* EBX. */ -#define bit_cpu_AESKLE (1u << 0) -#define bit_cpu_WIDE_KL (1u << 2) - -/* COMMON_CPUID_INDEX_1. */ - -/* ECX. */ -#define index_cpu_SSE3 COMMON_CPUID_INDEX_1 -#define index_cpu_PCLMULQDQ COMMON_CPUID_INDEX_1 -#define index_cpu_DTES64 COMMON_CPUID_INDEX_1 -#define index_cpu_MONITOR COMMON_CPUID_INDEX_1 -#define index_cpu_DS_CPL COMMON_CPUID_INDEX_1 -#define index_cpu_VMX COMMON_CPUID_INDEX_1 -#define index_cpu_SMX COMMON_CPUID_INDEX_1 -#define index_cpu_EIST COMMON_CPUID_INDEX_1 -#define index_cpu_TM2 COMMON_CPUID_INDEX_1 -#define index_cpu_SSSE3 COMMON_CPUID_INDEX_1 -#define index_cpu_CNXT_ID COMMON_CPUID_INDEX_1 -#define index_cpu_SDBG COMMON_CPUID_INDEX_1 -#define index_cpu_FMA COMMON_CPUID_INDEX_1 -#define index_cpu_CMPXCHG16B COMMON_CPUID_INDEX_1 -#define index_cpu_XTPRUPDCTRL COMMON_CPUID_INDEX_1 -#define index_cpu_PDCM COMMON_CPUID_INDEX_1 -#define index_cpu_INDEX_1_ECX_16 COMMON_CPUID_INDEX_1 -#define index_cpu_PCID COMMON_CPUID_INDEX_1 -#define index_cpu_DCA COMMON_CPUID_INDEX_1 -#define index_cpu_SSE4_1 COMMON_CPUID_INDEX_1 -#define index_cpu_SSE4_2 COMMON_CPUID_INDEX_1 -#define index_cpu_X2APIC COMMON_CPUID_INDEX_1 -#define index_cpu_MOVBE COMMON_CPUID_INDEX_1 -#define index_cpu_POPCNT COMMON_CPUID_INDEX_1 -#define index_cpu_TSC_DEADLINE COMMON_CPUID_INDEX_1 -#define index_cpu_AES COMMON_CPUID_INDEX_1 -#define index_cpu_XSAVE COMMON_CPUID_INDEX_1 -#define index_cpu_OSXSAVE COMMON_CPUID_INDEX_1 -#define index_cpu_AVX COMMON_CPUID_INDEX_1 -#define index_cpu_F16C COMMON_CPUID_INDEX_1 -#define index_cpu_RDRAND COMMON_CPUID_INDEX_1 -#define index_cpu_INDEX_1_ECX_31 COMMON_CPUID_INDEX_1 - -/* ECX. */ -#define index_cpu_FPU COMMON_CPUID_INDEX_1 -#define index_cpu_VME COMMON_CPUID_INDEX_1 -#define index_cpu_DE COMMON_CPUID_INDEX_1 -#define index_cpu_PSE COMMON_CPUID_INDEX_1 -#define index_cpu_TSC COMMON_CPUID_INDEX_1 -#define index_cpu_MSR COMMON_CPUID_INDEX_1 -#define index_cpu_PAE COMMON_CPUID_INDEX_1 -#define index_cpu_MCE COMMON_CPUID_INDEX_1 -#define index_cpu_CX8 COMMON_CPUID_INDEX_1 -#define index_cpu_APIC COMMON_CPUID_INDEX_1 -#define index_cpu_INDEX_1_EDX_10 COMMON_CPUID_INDEX_1 -#define index_cpu_SEP COMMON_CPUID_INDEX_1 -#define index_cpu_MTRR COMMON_CPUID_INDEX_1 -#define index_cpu_PGE COMMON_CPUID_INDEX_1 -#define index_cpu_MCA COMMON_CPUID_INDEX_1 -#define index_cpu_CMOV COMMON_CPUID_INDEX_1 -#define index_cpu_PAT COMMON_CPUID_INDEX_1 -#define index_cpu_PSE_36 COMMON_CPUID_INDEX_1 -#define index_cpu_PSN COMMON_CPUID_INDEX_1 -#define index_cpu_CLFSH COMMON_CPUID_INDEX_1 -#define index_cpu_INDEX_1_EDX_20 COMMON_CPUID_INDEX_1 -#define index_cpu_DS COMMON_CPUID_INDEX_1 -#define index_cpu_ACPI COMMON_CPUID_INDEX_1 -#define index_cpu_MMX COMMON_CPUID_INDEX_1 -#define index_cpu_FXSR COMMON_CPUID_INDEX_1 -#define index_cpu_SSE COMMON_CPUID_INDEX_1 -#define index_cpu_SSE2 COMMON_CPUID_INDEX_1 -#define index_cpu_SS COMMON_CPUID_INDEX_1 -#define index_cpu_HTT COMMON_CPUID_INDEX_1 -#define index_cpu_TM COMMON_CPUID_INDEX_1 -#define index_cpu_INDEX_1_EDX_30 COMMON_CPUID_INDEX_1 -#define index_cpu_PBE COMMON_CPUID_INDEX_1 - -/* COMMON_CPUID_INDEX_7. */ - -/* EBX. */ -#define index_cpu_FSGSBASE COMMON_CPUID_INDEX_7 -#define index_cpu_TSC_ADJUST COMMON_CPUID_INDEX_7 -#define index_cpu_SGX COMMON_CPUID_INDEX_7 -#define index_cpu_BMI1 COMMON_CPUID_INDEX_7 -#define index_cpu_HLE COMMON_CPUID_INDEX_7 -#define index_cpu_AVX2 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EBX_6 COMMON_CPUID_INDEX_7 -#define index_cpu_SMEP COMMON_CPUID_INDEX_7 -#define index_cpu_BMI2 COMMON_CPUID_INDEX_7 -#define index_cpu_ERMS COMMON_CPUID_INDEX_7 -#define index_cpu_INVPCID COMMON_CPUID_INDEX_7 -#define index_cpu_RTM COMMON_CPUID_INDEX_7 -#define index_cpu_RDT_M COMMON_CPUID_INDEX_7 -#define index_cpu_DEPR_FPU_CS_DS COMMON_CPUID_INDEX_7 -#define index_cpu_MPX COMMON_CPUID_INDEX_7 -#define index_cpu_RDT_A COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512F COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512DQ COMMON_CPUID_INDEX_7 -#define index_cpu_RDSEED COMMON_CPUID_INDEX_7 -#define index_cpu_ADX COMMON_CPUID_INDEX_7 -#define index_cpu_SMAP COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_IFMA COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EBX_22 COMMON_CPUID_INDEX_7 -#define index_cpu_CLFLUSHOPT COMMON_CPUID_INDEX_7 -#define index_cpu_CLWB COMMON_CPUID_INDEX_7 -#define index_cpu_TRACE COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512PF COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512ER COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512CD COMMON_CPUID_INDEX_7 -#define index_cpu_SHA COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512BW COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512VL COMMON_CPUID_INDEX_7 - -/* ECX. */ -#define index_cpu_PREFETCHWT1 COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_VBMI COMMON_CPUID_INDEX_7 -#define index_cpu_UMIP COMMON_CPUID_INDEX_7 -#define index_cpu_PKU COMMON_CPUID_INDEX_7 -#define index_cpu_OSPKE COMMON_CPUID_INDEX_7 -#define index_cpu_WAITPKG COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_VBMI2 COMMON_CPUID_INDEX_7 -#define index_cpu_SHSTK COMMON_CPUID_INDEX_7 -#define index_cpu_GFNI COMMON_CPUID_INDEX_7 -#define index_cpu_VAES COMMON_CPUID_INDEX_7 -#define index_cpu_VPCLMULQDQ COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_VNNI COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_BITALG COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_ECX_13 COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_VPOPCNTDQ COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_ECX_15 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_ECX_16 COMMON_CPUID_INDEX_7 -#define index_cpu_RDPID COMMON_CPUID_INDEX_7 -#define index_cpu_KL COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_ECX_24 COMMON_CPUID_INDEX_7 -#define index_cpu_CLDEMOTE COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_ECX_26 COMMON_CPUID_INDEX_7 -#define index_cpu_MOVDIRI COMMON_CPUID_INDEX_7 -#define index_cpu_MOVDIR64B COMMON_CPUID_INDEX_7 -#define index_cpu_ENQCMD COMMON_CPUID_INDEX_7 -#define index_cpu_SGX_LC COMMON_CPUID_INDEX_7 -#define index_cpu_PKS COMMON_CPUID_INDEX_7 - -/* EDX. */ -#define index_cpu_INDEX_7_EDX_0 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_1 COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_4VNNIW COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_4FMAPS COMMON_CPUID_INDEX_7 -#define index_cpu_FSRM COMMON_CPUID_INDEX_7 -#define index_cpu_UINTR COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_6 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_7 COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_VP2INTERSECT COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_9 COMMON_CPUID_INDEX_7 -#define index_cpu_MD_CLEAR COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_11 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_12 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_13 COMMON_CPUID_INDEX_7 -#define index_cpu_SERIALIZE COMMON_CPUID_INDEX_7 -#define index_cpu_HYBRID COMMON_CPUID_INDEX_7 -#define index_cpu_TSXLDTRK COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_17 COMMON_CPUID_INDEX_7 -#define index_cpu_PCONFIG COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_19 COMMON_CPUID_INDEX_7 -#define index_cpu_IBT COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_21 COMMON_CPUID_INDEX_7 -#define index_cpu_AMX_BF16 COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_FP16 COMMON_CPUID_INDEX_7 -#define index_cpu_AMX_TILE COMMON_CPUID_INDEX_7 -#define index_cpu_AMX_INT8 COMMON_CPUID_INDEX_7 -#define index_cpu_IBRS_IBPB COMMON_CPUID_INDEX_7 -#define index_cpu_STIBP COMMON_CPUID_INDEX_7 -#define index_cpu_L1D_FLUSH COMMON_CPUID_INDEX_7 -#define index_cpu_ARCH_CAPABILITIES COMMON_CPUID_INDEX_7 -#define index_cpu_CORE_CAPABILITIES COMMON_CPUID_INDEX_7 -#define index_cpu_SSBD COMMON_CPUID_INDEX_7 - -/* COMMON_CPUID_INDEX_80000001. */ - -/* ECX. */ -#define index_cpu_LAHF64_SAHF64 COMMON_CPUID_INDEX_80000001 -#define index_cpu_SVM COMMON_CPUID_INDEX_80000001 -#define index_cpu_LZCNT COMMON_CPUID_INDEX_80000001 -#define index_cpu_SSE4A COMMON_CPUID_INDEX_80000001 -#define index_cpu_PREFETCHW COMMON_CPUID_INDEX_80000001 -#define index_cpu_XOP COMMON_CPUID_INDEX_80000001 -#define index_cpu_LWP COMMON_CPUID_INDEX_80000001 -#define index_cpu_FMA4 COMMON_CPUID_INDEX_80000001 -#define index_cpu_TBM COMMON_CPUID_INDEX_80000001 - -/* EDX. */ -#define index_cpu_SYSCALL_SYSRET COMMON_CPUID_INDEX_80000001 -#define index_cpu_NX COMMON_CPUID_INDEX_80000001 -#define index_cpu_PAGE1GB COMMON_CPUID_INDEX_80000001 -#define index_cpu_RDTSCP COMMON_CPUID_INDEX_80000001 -#define index_cpu_LM COMMON_CPUID_INDEX_80000001 - -/* COMMON_CPUID_INDEX_D_ECX_1. */ - -/* EAX. */ -#define index_cpu_XSAVEOPT COMMON_CPUID_INDEX_D_ECX_1 -#define index_cpu_XSAVEC COMMON_CPUID_INDEX_D_ECX_1 -#define index_cpu_XGETBV_ECX_1 COMMON_CPUID_INDEX_D_ECX_1 -#define index_cpu_XSAVES COMMON_CPUID_INDEX_D_ECX_1 -#define index_cpu_XFD COMMON_CPUID_INDEX_D_ECX_1 - -/* COMMON_CPUID_INDEX_80000007. */ - -/* EDX. */ -#define index_cpu_INVARIANT_TSC COMMON_CPUID_INDEX_80000007 - -/* COMMON_CPUID_INDEX_80000008. */ - -/* EBX. */ -#define index_cpu_WBNOINVD COMMON_CPUID_INDEX_80000008 - -/* COMMON_CPUID_INDEX_7_ECX_1. */ - -/* EAX. */ -#define index_cpu_AVX_VNNI COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_AVX512_BF16 COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_FZLRM COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_FSRS COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_FSRCS COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_HRESET COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_LAM COMMON_CPUID_INDEX_7_ECX_1 - -/* COMMON_CPUID_INDEX_19. */ - -/* EBX. */ -#define index_cpu_AESKLE COMMON_CPUID_INDEX_19 -#define index_cpu_WIDE_KL COMMON_CPUID_INDEX_19 - -/* COMMON_CPUID_INDEX_1. */ - -/* ECX. */ -#define reg_SSE3 ecx -#define reg_PCLMULQDQ ecx -#define reg_DTES64 ecx -#define reg_MONITOR ecx -#define reg_DS_CPL ecx -#define reg_VMX ecx -#define reg_SMX ecx -#define reg_EIST ecx -#define reg_TM2 ecx -#define reg_SSSE3 ecx -#define reg_CNXT_ID ecx -#define reg_SDBG ecx -#define reg_FMA ecx -#define reg_CMPXCHG16B ecx -#define reg_XTPRUPDCTRL ecx -#define reg_PDCM ecx -#define reg_INDEX_1_ECX_16 ecx -#define reg_PCID ecx -#define reg_DCA ecx -#define reg_SSE4_1 ecx -#define reg_SSE4_2 ecx -#define reg_X2APIC ecx -#define reg_MOVBE ecx -#define reg_POPCNT ecx -#define reg_TSC_DEADLINE ecx -#define reg_AES ecx -#define reg_XSAVE ecx -#define reg_OSXSAVE ecx -#define reg_AVX ecx -#define reg_F16C ecx -#define reg_RDRAND ecx -#define reg_INDEX_1_ECX_31 ecx - -/* EDX. */ -#define reg_FPU edx -#define reg_VME edx -#define reg_DE edx -#define reg_PSE edx -#define reg_TSC edx -#define reg_MSR edx -#define reg_PAE edx -#define reg_MCE edx -#define reg_CX8 edx -#define reg_APIC edx -#define reg_INDEX_1_EDX_10 edx -#define reg_SEP edx -#define reg_MTRR edx -#define reg_PGE edx -#define reg_MCA edx -#define reg_CMOV edx -#define reg_PAT edx -#define reg_PSE_36 edx -#define reg_PSN edx -#define reg_CLFSH edx -#define reg_INDEX_1_EDX_20 edx -#define reg_DS edx -#define reg_ACPI edx -#define reg_MMX edx -#define reg_FXSR edx -#define reg_SSE edx -#define reg_SSE2 edx -#define reg_SS edx -#define reg_HTT edx -#define reg_TM edx -#define reg_INDEX_1_EDX_30 edx -#define reg_PBE edx - -/* COMMON_CPUID_INDEX_7. */ - -/* EBX. */ -#define reg_FSGSBASE ebx -#define reg_TSC_ADJUST ebx -#define reg_SGX ebx -#define reg_BMI1 ebx -#define reg_HLE ebx -#define reg_BMI2 ebx -#define reg_AVX2 ebx -#define reg_INDEX_7_EBX_6 ebx -#define reg_SMEP ebx -#define reg_ERMS ebx -#define reg_INVPCID ebx -#define reg_RTM ebx -#define reg_RDT_M ebx -#define reg_DEPR_FPU_CS_DS ebx -#define reg_MPX ebx -#define reg_RDT_A ebx -#define reg_AVX512F ebx -#define reg_AVX512DQ ebx -#define reg_RDSEED ebx -#define reg_ADX ebx -#define reg_SMAP ebx -#define reg_AVX512_IFMA ebx -#define reg_INDEX_7_EBX_22 ebx -#define reg_CLFLUSHOPT ebx -#define reg_CLWB ebx -#define reg_TRACE ebx -#define reg_AVX512PF ebx -#define reg_AVX512ER ebx -#define reg_AVX512CD ebx -#define reg_SHA ebx -#define reg_AVX512BW ebx -#define reg_AVX512VL ebx - -/* ECX. */ -#define reg_PREFETCHWT1 ecx -#define reg_AVX512_VBMI ecx -#define reg_UMIP ecx -#define reg_PKU ecx -#define reg_OSPKE ecx -#define reg_WAITPKG ecx -#define reg_AVX512_VBMI2 ecx -#define reg_SHSTK ecx -#define reg_GFNI ecx -#define reg_VAES ecx -#define reg_VPCLMULQDQ ecx -#define reg_AVX512_VNNI ecx -#define reg_AVX512_BITALG ecx -#define reg_INDEX_7_ECX_13 ecx -#define reg_AVX512_VPOPCNTDQ ecx -#define reg_INDEX_7_ECX_15 ecx -#define reg_INDEX_7_ECX_16 ecx -#define reg_RDPID ecx -#define reg_KL ecx -#define reg_INDEX_7_ECX_24 ecx -#define reg_CLDEMOTE ecx -#define reg_INDEX_7_ECX_26 ecx -#define reg_MOVDIRI ecx -#define reg_MOVDIR64B ecx -#define reg_ENQCMD ecx -#define reg_SGX_LC ecx -#define reg_PKS ecx - -/* EDX. */ -#define reg_INDEX_7_EDX_0 edx -#define reg_INDEX_7_EDX_1 edx -#define reg_AVX512_4VNNIW edx -#define reg_AVX512_4FMAPS edx -#define reg_FSRM edx -#define reg_UINTR edx -#define reg_INDEX_7_EDX_6 edx -#define reg_INDEX_7_EDX_7 edx -#define reg_AVX512_VP2INTERSECT edx -#define reg_INDEX_7_EDX_9 edx -#define reg_MD_CLEAR edx -#define reg_INDEX_7_EDX_11 edx -#define reg_INDEX_7_EDX_12 edx -#define reg_INDEX_7_EDX_13 edx -#define reg_SERIALIZE edx -#define reg_HYBRID edx -#define reg_TSXLDTRK edx -#define reg_INDEX_7_EDX_17 edx -#define reg_PCONFIG edx -#define reg_INDEX_7_EDX_19 edx -#define reg_IBT edx -#define reg_INDEX_7_EDX_21 edx -#define reg_AMX_BF16 edx -#define reg_AVX512_FP16 edx -#define reg_AMX_TILE edx -#define reg_AMX_INT8 edx -#define reg_IBRS_IBPB edx -#define reg_STIBP edx -#define reg_L1D_FLUSH edx -#define reg_ARCH_CAPABILITIES edx -#define reg_CORE_CAPABILITIES edx -#define reg_SSBD edx - -/* COMMON_CPUID_INDEX_80000001. */ - -/* ECX. */ -#define reg_LAHF64_SAHF64 ecx -#define reg_SVM ecx -#define reg_LZCNT ecx -#define reg_SSE4A ecx -#define reg_PREFETCHW ecx -#define reg_XOP ecx -#define reg_LWP ecx -#define reg_FMA4 ecx -#define reg_TBM ecx - -/* EDX. */ -#define reg_SYSCALL_SYSRET edx -#define reg_NX edx -#define reg_PAGE1GB edx -#define reg_RDTSCP edx -#define reg_LM edx - -/* COMMON_CPUID_INDEX_D_ECX_1. */ - -/* EAX. */ -#define reg_XSAVEOPT eax -#define reg_XSAVEC eax -#define reg_XGETBV_ECX_1 eax -#define reg_XSAVES eax -#define reg_XFD eax - -/* COMMON_CPUID_INDEX_80000007. */ - -/* EDX. */ -#define reg_INVARIANT_TSC edx - -/* COMMON_CPUID_INDEX_80000008. */ - -/* EBX. */ -#define reg_WBNOINVD ebx - -/* COMMON_CPUID_INDEX_7_ECX_1. */ - -/* EAX. */ -#define reg_AVX_VNNI eax -#define reg_AVX512_BF16 eax -#define reg_FZLRM eax -#define reg_FSRS eax -#define reg_FSRCS eax -#define reg_HRESET eax -#define reg_LAM eax + return __x86_cpu_array_index (__index | 32u) & (1u << (__index % 32)); +} -/* COMMON_CPUID_INDEX_19. */ +#endif /* __extern_always_inline */ -/* EBX. */ -#define reg_AESKLE ebx -#define reg_WIDE_KL ebx +__END_DECLS #endif /* _SYS_PLATFORM_X86_H */ diff --git a/sysdeps/x86/tst-cpu-features-cpuinfo.c b/sysdeps/x86/tst-cpu-features-cpuinfo.c index 96277284d1..92a2a169fd 100644 --- a/sysdeps/x86/tst-cpu-features-cpuinfo.c +++ b/sysdeps/x86/tst-cpu-features-cpuinfo.c @@ -74,8 +74,9 @@ check_proc (const char *proc_name, int flag, int usable, const char *name) return (found != flag); } -#define CHECK_PROC(str, name) \ - check_proc (#str, HAS_CPU_FEATURE (name), CPU_FEATURE_USABLE (name), \ +#define CHECK_PROC(str, name) \ + check_proc (#str, x86_cpu_has_feature (x86_cpu_##name), \ + x86_cpu_is_usable (x86_cpu_##name), \ "HAS_CPU_FEATURE (" #name ")"); static int diff --git a/sysdeps/x86/tst-cpu-features-supports.c b/sysdeps/x86/tst-cpu-features-supports.c index 287cf01fbd..0844b1f400 100644 --- a/sysdeps/x86/tst-cpu-features-supports.c +++ b/sysdeps/x86/tst-cpu-features-supports.c @@ -37,15 +37,15 @@ check_supports (int supports, int usable, const char *supports_name, return 0; } -#define CHECK_SUPPORTS(str, name) \ - check_supports (__builtin_cpu_supports (#str), \ - CPU_FEATURE_USABLE (name), \ - #str, "CPU_FEATURE_USABLE (" #name ")"); +#define CHECK_SUPPORTS(str, name) \ + check_supports (__builtin_cpu_supports (#str), \ + x86_cpu_is_usable (x86_cpu_##name), \ + #str, "x86_cpu_is_usable (x86_cpu_" #name ")"); -#define CHECK_CPU_SUPPORTS(str, name) \ - check_supports (__builtin_cpu_supports (#str), \ - HAS_CPU_FEATURE (name), \ - #str, "HAS_CPU_FEATURE (" #name ")"); +#define CHECK_CPU_SUPPORTS(str, name) \ + check_supports (__builtin_cpu_supports (#str), \ + x86_cpu_has_feature (x86_cpu_##name), \ + #str, "x86_cpu_has_feature (" #name ")"); static int do_test (int argc, char **argv) diff --git a/sysdeps/x86/tst-get-cpu-features.c b/sysdeps/x86/tst-get-cpu-features.c index 6f1e925a6a..e17939e77c 100644 --- a/sysdeps/x86/tst-get-cpu-features.c +++ b/sysdeps/x86/tst-get-cpu-features.c @@ -21,50 +21,23 @@ #include #include -#define CHECK_CPU_FEATURE(name) \ - { \ - if (HAS_CPU_FEATURE (name)) \ - printf (" " #name "\n"); \ +#define CHECK_CPU_FEATURE(name) \ + { \ + if (x86_cpu_has_feature (x86_cpu_##name)) \ + printf (" " #name "\n"); \ } -#define CHECK_CPU_FEATURE_USABLE(name) \ - { \ - if (CPU_FEATURE_USABLE(name)) \ - printf (" " #name "\n"); \ +#define CHECK_CPU_FEATURE_USABLE(name) \ + { \ + if (x86_cpu_is_usable (x86_cpu_##name)) \ + printf (" " #name "\n"); \ } -static const char * const cpu_kinds[] = -{ - "Unknown", - "Intel", - "AMD", - "ZHAOXIN", - "Other", -}; - static int do_test (void) { - const struct cpu_features *cpu_features = __x86_get_cpu_features (0); - - switch (cpu_features->basic.kind) - { - case arch_kind_intel: - case arch_kind_amd: - case arch_kind_zhaoxin: - case arch_kind_other: - printf ("Vendor: %s\n", cpu_kinds[cpu_features->basic.kind]); - printf ("Family: 0x%x\n", cpu_features->basic.family); - printf ("Model: 0x%x\n", cpu_features->basic.model); - printf ("Stepping: 0x%x\n", cpu_features->basic.stepping); - break; - - default: - abort (); - } - #ifdef __SSE2__ - TEST_VERIFY_EXIT (HAS_CPU_FEATURE (SSE2)); + TEST_VERIFY_EXIT (x86_cpu_has_feature (x86_cpu_SSE2)); #endif printf ("CPU features:\n"); diff --git a/sysdeps/x86/x86_cpu_array.c b/sysdeps/x86/x86_cpu_array.c new file mode 100644 index 0000000000..1561d43686 --- /dev/null +++ b/sysdeps/x86/x86_cpu_array.c @@ -0,0 +1,31 @@ +/* x86 CPU feature data. Copy in libc.so.6. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +struct x86_cpu_array __x86_cpu_array + attribute_relro __attribute__ ((nocommon)) = + { + /* In the dynamically linked case, libc.so.6 is initialized prior + to relocation via _dl_call_libc_vars_init_cpu. */ +#ifndef SHARED + .__x86_word = __x86_cpu_array_private, + .__x86_count = x86_cpu_array_size, +#endif + }; +libc_hidden_def (__x86_cpu_array) diff --git a/sysdeps/x86/x86_cpu_array_private.c b/sysdeps/x86/x86_cpu_array_private.c new file mode 100644 index 0000000000..550672b0ec --- /dev/null +++ b/sysdeps/x86/x86_cpu_array_private.c @@ -0,0 +1,21 @@ +/* x86 CPU feature data. Version for libc.a or ld.so. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +unsigned int __x86_cpu_array_private[x86_cpu_array_size]; diff --git a/sysdeps/x86_64/dl-hwcaps-subdirs.c b/sysdeps/x86_64/dl-hwcaps-subdirs.c index 8810a822ef..a16fc056f6 100644 --- a/sysdeps/x86_64/dl-hwcaps-subdirs.c +++ b/sysdeps/x86_64/dl-hwcaps-subdirs.c @@ -30,35 +30,35 @@ _dl_hwcaps_subdirs_active (void) /* Test in reverse preference order. */ /* x86-64-v2. */ - if (!(CPU_FEATURE_USABLE (CMPXCHG16B) - && CPU_FEATURE_USABLE (LAHF64_SAHF64) - && CPU_FEATURE_USABLE (POPCNT) - && CPU_FEATURE_USABLE (SSE3) - && CPU_FEATURE_USABLE (SSE4_1) - && CPU_FEATURE_USABLE (SSE4_2) - && CPU_FEATURE_USABLE (SSSE3))) + if (!(x86_cpu_is_usable (x86_cpu_CMPXCHG16B) + && x86_cpu_is_usable (x86_cpu_LAHF64_SAHF64) + && x86_cpu_is_usable (x86_cpu_POPCNT) + && x86_cpu_is_usable (x86_cpu_SSE3) + && x86_cpu_is_usable (x86_cpu_SSE4_1) + && x86_cpu_is_usable (x86_cpu_SSE4_2) + && x86_cpu_is_usable (x86_cpu_SSSE3))) return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active); ++active; /* x86-64-v3. */ - if (!(CPU_FEATURE_USABLE (AVX) - && CPU_FEATURE_USABLE (AVX2) - && CPU_FEATURE_USABLE (BMI1) - && CPU_FEATURE_USABLE (BMI2) - && CPU_FEATURE_USABLE (F16C) - && CPU_FEATURE_USABLE (FMA) - && CPU_FEATURE_USABLE (LZCNT) - && CPU_FEATURE_USABLE (MOVBE) - && CPU_FEATURE_USABLE (OSXSAVE))) + if (!(x86_cpu_is_usable (x86_cpu_AVX) + && x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_BMI1) + && x86_cpu_is_usable (x86_cpu_BMI2) + && x86_cpu_is_usable (x86_cpu_F16C) + && x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_LZCNT) + && x86_cpu_is_usable (x86_cpu_MOVBE) + && x86_cpu_is_usable (x86_cpu_OSXSAVE))) return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active); ++active; /* x86-64-v4. */ - if (!(CPU_FEATURE_USABLE (AVX512F) - && CPU_FEATURE_USABLE (AVX512BW) - && CPU_FEATURE_USABLE (AVX512CD) - && CPU_FEATURE_USABLE (AVX512DQ) - && CPU_FEATURE_USABLE (AVX512VL))) + if (!(x86_cpu_is_usable (x86_cpu_AVX512F) + && x86_cpu_is_usable (x86_cpu_AVX512BW) + && x86_cpu_is_usable (x86_cpu_AVX512CD) + && x86_cpu_is_usable (x86_cpu_AVX512DQ) + && x86_cpu_is_usable (x86_cpu_AVX512VL))) return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active); ++active; diff --git a/sysdeps/x86_64/dl-machine.h b/sysdeps/x86_64/dl-machine.h index bb93c7c6ab..ec3312edcd 100644 --- a/sysdeps/x86_64/dl-machine.h +++ b/sysdeps/x86_64/dl-machine.h @@ -98,9 +98,9 @@ elf_machine_runtime_setup (struct link_map *l, int lazy, int profile) end in this function. */ if (__glibc_unlikely (profile)) { - if (CPU_FEATURE_USABLE (AVX512F)) + if (x86_cpu_is_usable (x86_cpu_AVX512F)) *(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_profile_avx512; - else if (CPU_FEATURE_USABLE (AVX)) + else if (x86_cpu_is_usable (x86_cpu_AVX)) *(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_profile_avx; else *(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_profile_sse; @@ -118,7 +118,7 @@ elf_machine_runtime_setup (struct link_map *l, int lazy, int profile) the resolved address. */ if (GLRO(dl_x86_cpu_features).xsave_state_size != 0) *(ElfW(Addr) *) (got + 2) - = (CPU_FEATURE_USABLE (XSAVEC) + = (x86_cpu_is_usable (x86_cpu_XSAVEC) ? (ElfW(Addr)) &_dl_runtime_resolve_xsavec : (ElfW(Addr)) &_dl_runtime_resolve_xsave); else diff --git a/sysdeps/x86_64/fpu/math-tests-arch.h b/sysdeps/x86_64/fpu/math-tests-arch.h index cc3c2b0c11..724b174124 100644 --- a/sysdeps/x86_64/fpu/math-tests-arch.h +++ b/sysdeps/x86_64/fpu/math-tests-arch.h @@ -24,7 +24,7 @@ # define CHECK_ARCH_EXT \ do \ { \ - if (!CPU_FEATURE_USABLE (AVX)) return; \ + if (!x86_cpu_is_usable (x86_cpu_AVX)) return; \ } \ while (0) @@ -34,7 +34,7 @@ # define CHECK_ARCH_EXT \ do \ { \ - if (!CPU_FEATURE_USABLE (AVX2)) return; \ + if (!x86_cpu_is_usable (x86_cpu_AVX2)) return; \ } \ while (0) @@ -44,7 +44,7 @@ # define CHECK_ARCH_EXT \ do \ { \ - if (!CPU_FEATURE_USABLE (AVX512F)) return; \ + if (!x86_cpu_is_usable (x86_cpu_AVX512F)) return; \ } \ while (0) diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-avx-fma4.h b/sysdeps/x86_64/fpu/multiarch/ifunc-avx-fma4.h index 95fe2f4d70..ae344be0be 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-avx-fma4.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-avx-fma4.h @@ -27,16 +27,14 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (fma4) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2)) + if (x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_AVX2)) return OPTIMIZE (fma); - if (CPU_FEATURE_USABLE_P (cpu_features, FMA4)) + if (x86_cpu_is_usable (x86_cpu_FMA4)) return OPTIMIZE (fma4); - if (CPU_FEATURE_USABLE_P (cpu_features, AVX)) + if (x86_cpu_is_usable (x86_cpu_AVX)) return OPTIMIZE (avx); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-fma.h b/sysdeps/x86_64/fpu/multiarch/ifunc-fma.h index 0a25a44ab0..2c8efe993c 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-fma.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-fma.h @@ -24,10 +24,8 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (fma) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2)) + if (x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_AVX2)) return OPTIMIZE (fma); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-fma4.h b/sysdeps/x86_64/fpu/multiarch/ifunc-fma4.h index e5fd5ac9cb..57db900e6e 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-fma4.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-fma4.h @@ -26,13 +26,11 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (fma4) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2)) + if (x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_AVX2)) return OPTIMIZE (fma); - if (CPU_FEATURE_USABLE_P (cpu_features, FMA4)) + if (x86_cpu_is_usable (x86_cpu_FMA4)) return OPTIMIZE (fma4); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx2.h b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx2.h index 2655e55444..410f80fa13 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx2.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx2.h @@ -29,10 +29,8 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (avx2) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2)) + if (x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_AVX2)) return OPTIMIZE (avx2); return OPTIMIZE (sse_wrapper); diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx512.h b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx512.h index 5f8326503b..462ef20d7f 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx512.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx512.h @@ -34,10 +34,10 @@ IFUNC_SELECTOR (void) if (!CPU_FEATURES_ARCH_P (cpu_features, MathVec_Prefer_No_AVX512)) { - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512DQ)) + if (x86_cpu_is_usable (x86_cpu_AVX512DQ)) return OPTIMIZE (skx); - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512F)) + if (x86_cpu_is_usable (x86_cpu_AVX512F)) return OPTIMIZE (knl); } diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-sse4_1.h b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-sse4_1.h index 7240e554c9..700331d5a5 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-sse4_1.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-sse4_1.h @@ -29,9 +29,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse4) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_1)) + if (x86_cpu_is_usable (x86_cpu_SSE4_1)) return OPTIMIZE (sse4); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-sse4_1.h b/sysdeps/x86_64/fpu/multiarch/ifunc-sse4_1.h index e5d8a6f932..65972d0923 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-sse4_1.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-sse4_1.h @@ -24,9 +24,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse41) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_1)) + if (x86_cpu_is_usable (x86_cpu_SSE4_1)) return OPTIMIZE (sse41); return OPTIMIZE (c); diff --git a/sysdeps/x86_64/fpu/multiarch/s_fma.c b/sysdeps/x86_64/fpu/multiarch/s_fma.c index 0d8c0b7911..27a35fb7e0 100644 --- a/sysdeps/x86_64/fpu/multiarch/s_fma.c +++ b/sysdeps/x86_64/fpu/multiarch/s_fma.c @@ -41,8 +41,8 @@ __fma_fma4 (double x, double y, double z) } -libm_ifunc (__fma, CPU_FEATURE_USABLE (FMA) - ? __fma_fma3 : (CPU_FEATURE_USABLE (FMA4) +libm_ifunc (__fma, x86_cpu_is_usable (x86_cpu_FMA) + ? __fma_fma3 : (x86_cpu_is_usable (x86_cpu_FMA4) ? __fma_fma4 : __fma_sse2)); libm_alias_double (__fma, fma) diff --git a/sysdeps/x86_64/fpu/multiarch/s_fmaf.c b/sysdeps/x86_64/fpu/multiarch/s_fmaf.c index c01e5a21d4..fedddd1346 100644 --- a/sysdeps/x86_64/fpu/multiarch/s_fmaf.c +++ b/sysdeps/x86_64/fpu/multiarch/s_fmaf.c @@ -40,8 +40,8 @@ __fmaf_fma4 (float x, float y, float z) } -libm_ifunc (__fmaf, CPU_FEATURE_USABLE (FMA) - ? __fmaf_fma3 : (CPU_FEATURE_USABLE (FMA4) +libm_ifunc (__fmaf, x86_cpu_is_usable (x86_cpu_FMA) + ? __fmaf_fma3 : (x86_cpu_is_usable (x86_cpu_FMA4) ? __fmaf_fma4 : __fmaf_sse2)); libm_alias_float (__fma, fma) diff --git a/sysdeps/x86_64/multiarch/ifunc-avx2.h b/sysdeps/x86_64/multiarch/ifunc-avx2.h index f4e311d470..af3afefa32 100644 --- a/sysdeps/x86_64/multiarch/ifunc-avx2.h +++ b/sysdeps/x86_64/multiarch/ifunc-avx2.h @@ -28,7 +28,7 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); diff --git a/sysdeps/x86_64/multiarch/ifunc-impl-list.c b/sysdeps/x86_64/multiarch/ifunc-impl-list.c index f93ec39d98..e9d8c60da0 100644 --- a/sysdeps/x86_64/multiarch/ifunc-impl-list.c +++ b/sysdeps/x86_64/multiarch/ifunc-impl-list.c @@ -41,19 +41,19 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/memchr.c. */ IFUNC_IMPL (i, name, memchr, IFUNC_IMPL_ADD (array, i, memchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memchr_avx2) IFUNC_IMPL_ADD (array, i, memchr, 1, __memchr_sse2)) /* Support sysdeps/x86_64/multiarch/memcmp.c. */ IFUNC_IMPL (i, name, memcmp, IFUNC_IMPL_ADD (array, i, memcmp, - (CPU_FEATURE_USABLE (AVX2) - && CPU_FEATURE_USABLE (MOVBE)), + (x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_MOVBE)), __memcmp_avx2_movbe) - IFUNC_IMPL_ADD (array, i, memcmp, CPU_FEATURE_USABLE (SSE4_1), + IFUNC_IMPL_ADD (array, i, memcmp, x86_cpu_is_usable (x86_cpu_SSE4_1), __memcmp_sse4_1) - IFUNC_IMPL_ADD (array, i, memcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __memcmp_ssse3) IFUNC_IMPL_ADD (array, i, memcmp, 1, __memcmp_sse2)) @@ -61,25 +61,25 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/memmove_chk.c. */ IFUNC_IMPL (i, name, __memmove_chk, IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_chk_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_chk_avx512_unaligned) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_chk_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memmove_chk_avx_unaligned) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memmove_chk_avx_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_chk_ssse3_back) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_chk_ssse3) IFUNC_IMPL_ADD (array, i, __memmove_chk, 1, __memmove_chk_sse2_unaligned) @@ -92,23 +92,23 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/memmove.c. */ IFUNC_IMPL (i, name, memmove, IFUNC_IMPL_ADD (array, i, memmove, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memmove_avx_unaligned) IFUNC_IMPL_ADD (array, i, memmove, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memmove_avx_unaligned_erms) IFUNC_IMPL_ADD (array, i, memmove, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, memmove, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_avx512_unaligned) IFUNC_IMPL_ADD (array, i, memmove, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_avx512_unaligned_erms) - IFUNC_IMPL_ADD (array, i, memmove, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memmove, x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_ssse3_back) - IFUNC_IMPL_ADD (array, i, memmove, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memmove, x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_ssse3) IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_erms) IFUNC_IMPL_ADD (array, i, memmove, 1, @@ -119,7 +119,7 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/memrchr.c. */ IFUNC_IMPL (i, name, memrchr, IFUNC_IMPL_ADD (array, i, memrchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memrchr_avx2) IFUNC_IMPL_ADD (array, i, memrchr, 1, __memrchr_sse2)) @@ -133,19 +133,19 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, __memset_chk, 1, __memset_chk_sse2_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memset_chk_avx2_unaligned) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memset_chk_avx2_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_chk_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_chk_avx512_unaligned) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_chk_avx512_no_vzeroupper) ) #endif @@ -158,48 +158,48 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, __memset_sse2_unaligned_erms) IFUNC_IMPL_ADD (array, i, memset, 1, __memset_erms) IFUNC_IMPL_ADD (array, i, memset, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memset_avx2_unaligned) IFUNC_IMPL_ADD (array, i, memset, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memset_avx2_unaligned_erms) IFUNC_IMPL_ADD (array, i, memset, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, memset, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_avx512_unaligned) IFUNC_IMPL_ADD (array, i, memset, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_avx512_no_vzeroupper) ) /* Support sysdeps/x86_64/multiarch/rawmemchr.c. */ IFUNC_IMPL (i, name, rawmemchr, IFUNC_IMPL_ADD (array, i, rawmemchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __rawmemchr_avx2) IFUNC_IMPL_ADD (array, i, rawmemchr, 1, __rawmemchr_sse2)) /* Support sysdeps/x86_64/multiarch/strlen.c. */ IFUNC_IMPL (i, name, strlen, IFUNC_IMPL_ADD (array, i, strlen, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strlen_avx2) IFUNC_IMPL_ADD (array, i, strlen, 1, __strlen_sse2)) /* Support sysdeps/x86_64/multiarch/strnlen.c. */ IFUNC_IMPL (i, name, strnlen, IFUNC_IMPL_ADD (array, i, strnlen, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strnlen_avx2) IFUNC_IMPL_ADD (array, i, strnlen, 1, __strnlen_sse2)) /* Support sysdeps/x86_64/multiarch/stpncpy.c. */ IFUNC_IMPL (i, name, stpncpy, - IFUNC_IMPL_ADD (array, i, stpncpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, stpncpy, x86_cpu_is_usable (x86_cpu_SSSE3), __stpncpy_ssse3) - IFUNC_IMPL_ADD (array, i, stpncpy, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, stpncpy, x86_cpu_is_usable (x86_cpu_AVX2), __stpncpy_avx2) IFUNC_IMPL_ADD (array, i, stpncpy, 1, __stpncpy_sse2_unaligned) @@ -207,9 +207,9 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/stpcpy.c. */ IFUNC_IMPL (i, name, stpcpy, - IFUNC_IMPL_ADD (array, i, stpcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, stpcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __stpcpy_ssse3) - IFUNC_IMPL_ADD (array, i, stpcpy, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, stpcpy, x86_cpu_is_usable (x86_cpu_AVX2), __stpcpy_avx2) IFUNC_IMPL_ADD (array, i, stpcpy, 1, __stpcpy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, stpcpy, 1, __stpcpy_sse2)) @@ -217,35 +217,35 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strcasecmp_l.c. */ IFUNC_IMPL (i, name, strcasecmp, IFUNC_IMPL_ADD (array, i, strcasecmp, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __strcasecmp_avx) IFUNC_IMPL_ADD (array, i, strcasecmp, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strcasecmp_sse42) IFUNC_IMPL_ADD (array, i, strcasecmp, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strcasecmp_ssse3) IFUNC_IMPL_ADD (array, i, strcasecmp, 1, __strcasecmp_sse2)) /* Support sysdeps/x86_64/multiarch/strcasecmp_l.c. */ IFUNC_IMPL (i, name, strcasecmp_l, IFUNC_IMPL_ADD (array, i, strcasecmp_l, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __strcasecmp_l_avx) IFUNC_IMPL_ADD (array, i, strcasecmp_l, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strcasecmp_l_sse42) IFUNC_IMPL_ADD (array, i, strcasecmp_l, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strcasecmp_l_ssse3) IFUNC_IMPL_ADD (array, i, strcasecmp_l, 1, __strcasecmp_l_sse2)) /* Support sysdeps/x86_64/multiarch/strcat.c. */ IFUNC_IMPL (i, name, strcat, - IFUNC_IMPL_ADD (array, i, strcat, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, strcat, x86_cpu_is_usable (x86_cpu_AVX2), __strcat_avx2) - IFUNC_IMPL_ADD (array, i, strcat, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcat, x86_cpu_is_usable (x86_cpu_SSSE3), __strcat_ssse3) IFUNC_IMPL_ADD (array, i, strcat, 1, __strcat_sse2_unaligned) IFUNC_IMPL_ADD (array, i, strcat, 1, __strcat_sse2)) @@ -253,7 +253,7 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strchr.c. */ IFUNC_IMPL (i, name, strchr, IFUNC_IMPL_ADD (array, i, strchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strchr_avx2) IFUNC_IMPL_ADD (array, i, strchr, 1, __strchr_sse2_no_bsf) IFUNC_IMPL_ADD (array, i, strchr, 1, __strchr_sse2)) @@ -261,54 +261,54 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strchrnul.c. */ IFUNC_IMPL (i, name, strchrnul, IFUNC_IMPL_ADD (array, i, strchrnul, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strchrnul_avx2) IFUNC_IMPL_ADD (array, i, strchrnul, 1, __strchrnul_sse2)) /* Support sysdeps/x86_64/multiarch/strrchr.c. */ IFUNC_IMPL (i, name, strrchr, IFUNC_IMPL_ADD (array, i, strrchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strrchr_avx2) IFUNC_IMPL_ADD (array, i, strrchr, 1, __strrchr_sse2)) /* Support sysdeps/x86_64/multiarch/strcmp.c. */ IFUNC_IMPL (i, name, strcmp, IFUNC_IMPL_ADD (array, i, strcmp, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strcmp_avx2) - IFUNC_IMPL_ADD (array, i, strcmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strcmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __strcmp_sse42) - IFUNC_IMPL_ADD (array, i, strcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __strcmp_ssse3) IFUNC_IMPL_ADD (array, i, strcmp, 1, __strcmp_sse2_unaligned) IFUNC_IMPL_ADD (array, i, strcmp, 1, __strcmp_sse2)) /* Support sysdeps/x86_64/multiarch/strcpy.c. */ IFUNC_IMPL (i, name, strcpy, - IFUNC_IMPL_ADD (array, i, strcpy, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, strcpy, x86_cpu_is_usable (x86_cpu_AVX2), __strcpy_avx2) - IFUNC_IMPL_ADD (array, i, strcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __strcpy_ssse3) IFUNC_IMPL_ADD (array, i, strcpy, 1, __strcpy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, strcpy, 1, __strcpy_sse2)) /* Support sysdeps/x86_64/multiarch/strcspn.c. */ IFUNC_IMPL (i, name, strcspn, - IFUNC_IMPL_ADD (array, i, strcspn, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strcspn, x86_cpu_is_usable (x86_cpu_SSE4_2), __strcspn_sse42) IFUNC_IMPL_ADD (array, i, strcspn, 1, __strcspn_sse2)) /* Support sysdeps/x86_64/multiarch/strncase_l.c. */ IFUNC_IMPL (i, name, strncasecmp, IFUNC_IMPL_ADD (array, i, strncasecmp, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __strncasecmp_avx) IFUNC_IMPL_ADD (array, i, strncasecmp, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strncasecmp_sse42) IFUNC_IMPL_ADD (array, i, strncasecmp, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strncasecmp_ssse3) IFUNC_IMPL_ADD (array, i, strncasecmp, 1, __strncasecmp_sse2)) @@ -316,22 +316,22 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strncase_l.c. */ IFUNC_IMPL (i, name, strncasecmp_l, IFUNC_IMPL_ADD (array, i, strncasecmp_l, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __strncasecmp_l_avx) IFUNC_IMPL_ADD (array, i, strncasecmp_l, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strncasecmp_l_sse42) IFUNC_IMPL_ADD (array, i, strncasecmp_l, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strncasecmp_l_ssse3) IFUNC_IMPL_ADD (array, i, strncasecmp_l, 1, __strncasecmp_l_sse2)) /* Support sysdeps/x86_64/multiarch/strncat.c. */ IFUNC_IMPL (i, name, strncat, - IFUNC_IMPL_ADD (array, i, strncat, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, strncat, x86_cpu_is_usable (x86_cpu_AVX2), __strncat_avx2) - IFUNC_IMPL_ADD (array, i, strncat, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncat, x86_cpu_is_usable (x86_cpu_SSSE3), __strncat_ssse3) IFUNC_IMPL_ADD (array, i, strncat, 1, __strncat_sse2_unaligned) @@ -339,9 +339,9 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strncpy.c. */ IFUNC_IMPL (i, name, strncpy, - IFUNC_IMPL_ADD (array, i, strncpy, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, strncpy, x86_cpu_is_usable (x86_cpu_AVX2), __strncpy_avx2) - IFUNC_IMPL_ADD (array, i, strncpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncpy, x86_cpu_is_usable (x86_cpu_SSSE3), __strncpy_ssse3) IFUNC_IMPL_ADD (array, i, strncpy, 1, __strncpy_sse2_unaligned) @@ -349,14 +349,14 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strpbrk.c. */ IFUNC_IMPL (i, name, strpbrk, - IFUNC_IMPL_ADD (array, i, strpbrk, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strpbrk, x86_cpu_is_usable (x86_cpu_SSE4_2), __strpbrk_sse42) IFUNC_IMPL_ADD (array, i, strpbrk, 1, __strpbrk_sse2)) /* Support sysdeps/x86_64/multiarch/strspn.c. */ IFUNC_IMPL (i, name, strspn, - IFUNC_IMPL_ADD (array, i, strspn, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strspn, x86_cpu_is_usable (x86_cpu_SSE4_2), __strspn_sse42) IFUNC_IMPL_ADD (array, i, strspn, 1, __strspn_sse2)) @@ -368,70 +368,70 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/wcschr.c. */ IFUNC_IMPL (i, name, wcschr, IFUNC_IMPL_ADD (array, i, wcschr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcschr_avx2) IFUNC_IMPL_ADD (array, i, wcschr, 1, __wcschr_sse2)) /* Support sysdeps/x86_64/multiarch/wcsrchr.c. */ IFUNC_IMPL (i, name, wcsrchr, IFUNC_IMPL_ADD (array, i, wcsrchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcsrchr_avx2) IFUNC_IMPL_ADD (array, i, wcsrchr, 1, __wcsrchr_sse2)) /* Support sysdeps/x86_64/multiarch/wcscmp.c. */ IFUNC_IMPL (i, name, wcscmp, IFUNC_IMPL_ADD (array, i, wcscmp, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcscmp_avx2) IFUNC_IMPL_ADD (array, i, wcscmp, 1, __wcscmp_sse2)) /* Support sysdeps/x86_64/multiarch/wcsncmp.c. */ IFUNC_IMPL (i, name, wcsncmp, IFUNC_IMPL_ADD (array, i, wcsncmp, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcsncmp_avx2) IFUNC_IMPL_ADD (array, i, wcsncmp, 1, __wcsncmp_sse2)) /* Support sysdeps/x86_64/multiarch/wcscpy.c. */ IFUNC_IMPL (i, name, wcscpy, - IFUNC_IMPL_ADD (array, i, wcscpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, wcscpy, x86_cpu_is_usable (x86_cpu_SSSE3), __wcscpy_ssse3) IFUNC_IMPL_ADD (array, i, wcscpy, 1, __wcscpy_sse2)) /* Support sysdeps/x86_64/multiarch/wcslen.c. */ IFUNC_IMPL (i, name, wcslen, IFUNC_IMPL_ADD (array, i, wcslen, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcslen_avx2) IFUNC_IMPL_ADD (array, i, wcslen, 1, __wcslen_sse2)) /* Support sysdeps/x86_64/multiarch/wcsnlen.c. */ IFUNC_IMPL (i, name, wcsnlen, IFUNC_IMPL_ADD (array, i, wcsnlen, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcsnlen_avx2) IFUNC_IMPL_ADD (array, i, wcsnlen, - CPU_FEATURE_USABLE (SSE4_1), + x86_cpu_is_usable (x86_cpu_SSE4_1), __wcsnlen_sse4_1) IFUNC_IMPL_ADD (array, i, wcsnlen, 1, __wcsnlen_sse2)) /* Support sysdeps/x86_64/multiarch/wmemchr.c. */ IFUNC_IMPL (i, name, wmemchr, IFUNC_IMPL_ADD (array, i, wmemchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wmemchr_avx2) IFUNC_IMPL_ADD (array, i, wmemchr, 1, __wmemchr_sse2)) /* Support sysdeps/x86_64/multiarch/wmemcmp.c. */ IFUNC_IMPL (i, name, wmemcmp, IFUNC_IMPL_ADD (array, i, wmemcmp, - (CPU_FEATURE_USABLE (AVX2) - && CPU_FEATURE_USABLE (MOVBE)), + (x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_MOVBE)), __wmemcmp_avx2_movbe) - IFUNC_IMPL_ADD (array, i, wmemcmp, CPU_FEATURE_USABLE (SSE4_1), + IFUNC_IMPL_ADD (array, i, wmemcmp, x86_cpu_is_usable (x86_cpu_SSE4_1), __wmemcmp_sse4_1) - IFUNC_IMPL_ADD (array, i, wmemcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, wmemcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __wmemcmp_ssse3) IFUNC_IMPL_ADD (array, i, wmemcmp, 1, __wmemcmp_sse2)) @@ -440,35 +440,35 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, wmemset, 1, __wmemset_sse2_unaligned) IFUNC_IMPL_ADD (array, i, wmemset, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wmemset_avx2_unaligned) IFUNC_IMPL_ADD (array, i, wmemset, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __wmemset_avx512_unaligned)) #ifdef SHARED /* Support sysdeps/x86_64/multiarch/memcpy_chk.c. */ IFUNC_IMPL (i, name, __memcpy_chk, IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_chk_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_chk_avx512_unaligned) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_chk_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memcpy_chk_avx_unaligned) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memcpy_chk_avx_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_chk_ssse3_back) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_chk_ssse3) IFUNC_IMPL_ADD (array, i, __memcpy_chk, 1, __memcpy_chk_sse2_unaligned) @@ -481,23 +481,23 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/memcpy.c. */ IFUNC_IMPL (i, name, memcpy, IFUNC_IMPL_ADD (array, i, memcpy, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memcpy_avx_unaligned) IFUNC_IMPL_ADD (array, i, memcpy, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memcpy_avx_unaligned_erms) - IFUNC_IMPL_ADD (array, i, memcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_ssse3_back) - IFUNC_IMPL_ADD (array, i, memcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_ssse3) IFUNC_IMPL_ADD (array, i, memcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, memcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_avx512_unaligned) IFUNC_IMPL_ADD (array, i, memcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, memcpy, 1, __memcpy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, memcpy, 1, @@ -508,25 +508,25 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/mempcpy_chk.c. */ IFUNC_IMPL (i, name, __mempcpy_chk, IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_chk_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_chk_avx512_unaligned) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_chk_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __mempcpy_chk_avx_unaligned) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __mempcpy_chk_avx_unaligned_erms) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_chk_ssse3_back) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_chk_ssse3) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, 1, __mempcpy_chk_sse2_unaligned) @@ -539,23 +539,23 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/mempcpy.c. */ IFUNC_IMPL (i, name, mempcpy, IFUNC_IMPL_ADD (array, i, mempcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, mempcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_avx512_unaligned) IFUNC_IMPL_ADD (array, i, mempcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, mempcpy, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __mempcpy_avx_unaligned) IFUNC_IMPL_ADD (array, i, mempcpy, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __mempcpy_avx_unaligned_erms) - IFUNC_IMPL_ADD (array, i, mempcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, mempcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_ssse3_back) - IFUNC_IMPL_ADD (array, i, mempcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, mempcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_ssse3) IFUNC_IMPL_ADD (array, i, mempcpy, 1, __mempcpy_sse2_unaligned) @@ -566,11 +566,11 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strncmp.c. */ IFUNC_IMPL (i, name, strncmp, IFUNC_IMPL_ADD (array, i, strncmp, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strncmp_avx2) - IFUNC_IMPL_ADD (array, i, strncmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strncmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __strncmp_sse42) - IFUNC_IMPL_ADD (array, i, strncmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncmp, x86_cpu_is_usable (x86_cpu_SSSE3), __strncmp_ssse3) IFUNC_IMPL_ADD (array, i, strncmp, 1, __strncmp_sse2)) @@ -580,10 +580,10 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, __wmemset_chk, 1, __wmemset_chk_sse2_unaligned) IFUNC_IMPL_ADD (array, i, __wmemset_chk, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wmemset_chk_avx2_unaligned) IFUNC_IMPL_ADD (array, i, __wmemset_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __wmemset_chk_avx512_unaligned)) #endif diff --git a/sysdeps/x86_64/multiarch/ifunc-memcmp.h b/sysdeps/x86_64/multiarch/ifunc-memcmp.h index 0e21b3a628..592186b4e4 100644 --- a/sysdeps/x86_64/multiarch/ifunc-memcmp.h +++ b/sysdeps/x86_64/multiarch/ifunc-memcmp.h @@ -30,15 +30,15 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) - && CPU_FEATURE_USABLE_P (cpu_features, MOVBE) + && x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_MOVBE) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2_movbe); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_1)) + if (x86_cpu_is_usable (x86_cpu_SSE4_1)) return OPTIMIZE (sse4_1); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/ifunc-memmove.h b/sysdeps/x86_64/multiarch/ifunc-memmove.h index 9ada03aa43..bdba95022f 100644 --- a/sysdeps/x86_64/multiarch/ifunc-memmove.h +++ b/sysdeps/x86_64/multiarch/ifunc-memmove.h @@ -45,13 +45,13 @@ IFUNC_SELECTOR (void) || CPU_FEATURES_ARCH_P (cpu_features, Prefer_FSRM)) return OPTIMIZE (erms); - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512F) + if (x86_cpu_is_usable (x86_cpu_AVX512F) && !CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_AVX512)) { if (CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER)) return OPTIMIZE (avx512_no_vzeroupper); - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (avx512_unaligned_erms); return OPTIMIZE (avx512_unaligned); @@ -59,16 +59,16 @@ IFUNC_SELECTOR (void) if (CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) { - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (avx_unaligned_erms); return OPTIMIZE (avx_unaligned); } - if (!CPU_FEATURE_USABLE_P (cpu_features, SSSE3) + if (!x86_cpu_is_usable (x86_cpu_SSSE3) || CPU_FEATURES_ARCH_P (cpu_features, Fast_Unaligned_Copy)) { - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (sse2_unaligned_erms); return OPTIMIZE (sse2_unaligned); diff --git a/sysdeps/x86_64/multiarch/ifunc-memset.h b/sysdeps/x86_64/multiarch/ifunc-memset.h index f52613d372..b3995046e8 100644 --- a/sysdeps/x86_64/multiarch/ifunc-memset.h +++ b/sysdeps/x86_64/multiarch/ifunc-memset.h @@ -42,27 +42,27 @@ IFUNC_SELECTOR (void) if (CPU_FEATURES_ARCH_P (cpu_features, Prefer_ERMS)) return OPTIMIZE (erms); - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512F) + if (x86_cpu_is_usable (x86_cpu_AVX512F) && !CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_AVX512)) { if (CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER)) return OPTIMIZE (avx512_no_vzeroupper); - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (avx512_unaligned_erms); return OPTIMIZE (avx512_unaligned); } - if (CPU_FEATURE_USABLE_P (cpu_features, AVX2)) + if (x86_cpu_is_usable (x86_cpu_AVX2)) { - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (avx2_unaligned_erms); else return OPTIMIZE (avx2_unaligned); } - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (sse2_unaligned_erms); return OPTIMIZE (sse2_unaligned); diff --git a/sysdeps/x86_64/multiarch/ifunc-sse4_2.h b/sysdeps/x86_64/multiarch/ifunc-sse4_2.h index cbf18385d3..1aa71caf21 100644 --- a/sysdeps/x86_64/multiarch/ifunc-sse4_2.h +++ b/sysdeps/x86_64/multiarch/ifunc-sse4_2.h @@ -25,9 +25,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse42) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_2)) + if (x86_cpu_is_usable (x86_cpu_SSE4_2)) return OPTIMIZE (sse42); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/ifunc-strcasecmp.h b/sysdeps/x86_64/multiarch/ifunc-strcasecmp.h index 0818333931..0f5ab23047 100644 --- a/sysdeps/x86_64/multiarch/ifunc-strcasecmp.h +++ b/sysdeps/x86_64/multiarch/ifunc-strcasecmp.h @@ -29,14 +29,14 @@ IFUNC_SELECTOR (void) { const struct cpu_features* cpu_features = __get_cpu_features (); - if (CPU_FEATURE_USABLE_P (cpu_features, AVX)) + if (x86_cpu_is_usable (x86_cpu_AVX)) return OPTIMIZE (avx); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_2) + if (x86_cpu_is_usable (x86_cpu_SSE4_2) && !CPU_FEATURES_ARCH_P (cpu_features, Slow_SSE4_2)) return OPTIMIZE (sse42); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/ifunc-strcpy.h b/sysdeps/x86_64/multiarch/ifunc-strcpy.h index 63b0dc0d96..0ae6dcc704 100644 --- a/sysdeps/x86_64/multiarch/ifunc-strcpy.h +++ b/sysdeps/x86_64/multiarch/ifunc-strcpy.h @@ -32,14 +32,14 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); if (CPU_FEATURES_ARCH_P (cpu_features, Fast_Unaligned_Load)) return OPTIMIZE (sse2_unaligned); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/ifunc-wmemset.h b/sysdeps/x86_64/multiarch/ifunc-wmemset.h index 8cfce562fc..d7095301dd 100644 --- a/sysdeps/x86_64/multiarch/ifunc-wmemset.h +++ b/sysdeps/x86_64/multiarch/ifunc-wmemset.h @@ -28,10 +28,10 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) { - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512F) + if (x86_cpu_is_usable (x86_cpu_AVX512F) && !CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_AVX512)) return OPTIMIZE (avx512_unaligned); else diff --git a/sysdeps/x86_64/multiarch/sched_cpucount.c b/sysdeps/x86_64/multiarch/sched_cpucount.c index 074c663cf6..f1aba1462b 100644 --- a/sysdeps/x86_64/multiarch/sched_cpucount.c +++ b/sysdeps/x86_64/multiarch/sched_cpucount.c @@ -33,4 +33,5 @@ #undef __sched_cpucount libc_ifunc (__sched_cpucount, - CPU_FEATURE_USABLE (POPCNT) ? popcount_cpucount : generic_cpucount); + (x86_cpu_is_usable (x86_cpu_POPCNT) + ? popcount_cpucount : generic_cpucount)); diff --git a/sysdeps/x86_64/multiarch/strchr.c b/sysdeps/x86_64/multiarch/strchr.c index 8df4609bf8..9e570f1d4e 100644 --- a/sysdeps/x86_64/multiarch/strchr.c +++ b/sysdeps/x86_64/multiarch/strchr.c @@ -36,7 +36,7 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); diff --git a/sysdeps/x86_64/multiarch/strcmp.c b/sysdeps/x86_64/multiarch/strcmp.c index 16ae72a4c8..fdc29f9d7b 100644 --- a/sysdeps/x86_64/multiarch/strcmp.c +++ b/sysdeps/x86_64/multiarch/strcmp.c @@ -37,14 +37,14 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); if (CPU_FEATURES_ARCH_P (cpu_features, Fast_Unaligned_Load)) return OPTIMIZE (sse2_unaligned); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/strncmp.c b/sysdeps/x86_64/multiarch/strncmp.c index 3c94b3ffd9..97b8c58bed 100644 --- a/sysdeps/x86_64/multiarch/strncmp.c +++ b/sysdeps/x86_64/multiarch/strncmp.c @@ -37,15 +37,15 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_2) + if (x86_cpu_is_usable (x86_cpu_SSE4_2) && !CPU_FEATURES_ARCH_P (cpu_features, Slow_SSE4_2)) return OPTIMIZE (sse42); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/wcscpy.c b/sysdeps/x86_64/multiarch/wcscpy.c index e08536c593..d693985f7b 100644 --- a/sysdeps/x86_64/multiarch/wcscpy.c +++ b/sysdeps/x86_64/multiarch/wcscpy.c @@ -32,9 +32,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (ssse3) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/wcsnlen.c b/sysdeps/x86_64/multiarch/wcsnlen.c index 52e7e5d4f3..f17c25a0fc 100644 --- a/sysdeps/x86_64/multiarch/wcsnlen.c +++ b/sysdeps/x86_64/multiarch/wcsnlen.c @@ -36,11 +36,11 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_1)) + if (x86_cpu_is_usable (x86_cpu_SSE4_1)) return OPTIMIZE (sse4_1); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/tst-glibc-hwcaps.c b/sysdeps/x86_64/tst-glibc-hwcaps.c index 3075a8286d..9df70e58f2 100644 --- a/sysdeps/x86_64/tst-glibc-hwcaps.c +++ b/sysdeps/x86_64/tst-glibc-hwcaps.c @@ -29,35 +29,32 @@ extern int marker4 (void); static int compute_level (void) { - const struct cpu_features *cpu_features - = __x86_get_cpu_features (COMMON_CPUID_INDEX_MAX); - - if (!(CPU_FEATURE_USABLE_P (cpu_features, CMPXCHG16B) - && CPU_FEATURE_USABLE_P (cpu_features, LAHF64_SAHF64) - && CPU_FEATURE_USABLE_P (cpu_features, POPCNT) - && CPU_FEATURE_USABLE_P (cpu_features, MMX) - && CPU_FEATURE_USABLE_P (cpu_features, SSE) - && CPU_FEATURE_USABLE_P (cpu_features, SSE2) - && CPU_FEATURE_USABLE_P (cpu_features, SSE3) - && CPU_FEATURE_USABLE_P (cpu_features, SSSE3) - && CPU_FEATURE_USABLE_P (cpu_features, SSE4_1) - && CPU_FEATURE_USABLE_P (cpu_features, SSE4_2))) + if (!(x86_cpu_is_usable (x86_cpu_CMPXCHG16B) + && x86_cpu_is_usable (x86_cpu_LAHF64_SAHF64) + && x86_cpu_is_usable (x86_cpu_POPCNT) + && x86_cpu_is_usable (x86_cpu_MMX) + && x86_cpu_is_usable (x86_cpu_SSE) + && x86_cpu_is_usable (x86_cpu_SSE2) + && x86_cpu_is_usable (x86_cpu_SSE3) + && x86_cpu_is_usable (x86_cpu_SSSE3) + && x86_cpu_is_usable (x86_cpu_SSE4_1) + && x86_cpu_is_usable (x86_cpu_SSE4_2))) return 1; - if (!(CPU_FEATURE_USABLE_P (cpu_features, AVX) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) - && CPU_FEATURE_USABLE_P (cpu_features, BMI1) - && CPU_FEATURE_USABLE_P (cpu_features, BMI2) - && CPU_FEATURE_USABLE_P (cpu_features, F16C) - && CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, LZCNT) - && CPU_FEATURE_USABLE_P (cpu_features, MOVBE) - && CPU_FEATURE_USABLE_P (cpu_features, OSXSAVE))) + if (!(x86_cpu_is_usable (x86_cpu_AVX) + && x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_BMI1) + && x86_cpu_is_usable (x86_cpu_BMI2) + && x86_cpu_is_usable (x86_cpu_F16C) + && x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_LZCNT) + && x86_cpu_is_usable (x86_cpu_MOVBE) + && x86_cpu_is_usable (x86_cpu_OSXSAVE))) return 2; - if (!(CPU_FEATURE_USABLE_P (cpu_features, AVX512F) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512BW) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512CD) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512DQ) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512VL))) + if (!(x86_cpu_is_usable (x86_cpu_AVX512F) + && x86_cpu_is_usable (x86_cpu_AVX512BW) + && x86_cpu_is_usable (x86_cpu_AVX512CD) + && x86_cpu_is_usable (x86_cpu_AVX512DQ) + && x86_cpu_is_usable (x86_cpu_AVX512VL))) return 3; return 4; }