From patchwork Wed Dec 23 22:07:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 41541 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 87A943854813; Wed, 23 Dec 2020 22:08:07 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 87A943854813 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1608761287; bh=xNyD2AIf9PbGiOiWTwzKlV05jUq5dKkIVDuxf4Ith/Y=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=RMRAyLrMEe9Nl9eXs+wl0efjvUFm48vdOFM12KODTj331O2EspfakbfUbLzeLoyml lYuocefJG8tq+z9r8SpWo9ZGonvC3gOcNSs6X/io7hTjzZUQzCQrr4grbHNGZOOL3e eGwyICJHvMFozyStyKGz3hQ8kdWoSaQ4tYsQGZEg= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by sourceware.org (Postfix) with ESMTP id 6FDB13857004 for ; Wed, 23 Dec 2020 22:08:03 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 6FDB13857004 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-236-lpyQQNl6OWuvht7lhCy9lw-1; Wed, 23 Dec 2020 17:07:59 -0500 X-MC-Unique: lpyQQNl6OWuvht7lhCy9lw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9535D59 for ; Wed, 23 Dec 2020 22:07:58 +0000 (UTC) Received: from oldenburg2.str.redhat.com (ovpn-112-120.ams2.redhat.com [10.36.112.120]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8F74960C04 for ; Wed, 23 Dec 2020 22:07:57 +0000 (UTC) To: libc-alpha@sourceware.org Subject: [PATCH 1/2] elf: Support getauxval after static dlopen (bug 20802) Date: Wed, 23 Dec 2020 23:07:55 +0100 Message-ID: <87tuscp45g.fsf@oldenburg2.str.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-10.5 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, KAM_STOCKGEN, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP, URIBL_BLACK autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Florian Weimer via Libc-alpha From: Florian Weimer Reply-To: Florian Weimer Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" Variables which should be available after static dlopen are stored in __libc_vars and initialized from the outer libc. This mechanism is intended to replace _dl_var_init eventually. The main differences are that it is used in both the dynamic and static dlopen cases, and that the initialization happens before relocation, so that no special steps are necessary for RELRO compatibility. --- elf/Makefile | 15 ++++++-- elf/Versions | 2 + elf/dl-call-libc-early-init.c | 23 +++++++++++ elf/dl-libc_vars.c | 19 ++++++++++ elf/dl-libc_vars_init.c | 31 +++++++++++++++ elf/dl-open.c | 10 +++++ elf/dl-support.c | 4 +- elf/dl-sysdep.c | 9 +++-- elf/libc_vars.c | 22 +++++++++++ elf/rtld.c | 5 +++ elf/tst-auxvalmod.c | 31 +++++++++++++++ elf/tst-auxvalmod.h | 23 +++++++++++ elf/tst-getauxval-static.c | 54 ++++++++++++++++++++++++++ misc/getauxval.c | 3 +- sysdeps/generic/ldsodefs.h | 5 --- sysdeps/generic/libc-vars-cpu.h | 28 ++++++++++++++ sysdeps/generic/libc-vars-init.h | 32 ++++++++++++++++ sysdeps/generic/libc-vars.h | 59 +++++++++++++++++++++++++++++ sysdeps/unix/sysv/linux/powerpc/dl-static.c | 21 +++------- 19 files changed, 365 insertions(+), 31 deletions(-) diff --git a/elf/Makefile b/elf/Makefile index 0b4d78c874..93b378e461 100644 --- a/elf/Makefile +++ b/elf/Makefile @@ -25,7 +25,7 @@ headers = elf.h bits/elfclass.h link.h bits/link.h routines = $(all-dl-routines) dl-support dl-iteratephdr \ dl-addr dl-addr-obj enbl-secure dl-profstub \ dl-origin dl-libc dl-sym dl-sysdep dl-error \ - dl-reloc-static-pie libc_early_init + dl-reloc-static-pie libc_early_init libc_vars # The core dynamic linking functions are in libc for the static and # profiled libraries. @@ -35,7 +35,7 @@ dl-routines = $(addprefix dl-,load lookup object reloc deps \ execstack open close trampoline \ exception sort-maps lookup-direct \ call-libc-early-init write \ - thread_gscope_wait) + thread_gscope_wait libc_vars libc_vars_init) ifeq (yes,$(use-ldconfig)) dl-routines += dl-cache endif @@ -186,9 +186,14 @@ tests-internal := tst-tls1 tst-tls2 $(tests-static-internal) tests-static := $(tests-static-normal) $(tests-static-internal) ifeq (yes,$(build-shared)) -tests-static += tst-tls9-static tst-single_threaded-static-dlopen +tests-static += \ + tst-getauxval-static \ + tst-single_threaded-static-dlopen \ + tst-tls9-static \ + static-dlopen-environment = \ LD_LIBRARY_PATH=$(ld-library-path):$(common-objpfx)dlfcn +tst-getauxval-static-ENV = $(static-dlopen-environment) tst-tls9-static-ENV = $(static-dlopen-environment) tst-single_threaded-static-dlopen-ENV = $(static-dlopen-environment) @@ -343,6 +348,7 @@ modules-names = testobj1 testobj2 testobj3 testobj4 testobj5 testobj6 \ libmarkermod2-1 libmarkermod2-2 \ libmarkermod3-1 libmarkermod3-2 libmarkermod3-3 \ libmarkermod4-1 libmarkermod4-2 libmarkermod4-3 libmarkermod4-4 \ + tst-auxvalmod \ # Most modules build with _ISOMAC defined, but those filtered out # depend on internal headers. @@ -1894,3 +1900,6 @@ $(objpfx)tst-glibc-hwcaps-mask.out: \ # Generic dependency for sysdeps implementation of # tst-glibc-hwcaps-cache. $(objpfx)tst-glibc-hwcaps-cache.out: $(objpfx)tst-glibc-hwcaps + +$(objpfx)tst-getauxval-static: $(common-objpfx)dlfcn/libdl.a +$(objpfx)tst-getauxval-static.out: $(objpfx)tst-auxvalmod.so diff --git a/elf/Versions b/elf/Versions index be88c48e6d..94e6daadf5 100644 --- a/elf/Versions +++ b/elf/Versions @@ -31,6 +31,8 @@ libc { # Internal error handling support. Interposes the functions in ld.so. _dl_signal_exception; _dl_catch_exception; _dl_signal_error; _dl_catch_error; + + __libc_vars; } } diff --git a/elf/dl-call-libc-early-init.c b/elf/dl-call-libc-early-init.c index 9a84680a1c..135fa57bed 100644 --- a/elf/dl-call-libc-early-init.c +++ b/elf/dl-call-libc-early-init.c @@ -19,9 +19,12 @@ #include #include #include +#include #include #include +#include + void _dl_call_libc_early_init (struct link_map *libc_map, _Bool initial) { @@ -39,3 +42,23 @@ _dl_call_libc_early_init (struct link_map *libc_map, _Bool initial) = DL_SYMBOL_ADDRESS (libc_map, sym); early_init (initial); } + +void +_dl_call_libc_vars_init (struct link_map *libc_map) +{ + /* There is nothing to do if we did not actually load libc.so. */ + if (libc_map == NULL) + return; + + const ElfW (Sym) *sym + = _dl_lookup_direct (libc_map, "__libc_vars", + 0x59f05378, /* dl_new_hash output. */ + "GLIBC_PRIVATE", + 0x0963cf85); /* _dl_elf_hash output. */ + assert (sym != NULL); + assert (sym->st_size == sizeof (struct libc_vars)); + struct libc_vars *vars = DL_SYMBOL_ADDRESS (libc_map, sym); + + _dl_libc_vars_init (vars); + _dl_call_libc_vars_init_cpu (libc_map); +} diff --git a/elf/dl-libc_vars.c b/elf/dl-libc_vars.c new file mode 100644 index 0000000000..6d9d96ec63 --- /dev/null +++ b/elf/dl-libc_vars.c @@ -0,0 +1,19 @@ +/* Local copy of __libc_vars for ld.so. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include "libc_vars.c" diff --git a/elf/dl-libc_vars_init.c b/elf/dl-libc_vars_init.c new file mode 100644 index 0000000000..71cb0446fb --- /dev/null +++ b/elf/dl-libc_vars_init.c @@ -0,0 +1,31 @@ +/* Update __libc_vars in libc.so with the data from ld.so. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#include + +void +_dl_libc_vars_init (struct libc_vars *vars) +{ +#ifdef HAVE_AUX_VECTOR + vars->auxv = __libc_vars.auxv; +#endif + + _dl_libc_vars_init_cpu (vars); +} diff --git a/elf/dl-open.c b/elf/dl-open.c index 6710ea04cd..d5f9e61858 100644 --- a/elf/dl-open.c +++ b/elf/dl-open.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -590,6 +591,15 @@ dl_open_worker (void *a) (void) _dl_check_map_versions (new->l_searchlist.r_list[i]->l_real, 0, 0); + /* Initialize __libc_vars before relocation. This has to come after + the call to _dl_check_map_versions, so that lookups of versioned + symbols work. */ + if (!args->libc_already_loaded) + { + struct link_map *libc_map = GL (dl_ns)[args->nsid].libc_map; + _dl_call_libc_vars_init (libc_map); + } + #ifdef SHARED /* Auditing checkpoint: we have added all objects. */ if (__glibc_unlikely (GLRO(dl_naudit) > 0)) diff --git a/elf/dl-support.c b/elf/dl-support.c index 250e4cd092..6f6b20cb4e 100644 --- a/elf/dl-support.c +++ b/elf/dl-support.c @@ -37,6 +37,7 @@ #include #include #include +#include extern char *__progname; char **_dl_argv = &__progname; /* This is checked for some error messages. */ @@ -154,7 +155,6 @@ int _dl_debug_fd = STDERR_FILENO; int _dl_correct_cache_id = _DL_CACHE_DEFAULT_ID; -ElfW(auxv_t) *_dl_auxv; const ElfW(Phdr) *_dl_phdr; size_t _dl_phnum; uint64_t _dl_hwcap __attribute__ ((nocommon)); @@ -232,7 +232,7 @@ _dl_aux_init (ElfW(auxv_t) *av) uid_t uid = 0; gid_t gid = 0; - _dl_auxv = av; + __libc_vars.auxv = av; for (; av->a_type != AT_NULL; ++av) switch (av->a_type) { diff --git a/elf/dl-sysdep.c b/elf/dl-sysdep.c index 6cc4a76560..0969773127 100644 --- a/elf/dl-sysdep.c +++ b/elf/dl-sysdep.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include @@ -110,12 +111,12 @@ _dl_sysdep_start (void **start_argptr, __libc_stack_end = DL_STACK_END (start_argptr); DL_FIND_ARG_COMPONENTS (start_argptr, _dl_argc, _dl_argv, _environ, - GLRO(dl_auxv)); + __libc_vars.auxv); user_entry = (ElfW(Addr)) ENTRY_POINT; GLRO(dl_platform) = NULL; /* Default to nothing known about the platform. */ - for (av = GLRO(dl_auxv); av->a_type != AT_NULL; set_seen (av++)) + for (av = __libc_vars.auxv; av->a_type != AT_NULL; set_seen (av++)) switch (av->a_type) { case AT_PHDR: @@ -247,7 +248,7 @@ _dl_sysdep_start (void **start_argptr, if (__builtin_expect (__libc_enable_secure, 0)) __libc_check_standard_fds (); - (*dl_main) (phdr, phnum, &user_entry, GLRO(dl_auxv)); + (*dl_main) (phdr, phnum, &user_entry, __libc_vars.auxv); return user_entry; } @@ -270,7 +271,7 @@ _dl_show_auxv (void) close by (otherwise the array will be too large). In case we have to support a platform where these requirements are not fulfilled some alternative implementation has to be used. */ - for (av = GLRO(dl_auxv); av->a_type != AT_NULL; ++av) + for (av = __libc_vars.auxv; av->a_type != AT_NULL; ++av) { static const struct { diff --git a/elf/libc_vars.c b/elf/libc_vars.c new file mode 100644 index 0000000000..538de7fa56 --- /dev/null +++ b/elf/libc_vars.c @@ -0,0 +1,22 @@ +/* Definition of __libc_vars for libc. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +struct libc_vars __libc_vars attribute_relro; +hidden_def (__libc_vars) diff --git a/elf/rtld.c b/elf/rtld.c index 38f11f5b73..69b5b114ed 100644 --- a/elf/rtld.c +++ b/elf/rtld.c @@ -50,6 +50,7 @@ #include #include #include +#include #include @@ -2311,6 +2312,10 @@ dl_main (const ElfW(Phdr) *phdr, _rtld_main_check (main_map, _dl_argv[0]); + /* We are about to perform relocation. Make sure that __libc_vars + has been initialized. */ + _dl_call_libc_vars_init (GL (dl_ns)[LM_ID_BASE].libc_map); + if (prelinked) { if (main_map->l_info [ADDRIDX (DT_GNU_CONFLICT)] != NULL) diff --git a/elf/tst-auxvalmod.c b/elf/tst-auxvalmod.c new file mode 100644 index 0000000000..d695c9df91 --- /dev/null +++ b/elf/tst-auxvalmod.c @@ -0,0 +1,31 @@ +/* Wrapper for getauxval testing. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +#include "tst-auxvalmod.h" + +unsigned long int +getauxval_wrapper (unsigned long int type, int *errnop) +{ + errno = *errnop; + unsigned long int result = getauxval (type); + *errnop = errno; + return result; +} diff --git a/elf/tst-auxvalmod.h b/elf/tst-auxvalmod.h new file mode 100644 index 0000000000..f030326394 --- /dev/null +++ b/elf/tst-auxvalmod.h @@ -0,0 +1,23 @@ +/* Wrapper for getauxval testing. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* Call getauxval, write errno value after the call to ERRNOP, and + return the getauxval result. ERRNOP is needed because across a + static dlopen boundary, errno might have a different address, and + the caller would never see the intended value. */ +unsigned long int getauxval_wrapper (unsigned long int type, int *errnop); diff --git a/elf/tst-getauxval-static.c b/elf/tst-getauxval-static.c new file mode 100644 index 0000000000..ee5ce54629 --- /dev/null +++ b/elf/tst-getauxval-static.c @@ -0,0 +1,54 @@ +/* Test getauxval from a dynamic library after static dlopen. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include +#include + +#include "tst-auxvalmod.h" + +static int +do_test (void) +{ + unsigned long missing_auxv_type; + for (missing_auxv_type = 0; ; ++missing_auxv_type) + { + errno = 0; + if (getauxval (missing_auxv_type) == 0 && errno != 0) + break; + } + TEST_COMPARE (errno, ENOENT); + + void *handle = xdlopen ("tst-auxvalmod.so", RTLD_LAZY); + + __typeof__ (getauxval_wrapper) *wrapper + = xdlsym (handle, "getauxval_wrapper"); + int inner_errno = 0; + TEST_COMPARE (getauxval (AT_RANDOM), wrapper (AT_RANDOM, &inner_errno)); + + inner_errno = 0; + TEST_COMPARE (wrapper (missing_auxv_type, &inner_errno), 0); + TEST_COMPARE (inner_errno, ENOENT); + + xdlclose (handle); + return 0; +} + +#include diff --git a/misc/getauxval.c b/misc/getauxval.c index e96d4dfa20..5820884ad1 100644 --- a/misc/getauxval.c +++ b/misc/getauxval.c @@ -19,6 +19,7 @@ #include #include #include +#include bool __getauxval2 (unsigned long int type, unsigned long int *result) @@ -39,7 +40,7 @@ __getauxval2 (unsigned long int type, unsigned long int *result) } #ifdef HAVE_AUX_VECTOR - for (p = GLRO(dl_auxv); p->a_type != AT_NULL; p++) + for (p = __libc_vars.auxv; p->a_type != AT_NULL; p++) if (p->a_type == type) { *result = p->a_un.a_val; diff --git a/sysdeps/generic/ldsodefs.h b/sysdeps/generic/ldsodefs.h index 933cda117d..48837434b5 100644 --- a/sysdeps/generic/ldsodefs.h +++ b/sysdeps/generic/ldsodefs.h @@ -575,11 +575,6 @@ struct rtld_global_ro EXTERN uint64_t _dl_hwcap_mask; #endif -#ifdef HAVE_AUX_VECTOR - /* Pointer to the auxv list supplied to the program at startup. */ - EXTERN ElfW(auxv_t) *_dl_auxv; -#endif - /* Get architecture specific definitions. */ #include diff --git a/sysdeps/generic/libc-vars-cpu.h b/sysdeps/generic/libc-vars-cpu.h new file mode 100644 index 0000000000..fc9065b195 --- /dev/null +++ b/sysdeps/generic/libc-vars-cpu.h @@ -0,0 +1,28 @@ +/* CPU-specific extensions for __libc_vars. Generic version. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _LIBC_VARS_CPU_H +#define _LIBC_VARS_CPU_H + +/* The generic version does not have CPU-specific extensions. */ +#define HAVE_LIBC_VARS_CPU 0 + +struct libc_vars; +struct link_map; + +#endif /* _LIBC_VARS_CPU_H */ diff --git a/sysdeps/generic/libc-vars-init.h b/sysdeps/generic/libc-vars-init.h new file mode 100644 index 0000000000..e5cd6ba659 --- /dev/null +++ b/sysdeps/generic/libc-vars-init.h @@ -0,0 +1,32 @@ +/* Initialization of __libc_vars. Generic version. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* This file is included from elf/dl-libc_vars_init.c + and elf/dl-call-libc-early-init.c. + + The generic version does not perform any extra initialization. */ + +static inline void +_dl_libc_vars_init_cpu (struct libc_vars *vars) +{ +} + +static inline void +_dl_call_libc_vars_init_cpu (struct link_map *libc_map) +{ +} diff --git a/sysdeps/generic/libc-vars.h b/sysdeps/generic/libc-vars.h new file mode 100644 index 0000000000..378c51cc82 --- /dev/null +++ b/sysdeps/generic/libc-vars.h @@ -0,0 +1,59 @@ +/* Definition of __libc_vars and related update functionality. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _LIBC_VARS_H +#define _LIBC_VARS_H + +#include +#include +#include +#include + +struct libc_vars +{ +#ifdef HAVE_AUX_VECTOR + /* Pointer to the auxv list supplied to the program at startup. */ + ElfW (auxv_t) *auxv; +#endif +#if HAVE_LIBC_VARS_CPU + struct libc_vars_cpu cpu; +#endif +}; + +/* Initialized by ld.so immediate after loading libc.so.6, prior to + relocation. The variable is duplicated in ld.so, where it is set + up by the startup code. It is only exported (under GLIBC_PRIVATE) + in libc.so, and this copy is updated in _dl_libc_vars_init. + + In the static case, the initialization has to be put into the + startup code, __libc_vars is part of libc.a. */ +extern struct libc_vars __libc_vars attribute_relro; +#if IS_IN (libc) || IS_IN (rtld) +hidden_proto (__libc_vars) +#endif + +/* Called by the dynamic linker to initialize __libc_vars prior to + relocation. This function resides in ld.so, but updates the data + in libc.so. The similar function __libc_early_init resides in + libc.so and performs post-relocation initialization. */ +void _dl_libc_vars_init (struct libc_vars *vars) attribute_hidden; + +/* Invoke _dl_libc_vars_init for __libc_vars in LIBC_MAP. */ +void _dl_call_libc_vars_init (struct link_map *libc_map) attribute_hidden; + +#endif /* _LIBC_VARS_H */ diff --git a/sysdeps/unix/sysv/linux/powerpc/dl-static.c b/sysdeps/unix/sysv/linux/powerpc/dl-static.c index a77e07b503..cab84e5753 100644 --- a/sysdeps/unix/sysv/linux/powerpc/dl-static.c +++ b/sysdeps/unix/sysv/linux/powerpc/dl-static.c @@ -23,21 +23,11 @@ void _dl_var_init (void *array[]) { - /* It has to match "variables" below. */ - enum - { - DL_PAGESIZE = 0, - DL_AUXV = 1, - DL_HWCAP = 2, - DL_HWCAP2 = 3, - DL_CACHE_LINE_SIZE = 4 - }; - - GLRO(dl_pagesize) = *((size_t *) array[DL_PAGESIZE]); - GLRO(dl_auxv) = (ElfW(auxv_t) *) *((size_t *) array[DL_AUXV]); - GLRO(dl_hwcap) = *((unsigned long int *) array[DL_HWCAP]); - GLRO(dl_hwcap2) = *((unsigned long int *) array[DL_HWCAP2]); - GLRO(dl_cache_line_size) = (int) *((int *) array[DL_CACHE_LINE_SIZE]); + int index = 0; + GLRO(dl_pagesize) = *((size_t *) array[index++]); + GLRO(dl_hwcap) = *((unsigned long int *) array[index++]); + GLRO(dl_hwcap2) = *((unsigned long int *) array[index++]); + GLRO(dl_cache_line_size) = (int) *((int *) array[index++]); } #else @@ -45,7 +35,6 @@ _dl_var_init (void *array[]) static void *variables[] = { &GLRO(dl_pagesize), - &GLRO(dl_auxv), &GLRO(dl_hwcap), &GLRO(dl_hwcap2), &GLRO(dl_cache_line_size) From patchwork Wed Dec 23 22:15:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 41542 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 9FE463850401; Wed, 23 Dec 2020 22:15:22 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9FE463850401 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1608761722; bh=9nuaQTlZ/5yLmSC1xq510hiRAB0uZnvE0K3di7r/j78=; h=To:Subject:References:Date:In-Reply-To:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=a4Rd2vGPQfgzDh/MQWqhhmOcXY83G53V0eDP1vDxeC6UfaQnHpjzIXAI/D1jDUX/W gOPYIQDjKPQU2APMZuptwJUPGeW3XKj2ZU3ovF6kfm0VGEnvdvOkXEM9L3FyKV7ugs 7dnZtgysFhPbMB5A559gja8AkJg00n8g0kM3QVUU= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by sourceware.org (Postfix) with ESMTP id 1FF3A3857004 for ; Wed, 23 Dec 2020 22:15:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 1FF3A3857004 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-424-SfCVnBNPPRqbSln1BnX_lA-1; Wed, 23 Dec 2020 17:15:06 -0500 X-MC-Unique: SfCVnBNPPRqbSln1BnX_lA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 50628180A087 for ; Wed, 23 Dec 2020 22:15:05 +0000 (UTC) Received: from oldenburg2.str.redhat.com (ovpn-112-120.ams2.redhat.com [10.36.112.120]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9083460C04 for ; Wed, 23 Dec 2020 22:15:03 +0000 (UTC) To: libc-alpha@sourceware.org Subject: [RFC PATCH 2/2] x86: Alternative implementation References: <87tuscp45g.fsf@oldenburg2.str.redhat.com> Date: Wed, 23 Dec 2020 23:15:01 +0100 In-Reply-To: <87tuscp45g.fsf@oldenburg2.str.redhat.com> (Florian Weimer's message of "Wed, 23 Dec 2020 23:07:55 +0100") Message-ID: <87mty4p3tm.fsf@oldenburg2.str.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-11.3 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, KAM_STOCKGEN, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Florian Weimer via Libc-alpha From: Florian Weimer Reply-To: Florian Weimer Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" This implementation uses a data symbol containing a pointer to the bitmask array and an array length, not a function call. The expectation is that this is the final link-time ABI for this feature. (The run-time ABI will have to change once more, to support use of this facility in IFUNC resolvers.) The __libc_vars initialization mechanism is used to set up the copy in libc.so.6 before relocation, so that it can be used by IFUNC resolvers. Usage of the C preprocessor is greatly reduced, making it easier to wrap this functionality in other languages. This is still a preview. There are further cleanups possible, including removal of the function symbol. The manual still needs updating, and there are a few overlong lines. I'd like to receive feedback if this is the direction in which we want to move. I think it should be possible to hack in IFUNC resolver support using a custom dynamic section entry that points to a hidden __x86_cpu_array variable. It would be cleaner to use a new run-time-only relocation for the initialization. The dynamic section hack would not work with --gc-sections, for instance. --- sysdeps/i386/fpu/fclrexcpt.c | 2 +- sysdeps/i386/fpu/fedisblxcpt.c | 2 +- sysdeps/i386/fpu/feenablxcpt.c | 2 +- sysdeps/i386/fpu/fegetenv.c | 2 +- sysdeps/i386/fpu/fegetmode.c | 2 +- sysdeps/i386/fpu/feholdexcpt.c | 2 +- sysdeps/i386/fpu/fesetenv.c | 2 +- sysdeps/i386/fpu/fesetmode.c | 2 +- sysdeps/i386/fpu/fesetround.c | 2 +- sysdeps/i386/fpu/feupdateenv.c | 2 +- sysdeps/i386/fpu/fgetexcptflg.c | 2 +- sysdeps/i386/fpu/fsetexcptflg.c | 2 +- sysdeps/i386/fpu/ftestexcept.c | 2 +- sysdeps/i386/i686/fpu/multiarch/s_cosf.c | 3 +- sysdeps/i386/i686/fpu/multiarch/s_sincosf.c | 4 +- sysdeps/i386/i686/fpu/multiarch/s_sinf.c | 3 +- sysdeps/i386/i686/multiarch/ifunc-impl-list.c | 152 ++-- sysdeps/i386/i686/multiarch/ifunc-memmove.h | 2 +- sysdeps/i386/i686/multiarch/ifunc-memset.h | 2 +- sysdeps/i386/i686/multiarch/ifunc-sse2-bsf.h | 2 +- sysdeps/i386/i686/multiarch/ifunc-sse2-ssse3.h | 4 +- sysdeps/i386/i686/multiarch/ifunc-sse2.h | 4 +- sysdeps/i386/i686/multiarch/ifunc-sse4_2.h | 4 +- sysdeps/i386/i686/multiarch/ifunc-ssse3-sse4_2.h | 6 +- sysdeps/i386/i686/multiarch/s_fma.c | 2 +- sysdeps/i386/i686/multiarch/s_fmaf.c | 2 +- sysdeps/i386/i686/multiarch/wcscpy.c | 4 +- sysdeps/i386/setfpucw.c | 2 +- sysdeps/mach/hurd/i386/libc.abilist | 1 + sysdeps/unix/sysv/linux/i386/libc.abilist | 1 + sysdeps/unix/sysv/linux/x86/elision-conf.c | 3 +- sysdeps/unix/sysv/linux/x86_64/64/libc.abilist | 1 + sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist | 1 + sysdeps/x86/Makefile | 5 +- sysdeps/x86/Versions | 6 + sysdeps/x86/bits/platform/x86.h | 245 +++++++ sysdeps/x86/cacheinfo.h | 12 +- sysdeps/x86/cpu-features.c | 358 +++++----- sysdeps/x86/cpu-tunables.c | 6 +- sysdeps/x86/include/cpu-features.h | 62 +- sysdeps/x86/include/sys/platform/x86.h | 62 ++ sysdeps/x86/libc-vars-init.h | 42 ++ sysdeps/x86/sys/platform/x86.h | 788 +-------------------- sysdeps/x86/tst-cpu-features-cpuinfo.c | 5 +- sysdeps/x86/tst-cpu-features-supports.c | 16 +- sysdeps/x86/tst-get-cpu-features.c | 45 +- sysdeps/x86/x86_cpu_array.c | 31 + sysdeps/x86/x86_cpu_array_private.c | 21 + sysdeps/x86_64/dl-hwcaps-subdirs.c | 42 +- sysdeps/x86_64/dl-machine.h | 6 +- sysdeps/x86_64/fpu/math-tests-arch.h | 6 +- sysdeps/x86_64/fpu/multiarch/ifunc-avx-fma4.h | 10 +- sysdeps/x86_64/fpu/multiarch/ifunc-fma.h | 6 +- sysdeps/x86_64/fpu/multiarch/ifunc-fma4.h | 8 +- sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx2.h | 6 +- .../x86_64/fpu/multiarch/ifunc-mathvec-avx512.h | 4 +- .../x86_64/fpu/multiarch/ifunc-mathvec-sse4_1.h | 4 +- sysdeps/x86_64/fpu/multiarch/ifunc-sse4_1.h | 4 +- sysdeps/x86_64/fpu/multiarch/s_fma.c | 4 +- sysdeps/x86_64/fpu/multiarch/s_fmaf.c | 4 +- sysdeps/x86_64/multiarch/ifunc-avx2.h | 2 +- sysdeps/x86_64/multiarch/ifunc-impl-list.c | 228 +++--- sysdeps/x86_64/multiarch/ifunc-memcmp.h | 8 +- sysdeps/x86_64/multiarch/ifunc-memmove.h | 10 +- sysdeps/x86_64/multiarch/ifunc-memset.h | 10 +- sysdeps/x86_64/multiarch/ifunc-sse4_2.h | 4 +- sysdeps/x86_64/multiarch/ifunc-strcasecmp.h | 6 +- sysdeps/x86_64/multiarch/ifunc-strcpy.h | 4 +- sysdeps/x86_64/multiarch/ifunc-wmemset.h | 4 +- sysdeps/x86_64/multiarch/sched_cpucount.c | 3 +- sysdeps/x86_64/multiarch/strchr.c | 2 +- sysdeps/x86_64/multiarch/strcmp.c | 4 +- sysdeps/x86_64/multiarch/strncmp.c | 6 +- sysdeps/x86_64/multiarch/wcscpy.c | 4 +- sysdeps/x86_64/multiarch/wcsnlen.c | 4 +- sysdeps/x86_64/tst-glibc-hwcaps.c | 51 +- 76 files changed, 1032 insertions(+), 1355 deletions(-) diff --git a/sysdeps/i386/fpu/fclrexcpt.c b/sysdeps/i386/fpu/fclrexcpt.c index 7dc357f2d6..79379f78ef 100644 --- a/sysdeps/i386/fpu/fclrexcpt.c +++ b/sysdeps/i386/fpu/fclrexcpt.c @@ -41,7 +41,7 @@ __feclearexcept (int excepts) __asm__ ("fldenv %0" : : "m" (*&temp)); /* If the CPU supports SSE, we clear the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xnew_exc; diff --git a/sysdeps/i386/fpu/fedisblxcpt.c b/sysdeps/i386/fpu/fedisblxcpt.c index 5399bc1f25..e752a9f6a3 100644 --- a/sysdeps/i386/fpu/fedisblxcpt.c +++ b/sysdeps/i386/fpu/fedisblxcpt.c @@ -38,7 +38,7 @@ fedisableexcept (int excepts) __asm__ ("fldcw %0" : : "m" (*&new_exc)); /* If the CPU supports SSE we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xnew_exc; diff --git a/sysdeps/i386/fpu/feenablxcpt.c b/sysdeps/i386/fpu/feenablxcpt.c index b9d7e65668..7f8156b523 100644 --- a/sysdeps/i386/fpu/feenablxcpt.c +++ b/sysdeps/i386/fpu/feenablxcpt.c @@ -38,7 +38,7 @@ feenableexcept (int excepts) __asm__ ("fldcw %0" : : "m" (*&new_exc)); /* If the CPU supports SSE we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xnew_exc; diff --git a/sysdeps/i386/fpu/fegetenv.c b/sysdeps/i386/fpu/fegetenv.c index 637bc85454..1cc4ded6e9 100644 --- a/sysdeps/i386/fpu/fegetenv.c +++ b/sysdeps/i386/fpu/fegetenv.c @@ -31,7 +31,7 @@ __fegetenv (fenv_t *envp) would block all exceptions. */ __asm__ ("fldenv %0" : : "m" (*envp)); - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) __asm__ ("stmxcsr %0" : "=m" (envp->__eip)); /* Success. */ diff --git a/sysdeps/i386/fpu/fegetmode.c b/sysdeps/i386/fpu/fegetmode.c index e5154eab02..ca09eb5e45 100644 --- a/sysdeps/i386/fpu/fegetmode.c +++ b/sysdeps/i386/fpu/fegetmode.c @@ -26,7 +26,7 @@ int fegetmode (femode_t *modep) { _FPU_GETCW (modep->__control_word); - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) __asm__ ("stmxcsr %0" : "=m" (modep->__mxcsr)); return 0; } diff --git a/sysdeps/i386/fpu/feholdexcpt.c b/sysdeps/i386/fpu/feholdexcpt.c index 8d2d0ee275..94672a1a81 100644 --- a/sysdeps/i386/fpu/feholdexcpt.c +++ b/sysdeps/i386/fpu/feholdexcpt.c @@ -30,7 +30,7 @@ __feholdexcept (fenv_t *envp) __asm__ volatile ("fnstenv %0; fnclex" : "=m" (*envp)); /* If the CPU supports SSE we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xwork; diff --git a/sysdeps/i386/fpu/fesetenv.c b/sysdeps/i386/fpu/fesetenv.c index cd9afeae28..5d7c0179ad 100644 --- a/sysdeps/i386/fpu/fesetenv.c +++ b/sysdeps/i386/fpu/fesetenv.c @@ -79,7 +79,7 @@ __fesetenv (const fenv_t *envp) __asm__ ("fldenv %0" : : "m" (temp)); - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int mxcsr; __asm__ ("stmxcsr %0" : "=m" (mxcsr)); diff --git a/sysdeps/i386/fpu/fesetmode.c b/sysdeps/i386/fpu/fesetmode.c index e3b30657b1..ae866f1e41 100644 --- a/sysdeps/i386/fpu/fesetmode.c +++ b/sysdeps/i386/fpu/fesetmode.c @@ -35,7 +35,7 @@ fesetmode (const femode_t *modep) else cw = modep->__control_word; _FPU_SETCW (cw); - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int mxcsr; __asm__ ("stmxcsr %0" : "=m" (mxcsr)); diff --git a/sysdeps/i386/fpu/fesetround.c b/sysdeps/i386/fpu/fesetround.c index 5c3fd34cd4..8b53c6fd63 100644 --- a/sysdeps/i386/fpu/fesetround.c +++ b/sysdeps/i386/fpu/fesetround.c @@ -37,7 +37,7 @@ __fesetround (int round) __asm__ ("fldcw %0" : : "m" (*&cw)); /* If the CPU supports SSE we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xcw; diff --git a/sysdeps/i386/fpu/feupdateenv.c b/sysdeps/i386/fpu/feupdateenv.c index ef7132e4f0..8387fd2588 100644 --- a/sysdeps/i386/fpu/feupdateenv.c +++ b/sysdeps/i386/fpu/feupdateenv.c @@ -32,7 +32,7 @@ __feupdateenv (const fenv_t *envp) __asm__ ("fnstsw %0" : "=m" (*&temp)); /* If the CPU supports SSE we test the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) __asm__ ("stmxcsr %0" : "=m" (*&xtemp)); temp = (temp | xtemp) & FE_ALL_EXCEPT; diff --git a/sysdeps/i386/fpu/fgetexcptflg.c b/sysdeps/i386/fpu/fgetexcptflg.c index 2c32c83636..9efbe09802 100644 --- a/sysdeps/i386/fpu/fgetexcptflg.c +++ b/sysdeps/i386/fpu/fgetexcptflg.c @@ -34,7 +34,7 @@ __fegetexceptflag (fexcept_t *flagp, int excepts) *flagp = temp & excepts & FE_ALL_EXCEPT; /* If the CPU supports SSE, we clear the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int sse_exc; diff --git a/sysdeps/i386/fpu/fsetexcptflg.c b/sysdeps/i386/fpu/fsetexcptflg.c index 02a1bd526d..abf9f0c8d5 100644 --- a/sysdeps/i386/fpu/fsetexcptflg.c +++ b/sysdeps/i386/fpu/fsetexcptflg.c @@ -41,7 +41,7 @@ __fesetexceptflag (const fexcept_t *flagp, int excepts) __asm__ ("fldenv %0" : : "m" (*&temp)); /* If the CPU supports SSE, we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) { unsigned int xnew_exc; diff --git a/sysdeps/i386/fpu/ftestexcept.c b/sysdeps/i386/fpu/ftestexcept.c index a00c44e6db..bd8538de58 100644 --- a/sysdeps/i386/fpu/ftestexcept.c +++ b/sysdeps/i386/fpu/ftestexcept.c @@ -32,7 +32,7 @@ fetestexcept (int excepts) __asm__ ("fnstsw %0" : "=a" (temp)); /* If the CPU supports SSE we test the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_is_usable (x86_cpu_SSE)) __asm__ ("stmxcsr %0" : "=m" (*&xtemp)); return (temp | xtemp) & excepts & FE_ALL_EXCEPT; diff --git a/sysdeps/i386/i686/fpu/multiarch/s_cosf.c b/sysdeps/i386/i686/fpu/multiarch/s_cosf.c index 9cd14a103b..77f1939c54 100644 --- a/sysdeps/i386/i686/fpu/multiarch/s_cosf.c +++ b/sysdeps/i386/i686/fpu/multiarch/s_cosf.c @@ -23,7 +23,8 @@ extern float __cosf_sse2 (float); extern float __cosf_ia32 (float); -libm_ifunc (__cosf, CPU_FEATURE_USABLE (SSE2) ? __cosf_sse2 : __cosf_ia32); +libm_ifunc (__cosf, (x86_cpu_is_usable (x86_cpu_SSE2) + ? __cosf_sse2 : __cosf_ia32)); libm_alias_float (__cos, cos); #define COSF __cosf_ia32 diff --git a/sysdeps/i386/i686/fpu/multiarch/s_sincosf.c b/sysdeps/i386/i686/fpu/multiarch/s_sincosf.c index 9b479142d0..3a35ec7b5a 100644 --- a/sysdeps/i386/i686/fpu/multiarch/s_sincosf.c +++ b/sysdeps/i386/i686/fpu/multiarch/s_sincosf.c @@ -23,8 +23,8 @@ extern void __sincosf_sse2 (float, float *, float *); extern void __sincosf_ia32 (float, float *, float *); -libm_ifunc (__sincosf, - CPU_FEATURE_USABLE (SSE2) ? __sincosf_sse2 : __sincosf_ia32); +libm_ifunc (__sincosf, (x86_cpu_is_usable (x86_cpu_SSE2) + ? __sincosf_sse2 : __sincosf_ia32)); libm_alias_float (__sincos, sincos); #define SINCOSF __sincosf_ia32 diff --git a/sysdeps/i386/i686/fpu/multiarch/s_sinf.c b/sysdeps/i386/i686/fpu/multiarch/s_sinf.c index 84977e63e8..2f70bc30f8 100644 --- a/sysdeps/i386/i686/fpu/multiarch/s_sinf.c +++ b/sysdeps/i386/i686/fpu/multiarch/s_sinf.c @@ -23,7 +23,8 @@ extern float __sinf_sse2 (float); extern float __sinf_ia32 (float); -libm_ifunc (__sinf, CPU_FEATURE_USABLE (SSE2) ? __sinf_sse2 : __sinf_ia32); +libm_ifunc (__sinf, (x86_cpu_is_usable (x86_cpu_SSE2) + ? __sinf_sse2 : __sinf_ia32)); libm_alias_float (__sin, sin); #define SINF __sinf_ia32 #include diff --git a/sysdeps/i386/i686/multiarch/ifunc-impl-list.c b/sysdeps/i386/i686/multiarch/ifunc-impl-list.c index 89afdc0326..a1293bf30e 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-impl-list.c +++ b/sysdeps/i386/i686/multiarch/ifunc-impl-list.c @@ -38,35 +38,35 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/bcopy.S. */ IFUNC_IMPL (i, name, bcopy, - IFUNC_IMPL_ADD (array, i, bcopy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, bcopy, x86_cpu_is_usable (x86_cpu_SSSE3), __bcopy_ssse3_rep) - IFUNC_IMPL_ADD (array, i, bcopy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, bcopy, x86_cpu_is_usable (x86_cpu_SSSE3), __bcopy_ssse3) - IFUNC_IMPL_ADD (array, i, bcopy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, bcopy, x86_cpu_is_usable (x86_cpu_SSE2), __bcopy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, bcopy, 1, __bcopy_ia32)) /* Support sysdeps/i386/i686/multiarch/bzero.S. */ IFUNC_IMPL (i, name, bzero, - IFUNC_IMPL_ADD (array, i, bzero, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, bzero, x86_cpu_is_usable (x86_cpu_SSE2), __bzero_sse2_rep) - IFUNC_IMPL_ADD (array, i, bzero, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, bzero, x86_cpu_is_usable (x86_cpu_SSE2), __bzero_sse2) IFUNC_IMPL_ADD (array, i, bzero, 1, __bzero_ia32)) /* Support sysdeps/i386/i686/multiarch/memchr.S. */ IFUNC_IMPL (i, name, memchr, - IFUNC_IMPL_ADD (array, i, memchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memchr, x86_cpu_is_usable (x86_cpu_SSE2), __memchr_sse2_bsf) - IFUNC_IMPL_ADD (array, i, memchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memchr, x86_cpu_is_usable (x86_cpu_SSE2), __memchr_sse2) IFUNC_IMPL_ADD (array, i, memchr, 1, __memchr_ia32)) /* Support sysdeps/i386/i686/multiarch/memcmp.S. */ IFUNC_IMPL (i, name, memcmp, - IFUNC_IMPL_ADD (array, i, memcmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, memcmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __memcmp_sse4_2) - IFUNC_IMPL_ADD (array, i, memcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __memcmp_ssse3) IFUNC_IMPL_ADD (array, i, memcmp, 1, __memcmp_ia32)) @@ -74,13 +74,13 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/memmove_chk.S. */ IFUNC_IMPL (i, name, __memmove_chk, IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_chk_ssse3_rep) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_chk_ssse3) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (SSE2), + x86_cpu_is_usable (x86_cpu_SSE2), __memmove_chk_sse2_unaligned) IFUNC_IMPL_ADD (array, i, __memmove_chk, 1, __memmove_chk_ia32)) @@ -88,19 +88,19 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/memmove.S. */ IFUNC_IMPL (i, name, memmove, - IFUNC_IMPL_ADD (array, i, memmove, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memmove, x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_ssse3_rep) - IFUNC_IMPL_ADD (array, i, memmove, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memmove, x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_ssse3) - IFUNC_IMPL_ADD (array, i, memmove, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memmove, x86_cpu_is_usable (x86_cpu_SSE2), __memmove_sse2_unaligned) IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_ia32)) /* Support sysdeps/i386/i686/multiarch/memrchr.S. */ IFUNC_IMPL (i, name, memrchr, - IFUNC_IMPL_ADD (array, i, memrchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memrchr, x86_cpu_is_usable (x86_cpu_SSE2), __memrchr_sse2_bsf) - IFUNC_IMPL_ADD (array, i, memrchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memrchr, x86_cpu_is_usable (x86_cpu_SSE2), __memrchr_sse2) IFUNC_IMPL_ADD (array, i, memrchr, 1, __memrchr_ia32)) @@ -108,10 +108,10 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/memset_chk.S. */ IFUNC_IMPL (i, name, __memset_chk, IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (SSE2), + x86_cpu_is_usable (x86_cpu_SSE2), __memset_chk_sse2_rep) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (SSE2), + x86_cpu_is_usable (x86_cpu_SSE2), __memset_chk_sse2) IFUNC_IMPL_ADD (array, i, __memset_chk, 1, __memset_chk_ia32)) @@ -119,102 +119,102 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/memset.S. */ IFUNC_IMPL (i, name, memset, - IFUNC_IMPL_ADD (array, i, memset, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memset, x86_cpu_is_usable (x86_cpu_SSE2), __memset_sse2_rep) - IFUNC_IMPL_ADD (array, i, memset, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memset, x86_cpu_is_usable (x86_cpu_SSE2), __memset_sse2) IFUNC_IMPL_ADD (array, i, memset, 1, __memset_ia32)) /* Support sysdeps/i386/i686/multiarch/rawmemchr.S. */ IFUNC_IMPL (i, name, rawmemchr, - IFUNC_IMPL_ADD (array, i, rawmemchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, rawmemchr, x86_cpu_is_usable (x86_cpu_SSE2), __rawmemchr_sse2_bsf) - IFUNC_IMPL_ADD (array, i, rawmemchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, rawmemchr, x86_cpu_is_usable (x86_cpu_SSE2), __rawmemchr_sse2) IFUNC_IMPL_ADD (array, i, rawmemchr, 1, __rawmemchr_ia32)) /* Support sysdeps/i386/i686/multiarch/stpncpy.S. */ IFUNC_IMPL (i, name, stpncpy, - IFUNC_IMPL_ADD (array, i, stpncpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, stpncpy, x86_cpu_is_usable (x86_cpu_SSSE3), __stpncpy_ssse3) - IFUNC_IMPL_ADD (array, i, stpncpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, stpncpy, x86_cpu_is_usable (x86_cpu_SSE2), __stpncpy_sse2) IFUNC_IMPL_ADD (array, i, stpncpy, 1, __stpncpy_ia32)) /* Support sysdeps/i386/i686/multiarch/stpcpy.S. */ IFUNC_IMPL (i, name, stpcpy, - IFUNC_IMPL_ADD (array, i, stpcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, stpcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __stpcpy_ssse3) - IFUNC_IMPL_ADD (array, i, stpcpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, stpcpy, x86_cpu_is_usable (x86_cpu_SSE2), __stpcpy_sse2) IFUNC_IMPL_ADD (array, i, stpcpy, 1, __stpcpy_ia32)) /* Support sysdeps/i386/i686/multiarch/strcasecmp.S. */ IFUNC_IMPL (i, name, strcasecmp, IFUNC_IMPL_ADD (array, i, strcasecmp, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strcasecmp_sse4_2) IFUNC_IMPL_ADD (array, i, strcasecmp, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strcasecmp_ssse3) IFUNC_IMPL_ADD (array, i, strcasecmp, 1, __strcasecmp_ia32)) /* Support sysdeps/i386/i686/multiarch/strcasecmp_l.S. */ IFUNC_IMPL (i, name, strcasecmp_l, IFUNC_IMPL_ADD (array, i, strcasecmp_l, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strcasecmp_l_sse4_2) IFUNC_IMPL_ADD (array, i, strcasecmp_l, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strcasecmp_l_ssse3) IFUNC_IMPL_ADD (array, i, strcasecmp_l, 1, __strcasecmp_l_ia32)) /* Support sysdeps/i386/i686/multiarch/strcat.S. */ IFUNC_IMPL (i, name, strcat, - IFUNC_IMPL_ADD (array, i, strcat, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcat, x86_cpu_is_usable (x86_cpu_SSSE3), __strcat_ssse3) - IFUNC_IMPL_ADD (array, i, strcat, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strcat, x86_cpu_is_usable (x86_cpu_SSE2), __strcat_sse2) IFUNC_IMPL_ADD (array, i, strcat, 1, __strcat_ia32)) /* Support sysdeps/i386/i686/multiarch/strchr.S. */ IFUNC_IMPL (i, name, strchr, - IFUNC_IMPL_ADD (array, i, strchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strchr, x86_cpu_is_usable (x86_cpu_SSE2), __strchr_sse2_bsf) - IFUNC_IMPL_ADD (array, i, strchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strchr, x86_cpu_is_usable (x86_cpu_SSE2), __strchr_sse2) IFUNC_IMPL_ADD (array, i, strchr, 1, __strchr_ia32)) /* Support sysdeps/i386/i686/multiarch/strcmp.S. */ IFUNC_IMPL (i, name, strcmp, - IFUNC_IMPL_ADD (array, i, strcmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strcmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __strcmp_sse4_2) - IFUNC_IMPL_ADD (array, i, strcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __strcmp_ssse3) IFUNC_IMPL_ADD (array, i, strcmp, 1, __strcmp_ia32)) /* Support sysdeps/i386/i686/multiarch/strcpy.S. */ IFUNC_IMPL (i, name, strcpy, - IFUNC_IMPL_ADD (array, i, strcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __strcpy_ssse3) - IFUNC_IMPL_ADD (array, i, strcpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strcpy, x86_cpu_is_usable (x86_cpu_SSE2), __strcpy_sse2) IFUNC_IMPL_ADD (array, i, strcpy, 1, __strcpy_ia32)) /* Support sysdeps/i386/i686/multiarch/strcspn.S. */ IFUNC_IMPL (i, name, strcspn, - IFUNC_IMPL_ADD (array, i, strcspn, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strcspn, x86_cpu_is_usable (x86_cpu_SSE4_2), __strcspn_sse42) IFUNC_IMPL_ADD (array, i, strcspn, 1, __strcspn_ia32)) /* Support sysdeps/i386/i686/multiarch/strncase.S. */ IFUNC_IMPL (i, name, strncasecmp, IFUNC_IMPL_ADD (array, i, strncasecmp, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strncasecmp_sse4_2) IFUNC_IMPL_ADD (array, i, strncasecmp, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strncasecmp_ssse3) IFUNC_IMPL_ADD (array, i, strncasecmp, 1, __strncasecmp_ia32)) @@ -222,91 +222,91 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/strncase_l.S. */ IFUNC_IMPL (i, name, strncasecmp_l, IFUNC_IMPL_ADD (array, i, strncasecmp_l, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strncasecmp_l_sse4_2) IFUNC_IMPL_ADD (array, i, strncasecmp_l, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strncasecmp_l_ssse3) IFUNC_IMPL_ADD (array, i, strncasecmp_l, 1, __strncasecmp_l_ia32)) /* Support sysdeps/i386/i686/multiarch/strncat.S. */ IFUNC_IMPL (i, name, strncat, - IFUNC_IMPL_ADD (array, i, strncat, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncat, x86_cpu_is_usable (x86_cpu_SSSE3), __strncat_ssse3) - IFUNC_IMPL_ADD (array, i, strncat, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strncat, x86_cpu_is_usable (x86_cpu_SSE2), __strncat_sse2) IFUNC_IMPL_ADD (array, i, strncat, 1, __strncat_ia32)) /* Support sysdeps/i386/i686/multiarch/strncpy.S. */ IFUNC_IMPL (i, name, strncpy, - IFUNC_IMPL_ADD (array, i, strncpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncpy, x86_cpu_is_usable (x86_cpu_SSSE3), __strncpy_ssse3) - IFUNC_IMPL_ADD (array, i, strncpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strncpy, x86_cpu_is_usable (x86_cpu_SSE2), __strncpy_sse2) IFUNC_IMPL_ADD (array, i, strncpy, 1, __strncpy_ia32)) /* Support sysdeps/i386/i686/multiarch/strnlen.S. */ IFUNC_IMPL (i, name, strnlen, - IFUNC_IMPL_ADD (array, i, strnlen, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strnlen, x86_cpu_is_usable (x86_cpu_SSE2), __strnlen_sse2) IFUNC_IMPL_ADD (array, i, strnlen, 1, __strnlen_ia32)) /* Support sysdeps/i386/i686/multiarch/strpbrk.S. */ IFUNC_IMPL (i, name, strpbrk, - IFUNC_IMPL_ADD (array, i, strpbrk, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strpbrk, x86_cpu_is_usable (x86_cpu_SSE4_2), __strpbrk_sse42) IFUNC_IMPL_ADD (array, i, strpbrk, 1, __strpbrk_ia32)) /* Support sysdeps/i386/i686/multiarch/strrchr.S. */ IFUNC_IMPL (i, name, strrchr, - IFUNC_IMPL_ADD (array, i, strrchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strrchr, x86_cpu_is_usable (x86_cpu_SSE2), __strrchr_sse2_bsf) - IFUNC_IMPL_ADD (array, i, strrchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strrchr, x86_cpu_is_usable (x86_cpu_SSE2), __strrchr_sse2) IFUNC_IMPL_ADD (array, i, strrchr, 1, __strrchr_ia32)) /* Support sysdeps/i386/i686/multiarch/strspn.S. */ IFUNC_IMPL (i, name, strspn, - IFUNC_IMPL_ADD (array, i, strspn, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strspn, x86_cpu_is_usable (x86_cpu_SSE4_2), __strspn_sse42) IFUNC_IMPL_ADD (array, i, strspn, 1, __strspn_ia32)) /* Support sysdeps/i386/i686/multiarch/wcschr.S. */ IFUNC_IMPL (i, name, wcschr, - IFUNC_IMPL_ADD (array, i, wcschr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, wcschr, x86_cpu_is_usable (x86_cpu_SSE2), __wcschr_sse2) IFUNC_IMPL_ADD (array, i, wcschr, 1, __wcschr_ia32)) /* Support sysdeps/i386/i686/multiarch/wcscmp.S. */ IFUNC_IMPL (i, name, wcscmp, - IFUNC_IMPL_ADD (array, i, wcscmp, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, wcscmp, x86_cpu_is_usable (x86_cpu_SSE2), __wcscmp_sse2) IFUNC_IMPL_ADD (array, i, wcscmp, 1, __wcscmp_ia32)) /* Support sysdeps/i386/i686/multiarch/wcscpy.S. */ IFUNC_IMPL (i, name, wcscpy, - IFUNC_IMPL_ADD (array, i, wcscpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, wcscpy, x86_cpu_is_usable (x86_cpu_SSSE3), __wcscpy_ssse3) IFUNC_IMPL_ADD (array, i, wcscpy, 1, __wcscpy_ia32)) /* Support sysdeps/i386/i686/multiarch/wcslen.S. */ IFUNC_IMPL (i, name, wcslen, - IFUNC_IMPL_ADD (array, i, wcslen, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, wcslen, x86_cpu_is_usable (x86_cpu_SSE2), __wcslen_sse2) IFUNC_IMPL_ADD (array, i, wcslen, 1, __wcslen_ia32)) /* Support sysdeps/i386/i686/multiarch/wcsrchr.S. */ IFUNC_IMPL (i, name, wcsrchr, - IFUNC_IMPL_ADD (array, i, wcsrchr, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, wcsrchr, x86_cpu_is_usable (x86_cpu_SSE2), __wcsrchr_sse2) IFUNC_IMPL_ADD (array, i, wcsrchr, 1, __wcsrchr_ia32)) /* Support sysdeps/i386/i686/multiarch/wmemcmp.S. */ IFUNC_IMPL (i, name, wmemcmp, - IFUNC_IMPL_ADD (array, i, wmemcmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, wmemcmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __wmemcmp_sse4_2) - IFUNC_IMPL_ADD (array, i, wmemcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, wmemcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __wmemcmp_ssse3) IFUNC_IMPL_ADD (array, i, wmemcmp, 1, __wmemcmp_ia32)) @@ -314,64 +314,64 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/i386/i686/multiarch/memcpy_chk.S. */ IFUNC_IMPL (i, name, __memcpy_chk, IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_chk_ssse3_rep) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_chk_ssse3) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (SSE2), + x86_cpu_is_usable (x86_cpu_SSE2), __memcpy_chk_sse2_unaligned) IFUNC_IMPL_ADD (array, i, __memcpy_chk, 1, __memcpy_chk_ia32)) /* Support sysdeps/i386/i686/multiarch/memcpy.S. */ IFUNC_IMPL (i, name, memcpy, - IFUNC_IMPL_ADD (array, i, memcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_ssse3_rep) - IFUNC_IMPL_ADD (array, i, memcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_ssse3) - IFUNC_IMPL_ADD (array, i, memcpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, memcpy, x86_cpu_is_usable (x86_cpu_SSE2), __memcpy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, memcpy, 1, __memcpy_ia32)) /* Support sysdeps/i386/i686/multiarch/mempcpy_chk.S. */ IFUNC_IMPL (i, name, __mempcpy_chk, IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_chk_ssse3_rep) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_chk_ssse3) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (SSE2), + x86_cpu_is_usable (x86_cpu_SSE2), __mempcpy_chk_sse2_unaligned) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, 1, __mempcpy_chk_ia32)) /* Support sysdeps/i386/i686/multiarch/mempcpy.S. */ IFUNC_IMPL (i, name, mempcpy, - IFUNC_IMPL_ADD (array, i, mempcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, mempcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_ssse3_rep) - IFUNC_IMPL_ADD (array, i, mempcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, mempcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_ssse3) - IFUNC_IMPL_ADD (array, i, mempcpy, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, mempcpy, x86_cpu_is_usable (x86_cpu_SSE2), __mempcpy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, mempcpy, 1, __mempcpy_ia32)) /* Support sysdeps/i386/i686/multiarch/strlen.S. */ IFUNC_IMPL (i, name, strlen, - IFUNC_IMPL_ADD (array, i, strlen, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strlen, x86_cpu_is_usable (x86_cpu_SSE2), __strlen_sse2_bsf) - IFUNC_IMPL_ADD (array, i, strlen, CPU_FEATURE_USABLE (SSE2), + IFUNC_IMPL_ADD (array, i, strlen, x86_cpu_is_usable (x86_cpu_SSE2), __strlen_sse2) IFUNC_IMPL_ADD (array, i, strlen, 1, __strlen_ia32)) /* Support sysdeps/i386/i686/multiarch/strncmp.S. */ IFUNC_IMPL (i, name, strncmp, - IFUNC_IMPL_ADD (array, i, strncmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strncmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __strncmp_sse4_2) - IFUNC_IMPL_ADD (array, i, strncmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncmp, x86_cpu_is_usable (x86_cpu_SSSE3), __strncmp_ssse3) IFUNC_IMPL_ADD (array, i, strncmp, 1, __strncmp_ia32)) #endif diff --git a/sysdeps/i386/i686/multiarch/ifunc-memmove.h b/sysdeps/i386/i686/multiarch/ifunc-memmove.h index c05cb6dd4f..210b7713b1 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-memmove.h +++ b/sysdeps/i386/i686/multiarch/ifunc-memmove.h @@ -33,7 +33,7 @@ IFUNC_SELECTOR (void) if (CPU_FEATURES_ARCH_P (cpu_features, Fast_Unaligned_Load)) return OPTIMIZE (sse2_unaligned); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) { if (CPU_FEATURES_ARCH_P (cpu_features, Fast_Rep_String)) return OPTIMIZE (ssse3_rep); diff --git a/sysdeps/i386/i686/multiarch/ifunc-memset.h b/sysdeps/i386/i686/multiarch/ifunc-memset.h index bead331a9d..81476bf9a1 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-memset.h +++ b/sysdeps/i386/i686/multiarch/ifunc-memset.h @@ -28,7 +28,7 @@ IFUNC_SELECTOR (void) { const struct cpu_features* cpu_features = __get_cpu_features (); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE2)) + if (x86_cpu_is_usable (x86_cpu_SSE2)) { if (CPU_FEATURES_ARCH_P (cpu_features, Fast_Rep_String)) return OPTIMIZE (sse2_rep); diff --git a/sysdeps/i386/i686/multiarch/ifunc-sse2-bsf.h b/sysdeps/i386/i686/multiarch/ifunc-sse2-bsf.h index 0d302a3dcd..d60f965be4 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-sse2-bsf.h +++ b/sysdeps/i386/i686/multiarch/ifunc-sse2-bsf.h @@ -28,7 +28,7 @@ IFUNC_SELECTOR (void) { const struct cpu_features* cpu_features = __get_cpu_features (); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE2)) + if (x86_cpu_is_usable (x86_cpu_SSE2)) { if (CPU_FEATURES_ARCH_P (cpu_features, Slow_BSF)) return OPTIMIZE (sse2); diff --git a/sysdeps/i386/i686/multiarch/ifunc-sse2-ssse3.h b/sysdeps/i386/i686/multiarch/ifunc-sse2-ssse3.h index c10ca4a9df..eea46306ba 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-sse2-ssse3.h +++ b/sysdeps/i386/i686/multiarch/ifunc-sse2-ssse3.h @@ -29,11 +29,11 @@ IFUNC_SELECTOR (void) { const struct cpu_features* cpu_features = __get_cpu_features (); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE2) + if (x86_cpu_is_usable (x86_cpu_SSE2) && CPU_FEATURES_ARCH_P (cpu_features, Fast_Rep_String)) return OPTIMIZE (sse2); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (ia32); diff --git a/sysdeps/i386/i686/multiarch/ifunc-sse2.h b/sysdeps/i386/i686/multiarch/ifunc-sse2.h index 58794a2806..6a892890d8 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-sse2.h +++ b/sysdeps/i386/i686/multiarch/ifunc-sse2.h @@ -25,9 +25,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse2) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE2)) + if (x86_cpu_is_usable (x86_cpu_SSE2)) return OPTIMIZE (sse2); return OPTIMIZE (ia32); diff --git a/sysdeps/i386/i686/multiarch/ifunc-sse4_2.h b/sysdeps/i386/i686/multiarch/ifunc-sse4_2.h index 014be1d5f7..6f5f6582d5 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-sse4_2.h +++ b/sysdeps/i386/i686/multiarch/ifunc-sse4_2.h @@ -25,9 +25,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse42) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_2)) + if (x86_cpu_is_usable (x86_cpu_SSE4_2)) return OPTIMIZE (sse42); return OPTIMIZE (ia32); diff --git a/sysdeps/i386/i686/multiarch/ifunc-ssse3-sse4_2.h b/sysdeps/i386/i686/multiarch/ifunc-ssse3-sse4_2.h index 39bfea986d..02249ef10d 100644 --- a/sysdeps/i386/i686/multiarch/ifunc-ssse3-sse4_2.h +++ b/sysdeps/i386/i686/multiarch/ifunc-ssse3-sse4_2.h @@ -27,12 +27,10 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse4_2) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_2)) + if (x86_cpu_is_usable (x86_cpu_SSE4_2)) return OPTIMIZE (sse4_2); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (ia32); diff --git a/sysdeps/i386/i686/multiarch/s_fma.c b/sysdeps/i386/i686/multiarch/s_fma.c index 0729853e21..a70b6bc132 100644 --- a/sysdeps/i386/i686/multiarch/s_fma.c +++ b/sysdeps/i386/i686/multiarch/s_fma.c @@ -27,7 +27,7 @@ extern double __fma_ia32 (double x, double y, double z) attribute_hidden; extern double __fma_fma (double x, double y, double z) attribute_hidden; libm_ifunc (__fma, - CPU_FEATURE_USABLE (FMA) ? __fma_fma : __fma_ia32); + x86_cpu_is_usable (x86_cpu_FMA) ? __fma_fma : __fma_ia32); libm_alias_double (__fma, fma) #define __fma __fma_ia32 diff --git a/sysdeps/i386/i686/multiarch/s_fmaf.c b/sysdeps/i386/i686/multiarch/s_fmaf.c index 20f965c342..51a001b312 100644 --- a/sysdeps/i386/i686/multiarch/s_fmaf.c +++ b/sysdeps/i386/i686/multiarch/s_fmaf.c @@ -27,7 +27,7 @@ extern float __fmaf_ia32 (float x, float y, float z) attribute_hidden; extern float __fmaf_fma (float x, float y, float z) attribute_hidden; libm_ifunc (__fmaf, - CPU_FEATURE_USABLE (FMA) ? __fmaf_fma : __fmaf_ia32); + x86_cpu_is_usable (x86_cpu_FMA) ? __fmaf_fma : __fmaf_ia32); libm_alias_float (__fma, fma) #define __fmaf __fmaf_ia32 diff --git a/sysdeps/i386/i686/multiarch/wcscpy.c b/sysdeps/i386/i686/multiarch/wcscpy.c index f0038bc4a2..8d5120b0c6 100644 --- a/sysdeps/i386/i686/multiarch/wcscpy.c +++ b/sysdeps/i386/i686/multiarch/wcscpy.c @@ -32,9 +32,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (ssse3) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (ia32); diff --git a/sysdeps/i386/setfpucw.c b/sysdeps/i386/setfpucw.c index c640a72cc2..82c5aa55f7 100644 --- a/sysdeps/i386/setfpucw.c +++ b/sysdeps/i386/setfpucw.c @@ -39,7 +39,7 @@ __setfpucw (fpu_control_t set) __asm__ ("fldcw %0" : : "m" (*&cw)); /* If the CPU supports SSE, we set the MXCSR as well. */ - if (CPU_FEATURE_USABLE (SSE)) + if (x86_cpu_has_feature (x86_cpu_SSE)) { unsigned int xnew_exc; diff --git a/sysdeps/mach/hurd/i386/libc.abilist b/sysdeps/mach/hurd/i386/libc.abilist index 7a5eb66b85..36575c7d88 100644 --- a/sysdeps/mach/hurd/i386/libc.abilist +++ b/sysdeps/mach/hurd/i386/libc.abilist @@ -2191,6 +2191,7 @@ GLIBC_2.32 thrd_current F GLIBC_2.32 thrd_equal F GLIBC_2.32 thrd_sleep F GLIBC_2.32 thrd_yield F +GLIBC_2.33 __x86_cpu_array D 0x8 GLIBC_2.33 fstat F GLIBC_2.33 fstat64 F GLIBC_2.33 fstatat F diff --git a/sysdeps/unix/sysv/linux/i386/libc.abilist b/sysdeps/unix/sysv/linux/i386/libc.abilist index 4f0d3c1eb5..20b43a1f83 100644 --- a/sysdeps/unix/sysv/linux/i386/libc.abilist +++ b/sysdeps/unix/sysv/linux/i386/libc.abilist @@ -2229,6 +2229,7 @@ GLIBC_2.32 sigabbrev_np F GLIBC_2.32 sigdescr_np F GLIBC_2.32 strerrordesc_np F GLIBC_2.32 strerrorname_np F +GLIBC_2.33 __x86_cpu_array D 0x8 GLIBC_2.33 fstat F GLIBC_2.33 fstat64 F GLIBC_2.33 fstatat F diff --git a/sysdeps/unix/sysv/linux/x86/elision-conf.c b/sysdeps/unix/sysv/linux/x86/elision-conf.c index ecdb0378e3..64948b75b8 100644 --- a/sysdeps/unix/sysv/linux/x86/elision-conf.c +++ b/sysdeps/unix/sysv/linux/x86/elision-conf.c @@ -21,6 +21,7 @@ #include #include #include +#include #if HAVE_TUNABLES # define TUNABLE_NAMESPACE elision @@ -63,7 +64,7 @@ do_set_elision_enable (int32_t elision_enable) if __libc_enable_secure isn't enabled since elision_enable will be set according to the default, which is disabled. */ if (elision_enable == 1) - __pthread_force_elision = CPU_FEATURE_USABLE (RTM) ? 1 : 0; + __pthread_force_elision = x86_cpu_is_usable (x86_cpu_RTM) ? 1 : 0; } /* The pthread->elision_enable tunable is 0 or 1 indicating that elision diff --git a/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist b/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist index 4fff61818b..e66e13f59e 100644 --- a/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist +++ b/sysdeps/unix/sysv/linux/x86_64/64/libc.abilist @@ -2076,6 +2076,7 @@ GLIBC_2.32 sigabbrev_np F GLIBC_2.32 sigdescr_np F GLIBC_2.32 strerrordesc_np F GLIBC_2.32 strerrorname_np F +GLIBC_2.33 __x86_cpu_array D 0x10 GLIBC_2.33 fstat F GLIBC_2.33 fstat64 F GLIBC_2.33 fstatat F diff --git a/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist b/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist index 102ed47a9c..517b13e9fe 100644 --- a/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist +++ b/sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist @@ -2173,6 +2173,7 @@ GLIBC_2.32 sigabbrev_np F GLIBC_2.32 sigdescr_np F GLIBC_2.32 strerrordesc_np F GLIBC_2.32 strerrorname_np F +GLIBC_2.33 __x86_cpu_array D 0x8 GLIBC_2.33 fstat F GLIBC_2.33 fstat64 F GLIBC_2.33 fstatat F diff --git a/sysdeps/x86/Makefile b/sysdeps/x86/Makefile index 081cc72e93..d6f93fb45c 100644 --- a/sysdeps/x86/Makefile +++ b/sysdeps/x86/Makefile @@ -3,8 +3,9 @@ gen-as-const-headers += cpu-features-offsets.sym endif ifeq ($(subdir),elf) -sysdep-dl-routines += dl-get-cpu-features -sysdep_headers += sys/platform/x86.h +sysdep_routines += x86_cpu_array +sysdep-dl-routines += dl-get-cpu-features x86_cpu_array_private +sysdep_headers += bits/platform/x86.h sys/platform/x86.h tests += tst-get-cpu-features tst-get-cpu-features-static \ tst-cpu-features-cpuinfo tst-cpu-features-supports diff --git a/sysdeps/x86/Versions b/sysdeps/x86/Versions index 59db578a9d..64796eea6e 100644 --- a/sysdeps/x86/Versions +++ b/sysdeps/x86/Versions @@ -3,3 +3,9 @@ ld { __x86_get_cpu_features; } } + +libc { + GLIBC_2.33 { + __x86_cpu_array; + } +} \ No newline at end of file diff --git a/sysdeps/x86/bits/platform/x86.h b/sysdeps/x86/bits/platform/x86.h new file mode 100644 index 0000000000..dd9a273f5b --- /dev/null +++ b/sysdeps/x86/bits/platform/x86.h @@ -0,0 +1,245 @@ +/* Constants for x86 CPU features and struct x86_cpu_array definition. + This file is part of the GNU C Library. + Copyright (C) 2008-2020 Free Software Foundation, Inc. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _SYS_PLATFORM_X86_H +# error "Never include directly; use instead." +#endif + +struct x86_cpu_array +{ + /* Pointer to an array of __x86_count 32-bit values. */ + const unsigned int *__x86_word; + unsigned int __x86_count; +#ifdef __LP64__ + unsigned int __x86_padding; +#endif +}; + +enum +{ + /* CPUID.01H:ECX. */ + x86_cpu_SSE3 = 0u * 64u + 0u, + x86_cpu_PCLMULQDQ = 0u * 64u + 1u, + x86_cpu_DTES64 = 0u * 64u + 2u, + x86_cpu_MONITOR = 0u * 64u + 3u, + x86_cpu_DS_CPL = 0u * 64u + 4u, + x86_cpu_VMX = 0u * 64u + 5u, + x86_cpu_SMX = 0u * 64u + 6u, + x86_cpu_EIST = 0u * 64u + 7u, + x86_cpu_TM2 = 0u * 64u + 8u, + x86_cpu_SSSE3 = 0u * 64u + 9u, + x86_cpu_CNXT_ID = 0u * 64u + 10u, + x86_cpu_SDBG = 0u * 64u + 11u, + x86_cpu_FMA = 0u * 64u + 12u, + x86_cpu_CMPXCHG16B = 0u * 64u + 13u, + x86_cpu_XTPRUPDCTRL = 0u * 64u + 14u, + x86_cpu_PDCM = 0u * 64u + 15u, + x86_cpu_INDEX_1_ECX_16 = 0u * 64u + 16u, + x86_cpu_PCID = 0u * 64u + 17u, + x86_cpu_DCA = 0u * 64u + 18u, + x86_cpu_SSE4_1 = 0u * 64u + 19u, + x86_cpu_SSE4_2 = 0u * 64u + 20u, + x86_cpu_X2APIC = 0u * 64u + 21u, + x86_cpu_MOVBE = 0u * 64u + 22u, + x86_cpu_POPCNT = 0u * 64u + 23u, + x86_cpu_TSC_DEADLINE = 0u * 64u + 24u, + x86_cpu_AES = 0u * 64u + 25u, + x86_cpu_XSAVE = 0u * 64u + 26u, + x86_cpu_OSXSAVE = 0u * 64u + 27u, + x86_cpu_AVX = 0u * 64u + 28u, + x86_cpu_F16C = 0u * 64u + 29u, + x86_cpu_RDRAND = 0u * 64u + 30u, + x86_cpu_INDEX_1_ECX_31 = 0u * 64u + 31u, + + /* CPUID.01H:EDX. */ + x86_cpu_FPU = 1u * 64u + 0u, + x86_cpu_VME = 1u * 64u + 1u, + x86_cpu_DE = 1u * 64u + 2u, + x86_cpu_PSE = 1u * 64u + 3u, + x86_cpu_TSC = 1u * 64u + 4u, + x86_cpu_MSR = 1u * 64u + 5u, + x86_cpu_PAE = 1u * 64u + 6u, + x86_cpu_MCE = 1u * 64u + 7u, + x86_cpu_CX8 = 1u * 64u + 8u, + x86_cpu_APIC = 1u * 64u + 9u, + x86_cpu_INDEX_1_EDX_10 = 1u * 64u + 10u, + x86_cpu_SEP = 1u * 64u + 11u, + x86_cpu_MTRR = 1u * 64u + 12u, + x86_cpu_PGE = 1u * 64u + 13u, + x86_cpu_MCA = 1u * 64u + 14u, + x86_cpu_CMOV = 1u * 64u + 15u, + x86_cpu_PAT = 1u * 64u + 16u, + x86_cpu_PSE_36 = 1u * 64u + 17u, + x86_cpu_PSN = 1u * 64u + 18u, + x86_cpu_CLFSH = 1u * 64u + 19u, + x86_cpu_INDEX_1_EDX_20 = 1u * 64u + 20u, + x86_cpu_DS = 1u * 64u + 21u, + x86_cpu_ACPI = 1u * 64u + 22u, + x86_cpu_MMX = 1u * 64u + 23u, + x86_cpu_FXSR = 1u * 64u + 24u, + x86_cpu_SSE = 1u * 64u + 25u, + x86_cpu_SSE2 = 1u * 64u + 26u, + x86_cpu_SS = 1u * 64u + 27u, + x86_cpu_HTT = 1u * 64u + 28u, + x86_cpu_TM = 1u * 64u + 29u, + x86_cpu_INDEX_1_EDX_30 = 1u * 64u + 30u, + x86_cpu_PBE = 1u * 64u + 31u, + + /* CPUID.07H.0H:EBX. */ + x86_cpu_FSGSBASE = 2u * 64u + 0u, + x86_cpu_TSC_ADJUST = 2u * 64u + 1u, + x86_cpu_SGX = 2u * 64u + 2u, + x86_cpu_BMI1 = 2u * 64u + 3u, + x86_cpu_HLE = 2u * 64u + 4u, + x86_cpu_AVX2 = 2u * 64u + 5u, + x86_cpu_INDEX_7_EBX_6 = 2u * 64u + 6u, + x86_cpu_SMEP = 2u * 64u + 7u, + x86_cpu_BMI2 = 2u * 64u + 8u, + x86_cpu_ERMS = 2u * 64u + 9u, + x86_cpu_INVPCID = 2u * 64u + 10u, + x86_cpu_RTM = 2u * 64u + 11u, + x86_cpu_RDT_M = 2u * 64u + 12u, + x86_cpu_DEPR_FPU_CS_DS = 2u * 64u + 13u, + x86_cpu_MPX = 2u * 64u + 14u, + x86_cpu_RDT_A = 2u * 64u + 15u, + x86_cpu_AVX512F = 2u * 64u + 16u, + x86_cpu_AVX512DQ = 2u * 64u + 17u, + x86_cpu_RDSEED = 2u * 64u + 18u, + x86_cpu_ADX = 2u * 64u + 19u, + x86_cpu_SMAP = 2u * 64u + 20u, + x86_cpu_AVX512_IFMA = 2u * 64u + 21u, + x86_cpu_INDEX_7_EBX_22 = 2u * 64u + 22u, + x86_cpu_CLFLUSHOPT = 2u * 64u + 23u, + x86_cpu_CLWB = 2u * 64u + 24u, + x86_cpu_TRACE = 2u * 64u + 25u, + x86_cpu_AVX512PF = 2u * 64u + 26u, + x86_cpu_AVX512ER = 2u * 64u + 27u, + x86_cpu_AVX512CD = 2u * 64u + 28u, + x86_cpu_SHA = 2u * 64u + 29u, + x86_cpu_AVX512BW = 2u * 64u + 30u, + x86_cpu_AVX512VL = 2u * 64u + 31u, + + /* CPUID.07H.0H:ECX. */ + x86_cpu_PREFETCHWT1 = 3u * 64u + 0u, + x86_cpu_AVX512_VBMI = 3u * 64u + 1u, + x86_cpu_UMIP = 3u * 64u + 2u, + x86_cpu_PKU = 3u * 64u + 3u, + x86_cpu_OSPKE = 3u * 64u + 4u, + x86_cpu_WAITPKG = 3u * 64u + 5u, + x86_cpu_AVX512_VBMI2 = 3u * 64u + 6u, + x86_cpu_SHSTK = 3u * 64u + 7u, + x86_cpu_GFNI = 3u * 64u + 8u, + x86_cpu_VAES = 3u * 64u + 9u, + x86_cpu_VPCLMULQDQ = 3u * 64u + 10u, + x86_cpu_AVX512_VNNI = 3u * 64u + 11u, + x86_cpu_AVX512_BITALG = 3u * 64u + 12u, + x86_cpu_INDEX_7_ECX_13 = 3u * 64u + 13u, + x86_cpu_AVX512_VPOPCNTDQ = 3u * 64u + 14u, + x86_cpu_INDEX_7_ECX_15 = 3u * 64u + 15u, + x86_cpu_INDEX_7_ECX_16 = 3u * 64u + 16u, + /* Note: Bits 17-21: The value of MAWAU used by the BNDLDX and + BNDSTX instructions in 64-bit mode. */ + x86_cpu_RDPID = 3u * 64u + 22u, + x86_cpu_KL = 3u * 64u + 23u, + x86_cpu_INDEX_7_ECX_24 = 3u * 64u + 24u, + x86_cpu_CLDEMOTE = 3u * 64u + 25u, + x86_cpu_INDEX_7_ECX_26 = 3u * 64u + 26u, + x86_cpu_MOVDIRI = 3u * 64u + 27u, + x86_cpu_MOVDIR64B = 3u * 64u + 28u, + x86_cpu_ENQCMD = 3u * 64u + 29u, + x86_cpu_SGX_LC = 3u * 64u + 30u, + x86_cpu_PKS = 3u * 64u + 31u, + + /* CPUID.07H.0H:EDX. */ + x86_cpu_INDEX_7_EDX_0 = 4u * 64u + 0u, + x86_cpu_INDEX_7_EDX_1 = 4u * 64u + 1u, + x86_cpu_AVX512_4VNNIW = 4u * 64u + 2u, + x86_cpu_AVX512_4FMAPS = 4u * 64u + 3u, + x86_cpu_FSRM = 4u * 64u + 4u, + x86_cpu_UINTR = 4u * 64u + 5u, + x86_cpu_INDEX_7_EDX_6 = 4u * 64u + 6u, + x86_cpu_INDEX_7_EDX_7 = 4u * 64u + 7u, + x86_cpu_AVX512_VP2INTERSECT = 4u * 64u + 8u, + x86_cpu_INDEX_7_EDX_9 = 4u * 64u + 9u, + x86_cpu_MD_CLEAR = 4u * 64u + 10u, + x86_cpu_INDEX_7_EDX_11 = 4u * 64u + 11u, + x86_cpu_INDEX_7_EDX_12 = 4u * 64u + 12u, + x86_cpu_INDEX_7_EDX_13 = 4u * 64u + 13u, + x86_cpu_SERIALIZE = 4u * 64u + 14u, + x86_cpu_HYBRID = 4u * 64u + 15u, + x86_cpu_TSXLDTRK = 4u * 64u + 16u, + x86_cpu_INDEX_7_EDX_17 = 4u * 64u + 17u, + x86_cpu_PCONFIG = 4u * 64u + 18u, + x86_cpu_INDEX_7_EDX_19 = 4u * 64u + 19u, + x86_cpu_IBT = 4u * 64u + 20u, + x86_cpu_INDEX_7_EDX_21 = 4u * 64u + 21u, + x86_cpu_AMX_BF16 = 4u * 64u + 22u, + x86_cpu_AVX512_FP16 = 4u * 64u + 23u, + x86_cpu_AMX_TILE = 4u * 64u + 24u, + x86_cpu_AMX_INT8 = 4u * 64u + 25u, + x86_cpu_IBRS_IBPB = 4u * 64u + 26u, + x86_cpu_STIBP = 4u * 64u + 27u, + x86_cpu_L1D_FLUSH = 4u * 64u + 28u, + x86_cpu_ARCH_CAPABILITIES = 4u * 64u + 29u, + x86_cpu_CORE_CAPABILITIES = 4u * 64u + 30u, + x86_cpu_SSBD = 4u * 64u + 31u, + + /* CPUID.80000001H:ECX. */ + x86_cpu_LAHF64_SAHF64 = 5u * 64u + 0u, + x86_cpu_SVM = 5u * 64u + 2u, + x86_cpu_LZCNT = 5u * 64u + 5u, + x86_cpu_SSE4A = 5u * 64u + 6u, + x86_cpu_PREFETCHW = 5u * 64u + 8u, + x86_cpu_XOP = 5u * 64u + 11u, + x86_cpu_LWP = 5u * 64u + 15u, + x86_cpu_FMA4 = 5u * 64u + 16u, + x86_cpu_TBM = 5u * 64u + 21u, + + /* CPUID.80000001H:EDX. */ + x86_cpu_SYSCALL_SYSRET = 6u * 64u + 11u, + x86_cpu_NX = 6u * 64u + 20u, + x86_cpu_PAGE1GB = 6u * 64u + 26u, + x86_cpu_RDTSCP = 6u * 64u + 27u, + x86_cpu_LM = 6u * 64u + 29u, + + /* CPUID.(EAX=0DH,ECX=1):EAX. */ + x86_cpu_XSAVEOPT = 7u * 64u + 0u, + x86_cpu_XSAVEC = 7u * 64u + 1u, + x86_cpu_XGETBV_ECX_1 = 7u * 64u + 2u, + x86_cpu_XSAVES = 7u * 64u + 3u, + x86_cpu_XFD = 7u * 64u + 4u, + + /* CPUID.80000007H:EDX. */ + x86_cpu_INVARIANT_TSC = 8u * 64u + 8u, + + /* CPUID.80000008H:EBX. */ + x86_cpu_WBNOINVD = 9u * 64u + 9u, + + /* CPUID.(EAX=07H.,ECX=1):EAX. */ + x86_cpu_AVX_VNNI = 10u * 64u + 4u, + x86_cpu_AVX512_BF16 = 10u * 64u + 5u, + x86_cpu_FZLRM = 10u * 64u + 10u, + x86_cpu_FSRS = 10u * 64u + 11u, + x86_cpu_FSRCS = 10u * 64u + 12u, + x86_cpu_HRESET = 10u * 64u + 22u, + x86_cpu_LAM = 10u * 64u + 26u, + + /* CPUID.19H:EBX. */ + x86_cpu_AESKLE = 11u * 64u + 0u, + x86_cpu_WIDE_KL = 11u * 64u + 2u, +}; diff --git a/sysdeps/x86/cacheinfo.h b/sysdeps/x86/cacheinfo.h index 0aec0e2875..4a68072f3d 100644 --- a/sysdeps/x86/cacheinfo.h +++ b/sysdeps/x86/cacheinfo.h @@ -91,7 +91,7 @@ get_common_cache_info (long int *shared_ptr, unsigned int *threads_ptr, /* A value of 0 for the HTT bit indicates there is only a single logical processor. */ - if (HAS_CPU_FEATURE (HTT)) + if (x86_cpu_has_feature (x86_cpu_HTT)) { /* Figure out the number of logical threads that share the highest cache level. */ @@ -236,12 +236,12 @@ get_common_cache_info (long int *shared_ptr, unsigned int *threads_ptr, } else { -intel_bug_no_cache_info: +intel_bug_no_cache_info:; /* Assume that all logical threads share the highest cache level. */ - threads - = ((cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.ebx - >> 16) & 0xff); + unsigned int eax, ebx, ecx, edx; + __cpuid (1, eax, ebx, ecx, edx); + threads = (ebx >> 16) & 0xff; } /* Cap usage of highest cache level to the number of supported @@ -401,7 +401,7 @@ init_cacheinfo (void) unsigned int minimum_rep_movsb_threshold; /* NB: The default REP MOVSB threshold is 2048 * (VEC_SIZE / 16). */ unsigned int rep_movsb_threshold; - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512F) + if (x86_cpu_is_usable (x86_cpu_AVX512F) && !CPU_FEATURE_PREFERRED_P (cpu_features, Prefer_No_AVX512)) { rep_movsb_threshold = 2048 * (64 / 16); diff --git a/sysdeps/x86/cpu-features.c b/sysdeps/x86/cpu-features.c index fe080b63b2..f9cf6bbfba 100644 --- a/sysdeps/x86/cpu-features.c +++ b/sysdeps/x86/cpu-features.c @@ -46,65 +46,80 @@ extern void TUNABLE_CALLBACK (set_x86_shstk) (tunable_val_t *) # include #endif +/* Copy one indexed bit from the features word to the usable word. */ +static inline void +copy_usable_1 (unsigned int index) +{ + __x86_cpu_array_private[(index >> 5) + 1] + |= __x86_cpu_array_private[index >> 5] & (1u << (index % 32)); +} + +/* Mark one feature as usable. */ +static inline void +set_usable_1 (unsigned int index) +{ + __x86_cpu_array_private[(index >> 5) + 1] |= 1u << (index % 32); +} + static void update_usable (struct cpu_features *cpu_features) { /* Copy the cpuid bits to usable bits for CPU featuress whose usability in user space can be detected without additonal OS support. */ - CPU_FEATURE_SET_USABLE (cpu_features, SSE3); - CPU_FEATURE_SET_USABLE (cpu_features, PCLMULQDQ); - CPU_FEATURE_SET_USABLE (cpu_features, SSSE3); - CPU_FEATURE_SET_USABLE (cpu_features, CMPXCHG16B); - CPU_FEATURE_SET_USABLE (cpu_features, SSE4_1); - CPU_FEATURE_SET_USABLE (cpu_features, SSE4_2); - CPU_FEATURE_SET_USABLE (cpu_features, MOVBE); - CPU_FEATURE_SET_USABLE (cpu_features, POPCNT); - CPU_FEATURE_SET_USABLE (cpu_features, AES); - CPU_FEATURE_SET_USABLE (cpu_features, OSXSAVE); - CPU_FEATURE_SET_USABLE (cpu_features, TSC); - CPU_FEATURE_SET_USABLE (cpu_features, CX8); - CPU_FEATURE_SET_USABLE (cpu_features, CMOV); - CPU_FEATURE_SET_USABLE (cpu_features, CLFSH); - CPU_FEATURE_SET_USABLE (cpu_features, MMX); - CPU_FEATURE_SET_USABLE (cpu_features, FXSR); - CPU_FEATURE_SET_USABLE (cpu_features, SSE); - CPU_FEATURE_SET_USABLE (cpu_features, SSE2); - CPU_FEATURE_SET_USABLE (cpu_features, HTT); - CPU_FEATURE_SET_USABLE (cpu_features, BMI1); - CPU_FEATURE_SET_USABLE (cpu_features, HLE); - CPU_FEATURE_SET_USABLE (cpu_features, BMI2); - CPU_FEATURE_SET_USABLE (cpu_features, ERMS); - CPU_FEATURE_SET_USABLE (cpu_features, RTM); - CPU_FEATURE_SET_USABLE (cpu_features, RDSEED); - CPU_FEATURE_SET_USABLE (cpu_features, ADX); - CPU_FEATURE_SET_USABLE (cpu_features, CLFLUSHOPT); - CPU_FEATURE_SET_USABLE (cpu_features, CLWB); - CPU_FEATURE_SET_USABLE (cpu_features, SHA); - CPU_FEATURE_SET_USABLE (cpu_features, PREFETCHWT1); - CPU_FEATURE_SET_USABLE (cpu_features, OSPKE); - CPU_FEATURE_SET_USABLE (cpu_features, WAITPKG); - CPU_FEATURE_SET_USABLE (cpu_features, GFNI); - CPU_FEATURE_SET_USABLE (cpu_features, RDPID); - CPU_FEATURE_SET_USABLE (cpu_features, RDRAND); - CPU_FEATURE_SET_USABLE (cpu_features, CLDEMOTE); - CPU_FEATURE_SET_USABLE (cpu_features, MOVDIRI); - CPU_FEATURE_SET_USABLE (cpu_features, MOVDIR64B); - CPU_FEATURE_SET_USABLE (cpu_features, FSRM); - CPU_FEATURE_SET_USABLE (cpu_features, SERIALIZE); - CPU_FEATURE_SET_USABLE (cpu_features, TSXLDTRK); - CPU_FEATURE_SET_USABLE (cpu_features, LAHF64_SAHF64); - CPU_FEATURE_SET_USABLE (cpu_features, LZCNT); - CPU_FEATURE_SET_USABLE (cpu_features, SSE4A); - CPU_FEATURE_SET_USABLE (cpu_features, PREFETCHW); - CPU_FEATURE_SET_USABLE (cpu_features, TBM); - CPU_FEATURE_SET_USABLE (cpu_features, RDTSCP); - CPU_FEATURE_SET_USABLE (cpu_features, WBNOINVD); - CPU_FEATURE_SET_USABLE (cpu_features, FZLRM); - CPU_FEATURE_SET_USABLE (cpu_features, FSRS); - CPU_FEATURE_SET_USABLE (cpu_features, FSRCS); + copy_usable_1 (x86_cpu_SSE3); + copy_usable_1 (x86_cpu_PCLMULQDQ); + copy_usable_1 (x86_cpu_SSSE3); + copy_usable_1 (x86_cpu_CMPXCHG16B); + copy_usable_1 (x86_cpu_SSE4_1); + copy_usable_1 (x86_cpu_SSE4_2); + copy_usable_1 (x86_cpu_MOVBE); + copy_usable_1 (x86_cpu_POPCNT); + copy_usable_1 (x86_cpu_AES); + copy_usable_1 (x86_cpu_OSXSAVE); + copy_usable_1 (x86_cpu_TSC); + copy_usable_1 (x86_cpu_CX8); + copy_usable_1 (x86_cpu_CMOV); + copy_usable_1 (x86_cpu_CLFSH); + copy_usable_1 (x86_cpu_MMX); + copy_usable_1 (x86_cpu_FXSR); + copy_usable_1 (x86_cpu_SSE); + copy_usable_1 (x86_cpu_SSE2); + copy_usable_1 (x86_cpu_HTT); + copy_usable_1 (x86_cpu_BMI1); + copy_usable_1 (x86_cpu_HLE); + copy_usable_1 (x86_cpu_BMI2); + copy_usable_1 (x86_cpu_ERMS); + copy_usable_1 (x86_cpu_RTM); + copy_usable_1 (x86_cpu_RDSEED); + copy_usable_1 (x86_cpu_ADX); + copy_usable_1 (x86_cpu_CLFLUSHOPT); + copy_usable_1 (x86_cpu_CLWB); + copy_usable_1 (x86_cpu_SHA); + copy_usable_1 (x86_cpu_PREFETCHWT1); + copy_usable_1 (x86_cpu_OSPKE); + copy_usable_1 (x86_cpu_WAITPKG); + copy_usable_1 (x86_cpu_GFNI); + copy_usable_1 (x86_cpu_RDPID); + copy_usable_1 (x86_cpu_RDRAND); + copy_usable_1 (x86_cpu_CLDEMOTE); + copy_usable_1 (x86_cpu_MOVDIRI); + copy_usable_1 (x86_cpu_MOVDIR64B); + copy_usable_1 (x86_cpu_FSRM); + copy_usable_1 (x86_cpu_SERIALIZE); + copy_usable_1 (x86_cpu_TSXLDTRK); + copy_usable_1 (x86_cpu_LAHF64_SAHF64); + copy_usable_1 (x86_cpu_LZCNT); + copy_usable_1 (x86_cpu_SSE4A); + copy_usable_1 (x86_cpu_PREFETCHW); + copy_usable_1 (x86_cpu_TBM); + copy_usable_1 (x86_cpu_RDTSCP); + copy_usable_1 (x86_cpu_WBNOINVD); + copy_usable_1 (x86_cpu_FZLRM); + copy_usable_1 (x86_cpu_FSRS); + copy_usable_1 (x86_cpu_FSRCS); /* Can we call xgetbv? */ - if (CPU_FEATURES_CPU_P (cpu_features, OSXSAVE)) + if (x86_cpu_has_feature (x86_cpu_OSXSAVE)) { unsigned int xcrlow; unsigned int xcrhigh; @@ -114,14 +129,14 @@ update_usable (struct cpu_features *cpu_features) == (bit_YMM_state | bit_XMM_state)) { /* Determine if AVX is usable. */ - if (CPU_FEATURES_CPU_P (cpu_features, AVX)) + if (x86_cpu_has_feature (x86_cpu_AVX)) { - CPU_FEATURE_SET (cpu_features, AVX); + set_usable_1 (x86_cpu_AVX); /* The following features depend on AVX being usable. */ /* Determine if AVX2 is usable. */ - if (CPU_FEATURES_CPU_P (cpu_features, AVX2)) + if (x86_cpu_has_feature (x86_cpu_AVX2)) { - CPU_FEATURE_SET (cpu_features, AVX2); + set_usable_1 (x86_cpu_AVX2); /* Unaligned load with 256-bit AVX registers are faster on Intel/AMD processors with AVX2. */ @@ -129,17 +144,17 @@ update_usable (struct cpu_features *cpu_features) |= bit_arch_AVX_Fast_Unaligned_Load; } /* Determine if AVX-VNNI is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX_VNNI); + copy_usable_1 (x86_cpu_AVX_VNNI); /* Determine if FMA is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, FMA); + copy_usable_1 (x86_cpu_FMA); /* Determine if VAES is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, VAES); + copy_usable_1 (x86_cpu_VAES); /* Determine if VPCLMULQDQ is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, VPCLMULQDQ); + copy_usable_1 (x86_cpu_VPCLMULQDQ); /* Determine if XOP is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, XOP); + copy_usable_1 (x86_cpu_XOP); /* Determine if F16C is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, F16C); + copy_usable_1 (x86_cpu_F16C); } /* Check if OPMASK state, upper 256-bit of ZMM0-ZMM15 and @@ -149,45 +164,43 @@ update_usable (struct cpu_features *cpu_features) == (bit_Opmask_state | bit_ZMM0_15_state | bit_ZMM16_31_state)) { /* Determine if AVX512F is usable. */ - if (CPU_FEATURES_CPU_P (cpu_features, AVX512F)) + if (x86_cpu_has_feature (x86_cpu_AVX512F)) { - CPU_FEATURE_SET (cpu_features, AVX512F); + set_usable_1 (x86_cpu_AVX512F); /* Determine if AVX512CD is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512CD); + copy_usable_1 (x86_cpu_AVX512CD); /* Determine if AVX512ER is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512ER); + copy_usable_1 (x86_cpu_AVX512ER); /* Determine if AVX512PF is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512PF); + copy_usable_1 (x86_cpu_AVX512PF); /* Determine if AVX512VL is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512VL); + copy_usable_1 (x86_cpu_AVX512VL); /* Determine if AVX512DQ is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512DQ); + copy_usable_1 (x86_cpu_AVX512DQ); /* Determine if AVX512BW is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512BW); + copy_usable_1 (x86_cpu_AVX512BW); /* Determine if AVX512_4FMAPS is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_4FMAPS); + copy_usable_1 (x86_cpu_AVX512_4FMAPS); /* Determine if AVX512_4VNNIW is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_4VNNIW); + copy_usable_1 (x86_cpu_AVX512_4VNNIW); /* Determine if AVX512_BITALG is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_BITALG); + copy_usable_1 (x86_cpu_AVX512_BITALG); /* Determine if AVX512_IFMA is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_IFMA); + copy_usable_1 (x86_cpu_AVX512_IFMA); /* Determine if AVX512_VBMI is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_VBMI); + copy_usable_1 (x86_cpu_AVX512_VBMI); /* Determine if AVX512_VBMI2 is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_VBMI2); + copy_usable_1 (x86_cpu_AVX512_VBMI2); /* Determine if is AVX512_VNNI usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_VNNI); + copy_usable_1 (x86_cpu_AVX512_VNNI); /* Determine if AVX512_VPOPCNTDQ is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, - AVX512_VPOPCNTDQ); + copy_usable_1 (x86_cpu_AVX512_VPOPCNTDQ); /* Determine if AVX512_VP2INTERSECT is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, - AVX512_VP2INTERSECT); + copy_usable_1 (x86_cpu_AVX512_VP2INTERSECT); /* Determine if AVX512_BF16 is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_BF16); + copy_usable_1 (x86_cpu_AVX512_BF16); /* Determine if AVX512_FP16 is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AVX512_FP16); + copy_usable_1 (x86_cpu_AVX512_FP16); } } } @@ -197,19 +210,19 @@ update_usable (struct cpu_features *cpu_features) == (bit_XTILECFG_state | bit_XTILEDATA_state)) { /* Determine if AMX_BF16 is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AMX_BF16); + copy_usable_1 (x86_cpu_AMX_BF16); /* Determine if AMX_TILE is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AMX_TILE); + copy_usable_1 (x86_cpu_AMX_TILE); /* Determine if AMX_INT8 is usable. */ - CPU_FEATURE_SET_USABLE (cpu_features, AMX_INT8); + copy_usable_1 (x86_cpu_AMX_INT8); } /* These features are usable only when OSXSAVE is enabled. */ - CPU_FEATURE_SET (cpu_features, XSAVE); - CPU_FEATURE_SET_USABLE (cpu_features, XSAVEOPT); - CPU_FEATURE_SET_USABLE (cpu_features, XSAVEC); - CPU_FEATURE_SET_USABLE (cpu_features, XGETBV_ECX_1); - CPU_FEATURE_SET_USABLE (cpu_features, XFD); + set_usable_1 (x86_cpu_XSAVE); + copy_usable_1 (x86_cpu_XSAVEOPT); + copy_usable_1 (x86_cpu_XSAVEC); + copy_usable_1 (x86_cpu_XGETBV_ECX_1); + copy_usable_1 (x86_cpu_XFD); /* For _dl_runtime_resolve, set xsave_state_size to xsave area size + integer register save size and align it to 64 bytes. */ @@ -229,7 +242,7 @@ update_usable (struct cpu_features *cpu_features) = xsave_state_full_size; /* Check if XSAVEC is available. */ - if (CPU_FEATURES_CPU_P (cpu_features, XSAVEC)) + if (x86_cpu_has_feature (x86_cpu_XSAVEC)) { unsigned int xstate_comp_offsets[32]; unsigned int xstate_comp_sizes[32]; @@ -272,7 +285,7 @@ update_usable (struct cpu_features *cpu_features) { cpu_features->xsave_state_size = ALIGN_UP (size + STATE_SAVE_OFFSET, 64); - CPU_FEATURE_SET (cpu_features, XSAVEC); + set_usable_1 (x86_cpu_XSAVEC); } } } @@ -280,41 +293,40 @@ update_usable (struct cpu_features *cpu_features) } /* Determine if PKU is usable. */ - if (CPU_FEATURES_CPU_P (cpu_features, OSPKE)) - CPU_FEATURE_SET (cpu_features, PKU); + if (x86_cpu_has_feature (x86_cpu_OSPKE)) + set_usable_1 (x86_cpu_PKU); /* Determine if Key Locker instructions are usable. */ - if (CPU_FEATURES_CPU_P (cpu_features, AESKLE)) + if (x86_cpu_has_feature (x86_cpu_AESKLE)) { - CPU_FEATURE_SET (cpu_features, AESKLE); - CPU_FEATURE_SET_USABLE (cpu_features, KL); - CPU_FEATURE_SET_USABLE (cpu_features, WIDE_KL); + set_usable_1 (x86_cpu_AESKLE); + copy_usable_1 (x86_cpu_KL); + copy_usable_1 (x86_cpu_WIDE_KL); } } static void -get_extended_indices (struct cpu_features *cpu_features) +get_extended_indices (void) { unsigned int eax, ebx, ecx, edx; __cpuid (0x80000000, eax, ebx, ecx, edx); - if (eax >= 0x80000001) - __cpuid (0x80000001, - cpu_features->features[COMMON_CPUID_INDEX_80000001].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_80000001].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_80000001].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_80000001].cpuid.edx); - if (eax >= 0x80000007) - __cpuid (0x80000007, - cpu_features->features[COMMON_CPUID_INDEX_80000007].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_80000007].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_80000007].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_80000007].cpuid.edx); - if (eax >= 0x80000008) - __cpuid (0x80000008, - cpu_features->features[COMMON_CPUID_INDEX_80000008].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_80000008].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_80000008].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_80000008].cpuid.edx); + unsigned int level = eax; + if (level >= 0x80000001) + { + __cpuid (0x80000001, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_LAHF64_SAHF64 >> 5] = ecx; + __x86_cpu_array_private[x86_cpu_SYSCALL_SYSRET >> 5] = edx; + } + if (level >= 0x80000007) + { + __cpuid (0x80000007, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_INVARIANT_TSC >> 5] = edx; + } + if (level >= 0x80000008) + { + __cpuid (0x80000008, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_WBNOINVD >> 5] = ebx; + } } static void @@ -324,12 +336,8 @@ get_common_indices (struct cpu_features *cpu_features, { if (family) { - unsigned int eax; - __cpuid (1, eax, - cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.edx); - cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.eax = eax; + unsigned int eax, ebx, ecx, edx; + __cpuid (1, eax, ebx, ecx, edx); *family = (eax >> 8) & 0x0f; *model = (eax >> 4) & 0x0f; *extended_model = (eax >> 12) & 0xf0; @@ -339,35 +347,35 @@ get_common_indices (struct cpu_features *cpu_features, *family += (eax >> 20) & 0xff; *model += *extended_model; } + __x86_cpu_array_private[x86_cpu_SSE3 >> 5] = ecx; + __x86_cpu_array_private[x86_cpu_FPU >> 5] = edx; } if (cpu_features->basic.max_cpuid >= 7) { - __cpuid_count (7, 0, - cpu_features->features[COMMON_CPUID_INDEX_7].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_7].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_7].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_7].cpuid.edx); - __cpuid_count (7, 1, - cpu_features->features[COMMON_CPUID_INDEX_7_ECX_1].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_7_ECX_1].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_7_ECX_1].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_7_ECX_1].cpuid.edx); + unsigned int eax, ebx, ecx, edx; + __cpuid_count (7, 0, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_FSGSBASE >> 5] = ebx; + __x86_cpu_array_private[x86_cpu_PREFETCHWT1 >> 5] = ecx; + __x86_cpu_array_private[x86_cpu_INDEX_7_EDX_0 >> 5] = edx; + + __cpuid_count (7, 1, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_AVX_VNNI >> 5] = eax; } if (cpu_features->basic.max_cpuid >= 0xd) - __cpuid_count (0xd, 1, - cpu_features->features[COMMON_CPUID_INDEX_D_ECX_1].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_D_ECX_1].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_D_ECX_1].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_D_ECX_1].cpuid.edx); + { + unsigned int eax, ebx, ecx, edx; + __cpuid_count (0xd, 1, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_XSAVEOPT >> 5] = eax; + } if (cpu_features->basic.max_cpuid >= 0x19) - __cpuid_count (0x19, 0, - cpu_features->features[COMMON_CPUID_INDEX_19].cpuid.eax, - cpu_features->features[COMMON_CPUID_INDEX_19].cpuid.ebx, - cpu_features->features[COMMON_CPUID_INDEX_19].cpuid.ecx, - cpu_features->features[COMMON_CPUID_INDEX_19].cpuid.edx); + { + unsigned int eax, ebx, ecx, edx; + __cpuid_count (0x19, 0, eax, ebx, ecx, edx); + __x86_cpu_array_private[x86_cpu_AESKLE >> 5] = ebx; + } } _Static_assert (((index_arch_Fast_Unaligned_Load @@ -411,7 +419,7 @@ init_cpu_features (struct cpu_features *cpu_features) get_common_indices (cpu_features, &family, &model, &extended_model, &stepping); - get_extended_indices (cpu_features); + get_extended_indices (); update_usable (cpu_features); @@ -473,7 +481,7 @@ init_cpu_features (struct cpu_features *cpu_features) default: /* Unknown family 0x06 processors. Assuming this is one of Core i3/i5/i7 processors if AVX is available. */ - if (!CPU_FEATURES_CPU_P (cpu_features, AVX)) + if (!x86_cpu_has_feature (x86_cpu_AVX)) break; /* Fall through. */ @@ -511,7 +519,7 @@ init_cpu_features (struct cpu_features *cpu_features) with stepping >= 4) to avoid TSX on kernels that weren't updated with the latest microcode package (which disables broken feature by default). */ - CPU_FEATURE_UNSET (cpu_features, RTM); + x86_cpu_clear_usable (x86_cpu_RTM); break; } } @@ -520,7 +528,7 @@ init_cpu_features (struct cpu_features *cpu_features) /* Since AVX512ER is unique to Xeon Phi, set Prefer_No_VZEROUPPER if AVX512ER is available. Don't use AVX512 to avoid lower CPU frequency if AVX512ER isn't available. */ - if (CPU_FEATURES_CPU_P (cpu_features, AVX512ER)) + if (x86_cpu_has_feature (x86_cpu_AVX512ER)) cpu_features->preferred[index_arch_Prefer_No_VZEROUPPER] |= bit_arch_Prefer_No_VZEROUPPER; else @@ -538,17 +546,15 @@ init_cpu_features (struct cpu_features *cpu_features) get_common_indices (cpu_features, &family, &model, &extended_model, &stepping); - get_extended_indices (cpu_features); + get_extended_indices (); update_usable (cpu_features); - ecx = cpu_features->features[COMMON_CPUID_INDEX_1].cpuid.ecx; - - if (CPU_FEATURE_USABLE_P (cpu_features, AVX)) + if (x86_cpu_is_usable (x86_cpu_AVX)) { /* Since the FMA4 bit is in COMMON_CPUID_INDEX_80000001 and FMA4 requires AVX, determine if FMA4 is usable here. */ - CPU_FEATURE_SET_USABLE (cpu_features, FMA4); + copy_usable_1 (x86_cpu_FMA4); } if (family == 0x15) @@ -577,7 +583,7 @@ init_cpu_features (struct cpu_features *cpu_features) get_common_indices (cpu_features, &family, &model, &extended_model, &stepping); - get_extended_indices (cpu_features); + get_extended_indices (); update_usable (cpu_features); @@ -586,8 +592,8 @@ init_cpu_features (struct cpu_features *cpu_features) { if (model == 0xf || model == 0x19) { - CPU_FEATURE_UNSET (cpu_features, AVX); - CPU_FEATURE_UNSET (cpu_features, AVX2); + x86_cpu_clear_usable (x86_cpu_AVX); + x86_cpu_clear_usable (x86_cpu_AVX2); cpu_features->preferred[index_arch_Slow_SSE4_2] |= bit_arch_Slow_SSE4_2; @@ -600,8 +606,8 @@ init_cpu_features (struct cpu_features *cpu_features) { if (model == 0x1b) { - CPU_FEATURE_UNSET (cpu_features, AVX); - CPU_FEATURE_UNSET (cpu_features, AVX2); + x86_cpu_clear_usable (x86_cpu_AVX); + x86_cpu_clear_usable (x86_cpu_AVX2); cpu_features->preferred[index_arch_Slow_SSE4_2] |= bit_arch_Slow_SSE4_2; @@ -611,8 +617,8 @@ init_cpu_features (struct cpu_features *cpu_features) } else if (model == 0x3b) { - CPU_FEATURE_UNSET (cpu_features, AVX); - CPU_FEATURE_UNSET (cpu_features, AVX2); + x86_cpu_clear_usable (x86_cpu_AVX); + x86_cpu_clear_usable (x86_cpu_AVX2); cpu_features->preferred[index_arch_AVX_Fast_Unaligned_Load] &= ~bit_arch_AVX_Fast_Unaligned_Load; @@ -627,11 +633,11 @@ init_cpu_features (struct cpu_features *cpu_features) } /* Support i586 if CX8 is available. */ - if (CPU_FEATURES_CPU_P (cpu_features, CX8)) + if (x86_cpu_has_feature (x86_cpu_CX8)) cpu_features->preferred[index_arch_I586] |= bit_arch_I586; /* Support i686 if CMOV is available. */ - if (CPU_FEATURES_CPU_P (cpu_features, CMOV)) + if (x86_cpu_has_feature (x86_cpu_CMOV)) cpu_features->preferred[index_arch_I686] |= bit_arch_I686; #if !HAS_CPUID @@ -670,30 +676,30 @@ no_cpuid: { const char *platform = NULL; - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512CD)) + if (x86_cpu_is_usable (x86_cpu_AVX512CD)) { - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512ER)) + if (x86_cpu_is_usable (x86_cpu_AVX512ER)) { - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512PF)) + if (x86_cpu_is_usable (x86_cpu_AVX512PF)) platform = "xeon_phi"; } else { - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512BW) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512DQ) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512VL)) + if (x86_cpu_is_usable (x86_cpu_AVX512BW) + && x86_cpu_is_usable (x86_cpu_AVX512DQ) + && x86_cpu_is_usable (x86_cpu_AVX512VL)) GLRO(dl_hwcap) |= HWCAP_X86_AVX512_1; } } if (platform == NULL - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) - && CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, BMI1) - && CPU_FEATURE_USABLE_P (cpu_features, BMI2) - && CPU_FEATURE_USABLE_P (cpu_features, LZCNT) - && CPU_FEATURE_USABLE_P (cpu_features, MOVBE) - && CPU_FEATURE_USABLE_P (cpu_features, POPCNT)) + && x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_BMI1) + && x86_cpu_is_usable (x86_cpu_BMI2) + && x86_cpu_is_usable (x86_cpu_LZCNT) + && x86_cpu_is_usable (x86_cpu_MOVBE) + && x86_cpu_is_usable (x86_cpu_POPCNT)) platform = "haswell"; if (platform != NULL) @@ -701,7 +707,7 @@ no_cpuid: } #else GLRO(dl_hwcap) = 0; - if (CPU_FEATURE_USABLE_P (cpu_features, SSE2)) + if (x86_cpu_is_usable (x86_cpu_SSE2)) GLRO(dl_hwcap) |= HWCAP_X86_SSE2; if (CPU_FEATURES_ARCH_P (cpu_features, I686)) diff --git a/sysdeps/x86/cpu-tunables.c b/sysdeps/x86/cpu-tunables.c index 588bbf9448..3f904f7e55 100644 --- a/sysdeps/x86/cpu-tunables.c +++ b/sysdeps/x86/cpu-tunables.c @@ -43,7 +43,7 @@ extern __typeof (memcmp) DEFAULT_MEMCMP; _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \ if (!DEFAULT_MEMCMP (f, #name, len)) \ { \ - CPU_FEATURE_UNSET (cpu_features, name) \ + x86_cpu_clear_usable (x86_cpu_##name); \ break; \ } @@ -80,7 +80,7 @@ extern __typeof (memcmp) DEFAULT_MEMCMP; { \ if (disable) \ cpu_features->preferred[index_arch_##name] &= ~bit_arch_##name; \ - else if (CPU_FEATURE_USABLE_P (cpu_features, need)) \ + else if (x86_cpu_is_usable (x86_cpu_##need)) \ cpu_features->preferred[index_arch_##name] |= bit_arch_##name; \ break; \ } @@ -181,7 +181,7 @@ TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *valp) /* Update xsave_state_size to XSAVE state size. */ cpu_features->xsave_state_size = cpu_features->xsave_state_full_size; - CPU_FEATURE_UNSET (cpu_features, XSAVEC); + x86_cpu_clear_usable (x86_cpu_XSAVEC); } } break; diff --git a/sysdeps/x86/include/cpu-features.h b/sysdeps/x86/include/cpu-features.h index f62be0b9b3..bbc3afdaac 100644 --- a/sysdeps/x86/include/cpu-features.h +++ b/sysdeps/x86/include/cpu-features.h @@ -30,7 +30,7 @@ # define __x86_get_cpu_features __x86_get_cpu_features_public #endif -#include +#include #ifndef _ISOMAC @@ -48,33 +48,14 @@ enum }; /* Only used directly in cpu-features.c. */ -# define CPU_FEATURE_SET(ptr, name) \ - ptr->features[index_cpu_##name].usable.reg_##name |= bit_cpu_##name; -# define CPU_FEATURE_UNSET(ptr, name) \ - ptr->features[index_cpu_##name].usable.reg_##name &= ~bit_cpu_##name; -# define CPU_FEATURE_SET_USABLE(ptr, name) \ - ptr->features[index_cpu_##name].usable.reg_##name \ - |= ptr->features[index_cpu_##name].cpuid.reg_##name & bit_cpu_##name; # define CPU_FEATURE_PREFERRED_P(ptr, name) \ ((ptr->preferred[index_arch_##name] & bit_arch_##name) != 0) -# define CPU_FEATURE_CPU_P(ptr, name) \ - CPU_FEATURE_CHECK_P (ptr, name, cpuid) -/* HAS_CPU_FEATURE evaluates to true if CPU supports the feature. */ -# undef HAS_CPU_FEATURE -# define HAS_CPU_FEATURE(name) \ - CPU_FEATURE_CPU_P (__x86_get_cpu_features (0), name) -/* CPU_FEATURE_USABLE evaluates to true if the feature is usable. */ -# undef CPU_FEATURE_USABLE -# define CPU_FEATURE_USABLE(name) \ - CPU_FEATURE_USABLE_P (__x86_get_cpu_features (0), name) /* CPU_FEATURE_PREFER evaluates to true if we prefer the feature at runtime. */ # define CPU_FEATURE_PREFERRED(name) \ CPU_FEATURE_PREFERRED_P(__get_cpu_features (), name) -# define CPU_FEATURES_CPU_P(ptr, name) \ - CPU_FEATURE_CPU_P (ptr, name) # define CPU_FEATURES_ARCH_P(ptr, name) \ CPU_FEATURE_PREFERRED_P (ptr, name) # define HAS_ARCH_FEATURE(name) \ @@ -124,10 +105,49 @@ enum # define bit_XTILECFG_state (1u << 17) # define bit_XTILEDATA_state (1u << 18) +enum +{ + COMMON_CPUID_INDEX_1 = 0, + COMMON_CPUID_INDEX_7, + COMMON_CPUID_INDEX_80000001, + COMMON_CPUID_INDEX_D_ECX_1, + COMMON_CPUID_INDEX_80000007, + COMMON_CPUID_INDEX_80000008, + COMMON_CPUID_INDEX_7_ECX_1, + COMMON_CPUID_INDEX_19, + /* Keep the following line at the end. */ + COMMON_CPUID_INDEX_MAX +}; + +struct cpuid_registers +{ + unsigned int eax; + unsigned int ebx; + unsigned int ecx; + unsigned int edx; +}; + +enum cpu_features_kind +{ + arch_kind_unknown = 0, + arch_kind_intel, + arch_kind_amd, + arch_kind_zhaoxin, + arch_kind_other +}; + +struct cpu_features_basic +{ + enum cpu_features_kind kind; + int max_cpuid; + unsigned int family; + unsigned int model; + unsigned int stepping; +}; + struct cpu_features { struct cpu_features_basic basic; - struct cpuid_features features[COMMON_CPUID_INDEX_MAX]; unsigned int preferred[PREFERRED_FEATURE_INDEX_MAX]; /* The state size for XSAVEC or XSAVE. The type must be unsigned long int so that we use diff --git a/sysdeps/x86/include/sys/platform/x86.h b/sysdeps/x86/include/sys/platform/x86.h new file mode 100644 index 0000000000..7d024aa376 --- /dev/null +++ b/sysdeps/x86/include/sys/platform/x86.h @@ -0,0 +1,62 @@ +/* Wrapper header for . + This file is part of the GNU C Library. + Copyright (C) 2008-2020 Free Software Foundation, Inc. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _SYS_PLATFORM_X86_H +# ifdef _ISOMAC +# include +# else +# if IS_IN (rtld) || (! defined (SHARED) && IS_IN (libc)) +/* libc.a and ld.so are special because they have a local array of fixed + size. This is possible because internal users know the exact array + size. Statically linked application code still uses the + indirection through __x86_cpu_array, for ABI consistency with + dynamic linking. */ + +# define _SYS_PLATFORM_X86_H +# include + +enum { x86_cpu_array_size = 12 * 2 }; +extern unsigned int __x86_cpu_array_private[x86_cpu_array_size] + attribute_hidden; + +static inline _Bool +x86_cpu_has_feature (unsigned int __index) +{ + return __x86_cpu_array_private[__index >> 5] & (1u << (__index % 32)); +} + +static inline _Bool +x86_cpu_is_usable (unsigned int __index) +{ + return __x86_cpu_array_private[(__index >> 5) + 1] & (1u << (__index % 32)); +} + +/* Mark one feature as unavailable. */ +static inline void +x86_cpu_clear_usable (unsigned int index) +{ + __x86_cpu_array_private[(index >> 5) + 1] &= ~(1u << (index % 32)); +} + +# else /* !rtld */ +# include +libc_hidden_proto (__x86_cpu_array) +# endif /* !rtld */ + +# endif /* !ISOMAC */ +#endif /* !_SYS_PLATFORM_X86_H */ diff --git a/sysdeps/x86/libc-vars-init.h b/sysdeps/x86/libc-vars-init.h new file mode 100644 index 0000000000..98e908534f --- /dev/null +++ b/sysdeps/x86/libc-vars-init.h @@ -0,0 +1,42 @@ +/* Initialization of __libc_vars. x86 version. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +/* No CPU-specific data directly in __libc_vars. */ +static inline void +_dl_libc_vars_init_cpu (struct libc_vars *vars) +{ +} + +/* But we have an additional variable that needs initialization. */ +static inline void +_dl_call_libc_vars_init_cpu (struct link_map *libc_map) +{ + const ElfW (Sym) *sym + = _dl_lookup_direct (libc_map, "__x86_cpu_array", + 0x34bfb42e, /* dl_new_hash output. */ + "GLIBC_2.33", + 0x69691b3); /* _dl_elf_hash output. */ + assert (sym != NULL); + assert (sym->st_size == sizeof (struct x86_cpu_array)); + struct x86_cpu_array *array = DL_SYMBOL_ADDRESS (libc_map, sym); + array->__x86_word = __x86_cpu_array_private; + array->__x86_count = array_length (__x86_cpu_array_private); +} diff --git a/sysdeps/x86/sys/platform/x86.h b/sysdeps/x86/sys/platform/x86.h index 99d8c9b0ab..fb0821b994 100644 --- a/sysdeps/x86/sys/platform/x86.h +++ b/sysdeps/x86/sys/platform/x86.h @@ -19,760 +19,48 @@ #ifndef _SYS_PLATFORM_X86_H #define _SYS_PLATFORM_X86_H -enum +#include +#include +#include + +__BEGIN_DECLS + +/* Access to detected CPU features. */ +extern struct x86_cpu_array __x86_cpu_array; + +#ifdef __extern_always_inline +/* Return the word referenced by INDEX, or 0 if it is out of bounds. + INDEX be one of the enumeration constants defined below, + potentially ORed with 32u, to request a usable (instead of feature) + CPU flag. */ +__extern_always_inline unsigned int __attribute__ ((const)) +__x86_cpu_array_index (unsigned int __index) { - COMMON_CPUID_INDEX_1 = 0, - COMMON_CPUID_INDEX_7, - COMMON_CPUID_INDEX_80000001, - COMMON_CPUID_INDEX_D_ECX_1, - COMMON_CPUID_INDEX_80000007, - COMMON_CPUID_INDEX_80000008, - COMMON_CPUID_INDEX_7_ECX_1, - COMMON_CPUID_INDEX_19, - /* Keep the following line at the end. */ - COMMON_CPUID_INDEX_MAX -}; - -struct cpuid_registers + if (__index >> 5 < __x86_cpu_array.__x86_count) + return __x86_cpu_array.__x86_word[__index >> 5]; + else + return 0; +} + +/* Return true if the CPU has feature INDEX, one of the enumeration + constants defined below. */ +__extern_always_inline bool __attribute__ ((const)) +x86_cpu_has_feature (unsigned int __index) { - unsigned int eax; - unsigned int ebx; - unsigned int ecx; - unsigned int edx; -}; - -struct cpuid_features + return __x86_cpu_array_index (__index) & (1u << (__index % 32)); +} + +/* Return true if the CPU has feature INDEX, one of the enumeration + constants defined below, and the feature is supported by the + current executation environment. */ +__extern_always_inline bool __attribute__ ((const)) +x86_cpu_is_usable (unsigned int __index) { - struct cpuid_registers cpuid; - struct cpuid_registers usable; -}; - -enum cpu_features_kind -{ - arch_kind_unknown = 0, - arch_kind_intel, - arch_kind_amd, - arch_kind_zhaoxin, - arch_kind_other -}; - -struct cpu_features_basic -{ - enum cpu_features_kind kind; - int max_cpuid; - unsigned int family; - unsigned int model; - unsigned int stepping; -}; - -struct cpu_features -{ - struct cpu_features_basic basic; - struct cpuid_features features[COMMON_CPUID_INDEX_MAX]; -}; - -/* Get a pointer to the CPU features structure. */ -extern const struct cpu_features *__x86_get_cpu_features (unsigned int) - __attribute__ ((const)); - -#define CPU_FEATURE_CHECK_P(ptr, name, check) \ - ((ptr->features[index_cpu_##name].check.reg_##name \ - & bit_cpu_##name) != 0) -#define CPU_FEATURE_CPU_P(ptr, name) \ - CPU_FEATURE_CHECK_P (ptr, name, cpuid) -#define CPU_FEATURE_USABLE_P(ptr, name) \ - CPU_FEATURE_CHECK_P (ptr, name, usable) - -/* HAS_CPU_FEATURE evaluates to true if CPU supports the feature. */ -#define HAS_CPU_FEATURE(name) \ - (__extension__ \ - ({ const struct cpu_features *__ptr = \ - __x86_get_cpu_features (COMMON_CPUID_INDEX_MAX); \ - __ptr && CPU_FEATURE_CPU_P (__ptr, name); })) -/* CPU_FEATURE_USABLE evaluates to true if the feature is usable. */ -#define CPU_FEATURE_USABLE(name) \ - (__extension__ \ - ({ const struct cpu_features *__ptr = \ - __x86_get_cpu_features (COMMON_CPUID_INDEX_MAX); \ - __ptr && CPU_FEATURE_USABLE_P (__ptr, name); })) - -/* CPU features. */ - -/* COMMON_CPUID_INDEX_1. */ - -/* ECX. */ -#define bit_cpu_SSE3 (1u << 0) -#define bit_cpu_PCLMULQDQ (1u << 1) -#define bit_cpu_DTES64 (1u << 2) -#define bit_cpu_MONITOR (1u << 3) -#define bit_cpu_DS_CPL (1u << 4) -#define bit_cpu_VMX (1u << 5) -#define bit_cpu_SMX (1u << 6) -#define bit_cpu_EIST (1u << 7) -#define bit_cpu_TM2 (1u << 8) -#define bit_cpu_SSSE3 (1u << 9) -#define bit_cpu_CNXT_ID (1u << 10) -#define bit_cpu_SDBG (1u << 11) -#define bit_cpu_FMA (1u << 12) -#define bit_cpu_CMPXCHG16B (1u << 13) -#define bit_cpu_XTPRUPDCTRL (1u << 14) -#define bit_cpu_PDCM (1u << 15) -#define bit_cpu_INDEX_1_ECX_16 (1u << 16) -#define bit_cpu_PCID (1u << 17) -#define bit_cpu_DCA (1u << 18) -#define bit_cpu_SSE4_1 (1u << 19) -#define bit_cpu_SSE4_2 (1u << 20) -#define bit_cpu_X2APIC (1u << 21) -#define bit_cpu_MOVBE (1u << 22) -#define bit_cpu_POPCNT (1u << 23) -#define bit_cpu_TSC_DEADLINE (1u << 24) -#define bit_cpu_AES (1u << 25) -#define bit_cpu_XSAVE (1u << 26) -#define bit_cpu_OSXSAVE (1u << 27) -#define bit_cpu_AVX (1u << 28) -#define bit_cpu_F16C (1u << 29) -#define bit_cpu_RDRAND (1u << 30) -#define bit_cpu_INDEX_1_ECX_31 (1u << 31) - -/* EDX. */ -#define bit_cpu_FPU (1u << 0) -#define bit_cpu_VME (1u << 1) -#define bit_cpu_DE (1u << 2) -#define bit_cpu_PSE (1u << 3) -#define bit_cpu_TSC (1u << 4) -#define bit_cpu_MSR (1u << 5) -#define bit_cpu_PAE (1u << 6) -#define bit_cpu_MCE (1u << 7) -#define bit_cpu_CX8 (1u << 8) -#define bit_cpu_APIC (1u << 9) -#define bit_cpu_INDEX_1_EDX_10 (1u << 10) -#define bit_cpu_SEP (1u << 11) -#define bit_cpu_MTRR (1u << 12) -#define bit_cpu_PGE (1u << 13) -#define bit_cpu_MCA (1u << 14) -#define bit_cpu_CMOV (1u << 15) -#define bit_cpu_PAT (1u << 16) -#define bit_cpu_PSE_36 (1u << 17) -#define bit_cpu_PSN (1u << 18) -#define bit_cpu_CLFSH (1u << 19) -#define bit_cpu_INDEX_1_EDX_20 (1u << 20) -#define bit_cpu_DS (1u << 21) -#define bit_cpu_ACPI (1u << 22) -#define bit_cpu_MMX (1u << 23) -#define bit_cpu_FXSR (1u << 24) -#define bit_cpu_SSE (1u << 25) -#define bit_cpu_SSE2 (1u << 26) -#define bit_cpu_SS (1u << 27) -#define bit_cpu_HTT (1u << 28) -#define bit_cpu_TM (1u << 29) -#define bit_cpu_INDEX_1_EDX_30 (1u << 30) -#define bit_cpu_PBE (1u << 31) - -/* COMMON_CPUID_INDEX_7. */ - -/* EBX. */ -#define bit_cpu_FSGSBASE (1u << 0) -#define bit_cpu_TSC_ADJUST (1u << 1) -#define bit_cpu_SGX (1u << 2) -#define bit_cpu_BMI1 (1u << 3) -#define bit_cpu_HLE (1u << 4) -#define bit_cpu_AVX2 (1u << 5) -#define bit_cpu_INDEX_7_EBX_6 (1u << 6) -#define bit_cpu_SMEP (1u << 7) -#define bit_cpu_BMI2 (1u << 8) -#define bit_cpu_ERMS (1u << 9) -#define bit_cpu_INVPCID (1u << 10) -#define bit_cpu_RTM (1u << 11) -#define bit_cpu_RDT_M (1u << 12) -#define bit_cpu_DEPR_FPU_CS_DS (1u << 13) -#define bit_cpu_MPX (1u << 14) -#define bit_cpu_RDT_A (1u << 15) -#define bit_cpu_AVX512F (1u << 16) -#define bit_cpu_AVX512DQ (1u << 17) -#define bit_cpu_RDSEED (1u << 18) -#define bit_cpu_ADX (1u << 19) -#define bit_cpu_SMAP (1u << 20) -#define bit_cpu_AVX512_IFMA (1u << 21) -#define bit_cpu_INDEX_7_EBX_22 (1u << 22) -#define bit_cpu_CLFLUSHOPT (1u << 23) -#define bit_cpu_CLWB (1u << 24) -#define bit_cpu_TRACE (1u << 25) -#define bit_cpu_AVX512PF (1u << 26) -#define bit_cpu_AVX512ER (1u << 27) -#define bit_cpu_AVX512CD (1u << 28) -#define bit_cpu_SHA (1u << 29) -#define bit_cpu_AVX512BW (1u << 30) -#define bit_cpu_AVX512VL (1u << 31) - -/* ECX. */ -#define bit_cpu_PREFETCHWT1 (1u << 0) -#define bit_cpu_AVX512_VBMI (1u << 1) -#define bit_cpu_UMIP (1u << 2) -#define bit_cpu_PKU (1u << 3) -#define bit_cpu_OSPKE (1u << 4) -#define bit_cpu_WAITPKG (1u << 5) -#define bit_cpu_AVX512_VBMI2 (1u << 6) -#define bit_cpu_SHSTK (1u << 7) -#define bit_cpu_GFNI (1u << 8) -#define bit_cpu_VAES (1u << 9) -#define bit_cpu_VPCLMULQDQ (1u << 10) -#define bit_cpu_AVX512_VNNI (1u << 11) -#define bit_cpu_AVX512_BITALG (1u << 12) -#define bit_cpu_INDEX_7_ECX_13 (1u << 13) -#define bit_cpu_AVX512_VPOPCNTDQ (1u << 14) -#define bit_cpu_INDEX_7_ECX_15 (1u << 15) -#define bit_cpu_INDEX_7_ECX_16 (1u << 16) -/* Note: Bits 17-21: The value of MAWAU used by the BNDLDX and BNDSTX - instructions in 64-bit mode. */ -#define bit_cpu_RDPID (1u << 22) -#define bit_cpu_KL (1u << 23) -#define bit_cpu_INDEX_7_ECX_24 (1u << 24) -#define bit_cpu_CLDEMOTE (1u << 25) -#define bit_cpu_INDEX_7_ECX_26 (1u << 26) -#define bit_cpu_MOVDIRI (1u << 27) -#define bit_cpu_MOVDIR64B (1u << 28) -#define bit_cpu_ENQCMD (1u << 29) -#define bit_cpu_SGX_LC (1u << 30) -#define bit_cpu_PKS (1u << 31) - -/* EDX. */ -#define bit_cpu_INDEX_7_EDX_0 (1u << 0) -#define bit_cpu_INDEX_7_EDX_1 (1u << 1) -#define bit_cpu_AVX512_4VNNIW (1u << 2) -#define bit_cpu_AVX512_4FMAPS (1u << 3) -#define bit_cpu_FSRM (1u << 4) -#define bit_cpu_UINTR (1u << 5) -#define bit_cpu_INDEX_7_EDX_6 (1u << 6) -#define bit_cpu_INDEX_7_EDX_7 (1u << 7) -#define bit_cpu_AVX512_VP2INTERSECT (1u << 8) -#define bit_cpu_INDEX_7_EDX_9 (1u << 9) -#define bit_cpu_MD_CLEAR (1u << 10) -#define bit_cpu_INDEX_7_EDX_11 (1u << 11) -#define bit_cpu_INDEX_7_EDX_12 (1u << 12) -#define bit_cpu_INDEX_7_EDX_13 (1u << 13) -#define bit_cpu_SERIALIZE (1u << 14) -#define bit_cpu_HYBRID (1u << 15) -#define bit_cpu_TSXLDTRK (1u << 16) -#define bit_cpu_INDEX_7_EDX_17 (1u << 17) -#define bit_cpu_PCONFIG (1u << 18) -#define bit_cpu_INDEX_7_EDX_19 (1u << 19) -#define bit_cpu_IBT (1u << 20) -#define bit_cpu_INDEX_7_EDX_21 (1u << 21) -#define bit_cpu_AMX_BF16 (1u << 22) -#define bit_cpu_AVX512_FP16 (1u << 23) -#define bit_cpu_AMX_TILE (1u << 24) -#define bit_cpu_AMX_INT8 (1u << 25) -#define bit_cpu_IBRS_IBPB (1u << 26) -#define bit_cpu_STIBP (1u << 27) -#define bit_cpu_L1D_FLUSH (1u << 28) -#define bit_cpu_ARCH_CAPABILITIES (1u << 29) -#define bit_cpu_CORE_CAPABILITIES (1u << 30) -#define bit_cpu_SSBD (1u << 31) - -/* COMMON_CPUID_INDEX_80000001. */ - -/* ECX. */ -#define bit_cpu_LAHF64_SAHF64 (1u << 0) -#define bit_cpu_SVM (1u << 2) -#define bit_cpu_LZCNT (1u << 5) -#define bit_cpu_SSE4A (1u << 6) -#define bit_cpu_PREFETCHW (1u << 8) -#define bit_cpu_XOP (1u << 11) -#define bit_cpu_LWP (1u << 15) -#define bit_cpu_FMA4 (1u << 16) -#define bit_cpu_TBM (1u << 21) - -/* EDX. */ -#define bit_cpu_SYSCALL_SYSRET (1u << 11) -#define bit_cpu_NX (1u << 20) -#define bit_cpu_PAGE1GB (1u << 26) -#define bit_cpu_RDTSCP (1u << 27) -#define bit_cpu_LM (1u << 29) - -/* COMMON_CPUID_INDEX_D_ECX_1. */ - -/* EAX. */ -#define bit_cpu_XSAVEOPT (1u << 0) -#define bit_cpu_XSAVEC (1u << 1) -#define bit_cpu_XGETBV_ECX_1 (1u << 2) -#define bit_cpu_XSAVES (1u << 3) -#define bit_cpu_XFD (1u << 4) - -/* COMMON_CPUID_INDEX_80000007. */ - -/* EDX. */ -#define bit_cpu_INVARIANT_TSC (1u << 8) - -/* COMMON_CPUID_INDEX_80000008. */ - -/* EBX. */ -#define bit_cpu_WBNOINVD (1u << 9) - -/* COMMON_CPUID_INDEX_7_ECX_1. */ - -/* EAX. */ -#define bit_cpu_AVX_VNNI (1u << 4) -#define bit_cpu_AVX512_BF16 (1u << 5) -#define bit_cpu_FZLRM (1u << 10) -#define bit_cpu_FSRS (1u << 11) -#define bit_cpu_FSRCS (1u << 12) -#define bit_cpu_HRESET (1u << 22) -#define bit_cpu_LAM (1u << 26) - -/* COMMON_CPUID_INDEX_19. */ - -/* EBX. */ -#define bit_cpu_AESKLE (1u << 0) -#define bit_cpu_WIDE_KL (1u << 2) - -/* COMMON_CPUID_INDEX_1. */ - -/* ECX. */ -#define index_cpu_SSE3 COMMON_CPUID_INDEX_1 -#define index_cpu_PCLMULQDQ COMMON_CPUID_INDEX_1 -#define index_cpu_DTES64 COMMON_CPUID_INDEX_1 -#define index_cpu_MONITOR COMMON_CPUID_INDEX_1 -#define index_cpu_DS_CPL COMMON_CPUID_INDEX_1 -#define index_cpu_VMX COMMON_CPUID_INDEX_1 -#define index_cpu_SMX COMMON_CPUID_INDEX_1 -#define index_cpu_EIST COMMON_CPUID_INDEX_1 -#define index_cpu_TM2 COMMON_CPUID_INDEX_1 -#define index_cpu_SSSE3 COMMON_CPUID_INDEX_1 -#define index_cpu_CNXT_ID COMMON_CPUID_INDEX_1 -#define index_cpu_SDBG COMMON_CPUID_INDEX_1 -#define index_cpu_FMA COMMON_CPUID_INDEX_1 -#define index_cpu_CMPXCHG16B COMMON_CPUID_INDEX_1 -#define index_cpu_XTPRUPDCTRL COMMON_CPUID_INDEX_1 -#define index_cpu_PDCM COMMON_CPUID_INDEX_1 -#define index_cpu_INDEX_1_ECX_16 COMMON_CPUID_INDEX_1 -#define index_cpu_PCID COMMON_CPUID_INDEX_1 -#define index_cpu_DCA COMMON_CPUID_INDEX_1 -#define index_cpu_SSE4_1 COMMON_CPUID_INDEX_1 -#define index_cpu_SSE4_2 COMMON_CPUID_INDEX_1 -#define index_cpu_X2APIC COMMON_CPUID_INDEX_1 -#define index_cpu_MOVBE COMMON_CPUID_INDEX_1 -#define index_cpu_POPCNT COMMON_CPUID_INDEX_1 -#define index_cpu_TSC_DEADLINE COMMON_CPUID_INDEX_1 -#define index_cpu_AES COMMON_CPUID_INDEX_1 -#define index_cpu_XSAVE COMMON_CPUID_INDEX_1 -#define index_cpu_OSXSAVE COMMON_CPUID_INDEX_1 -#define index_cpu_AVX COMMON_CPUID_INDEX_1 -#define index_cpu_F16C COMMON_CPUID_INDEX_1 -#define index_cpu_RDRAND COMMON_CPUID_INDEX_1 -#define index_cpu_INDEX_1_ECX_31 COMMON_CPUID_INDEX_1 - -/* ECX. */ -#define index_cpu_FPU COMMON_CPUID_INDEX_1 -#define index_cpu_VME COMMON_CPUID_INDEX_1 -#define index_cpu_DE COMMON_CPUID_INDEX_1 -#define index_cpu_PSE COMMON_CPUID_INDEX_1 -#define index_cpu_TSC COMMON_CPUID_INDEX_1 -#define index_cpu_MSR COMMON_CPUID_INDEX_1 -#define index_cpu_PAE COMMON_CPUID_INDEX_1 -#define index_cpu_MCE COMMON_CPUID_INDEX_1 -#define index_cpu_CX8 COMMON_CPUID_INDEX_1 -#define index_cpu_APIC COMMON_CPUID_INDEX_1 -#define index_cpu_INDEX_1_EDX_10 COMMON_CPUID_INDEX_1 -#define index_cpu_SEP COMMON_CPUID_INDEX_1 -#define index_cpu_MTRR COMMON_CPUID_INDEX_1 -#define index_cpu_PGE COMMON_CPUID_INDEX_1 -#define index_cpu_MCA COMMON_CPUID_INDEX_1 -#define index_cpu_CMOV COMMON_CPUID_INDEX_1 -#define index_cpu_PAT COMMON_CPUID_INDEX_1 -#define index_cpu_PSE_36 COMMON_CPUID_INDEX_1 -#define index_cpu_PSN COMMON_CPUID_INDEX_1 -#define index_cpu_CLFSH COMMON_CPUID_INDEX_1 -#define index_cpu_INDEX_1_EDX_20 COMMON_CPUID_INDEX_1 -#define index_cpu_DS COMMON_CPUID_INDEX_1 -#define index_cpu_ACPI COMMON_CPUID_INDEX_1 -#define index_cpu_MMX COMMON_CPUID_INDEX_1 -#define index_cpu_FXSR COMMON_CPUID_INDEX_1 -#define index_cpu_SSE COMMON_CPUID_INDEX_1 -#define index_cpu_SSE2 COMMON_CPUID_INDEX_1 -#define index_cpu_SS COMMON_CPUID_INDEX_1 -#define index_cpu_HTT COMMON_CPUID_INDEX_1 -#define index_cpu_TM COMMON_CPUID_INDEX_1 -#define index_cpu_INDEX_1_EDX_30 COMMON_CPUID_INDEX_1 -#define index_cpu_PBE COMMON_CPUID_INDEX_1 - -/* COMMON_CPUID_INDEX_7. */ - -/* EBX. */ -#define index_cpu_FSGSBASE COMMON_CPUID_INDEX_7 -#define index_cpu_TSC_ADJUST COMMON_CPUID_INDEX_7 -#define index_cpu_SGX COMMON_CPUID_INDEX_7 -#define index_cpu_BMI1 COMMON_CPUID_INDEX_7 -#define index_cpu_HLE COMMON_CPUID_INDEX_7 -#define index_cpu_AVX2 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EBX_6 COMMON_CPUID_INDEX_7 -#define index_cpu_SMEP COMMON_CPUID_INDEX_7 -#define index_cpu_BMI2 COMMON_CPUID_INDEX_7 -#define index_cpu_ERMS COMMON_CPUID_INDEX_7 -#define index_cpu_INVPCID COMMON_CPUID_INDEX_7 -#define index_cpu_RTM COMMON_CPUID_INDEX_7 -#define index_cpu_RDT_M COMMON_CPUID_INDEX_7 -#define index_cpu_DEPR_FPU_CS_DS COMMON_CPUID_INDEX_7 -#define index_cpu_MPX COMMON_CPUID_INDEX_7 -#define index_cpu_RDT_A COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512F COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512DQ COMMON_CPUID_INDEX_7 -#define index_cpu_RDSEED COMMON_CPUID_INDEX_7 -#define index_cpu_ADX COMMON_CPUID_INDEX_7 -#define index_cpu_SMAP COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_IFMA COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EBX_22 COMMON_CPUID_INDEX_7 -#define index_cpu_CLFLUSHOPT COMMON_CPUID_INDEX_7 -#define index_cpu_CLWB COMMON_CPUID_INDEX_7 -#define index_cpu_TRACE COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512PF COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512ER COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512CD COMMON_CPUID_INDEX_7 -#define index_cpu_SHA COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512BW COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512VL COMMON_CPUID_INDEX_7 - -/* ECX. */ -#define index_cpu_PREFETCHWT1 COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_VBMI COMMON_CPUID_INDEX_7 -#define index_cpu_UMIP COMMON_CPUID_INDEX_7 -#define index_cpu_PKU COMMON_CPUID_INDEX_7 -#define index_cpu_OSPKE COMMON_CPUID_INDEX_7 -#define index_cpu_WAITPKG COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_VBMI2 COMMON_CPUID_INDEX_7 -#define index_cpu_SHSTK COMMON_CPUID_INDEX_7 -#define index_cpu_GFNI COMMON_CPUID_INDEX_7 -#define index_cpu_VAES COMMON_CPUID_INDEX_7 -#define index_cpu_VPCLMULQDQ COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_VNNI COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_BITALG COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_ECX_13 COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_VPOPCNTDQ COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_ECX_15 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_ECX_16 COMMON_CPUID_INDEX_7 -#define index_cpu_RDPID COMMON_CPUID_INDEX_7 -#define index_cpu_KL COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_ECX_24 COMMON_CPUID_INDEX_7 -#define index_cpu_CLDEMOTE COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_ECX_26 COMMON_CPUID_INDEX_7 -#define index_cpu_MOVDIRI COMMON_CPUID_INDEX_7 -#define index_cpu_MOVDIR64B COMMON_CPUID_INDEX_7 -#define index_cpu_ENQCMD COMMON_CPUID_INDEX_7 -#define index_cpu_SGX_LC COMMON_CPUID_INDEX_7 -#define index_cpu_PKS COMMON_CPUID_INDEX_7 - -/* EDX. */ -#define index_cpu_INDEX_7_EDX_0 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_1 COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_4VNNIW COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_4FMAPS COMMON_CPUID_INDEX_7 -#define index_cpu_FSRM COMMON_CPUID_INDEX_7 -#define index_cpu_UINTR COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_6 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_7 COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_VP2INTERSECT COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_9 COMMON_CPUID_INDEX_7 -#define index_cpu_MD_CLEAR COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_11 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_12 COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_13 COMMON_CPUID_INDEX_7 -#define index_cpu_SERIALIZE COMMON_CPUID_INDEX_7 -#define index_cpu_HYBRID COMMON_CPUID_INDEX_7 -#define index_cpu_TSXLDTRK COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_17 COMMON_CPUID_INDEX_7 -#define index_cpu_PCONFIG COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_19 COMMON_CPUID_INDEX_7 -#define index_cpu_IBT COMMON_CPUID_INDEX_7 -#define index_cpu_INDEX_7_EDX_21 COMMON_CPUID_INDEX_7 -#define index_cpu_AMX_BF16 COMMON_CPUID_INDEX_7 -#define index_cpu_AVX512_FP16 COMMON_CPUID_INDEX_7 -#define index_cpu_AMX_TILE COMMON_CPUID_INDEX_7 -#define index_cpu_AMX_INT8 COMMON_CPUID_INDEX_7 -#define index_cpu_IBRS_IBPB COMMON_CPUID_INDEX_7 -#define index_cpu_STIBP COMMON_CPUID_INDEX_7 -#define index_cpu_L1D_FLUSH COMMON_CPUID_INDEX_7 -#define index_cpu_ARCH_CAPABILITIES COMMON_CPUID_INDEX_7 -#define index_cpu_CORE_CAPABILITIES COMMON_CPUID_INDEX_7 -#define index_cpu_SSBD COMMON_CPUID_INDEX_7 - -/* COMMON_CPUID_INDEX_80000001. */ - -/* ECX. */ -#define index_cpu_LAHF64_SAHF64 COMMON_CPUID_INDEX_80000001 -#define index_cpu_SVM COMMON_CPUID_INDEX_80000001 -#define index_cpu_LZCNT COMMON_CPUID_INDEX_80000001 -#define index_cpu_SSE4A COMMON_CPUID_INDEX_80000001 -#define index_cpu_PREFETCHW COMMON_CPUID_INDEX_80000001 -#define index_cpu_XOP COMMON_CPUID_INDEX_80000001 -#define index_cpu_LWP COMMON_CPUID_INDEX_80000001 -#define index_cpu_FMA4 COMMON_CPUID_INDEX_80000001 -#define index_cpu_TBM COMMON_CPUID_INDEX_80000001 - -/* EDX. */ -#define index_cpu_SYSCALL_SYSRET COMMON_CPUID_INDEX_80000001 -#define index_cpu_NX COMMON_CPUID_INDEX_80000001 -#define index_cpu_PAGE1GB COMMON_CPUID_INDEX_80000001 -#define index_cpu_RDTSCP COMMON_CPUID_INDEX_80000001 -#define index_cpu_LM COMMON_CPUID_INDEX_80000001 - -/* COMMON_CPUID_INDEX_D_ECX_1. */ - -/* EAX. */ -#define index_cpu_XSAVEOPT COMMON_CPUID_INDEX_D_ECX_1 -#define index_cpu_XSAVEC COMMON_CPUID_INDEX_D_ECX_1 -#define index_cpu_XGETBV_ECX_1 COMMON_CPUID_INDEX_D_ECX_1 -#define index_cpu_XSAVES COMMON_CPUID_INDEX_D_ECX_1 -#define index_cpu_XFD COMMON_CPUID_INDEX_D_ECX_1 - -/* COMMON_CPUID_INDEX_80000007. */ - -/* EDX. */ -#define index_cpu_INVARIANT_TSC COMMON_CPUID_INDEX_80000007 - -/* COMMON_CPUID_INDEX_80000008. */ - -/* EBX. */ -#define index_cpu_WBNOINVD COMMON_CPUID_INDEX_80000008 - -/* COMMON_CPUID_INDEX_7_ECX_1. */ - -/* EAX. */ -#define index_cpu_AVX_VNNI COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_AVX512_BF16 COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_FZLRM COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_FSRS COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_FSRCS COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_HRESET COMMON_CPUID_INDEX_7_ECX_1 -#define index_cpu_LAM COMMON_CPUID_INDEX_7_ECX_1 - -/* COMMON_CPUID_INDEX_19. */ - -/* EBX. */ -#define index_cpu_AESKLE COMMON_CPUID_INDEX_19 -#define index_cpu_WIDE_KL COMMON_CPUID_INDEX_19 - -/* COMMON_CPUID_INDEX_1. */ - -/* ECX. */ -#define reg_SSE3 ecx -#define reg_PCLMULQDQ ecx -#define reg_DTES64 ecx -#define reg_MONITOR ecx -#define reg_DS_CPL ecx -#define reg_VMX ecx -#define reg_SMX ecx -#define reg_EIST ecx -#define reg_TM2 ecx -#define reg_SSSE3 ecx -#define reg_CNXT_ID ecx -#define reg_SDBG ecx -#define reg_FMA ecx -#define reg_CMPXCHG16B ecx -#define reg_XTPRUPDCTRL ecx -#define reg_PDCM ecx -#define reg_INDEX_1_ECX_16 ecx -#define reg_PCID ecx -#define reg_DCA ecx -#define reg_SSE4_1 ecx -#define reg_SSE4_2 ecx -#define reg_X2APIC ecx -#define reg_MOVBE ecx -#define reg_POPCNT ecx -#define reg_TSC_DEADLINE ecx -#define reg_AES ecx -#define reg_XSAVE ecx -#define reg_OSXSAVE ecx -#define reg_AVX ecx -#define reg_F16C ecx -#define reg_RDRAND ecx -#define reg_INDEX_1_ECX_31 ecx - -/* EDX. */ -#define reg_FPU edx -#define reg_VME edx -#define reg_DE edx -#define reg_PSE edx -#define reg_TSC edx -#define reg_MSR edx -#define reg_PAE edx -#define reg_MCE edx -#define reg_CX8 edx -#define reg_APIC edx -#define reg_INDEX_1_EDX_10 edx -#define reg_SEP edx -#define reg_MTRR edx -#define reg_PGE edx -#define reg_MCA edx -#define reg_CMOV edx -#define reg_PAT edx -#define reg_PSE_36 edx -#define reg_PSN edx -#define reg_CLFSH edx -#define reg_INDEX_1_EDX_20 edx -#define reg_DS edx -#define reg_ACPI edx -#define reg_MMX edx -#define reg_FXSR edx -#define reg_SSE edx -#define reg_SSE2 edx -#define reg_SS edx -#define reg_HTT edx -#define reg_TM edx -#define reg_INDEX_1_EDX_30 edx -#define reg_PBE edx - -/* COMMON_CPUID_INDEX_7. */ - -/* EBX. */ -#define reg_FSGSBASE ebx -#define reg_TSC_ADJUST ebx -#define reg_SGX ebx -#define reg_BMI1 ebx -#define reg_HLE ebx -#define reg_BMI2 ebx -#define reg_AVX2 ebx -#define reg_INDEX_7_EBX_6 ebx -#define reg_SMEP ebx -#define reg_ERMS ebx -#define reg_INVPCID ebx -#define reg_RTM ebx -#define reg_RDT_M ebx -#define reg_DEPR_FPU_CS_DS ebx -#define reg_MPX ebx -#define reg_RDT_A ebx -#define reg_AVX512F ebx -#define reg_AVX512DQ ebx -#define reg_RDSEED ebx -#define reg_ADX ebx -#define reg_SMAP ebx -#define reg_AVX512_IFMA ebx -#define reg_INDEX_7_EBX_22 ebx -#define reg_CLFLUSHOPT ebx -#define reg_CLWB ebx -#define reg_TRACE ebx -#define reg_AVX512PF ebx -#define reg_AVX512ER ebx -#define reg_AVX512CD ebx -#define reg_SHA ebx -#define reg_AVX512BW ebx -#define reg_AVX512VL ebx - -/* ECX. */ -#define reg_PREFETCHWT1 ecx -#define reg_AVX512_VBMI ecx -#define reg_UMIP ecx -#define reg_PKU ecx -#define reg_OSPKE ecx -#define reg_WAITPKG ecx -#define reg_AVX512_VBMI2 ecx -#define reg_SHSTK ecx -#define reg_GFNI ecx -#define reg_VAES ecx -#define reg_VPCLMULQDQ ecx -#define reg_AVX512_VNNI ecx -#define reg_AVX512_BITALG ecx -#define reg_INDEX_7_ECX_13 ecx -#define reg_AVX512_VPOPCNTDQ ecx -#define reg_INDEX_7_ECX_15 ecx -#define reg_INDEX_7_ECX_16 ecx -#define reg_RDPID ecx -#define reg_KL ecx -#define reg_INDEX_7_ECX_24 ecx -#define reg_CLDEMOTE ecx -#define reg_INDEX_7_ECX_26 ecx -#define reg_MOVDIRI ecx -#define reg_MOVDIR64B ecx -#define reg_ENQCMD ecx -#define reg_SGX_LC ecx -#define reg_PKS ecx - -/* EDX. */ -#define reg_INDEX_7_EDX_0 edx -#define reg_INDEX_7_EDX_1 edx -#define reg_AVX512_4VNNIW edx -#define reg_AVX512_4FMAPS edx -#define reg_FSRM edx -#define reg_UINTR edx -#define reg_INDEX_7_EDX_6 edx -#define reg_INDEX_7_EDX_7 edx -#define reg_AVX512_VP2INTERSECT edx -#define reg_INDEX_7_EDX_9 edx -#define reg_MD_CLEAR edx -#define reg_INDEX_7_EDX_11 edx -#define reg_INDEX_7_EDX_12 edx -#define reg_INDEX_7_EDX_13 edx -#define reg_SERIALIZE edx -#define reg_HYBRID edx -#define reg_TSXLDTRK edx -#define reg_INDEX_7_EDX_17 edx -#define reg_PCONFIG edx -#define reg_INDEX_7_EDX_19 edx -#define reg_IBT edx -#define reg_INDEX_7_EDX_21 edx -#define reg_AMX_BF16 edx -#define reg_AVX512_FP16 edx -#define reg_AMX_TILE edx -#define reg_AMX_INT8 edx -#define reg_IBRS_IBPB edx -#define reg_STIBP edx -#define reg_L1D_FLUSH edx -#define reg_ARCH_CAPABILITIES edx -#define reg_CORE_CAPABILITIES edx -#define reg_SSBD edx - -/* COMMON_CPUID_INDEX_80000001. */ - -/* ECX. */ -#define reg_LAHF64_SAHF64 ecx -#define reg_SVM ecx -#define reg_LZCNT ecx -#define reg_SSE4A ecx -#define reg_PREFETCHW ecx -#define reg_XOP ecx -#define reg_LWP ecx -#define reg_FMA4 ecx -#define reg_TBM ecx - -/* EDX. */ -#define reg_SYSCALL_SYSRET edx -#define reg_NX edx -#define reg_PAGE1GB edx -#define reg_RDTSCP edx -#define reg_LM edx - -/* COMMON_CPUID_INDEX_D_ECX_1. */ - -/* EAX. */ -#define reg_XSAVEOPT eax -#define reg_XSAVEC eax -#define reg_XGETBV_ECX_1 eax -#define reg_XSAVES eax -#define reg_XFD eax - -/* COMMON_CPUID_INDEX_80000007. */ - -/* EDX. */ -#define reg_INVARIANT_TSC edx - -/* COMMON_CPUID_INDEX_80000008. */ - -/* EBX. */ -#define reg_WBNOINVD ebx - -/* COMMON_CPUID_INDEX_7_ECX_1. */ - -/* EAX. */ -#define reg_AVX_VNNI eax -#define reg_AVX512_BF16 eax -#define reg_FZLRM eax -#define reg_FSRS eax -#define reg_FSRCS eax -#define reg_HRESET eax -#define reg_LAM eax + return __x86_cpu_array_index (__index | 32u) & (1u << (__index % 32)); +} -/* COMMON_CPUID_INDEX_19. */ +#endif /* __extern_always_inline */ -/* EBX. */ -#define reg_AESKLE ebx -#define reg_WIDE_KL ebx +__END_DECLS #endif /* _SYS_PLATFORM_X86_H */ diff --git a/sysdeps/x86/tst-cpu-features-cpuinfo.c b/sysdeps/x86/tst-cpu-features-cpuinfo.c index 96277284d1..92a2a169fd 100644 --- a/sysdeps/x86/tst-cpu-features-cpuinfo.c +++ b/sysdeps/x86/tst-cpu-features-cpuinfo.c @@ -74,8 +74,9 @@ check_proc (const char *proc_name, int flag, int usable, const char *name) return (found != flag); } -#define CHECK_PROC(str, name) \ - check_proc (#str, HAS_CPU_FEATURE (name), CPU_FEATURE_USABLE (name), \ +#define CHECK_PROC(str, name) \ + check_proc (#str, x86_cpu_has_feature (x86_cpu_##name), \ + x86_cpu_is_usable (x86_cpu_##name), \ "HAS_CPU_FEATURE (" #name ")"); static int diff --git a/sysdeps/x86/tst-cpu-features-supports.c b/sysdeps/x86/tst-cpu-features-supports.c index 287cf01fbd..0844b1f400 100644 --- a/sysdeps/x86/tst-cpu-features-supports.c +++ b/sysdeps/x86/tst-cpu-features-supports.c @@ -37,15 +37,15 @@ check_supports (int supports, int usable, const char *supports_name, return 0; } -#define CHECK_SUPPORTS(str, name) \ - check_supports (__builtin_cpu_supports (#str), \ - CPU_FEATURE_USABLE (name), \ - #str, "CPU_FEATURE_USABLE (" #name ")"); +#define CHECK_SUPPORTS(str, name) \ + check_supports (__builtin_cpu_supports (#str), \ + x86_cpu_is_usable (x86_cpu_##name), \ + #str, "x86_cpu_is_usable (x86_cpu_" #name ")"); -#define CHECK_CPU_SUPPORTS(str, name) \ - check_supports (__builtin_cpu_supports (#str), \ - HAS_CPU_FEATURE (name), \ - #str, "HAS_CPU_FEATURE (" #name ")"); +#define CHECK_CPU_SUPPORTS(str, name) \ + check_supports (__builtin_cpu_supports (#str), \ + x86_cpu_has_feature (x86_cpu_##name), \ + #str, "x86_cpu_has_feature (" #name ")"); static int do_test (int argc, char **argv) diff --git a/sysdeps/x86/tst-get-cpu-features.c b/sysdeps/x86/tst-get-cpu-features.c index 6f1e925a6a..e17939e77c 100644 --- a/sysdeps/x86/tst-get-cpu-features.c +++ b/sysdeps/x86/tst-get-cpu-features.c @@ -21,50 +21,23 @@ #include #include -#define CHECK_CPU_FEATURE(name) \ - { \ - if (HAS_CPU_FEATURE (name)) \ - printf (" " #name "\n"); \ +#define CHECK_CPU_FEATURE(name) \ + { \ + if (x86_cpu_has_feature (x86_cpu_##name)) \ + printf (" " #name "\n"); \ } -#define CHECK_CPU_FEATURE_USABLE(name) \ - { \ - if (CPU_FEATURE_USABLE(name)) \ - printf (" " #name "\n"); \ +#define CHECK_CPU_FEATURE_USABLE(name) \ + { \ + if (x86_cpu_is_usable (x86_cpu_##name)) \ + printf (" " #name "\n"); \ } -static const char * const cpu_kinds[] = -{ - "Unknown", - "Intel", - "AMD", - "ZHAOXIN", - "Other", -}; - static int do_test (void) { - const struct cpu_features *cpu_features = __x86_get_cpu_features (0); - - switch (cpu_features->basic.kind) - { - case arch_kind_intel: - case arch_kind_amd: - case arch_kind_zhaoxin: - case arch_kind_other: - printf ("Vendor: %s\n", cpu_kinds[cpu_features->basic.kind]); - printf ("Family: 0x%x\n", cpu_features->basic.family); - printf ("Model: 0x%x\n", cpu_features->basic.model); - printf ("Stepping: 0x%x\n", cpu_features->basic.stepping); - break; - - default: - abort (); - } - #ifdef __SSE2__ - TEST_VERIFY_EXIT (HAS_CPU_FEATURE (SSE2)); + TEST_VERIFY_EXIT (x86_cpu_has_feature (x86_cpu_SSE2)); #endif printf ("CPU features:\n"); diff --git a/sysdeps/x86/x86_cpu_array.c b/sysdeps/x86/x86_cpu_array.c new file mode 100644 index 0000000000..1561d43686 --- /dev/null +++ b/sysdeps/x86/x86_cpu_array.c @@ -0,0 +1,31 @@ +/* x86 CPU feature data. Copy in libc.so.6. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +struct x86_cpu_array __x86_cpu_array + attribute_relro __attribute__ ((nocommon)) = + { + /* In the dynamically linked case, libc.so.6 is initialized prior + to relocation via _dl_call_libc_vars_init_cpu. */ +#ifndef SHARED + .__x86_word = __x86_cpu_array_private, + .__x86_count = x86_cpu_array_size, +#endif + }; +libc_hidden_def (__x86_cpu_array) diff --git a/sysdeps/x86/x86_cpu_array_private.c b/sysdeps/x86/x86_cpu_array_private.c new file mode 100644 index 0000000000..550672b0ec --- /dev/null +++ b/sysdeps/x86/x86_cpu_array_private.c @@ -0,0 +1,21 @@ +/* x86 CPU feature data. Version for libc.a or ld.so. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +unsigned int __x86_cpu_array_private[x86_cpu_array_size]; diff --git a/sysdeps/x86_64/dl-hwcaps-subdirs.c b/sysdeps/x86_64/dl-hwcaps-subdirs.c index 8810a822ef..a16fc056f6 100644 --- a/sysdeps/x86_64/dl-hwcaps-subdirs.c +++ b/sysdeps/x86_64/dl-hwcaps-subdirs.c @@ -30,35 +30,35 @@ _dl_hwcaps_subdirs_active (void) /* Test in reverse preference order. */ /* x86-64-v2. */ - if (!(CPU_FEATURE_USABLE (CMPXCHG16B) - && CPU_FEATURE_USABLE (LAHF64_SAHF64) - && CPU_FEATURE_USABLE (POPCNT) - && CPU_FEATURE_USABLE (SSE3) - && CPU_FEATURE_USABLE (SSE4_1) - && CPU_FEATURE_USABLE (SSE4_2) - && CPU_FEATURE_USABLE (SSSE3))) + if (!(x86_cpu_is_usable (x86_cpu_CMPXCHG16B) + && x86_cpu_is_usable (x86_cpu_LAHF64_SAHF64) + && x86_cpu_is_usable (x86_cpu_POPCNT) + && x86_cpu_is_usable (x86_cpu_SSE3) + && x86_cpu_is_usable (x86_cpu_SSE4_1) + && x86_cpu_is_usable (x86_cpu_SSE4_2) + && x86_cpu_is_usable (x86_cpu_SSSE3))) return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active); ++active; /* x86-64-v3. */ - if (!(CPU_FEATURE_USABLE (AVX) - && CPU_FEATURE_USABLE (AVX2) - && CPU_FEATURE_USABLE (BMI1) - && CPU_FEATURE_USABLE (BMI2) - && CPU_FEATURE_USABLE (F16C) - && CPU_FEATURE_USABLE (FMA) - && CPU_FEATURE_USABLE (LZCNT) - && CPU_FEATURE_USABLE (MOVBE) - && CPU_FEATURE_USABLE (OSXSAVE))) + if (!(x86_cpu_is_usable (x86_cpu_AVX) + && x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_BMI1) + && x86_cpu_is_usable (x86_cpu_BMI2) + && x86_cpu_is_usable (x86_cpu_F16C) + && x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_LZCNT) + && x86_cpu_is_usable (x86_cpu_MOVBE) + && x86_cpu_is_usable (x86_cpu_OSXSAVE))) return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active); ++active; /* x86-64-v4. */ - if (!(CPU_FEATURE_USABLE (AVX512F) - && CPU_FEATURE_USABLE (AVX512BW) - && CPU_FEATURE_USABLE (AVX512CD) - && CPU_FEATURE_USABLE (AVX512DQ) - && CPU_FEATURE_USABLE (AVX512VL))) + if (!(x86_cpu_is_usable (x86_cpu_AVX512F) + && x86_cpu_is_usable (x86_cpu_AVX512BW) + && x86_cpu_is_usable (x86_cpu_AVX512CD) + && x86_cpu_is_usable (x86_cpu_AVX512DQ) + && x86_cpu_is_usable (x86_cpu_AVX512VL))) return _dl_hwcaps_subdirs_build_bitmask (subdirs_count, active); ++active; diff --git a/sysdeps/x86_64/dl-machine.h b/sysdeps/x86_64/dl-machine.h index bb93c7c6ab..ec3312edcd 100644 --- a/sysdeps/x86_64/dl-machine.h +++ b/sysdeps/x86_64/dl-machine.h @@ -98,9 +98,9 @@ elf_machine_runtime_setup (struct link_map *l, int lazy, int profile) end in this function. */ if (__glibc_unlikely (profile)) { - if (CPU_FEATURE_USABLE (AVX512F)) + if (x86_cpu_is_usable (x86_cpu_AVX512F)) *(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_profile_avx512; - else if (CPU_FEATURE_USABLE (AVX)) + else if (x86_cpu_is_usable (x86_cpu_AVX)) *(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_profile_avx; else *(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_profile_sse; @@ -118,7 +118,7 @@ elf_machine_runtime_setup (struct link_map *l, int lazy, int profile) the resolved address. */ if (GLRO(dl_x86_cpu_features).xsave_state_size != 0) *(ElfW(Addr) *) (got + 2) - = (CPU_FEATURE_USABLE (XSAVEC) + = (x86_cpu_is_usable (x86_cpu_XSAVEC) ? (ElfW(Addr)) &_dl_runtime_resolve_xsavec : (ElfW(Addr)) &_dl_runtime_resolve_xsave); else diff --git a/sysdeps/x86_64/fpu/math-tests-arch.h b/sysdeps/x86_64/fpu/math-tests-arch.h index cc3c2b0c11..724b174124 100644 --- a/sysdeps/x86_64/fpu/math-tests-arch.h +++ b/sysdeps/x86_64/fpu/math-tests-arch.h @@ -24,7 +24,7 @@ # define CHECK_ARCH_EXT \ do \ { \ - if (!CPU_FEATURE_USABLE (AVX)) return; \ + if (!x86_cpu_is_usable (x86_cpu_AVX)) return; \ } \ while (0) @@ -34,7 +34,7 @@ # define CHECK_ARCH_EXT \ do \ { \ - if (!CPU_FEATURE_USABLE (AVX2)) return; \ + if (!x86_cpu_is_usable (x86_cpu_AVX2)) return; \ } \ while (0) @@ -44,7 +44,7 @@ # define CHECK_ARCH_EXT \ do \ { \ - if (!CPU_FEATURE_USABLE (AVX512F)) return; \ + if (!x86_cpu_is_usable (x86_cpu_AVX512F)) return; \ } \ while (0) diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-avx-fma4.h b/sysdeps/x86_64/fpu/multiarch/ifunc-avx-fma4.h index 95fe2f4d70..ae344be0be 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-avx-fma4.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-avx-fma4.h @@ -27,16 +27,14 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (fma4) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2)) + if (x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_AVX2)) return OPTIMIZE (fma); - if (CPU_FEATURE_USABLE_P (cpu_features, FMA4)) + if (x86_cpu_is_usable (x86_cpu_FMA4)) return OPTIMIZE (fma4); - if (CPU_FEATURE_USABLE_P (cpu_features, AVX)) + if (x86_cpu_is_usable (x86_cpu_AVX)) return OPTIMIZE (avx); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-fma.h b/sysdeps/x86_64/fpu/multiarch/ifunc-fma.h index 0a25a44ab0..2c8efe993c 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-fma.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-fma.h @@ -24,10 +24,8 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (fma) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2)) + if (x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_AVX2)) return OPTIMIZE (fma); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-fma4.h b/sysdeps/x86_64/fpu/multiarch/ifunc-fma4.h index e5fd5ac9cb..57db900e6e 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-fma4.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-fma4.h @@ -26,13 +26,11 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (fma4) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2)) + if (x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_AVX2)) return OPTIMIZE (fma); - if (CPU_FEATURE_USABLE_P (cpu_features, FMA4)) + if (x86_cpu_is_usable (x86_cpu_FMA4)) return OPTIMIZE (fma4); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx2.h b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx2.h index 2655e55444..410f80fa13 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx2.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx2.h @@ -29,10 +29,8 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (avx2) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2)) + if (x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_AVX2)) return OPTIMIZE (avx2); return OPTIMIZE (sse_wrapper); diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx512.h b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx512.h index 5f8326503b..462ef20d7f 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx512.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-avx512.h @@ -34,10 +34,10 @@ IFUNC_SELECTOR (void) if (!CPU_FEATURES_ARCH_P (cpu_features, MathVec_Prefer_No_AVX512)) { - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512DQ)) + if (x86_cpu_is_usable (x86_cpu_AVX512DQ)) return OPTIMIZE (skx); - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512F)) + if (x86_cpu_is_usable (x86_cpu_AVX512F)) return OPTIMIZE (knl); } diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-sse4_1.h b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-sse4_1.h index 7240e554c9..700331d5a5 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-sse4_1.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-mathvec-sse4_1.h @@ -29,9 +29,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse4) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_1)) + if (x86_cpu_is_usable (x86_cpu_SSE4_1)) return OPTIMIZE (sse4); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/fpu/multiarch/ifunc-sse4_1.h b/sysdeps/x86_64/fpu/multiarch/ifunc-sse4_1.h index e5d8a6f932..65972d0923 100644 --- a/sysdeps/x86_64/fpu/multiarch/ifunc-sse4_1.h +++ b/sysdeps/x86_64/fpu/multiarch/ifunc-sse4_1.h @@ -24,9 +24,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse41) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_1)) + if (x86_cpu_is_usable (x86_cpu_SSE4_1)) return OPTIMIZE (sse41); return OPTIMIZE (c); diff --git a/sysdeps/x86_64/fpu/multiarch/s_fma.c b/sysdeps/x86_64/fpu/multiarch/s_fma.c index 0d8c0b7911..27a35fb7e0 100644 --- a/sysdeps/x86_64/fpu/multiarch/s_fma.c +++ b/sysdeps/x86_64/fpu/multiarch/s_fma.c @@ -41,8 +41,8 @@ __fma_fma4 (double x, double y, double z) } -libm_ifunc (__fma, CPU_FEATURE_USABLE (FMA) - ? __fma_fma3 : (CPU_FEATURE_USABLE (FMA4) +libm_ifunc (__fma, x86_cpu_is_usable (x86_cpu_FMA) + ? __fma_fma3 : (x86_cpu_is_usable (x86_cpu_FMA4) ? __fma_fma4 : __fma_sse2)); libm_alias_double (__fma, fma) diff --git a/sysdeps/x86_64/fpu/multiarch/s_fmaf.c b/sysdeps/x86_64/fpu/multiarch/s_fmaf.c index c01e5a21d4..fedddd1346 100644 --- a/sysdeps/x86_64/fpu/multiarch/s_fmaf.c +++ b/sysdeps/x86_64/fpu/multiarch/s_fmaf.c @@ -40,8 +40,8 @@ __fmaf_fma4 (float x, float y, float z) } -libm_ifunc (__fmaf, CPU_FEATURE_USABLE (FMA) - ? __fmaf_fma3 : (CPU_FEATURE_USABLE (FMA4) +libm_ifunc (__fmaf, x86_cpu_is_usable (x86_cpu_FMA) + ? __fmaf_fma3 : (x86_cpu_is_usable (x86_cpu_FMA4) ? __fmaf_fma4 : __fmaf_sse2)); libm_alias_float (__fma, fma) diff --git a/sysdeps/x86_64/multiarch/ifunc-avx2.h b/sysdeps/x86_64/multiarch/ifunc-avx2.h index f4e311d470..af3afefa32 100644 --- a/sysdeps/x86_64/multiarch/ifunc-avx2.h +++ b/sysdeps/x86_64/multiarch/ifunc-avx2.h @@ -28,7 +28,7 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); diff --git a/sysdeps/x86_64/multiarch/ifunc-impl-list.c b/sysdeps/x86_64/multiarch/ifunc-impl-list.c index f93ec39d98..e9d8c60da0 100644 --- a/sysdeps/x86_64/multiarch/ifunc-impl-list.c +++ b/sysdeps/x86_64/multiarch/ifunc-impl-list.c @@ -41,19 +41,19 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/memchr.c. */ IFUNC_IMPL (i, name, memchr, IFUNC_IMPL_ADD (array, i, memchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memchr_avx2) IFUNC_IMPL_ADD (array, i, memchr, 1, __memchr_sse2)) /* Support sysdeps/x86_64/multiarch/memcmp.c. */ IFUNC_IMPL (i, name, memcmp, IFUNC_IMPL_ADD (array, i, memcmp, - (CPU_FEATURE_USABLE (AVX2) - && CPU_FEATURE_USABLE (MOVBE)), + (x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_MOVBE)), __memcmp_avx2_movbe) - IFUNC_IMPL_ADD (array, i, memcmp, CPU_FEATURE_USABLE (SSE4_1), + IFUNC_IMPL_ADD (array, i, memcmp, x86_cpu_is_usable (x86_cpu_SSE4_1), __memcmp_sse4_1) - IFUNC_IMPL_ADD (array, i, memcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __memcmp_ssse3) IFUNC_IMPL_ADD (array, i, memcmp, 1, __memcmp_sse2)) @@ -61,25 +61,25 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/memmove_chk.c. */ IFUNC_IMPL (i, name, __memmove_chk, IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_chk_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_chk_avx512_unaligned) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_chk_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memmove_chk_avx_unaligned) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memmove_chk_avx_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_chk_ssse3_back) IFUNC_IMPL_ADD (array, i, __memmove_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_chk_ssse3) IFUNC_IMPL_ADD (array, i, __memmove_chk, 1, __memmove_chk_sse2_unaligned) @@ -92,23 +92,23 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/memmove.c. */ IFUNC_IMPL (i, name, memmove, IFUNC_IMPL_ADD (array, i, memmove, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memmove_avx_unaligned) IFUNC_IMPL_ADD (array, i, memmove, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memmove_avx_unaligned_erms) IFUNC_IMPL_ADD (array, i, memmove, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, memmove, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_avx512_unaligned) IFUNC_IMPL_ADD (array, i, memmove, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memmove_avx512_unaligned_erms) - IFUNC_IMPL_ADD (array, i, memmove, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memmove, x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_ssse3_back) - IFUNC_IMPL_ADD (array, i, memmove, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memmove, x86_cpu_is_usable (x86_cpu_SSSE3), __memmove_ssse3) IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_erms) IFUNC_IMPL_ADD (array, i, memmove, 1, @@ -119,7 +119,7 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/memrchr.c. */ IFUNC_IMPL (i, name, memrchr, IFUNC_IMPL_ADD (array, i, memrchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memrchr_avx2) IFUNC_IMPL_ADD (array, i, memrchr, 1, __memrchr_sse2)) @@ -133,19 +133,19 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, __memset_chk, 1, __memset_chk_sse2_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memset_chk_avx2_unaligned) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memset_chk_avx2_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_chk_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_chk_avx512_unaligned) IFUNC_IMPL_ADD (array, i, __memset_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_chk_avx512_no_vzeroupper) ) #endif @@ -158,48 +158,48 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, __memset_sse2_unaligned_erms) IFUNC_IMPL_ADD (array, i, memset, 1, __memset_erms) IFUNC_IMPL_ADD (array, i, memset, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memset_avx2_unaligned) IFUNC_IMPL_ADD (array, i, memset, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __memset_avx2_unaligned_erms) IFUNC_IMPL_ADD (array, i, memset, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, memset, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_avx512_unaligned) IFUNC_IMPL_ADD (array, i, memset, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memset_avx512_no_vzeroupper) ) /* Support sysdeps/x86_64/multiarch/rawmemchr.c. */ IFUNC_IMPL (i, name, rawmemchr, IFUNC_IMPL_ADD (array, i, rawmemchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __rawmemchr_avx2) IFUNC_IMPL_ADD (array, i, rawmemchr, 1, __rawmemchr_sse2)) /* Support sysdeps/x86_64/multiarch/strlen.c. */ IFUNC_IMPL (i, name, strlen, IFUNC_IMPL_ADD (array, i, strlen, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strlen_avx2) IFUNC_IMPL_ADD (array, i, strlen, 1, __strlen_sse2)) /* Support sysdeps/x86_64/multiarch/strnlen.c. */ IFUNC_IMPL (i, name, strnlen, IFUNC_IMPL_ADD (array, i, strnlen, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strnlen_avx2) IFUNC_IMPL_ADD (array, i, strnlen, 1, __strnlen_sse2)) /* Support sysdeps/x86_64/multiarch/stpncpy.c. */ IFUNC_IMPL (i, name, stpncpy, - IFUNC_IMPL_ADD (array, i, stpncpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, stpncpy, x86_cpu_is_usable (x86_cpu_SSSE3), __stpncpy_ssse3) - IFUNC_IMPL_ADD (array, i, stpncpy, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, stpncpy, x86_cpu_is_usable (x86_cpu_AVX2), __stpncpy_avx2) IFUNC_IMPL_ADD (array, i, stpncpy, 1, __stpncpy_sse2_unaligned) @@ -207,9 +207,9 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/stpcpy.c. */ IFUNC_IMPL (i, name, stpcpy, - IFUNC_IMPL_ADD (array, i, stpcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, stpcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __stpcpy_ssse3) - IFUNC_IMPL_ADD (array, i, stpcpy, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, stpcpy, x86_cpu_is_usable (x86_cpu_AVX2), __stpcpy_avx2) IFUNC_IMPL_ADD (array, i, stpcpy, 1, __stpcpy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, stpcpy, 1, __stpcpy_sse2)) @@ -217,35 +217,35 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strcasecmp_l.c. */ IFUNC_IMPL (i, name, strcasecmp, IFUNC_IMPL_ADD (array, i, strcasecmp, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __strcasecmp_avx) IFUNC_IMPL_ADD (array, i, strcasecmp, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strcasecmp_sse42) IFUNC_IMPL_ADD (array, i, strcasecmp, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strcasecmp_ssse3) IFUNC_IMPL_ADD (array, i, strcasecmp, 1, __strcasecmp_sse2)) /* Support sysdeps/x86_64/multiarch/strcasecmp_l.c. */ IFUNC_IMPL (i, name, strcasecmp_l, IFUNC_IMPL_ADD (array, i, strcasecmp_l, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __strcasecmp_l_avx) IFUNC_IMPL_ADD (array, i, strcasecmp_l, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strcasecmp_l_sse42) IFUNC_IMPL_ADD (array, i, strcasecmp_l, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strcasecmp_l_ssse3) IFUNC_IMPL_ADD (array, i, strcasecmp_l, 1, __strcasecmp_l_sse2)) /* Support sysdeps/x86_64/multiarch/strcat.c. */ IFUNC_IMPL (i, name, strcat, - IFUNC_IMPL_ADD (array, i, strcat, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, strcat, x86_cpu_is_usable (x86_cpu_AVX2), __strcat_avx2) - IFUNC_IMPL_ADD (array, i, strcat, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcat, x86_cpu_is_usable (x86_cpu_SSSE3), __strcat_ssse3) IFUNC_IMPL_ADD (array, i, strcat, 1, __strcat_sse2_unaligned) IFUNC_IMPL_ADD (array, i, strcat, 1, __strcat_sse2)) @@ -253,7 +253,7 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strchr.c. */ IFUNC_IMPL (i, name, strchr, IFUNC_IMPL_ADD (array, i, strchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strchr_avx2) IFUNC_IMPL_ADD (array, i, strchr, 1, __strchr_sse2_no_bsf) IFUNC_IMPL_ADD (array, i, strchr, 1, __strchr_sse2)) @@ -261,54 +261,54 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strchrnul.c. */ IFUNC_IMPL (i, name, strchrnul, IFUNC_IMPL_ADD (array, i, strchrnul, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strchrnul_avx2) IFUNC_IMPL_ADD (array, i, strchrnul, 1, __strchrnul_sse2)) /* Support sysdeps/x86_64/multiarch/strrchr.c. */ IFUNC_IMPL (i, name, strrchr, IFUNC_IMPL_ADD (array, i, strrchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strrchr_avx2) IFUNC_IMPL_ADD (array, i, strrchr, 1, __strrchr_sse2)) /* Support sysdeps/x86_64/multiarch/strcmp.c. */ IFUNC_IMPL (i, name, strcmp, IFUNC_IMPL_ADD (array, i, strcmp, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strcmp_avx2) - IFUNC_IMPL_ADD (array, i, strcmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strcmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __strcmp_sse42) - IFUNC_IMPL_ADD (array, i, strcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __strcmp_ssse3) IFUNC_IMPL_ADD (array, i, strcmp, 1, __strcmp_sse2_unaligned) IFUNC_IMPL_ADD (array, i, strcmp, 1, __strcmp_sse2)) /* Support sysdeps/x86_64/multiarch/strcpy.c. */ IFUNC_IMPL (i, name, strcpy, - IFUNC_IMPL_ADD (array, i, strcpy, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, strcpy, x86_cpu_is_usable (x86_cpu_AVX2), __strcpy_avx2) - IFUNC_IMPL_ADD (array, i, strcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __strcpy_ssse3) IFUNC_IMPL_ADD (array, i, strcpy, 1, __strcpy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, strcpy, 1, __strcpy_sse2)) /* Support sysdeps/x86_64/multiarch/strcspn.c. */ IFUNC_IMPL (i, name, strcspn, - IFUNC_IMPL_ADD (array, i, strcspn, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strcspn, x86_cpu_is_usable (x86_cpu_SSE4_2), __strcspn_sse42) IFUNC_IMPL_ADD (array, i, strcspn, 1, __strcspn_sse2)) /* Support sysdeps/x86_64/multiarch/strncase_l.c. */ IFUNC_IMPL (i, name, strncasecmp, IFUNC_IMPL_ADD (array, i, strncasecmp, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __strncasecmp_avx) IFUNC_IMPL_ADD (array, i, strncasecmp, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strncasecmp_sse42) IFUNC_IMPL_ADD (array, i, strncasecmp, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strncasecmp_ssse3) IFUNC_IMPL_ADD (array, i, strncasecmp, 1, __strncasecmp_sse2)) @@ -316,22 +316,22 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strncase_l.c. */ IFUNC_IMPL (i, name, strncasecmp_l, IFUNC_IMPL_ADD (array, i, strncasecmp_l, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __strncasecmp_l_avx) IFUNC_IMPL_ADD (array, i, strncasecmp_l, - CPU_FEATURE_USABLE (SSE4_2), + x86_cpu_is_usable (x86_cpu_SSE4_2), __strncasecmp_l_sse42) IFUNC_IMPL_ADD (array, i, strncasecmp_l, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __strncasecmp_l_ssse3) IFUNC_IMPL_ADD (array, i, strncasecmp_l, 1, __strncasecmp_l_sse2)) /* Support sysdeps/x86_64/multiarch/strncat.c. */ IFUNC_IMPL (i, name, strncat, - IFUNC_IMPL_ADD (array, i, strncat, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, strncat, x86_cpu_is_usable (x86_cpu_AVX2), __strncat_avx2) - IFUNC_IMPL_ADD (array, i, strncat, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncat, x86_cpu_is_usable (x86_cpu_SSSE3), __strncat_ssse3) IFUNC_IMPL_ADD (array, i, strncat, 1, __strncat_sse2_unaligned) @@ -339,9 +339,9 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strncpy.c. */ IFUNC_IMPL (i, name, strncpy, - IFUNC_IMPL_ADD (array, i, strncpy, CPU_FEATURE_USABLE (AVX2), + IFUNC_IMPL_ADD (array, i, strncpy, x86_cpu_is_usable (x86_cpu_AVX2), __strncpy_avx2) - IFUNC_IMPL_ADD (array, i, strncpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncpy, x86_cpu_is_usable (x86_cpu_SSSE3), __strncpy_ssse3) IFUNC_IMPL_ADD (array, i, strncpy, 1, __strncpy_sse2_unaligned) @@ -349,14 +349,14 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strpbrk.c. */ IFUNC_IMPL (i, name, strpbrk, - IFUNC_IMPL_ADD (array, i, strpbrk, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strpbrk, x86_cpu_is_usable (x86_cpu_SSE4_2), __strpbrk_sse42) IFUNC_IMPL_ADD (array, i, strpbrk, 1, __strpbrk_sse2)) /* Support sysdeps/x86_64/multiarch/strspn.c. */ IFUNC_IMPL (i, name, strspn, - IFUNC_IMPL_ADD (array, i, strspn, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strspn, x86_cpu_is_usable (x86_cpu_SSE4_2), __strspn_sse42) IFUNC_IMPL_ADD (array, i, strspn, 1, __strspn_sse2)) @@ -368,70 +368,70 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/wcschr.c. */ IFUNC_IMPL (i, name, wcschr, IFUNC_IMPL_ADD (array, i, wcschr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcschr_avx2) IFUNC_IMPL_ADD (array, i, wcschr, 1, __wcschr_sse2)) /* Support sysdeps/x86_64/multiarch/wcsrchr.c. */ IFUNC_IMPL (i, name, wcsrchr, IFUNC_IMPL_ADD (array, i, wcsrchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcsrchr_avx2) IFUNC_IMPL_ADD (array, i, wcsrchr, 1, __wcsrchr_sse2)) /* Support sysdeps/x86_64/multiarch/wcscmp.c. */ IFUNC_IMPL (i, name, wcscmp, IFUNC_IMPL_ADD (array, i, wcscmp, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcscmp_avx2) IFUNC_IMPL_ADD (array, i, wcscmp, 1, __wcscmp_sse2)) /* Support sysdeps/x86_64/multiarch/wcsncmp.c. */ IFUNC_IMPL (i, name, wcsncmp, IFUNC_IMPL_ADD (array, i, wcsncmp, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcsncmp_avx2) IFUNC_IMPL_ADD (array, i, wcsncmp, 1, __wcsncmp_sse2)) /* Support sysdeps/x86_64/multiarch/wcscpy.c. */ IFUNC_IMPL (i, name, wcscpy, - IFUNC_IMPL_ADD (array, i, wcscpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, wcscpy, x86_cpu_is_usable (x86_cpu_SSSE3), __wcscpy_ssse3) IFUNC_IMPL_ADD (array, i, wcscpy, 1, __wcscpy_sse2)) /* Support sysdeps/x86_64/multiarch/wcslen.c. */ IFUNC_IMPL (i, name, wcslen, IFUNC_IMPL_ADD (array, i, wcslen, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcslen_avx2) IFUNC_IMPL_ADD (array, i, wcslen, 1, __wcslen_sse2)) /* Support sysdeps/x86_64/multiarch/wcsnlen.c. */ IFUNC_IMPL (i, name, wcsnlen, IFUNC_IMPL_ADD (array, i, wcsnlen, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wcsnlen_avx2) IFUNC_IMPL_ADD (array, i, wcsnlen, - CPU_FEATURE_USABLE (SSE4_1), + x86_cpu_is_usable (x86_cpu_SSE4_1), __wcsnlen_sse4_1) IFUNC_IMPL_ADD (array, i, wcsnlen, 1, __wcsnlen_sse2)) /* Support sysdeps/x86_64/multiarch/wmemchr.c. */ IFUNC_IMPL (i, name, wmemchr, IFUNC_IMPL_ADD (array, i, wmemchr, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wmemchr_avx2) IFUNC_IMPL_ADD (array, i, wmemchr, 1, __wmemchr_sse2)) /* Support sysdeps/x86_64/multiarch/wmemcmp.c. */ IFUNC_IMPL (i, name, wmemcmp, IFUNC_IMPL_ADD (array, i, wmemcmp, - (CPU_FEATURE_USABLE (AVX2) - && CPU_FEATURE_USABLE (MOVBE)), + (x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_MOVBE)), __wmemcmp_avx2_movbe) - IFUNC_IMPL_ADD (array, i, wmemcmp, CPU_FEATURE_USABLE (SSE4_1), + IFUNC_IMPL_ADD (array, i, wmemcmp, x86_cpu_is_usable (x86_cpu_SSE4_1), __wmemcmp_sse4_1) - IFUNC_IMPL_ADD (array, i, wmemcmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, wmemcmp, x86_cpu_is_usable (x86_cpu_SSSE3), __wmemcmp_ssse3) IFUNC_IMPL_ADD (array, i, wmemcmp, 1, __wmemcmp_sse2)) @@ -440,35 +440,35 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, wmemset, 1, __wmemset_sse2_unaligned) IFUNC_IMPL_ADD (array, i, wmemset, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wmemset_avx2_unaligned) IFUNC_IMPL_ADD (array, i, wmemset, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __wmemset_avx512_unaligned)) #ifdef SHARED /* Support sysdeps/x86_64/multiarch/memcpy_chk.c. */ IFUNC_IMPL (i, name, __memcpy_chk, IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_chk_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_chk_avx512_unaligned) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_chk_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memcpy_chk_avx_unaligned) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memcpy_chk_avx_unaligned_erms) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_chk_ssse3_back) IFUNC_IMPL_ADD (array, i, __memcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_chk_ssse3) IFUNC_IMPL_ADD (array, i, __memcpy_chk, 1, __memcpy_chk_sse2_unaligned) @@ -481,23 +481,23 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/memcpy.c. */ IFUNC_IMPL (i, name, memcpy, IFUNC_IMPL_ADD (array, i, memcpy, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memcpy_avx_unaligned) IFUNC_IMPL_ADD (array, i, memcpy, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __memcpy_avx_unaligned_erms) - IFUNC_IMPL_ADD (array, i, memcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_ssse3_back) - IFUNC_IMPL_ADD (array, i, memcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, memcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __memcpy_ssse3) IFUNC_IMPL_ADD (array, i, memcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, memcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_avx512_unaligned) IFUNC_IMPL_ADD (array, i, memcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __memcpy_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, memcpy, 1, __memcpy_sse2_unaligned) IFUNC_IMPL_ADD (array, i, memcpy, 1, @@ -508,25 +508,25 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/mempcpy_chk.c. */ IFUNC_IMPL (i, name, __mempcpy_chk, IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_chk_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_chk_avx512_unaligned) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_chk_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __mempcpy_chk_avx_unaligned) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __mempcpy_chk_avx_unaligned_erms) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_chk_ssse3_back) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, - CPU_FEATURE_USABLE (SSSE3), + x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_chk_ssse3) IFUNC_IMPL_ADD (array, i, __mempcpy_chk, 1, __mempcpy_chk_sse2_unaligned) @@ -539,23 +539,23 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/mempcpy.c. */ IFUNC_IMPL (i, name, mempcpy, IFUNC_IMPL_ADD (array, i, mempcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_avx512_no_vzeroupper) IFUNC_IMPL_ADD (array, i, mempcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_avx512_unaligned) IFUNC_IMPL_ADD (array, i, mempcpy, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __mempcpy_avx512_unaligned_erms) IFUNC_IMPL_ADD (array, i, mempcpy, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __mempcpy_avx_unaligned) IFUNC_IMPL_ADD (array, i, mempcpy, - CPU_FEATURE_USABLE (AVX), + x86_cpu_is_usable (x86_cpu_AVX), __mempcpy_avx_unaligned_erms) - IFUNC_IMPL_ADD (array, i, mempcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, mempcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_ssse3_back) - IFUNC_IMPL_ADD (array, i, mempcpy, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, mempcpy, x86_cpu_is_usable (x86_cpu_SSSE3), __mempcpy_ssse3) IFUNC_IMPL_ADD (array, i, mempcpy, 1, __mempcpy_sse2_unaligned) @@ -566,11 +566,11 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/x86_64/multiarch/strncmp.c. */ IFUNC_IMPL (i, name, strncmp, IFUNC_IMPL_ADD (array, i, strncmp, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __strncmp_avx2) - IFUNC_IMPL_ADD (array, i, strncmp, CPU_FEATURE_USABLE (SSE4_2), + IFUNC_IMPL_ADD (array, i, strncmp, x86_cpu_is_usable (x86_cpu_SSE4_2), __strncmp_sse42) - IFUNC_IMPL_ADD (array, i, strncmp, CPU_FEATURE_USABLE (SSSE3), + IFUNC_IMPL_ADD (array, i, strncmp, x86_cpu_is_usable (x86_cpu_SSSE3), __strncmp_ssse3) IFUNC_IMPL_ADD (array, i, strncmp, 1, __strncmp_sse2)) @@ -580,10 +580,10 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, __wmemset_chk, 1, __wmemset_chk_sse2_unaligned) IFUNC_IMPL_ADD (array, i, __wmemset_chk, - CPU_FEATURE_USABLE (AVX2), + x86_cpu_is_usable (x86_cpu_AVX2), __wmemset_chk_avx2_unaligned) IFUNC_IMPL_ADD (array, i, __wmemset_chk, - CPU_FEATURE_USABLE (AVX512F), + x86_cpu_is_usable (x86_cpu_AVX512F), __wmemset_chk_avx512_unaligned)) #endif diff --git a/sysdeps/x86_64/multiarch/ifunc-memcmp.h b/sysdeps/x86_64/multiarch/ifunc-memcmp.h index 0e21b3a628..592186b4e4 100644 --- a/sysdeps/x86_64/multiarch/ifunc-memcmp.h +++ b/sysdeps/x86_64/multiarch/ifunc-memcmp.h @@ -30,15 +30,15 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) - && CPU_FEATURE_USABLE_P (cpu_features, MOVBE) + && x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_MOVBE) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2_movbe); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_1)) + if (x86_cpu_is_usable (x86_cpu_SSE4_1)) return OPTIMIZE (sse4_1); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/ifunc-memmove.h b/sysdeps/x86_64/multiarch/ifunc-memmove.h index 9ada03aa43..bdba95022f 100644 --- a/sysdeps/x86_64/multiarch/ifunc-memmove.h +++ b/sysdeps/x86_64/multiarch/ifunc-memmove.h @@ -45,13 +45,13 @@ IFUNC_SELECTOR (void) || CPU_FEATURES_ARCH_P (cpu_features, Prefer_FSRM)) return OPTIMIZE (erms); - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512F) + if (x86_cpu_is_usable (x86_cpu_AVX512F) && !CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_AVX512)) { if (CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER)) return OPTIMIZE (avx512_no_vzeroupper); - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (avx512_unaligned_erms); return OPTIMIZE (avx512_unaligned); @@ -59,16 +59,16 @@ IFUNC_SELECTOR (void) if (CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) { - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (avx_unaligned_erms); return OPTIMIZE (avx_unaligned); } - if (!CPU_FEATURE_USABLE_P (cpu_features, SSSE3) + if (!x86_cpu_is_usable (x86_cpu_SSSE3) || CPU_FEATURES_ARCH_P (cpu_features, Fast_Unaligned_Copy)) { - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (sse2_unaligned_erms); return OPTIMIZE (sse2_unaligned); diff --git a/sysdeps/x86_64/multiarch/ifunc-memset.h b/sysdeps/x86_64/multiarch/ifunc-memset.h index f52613d372..b3995046e8 100644 --- a/sysdeps/x86_64/multiarch/ifunc-memset.h +++ b/sysdeps/x86_64/multiarch/ifunc-memset.h @@ -42,27 +42,27 @@ IFUNC_SELECTOR (void) if (CPU_FEATURES_ARCH_P (cpu_features, Prefer_ERMS)) return OPTIMIZE (erms); - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512F) + if (x86_cpu_is_usable (x86_cpu_AVX512F) && !CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_AVX512)) { if (CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER)) return OPTIMIZE (avx512_no_vzeroupper); - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (avx512_unaligned_erms); return OPTIMIZE (avx512_unaligned); } - if (CPU_FEATURE_USABLE_P (cpu_features, AVX2)) + if (x86_cpu_is_usable (x86_cpu_AVX2)) { - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (avx2_unaligned_erms); else return OPTIMIZE (avx2_unaligned); } - if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) + if (x86_cpu_is_usable (x86_cpu_ERMS)) return OPTIMIZE (sse2_unaligned_erms); return OPTIMIZE (sse2_unaligned); diff --git a/sysdeps/x86_64/multiarch/ifunc-sse4_2.h b/sysdeps/x86_64/multiarch/ifunc-sse4_2.h index cbf18385d3..1aa71caf21 100644 --- a/sysdeps/x86_64/multiarch/ifunc-sse4_2.h +++ b/sysdeps/x86_64/multiarch/ifunc-sse4_2.h @@ -25,9 +25,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (sse42) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_2)) + if (x86_cpu_is_usable (x86_cpu_SSE4_2)) return OPTIMIZE (sse42); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/ifunc-strcasecmp.h b/sysdeps/x86_64/multiarch/ifunc-strcasecmp.h index 0818333931..0f5ab23047 100644 --- a/sysdeps/x86_64/multiarch/ifunc-strcasecmp.h +++ b/sysdeps/x86_64/multiarch/ifunc-strcasecmp.h @@ -29,14 +29,14 @@ IFUNC_SELECTOR (void) { const struct cpu_features* cpu_features = __get_cpu_features (); - if (CPU_FEATURE_USABLE_P (cpu_features, AVX)) + if (x86_cpu_is_usable (x86_cpu_AVX)) return OPTIMIZE (avx); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_2) + if (x86_cpu_is_usable (x86_cpu_SSE4_2) && !CPU_FEATURES_ARCH_P (cpu_features, Slow_SSE4_2)) return OPTIMIZE (sse42); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/ifunc-strcpy.h b/sysdeps/x86_64/multiarch/ifunc-strcpy.h index 63b0dc0d96..0ae6dcc704 100644 --- a/sysdeps/x86_64/multiarch/ifunc-strcpy.h +++ b/sysdeps/x86_64/multiarch/ifunc-strcpy.h @@ -32,14 +32,14 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); if (CPU_FEATURES_ARCH_P (cpu_features, Fast_Unaligned_Load)) return OPTIMIZE (sse2_unaligned); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/ifunc-wmemset.h b/sysdeps/x86_64/multiarch/ifunc-wmemset.h index 8cfce562fc..d7095301dd 100644 --- a/sysdeps/x86_64/multiarch/ifunc-wmemset.h +++ b/sysdeps/x86_64/multiarch/ifunc-wmemset.h @@ -28,10 +28,10 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) { - if (CPU_FEATURE_USABLE_P (cpu_features, AVX512F) + if (x86_cpu_is_usable (x86_cpu_AVX512F) && !CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_AVX512)) return OPTIMIZE (avx512_unaligned); else diff --git a/sysdeps/x86_64/multiarch/sched_cpucount.c b/sysdeps/x86_64/multiarch/sched_cpucount.c index 074c663cf6..f1aba1462b 100644 --- a/sysdeps/x86_64/multiarch/sched_cpucount.c +++ b/sysdeps/x86_64/multiarch/sched_cpucount.c @@ -33,4 +33,5 @@ #undef __sched_cpucount libc_ifunc (__sched_cpucount, - CPU_FEATURE_USABLE (POPCNT) ? popcount_cpucount : generic_cpucount); + (x86_cpu_is_usable (x86_cpu_POPCNT) + ? popcount_cpucount : generic_cpucount)); diff --git a/sysdeps/x86_64/multiarch/strchr.c b/sysdeps/x86_64/multiarch/strchr.c index 8df4609bf8..9e570f1d4e 100644 --- a/sysdeps/x86_64/multiarch/strchr.c +++ b/sysdeps/x86_64/multiarch/strchr.c @@ -36,7 +36,7 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); diff --git a/sysdeps/x86_64/multiarch/strcmp.c b/sysdeps/x86_64/multiarch/strcmp.c index 16ae72a4c8..fdc29f9d7b 100644 --- a/sysdeps/x86_64/multiarch/strcmp.c +++ b/sysdeps/x86_64/multiarch/strcmp.c @@ -37,14 +37,14 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); if (CPU_FEATURES_ARCH_P (cpu_features, Fast_Unaligned_Load)) return OPTIMIZE (sse2_unaligned); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/strncmp.c b/sysdeps/x86_64/multiarch/strncmp.c index 3c94b3ffd9..97b8c58bed 100644 --- a/sysdeps/x86_64/multiarch/strncmp.c +++ b/sysdeps/x86_64/multiarch/strncmp.c @@ -37,15 +37,15 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_2) + if (x86_cpu_is_usable (x86_cpu_SSE4_2) && !CPU_FEATURES_ARCH_P (cpu_features, Slow_SSE4_2)) return OPTIMIZE (sse42); - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/wcscpy.c b/sysdeps/x86_64/multiarch/wcscpy.c index e08536c593..d693985f7b 100644 --- a/sysdeps/x86_64/multiarch/wcscpy.c +++ b/sysdeps/x86_64/multiarch/wcscpy.c @@ -32,9 +32,7 @@ extern __typeof (REDIRECT_NAME) OPTIMIZE (ssse3) attribute_hidden; static inline void * IFUNC_SELECTOR (void) { - const struct cpu_features* cpu_features = __get_cpu_features (); - - if (CPU_FEATURE_USABLE_P (cpu_features, SSSE3)) + if (x86_cpu_is_usable (x86_cpu_SSSE3)) return OPTIMIZE (ssse3); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/multiarch/wcsnlen.c b/sysdeps/x86_64/multiarch/wcsnlen.c index 52e7e5d4f3..f17c25a0fc 100644 --- a/sysdeps/x86_64/multiarch/wcsnlen.c +++ b/sysdeps/x86_64/multiarch/wcsnlen.c @@ -36,11 +36,11 @@ IFUNC_SELECTOR (void) const struct cpu_features* cpu_features = __get_cpu_features (); if (!CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_VZEROUPPER) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) + && x86_cpu_is_usable (x86_cpu_AVX2) && CPU_FEATURES_ARCH_P (cpu_features, AVX_Fast_Unaligned_Load)) return OPTIMIZE (avx2); - if (CPU_FEATURE_USABLE_P (cpu_features, SSE4_1)) + if (x86_cpu_is_usable (x86_cpu_SSE4_1)) return OPTIMIZE (sse4_1); return OPTIMIZE (sse2); diff --git a/sysdeps/x86_64/tst-glibc-hwcaps.c b/sysdeps/x86_64/tst-glibc-hwcaps.c index 3075a8286d..9df70e58f2 100644 --- a/sysdeps/x86_64/tst-glibc-hwcaps.c +++ b/sysdeps/x86_64/tst-glibc-hwcaps.c @@ -29,35 +29,32 @@ extern int marker4 (void); static int compute_level (void) { - const struct cpu_features *cpu_features - = __x86_get_cpu_features (COMMON_CPUID_INDEX_MAX); - - if (!(CPU_FEATURE_USABLE_P (cpu_features, CMPXCHG16B) - && CPU_FEATURE_USABLE_P (cpu_features, LAHF64_SAHF64) - && CPU_FEATURE_USABLE_P (cpu_features, POPCNT) - && CPU_FEATURE_USABLE_P (cpu_features, MMX) - && CPU_FEATURE_USABLE_P (cpu_features, SSE) - && CPU_FEATURE_USABLE_P (cpu_features, SSE2) - && CPU_FEATURE_USABLE_P (cpu_features, SSE3) - && CPU_FEATURE_USABLE_P (cpu_features, SSSE3) - && CPU_FEATURE_USABLE_P (cpu_features, SSE4_1) - && CPU_FEATURE_USABLE_P (cpu_features, SSE4_2))) + if (!(x86_cpu_is_usable (x86_cpu_CMPXCHG16B) + && x86_cpu_is_usable (x86_cpu_LAHF64_SAHF64) + && x86_cpu_is_usable (x86_cpu_POPCNT) + && x86_cpu_is_usable (x86_cpu_MMX) + && x86_cpu_is_usable (x86_cpu_SSE) + && x86_cpu_is_usable (x86_cpu_SSE2) + && x86_cpu_is_usable (x86_cpu_SSE3) + && x86_cpu_is_usable (x86_cpu_SSSE3) + && x86_cpu_is_usable (x86_cpu_SSE4_1) + && x86_cpu_is_usable (x86_cpu_SSE4_2))) return 1; - if (!(CPU_FEATURE_USABLE_P (cpu_features, AVX) - && CPU_FEATURE_USABLE_P (cpu_features, AVX2) - && CPU_FEATURE_USABLE_P (cpu_features, BMI1) - && CPU_FEATURE_USABLE_P (cpu_features, BMI2) - && CPU_FEATURE_USABLE_P (cpu_features, F16C) - && CPU_FEATURE_USABLE_P (cpu_features, FMA) - && CPU_FEATURE_USABLE_P (cpu_features, LZCNT) - && CPU_FEATURE_USABLE_P (cpu_features, MOVBE) - && CPU_FEATURE_USABLE_P (cpu_features, OSXSAVE))) + if (!(x86_cpu_is_usable (x86_cpu_AVX) + && x86_cpu_is_usable (x86_cpu_AVX2) + && x86_cpu_is_usable (x86_cpu_BMI1) + && x86_cpu_is_usable (x86_cpu_BMI2) + && x86_cpu_is_usable (x86_cpu_F16C) + && x86_cpu_is_usable (x86_cpu_FMA) + && x86_cpu_is_usable (x86_cpu_LZCNT) + && x86_cpu_is_usable (x86_cpu_MOVBE) + && x86_cpu_is_usable (x86_cpu_OSXSAVE))) return 2; - if (!(CPU_FEATURE_USABLE_P (cpu_features, AVX512F) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512BW) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512CD) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512DQ) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512VL))) + if (!(x86_cpu_is_usable (x86_cpu_AVX512F) + && x86_cpu_is_usable (x86_cpu_AVX512BW) + && x86_cpu_is_usable (x86_cpu_AVX512CD) + && x86_cpu_is_usable (x86_cpu_AVX512DQ) + && x86_cpu_is_usable (x86_cpu_AVX512VL))) return 3; return 4; }