From patchwork Thu Jan 26 17:22:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 63751 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B6A693858412 for ; Thu, 26 Jan 2023 17:23:23 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B6A693858412 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1674753803; bh=kdccmXP7/5rsNonBkvOwBFBK3hxQa7XE8VD6xSini38=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=YYmSIbdH620IpF7NHCGalnPkTOsZrUqVlOKUktal+s//HpCF2oXjQei1O1FhnypIu ojgEI9O5WHuwVdyALqrs7eH7ZqOcmIRztTYoKGVvA9dhKCFYt0lR+tH9eci/KOfbf8 zJG+ZSz3vScLrX8jYEa2ecji8IPwaH6uynOKsSCI= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by sourceware.org (Postfix) with ESMTPS id 650AA3858D28 for ; Thu, 26 Jan 2023 17:23:00 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 650AA3858D28 Received: by mail-pj1-x1035.google.com with SMTP id h5-20020a17090a9c0500b0022bb85eb35dso2389257pjp.3 for ; Thu, 26 Jan 2023 09:23:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kdccmXP7/5rsNonBkvOwBFBK3hxQa7XE8VD6xSini38=; b=2Klat9x6N1AW2J8Ea60crSaVt5tEPtPVr1kh4uBG7prN7ZoEQysjPK6Gt/MNfWcNRh 04q9AzLjB+HE8bsmBsbBtrBV3gW/q640ctueUZC1arViOLXHjNVl/zjKKTqT+lLssePc rPMhhiD9e5TR9GdtvndcO5hjNMn4X5yx12wh3iGpJE/C1Ff720OP7hLsebxnSW5rhfTb kQov31MCWmikvKOwgbdkgVQDw8oNN/8SsIY/jwHs9EoCD0gTEp4lGnQWVgeOo8pSjvPs 2zGr1qz9TiFnXTojAHWqncp15f2YEGrisCjihE9DdknKQ3Cp7aiPCajwB0U/uBzv5MQW TWyA== X-Gm-Message-State: AO0yUKUqFDhipd+7OzpCYGqtW7hCUf0exO35uoejuivJ6FbkzP0m9zRQ hu1630IaIkO1J5yoKgbs2EgKAjPiZAU= X-Google-Smtp-Source: AK7set8VwxiFdKhIFGOlXTZ1xCuJDnMeRj/Le28V5MS0LDNwUmxYmVlCTafC7K9FtBL+I4jIepcNbw== X-Received: by 2002:a17:902:ec82:b0:196:125a:e4b8 with SMTP id x2-20020a170902ec8200b00196125ae4b8mr13989275plg.12.1674753778534; Thu, 26 Jan 2023 09:22:58 -0800 (PST) Received: from gnu-cfl-3.localdomain ([172.56.31.90]) by smtp.gmail.com with ESMTPSA id 26-20020aa7915a000000b0058d99337381sm1089797pfi.172.2023.01.26.09.22.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Jan 2023 09:22:58 -0800 (PST) Received: from gnu-cfl-3.. (localhost [IPv6:::1]) by gnu-cfl-3.localdomain (Postfix) with ESMTP id 8077E74015B for ; Thu, 26 Jan 2023 09:22:56 -0800 (PST) To: libc-alpha@sourceware.org Subject: [PATCH] x86-64: Restore LD_PREFER_MAP_32BIT_EXEC support [BZ #28656] Date: Thu, 26 Jan 2023 09:22:56 -0800 Message-Id: <20230126172256.829709-1-hjl.tools@gmail.com> X-Mailer: git-send-email 2.39.1 MIME-Version: 1.0 X-Spam-Status: No, score=-3023.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Libc-alpha" From: "H.J. Lu" Reply-To: "H.J. Lu" Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Crossing 2GB boundaries with indirect calls and jumps can use more branch prediction resources on Intel Golden Cove CPU (see the "Misprediction for Branches >2GB" section in Intel 64 and IA-32 Architectures Optimization Reference Manual.) There is visible performance improvement on workloads with many PLT calls when executable and shared libraries are mmapped below 2GB. Add the Prefer_MAP_32BIT_EXEC bit so that mmap will try to map executable or denywrite pages in shared libraries with MAP_32BIT first. NB: Prefer_MAP_32BIT_EXEC reduces bits available for address space layout randomization (ASLR), which is always disabled for SUID programs and can only be enabled by setting environment variable, LD_PREFER_MAP_32BIT_EXEC. LD_PREFER_MAP_32BIT_EXEC works only between shared libraries or between shared libraries and executables with addresses below 2GB. PIEs are usually mapped above 4GB by the kernel. --- elf/dl-support.c | 4 ++ elf/rtld.c | 18 +++++++- sysdeps/generic/dl-librecon.h | 24 +++++++++++ sysdeps/unix/sysv/linux/x86_64/64/Makefile | 19 ++++++++ .../unix/sysv/linux/x86_64/64/dl-librecon.h | 43 +++++++++++++++++++ .../unix/sysv/linux/x86_64/64/mmap_internal.h | 43 +++++++++++++++++++ .../sysv/linux/x86_64/64/tst-map-32bit-mod.c | 33 ++++++++++++++ .../unix/sysv/linux/x86_64/64/tst-map-32bit.c | 34 +++++++++++++++ sysdeps/x86/cpu-tunables.c | 7 +++ ...cpu-features-preferred_feature_index_1.def | 1 + 10 files changed, 225 insertions(+), 1 deletion(-) create mode 100644 sysdeps/generic/dl-librecon.h create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/dl-librecon.h create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit.c diff --git a/elf/dl-support.c b/elf/dl-support.c index 9714f75db0..39447d2fad 100644 --- a/elf/dl-support.c +++ b/elf/dl-support.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include #include @@ -320,6 +321,9 @@ _dl_non_dynamic_init (void) { static const char unsecure_envvars[] = UNSECURE_ENVVARS +#ifdef EXTRA_UNSECURE_ENVVARS + EXTRA_UNSECURE_ENVVARS +#endif ; const char *cp = unsecure_envvars; diff --git a/elf/rtld.c b/elf/rtld.c index b8467f37cf..dd7952efe4 100644 --- a/elf/rtld.c +++ b/elf/rtld.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -2679,6 +2680,14 @@ process_envvars (struct dl_main_state *state) = _dl_strtoul (&envline[21], NULL) > 1; } break; + + /* We might have some extra environment variable to handle. This + is tricky due to the pre-processing of the length of the name + in the switch statement here. The code here assumes that added + environment variables have a different length. */ +#ifdef EXTRA_LD_ENVVARS + EXTRA_LD_ENVVARS +#endif } } @@ -2686,7 +2695,14 @@ process_envvars (struct dl_main_state *state) variables. */ if (__glibc_unlikely (__libc_enable_secure)) { - const char *nextp = UNSECURE_ENVVARS; + static const char unsecure_envvars[] = +#ifdef EXTRA_UNSECURE_ENVVARS + EXTRA_UNSECURE_ENVVARS +#endif + UNSECURE_ENVVARS; + const char *nextp; + + nextp = unsecure_envvars; do { unsetenv (nextp); diff --git a/sysdeps/generic/dl-librecon.h b/sysdeps/generic/dl-librecon.h new file mode 100644 index 0000000000..823b200cb8 --- /dev/null +++ b/sysdeps/generic/dl-librecon.h @@ -0,0 +1,24 @@ +/* Optional code to distinguish library flavours. + Copyright (C) 1998-2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _DL_LIBRECON_H +#define _DL_LIBRECON_H 1 + +/* In the general case we don't do anything. */ + +#endif /* dl-librecon.h */ diff --git a/sysdeps/unix/sysv/linux/x86_64/64/Makefile b/sysdeps/unix/sysv/linux/x86_64/64/Makefile index a7b6dc5a53..53a401e35e 100644 --- a/sysdeps/unix/sysv/linux/x86_64/64/Makefile +++ b/sysdeps/unix/sysv/linux/x86_64/64/Makefile @@ -1,2 +1,21 @@ # The default ABI is 64. default-abi := 64 + +ifeq ($(subdir),elf) + +tests-map-32bit = \ + tst-map-32bit \ +# tests-map-32bit +tst-map-32bit-no-pie = yes +tests += $(tests-map-32bit) + +modules-map-32bit = \ + tst-map-32bit-mod \ +# modules-map-32bit +modules-names += $(modules-map-32bit) + +tst-map-32bit-ENV = LD_PREFER_MAP_32BIT_EXEC=1 +$(objpfx)tst-map-32bit-mod.so: $(libsupport) +$(objpfx)tst-map-32bit: $(objpfx)tst-map-32bit-mod.so + +endif diff --git a/sysdeps/unix/sysv/linux/x86_64/64/dl-librecon.h b/sysdeps/unix/sysv/linux/x86_64/64/dl-librecon.h new file mode 100644 index 0000000000..8e6ca0d999 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/dl-librecon.h @@ -0,0 +1,43 @@ +/* Optional code to distinguish library flavours. x86-64 version. + Copyright (C) 2015-2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _DL_LIBRECON_H + +/* Recognizing extra environment variables. For 64-bit applications, + branch prediction performance may be negatively impacted when the + target of a branch is more than 4GB away from the branch. Add the + Prefer_MAP_32BIT_EXEC bit so that mmap will try to map executable + pages with MAP_32BIT first. NB: MAP_32BIT will map to lower 2GB, + not lower 4GB, address. Prefer_MAP_32BIT_EXEC reduces bits available + for address space layout randomization (ASLR). Prefer_MAP_32BIT_EXEC + is always disabled for SUID programs and can be enabled by setting + environment variable, LD_PREFER_MAP_32BIT_EXEC. */ +#define EXTRA_LD_ENVVARS \ + case 21: \ + if (!__libc_enable_secure \ + && memcmp (envline, "PREFER_MAP_32BIT_EXEC", 21) == 0) \ + GLRO(dl_x86_cpu_features).preferred[index_arch_Prefer_MAP_32BIT_EXEC] \ + |= bit_arch_Prefer_MAP_32BIT_EXEC; \ + break; + +/* Extra unsecure variables. The names are all stuffed in a single + string which means they have to be terminated with a '\0' explicitly. */ +#define EXTRA_UNSECURE_ENVVARS \ + "LD_PREFER_MAP_32BIT_EXEC\0" + +#endif /* dl-librecon.h */ diff --git a/sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h b/sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h new file mode 100644 index 0000000000..33dec3f805 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h @@ -0,0 +1,43 @@ +/* Linux mmap system call. x86-64 version. + Copyright (C) 2015-2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef MMAP_X86_64_INTERNAL_H +#define MMAP_X86_64_INTERNAL_H + +#include + +/* If the Prefer_MAP_32BIT_EXEC bit is set, try to map executable or + denywrite pages with MAP_32BIT first. */ +#define MMAP_PREPARE(addr, len, prot, flags, fd, offset) \ + if ((addr) == NULL \ + && (((prot) & PROT_EXEC) != 0 \ + || ((flags) & MAP_DENYWRITE) != 0) \ + && HAS_ARCH_FEATURE (Prefer_MAP_32BIT_EXEC)) \ + { \ + void *ret = (void*) INLINE_SYSCALL_CALL (mmap, (addr), (len), \ + (prot), \ + (flags) | MAP_32BIT, \ + (fd), (offset)); \ + if (ret != MAP_FAILED) \ + return ret; \ + } + +#include_next + +#endif diff --git a/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c new file mode 100644 index 0000000000..78d4b6133c --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c @@ -0,0 +1,33 @@ +/* Check that LD_PREFER_MAP_32BIT_EXEC works in shared library. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +static void +dso_do_test (void) +{ +} + +void +dso_check_map_32bit (void) +{ + printf ("dso_do_test: %p\n", dso_do_test); + TEST_VERIFY ((uintptr_t) dso_do_test < 0xffffffffUL); +} diff --git a/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit.c b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit.c new file mode 100644 index 0000000000..abc396589e --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit.c @@ -0,0 +1,34 @@ +/* Check that LD_PREFER_MAP_32BIT_EXEC works in PDE and shared library. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +extern void dso_check_map_32bit (void); + +static int +do_test (void) +{ + printf ("do_test: %p\n", do_test); + TEST_VERIFY ((uintptr_t) do_test < 0xffffffffUL); + dso_check_map_32bit (); + return 0; +} + +#include diff --git a/sysdeps/x86/cpu-tunables.c b/sysdeps/x86/cpu-tunables.c index d3e1367bda..a381b5249c 100644 --- a/sysdeps/x86/cpu-tunables.c +++ b/sysdeps/x86/cpu-tunables.c @@ -259,6 +259,13 @@ TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *valp) 20); } break; + case 21: + { + CHECK_GLIBC_IFUNC_PREFERRED_BOTH (n, cpu_features, + Prefer_MAP_32BIT_EXEC, + disable, 21); + } + break; case 23: { CHECK_GLIBC_IFUNC_PREFERRED_NEED_BOTH diff --git a/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def b/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def index e45f9cb159..d20c5b3196 100644 --- a/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def +++ b/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def @@ -26,6 +26,7 @@ BIT (I586) BIT (I686) BIT (Slow_SSE4_2) BIT (AVX_Fast_Unaligned_Load) +BIT (Prefer_MAP_32BIT_EXEC) BIT (Prefer_No_VZEROUPPER) BIT (Prefer_ERMS) BIT (Prefer_No_AVX512)