From patchwork Mon Aug 1 19:51:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 56474 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 13A61385801E for ; Mon, 1 Aug 2022 19:52:22 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 13A61385801E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1659383542; bh=4VJe7tsxjpL608A4amvIEbrNr+mCFaV94IiRDS3vxKw=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=PhF7JmMuqMv2LvAk+hIBmN1nDOv4kUdrlN1c+a493rY/++6bINLLqqZ2UhVjrSoKz yL3q6/SCqQibulC05yqF2Mxskv5z1mQRRqdwFESUJW7esva0PbEI2qrvcY/x/H4xZN jA7psY8aiSkyuYtVMNBL9CBaPXrbswgCE5fk6weE= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) by sourceware.org (Postfix) with ESMTPS id 4E0A73858425 for ; Mon, 1 Aug 2022 19:51:59 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 4E0A73858425 Received: by mail-pl1-x62c.google.com with SMTP id z19so11450304plb.1 for ; Mon, 01 Aug 2022 12:51:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=4VJe7tsxjpL608A4amvIEbrNr+mCFaV94IiRDS3vxKw=; b=GS9Hs0yD/3JvlvBiUDqUYaHxhGtFrD9vTkfzKhGC2StO3ZOVaDPVtAi/jkiUGU0Hj9 kS5B/FOqnPKhjWurJiQ6mchoYn/YWuMTwp4Af72CZXuhIce8HUxG0rmwONnjU4nt100d 46nH5pXU7wLk704SKxRn0T0Y1fhknBprH8YvtLnBH1r1O39XehF+n4ib8AwfxUGtBbDY KUXAEwpbOD1lva2Dz6hApONsaYwQcNNzEJj+IXiuOEb3X5caLG9/3r0rh0ldZtQaraic mmjaAFhPJcqoxoKBBk8ebGSu1DMoWFUaGPCJyRoxXWK3j8XzJ/5Nhn9ctixS2xZAKlaF nRXQ== X-Gm-Message-State: ACgBeo0pnh5x8oisB8lP3EEonrexMwS1E3QZJ0d1+pu+NYzxI1crTUpU ta111q9s2Hd0QrADbgdCz7IcpPadlMA= X-Google-Smtp-Source: AA6agR45Jyw1Etw+rYYgUj6I0OuaE4BepL2nU+cVx9sHFO1YGkeFEwYEjqiqPt5aDJl9BWIPaLlqQw== X-Received: by 2002:a17:902:ef4d:b0:16d:ab14:cc7b with SMTP id e13-20020a170902ef4d00b0016dab14cc7bmr17972308plx.59.1659383517871; Mon, 01 Aug 2022 12:51:57 -0700 (PDT) Received: from gnu-tgl-3.localdomain ([172.58.37.102]) by smtp.gmail.com with ESMTPSA id u2-20020a626002000000b0052d3899f8c2sm4369322pfb.4.2022.08.01.12.51.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Aug 2022 12:51:57 -0700 (PDT) Received: from gnu-tgl-3.. (localhost [IPv6:::1]) by gnu-tgl-3.localdomain (Postfix) with ESMTP id 28BA8C0112 for ; Mon, 1 Aug 2022 12:51:50 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH] x86-64: Restore LD_PREFER_MAP_32BIT_EXEC support [BZ #28656] Date: Mon, 1 Aug 2022 12:51:50 -0700 Message-Id: <20220801195150.2160919-1-hjl.tools@gmail.com> X-Mailer: git-send-email 2.37.1 MIME-Version: 1.0 X-Spam-Status: No, score=-3027.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Libc-alpha" From: "H.J. Lu" Reply-To: "H.J. Lu" Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Crossing 2GB boundaries with indirect calls and jumps can use more branch prediction resources on several Intel CPUs. There is visible performance improvement on workloads with many PLT calls when executable and shared libraries are mmapped below 2GB. Add the Prefer_MAP_32BIT_EXEC bit so that mmap will try to map executable or denywrite pages with MAP_32BIT first. NB: Prefer_MAP_32BIT_EXEC reduces bits available for address space layout randomization (ASLR), which is always disabled for SUID programs and can only be enabled by setting environment variable, LD_PREFER_MAP_32BIT_EXEC. --- elf/dl-support.c | 4 ++ elf/rtld.c | 20 ++++++++- sysdeps/generic/dl-librecon.h | 24 +++++++++++ sysdeps/unix/sysv/linux/x86_64/64/Makefile | 19 ++++++++ .../unix/sysv/linux/x86_64/64/dl-librecon.h | 43 +++++++++++++++++++ .../unix/sysv/linux/x86_64/64/mmap_internal.h | 43 +++++++++++++++++++ .../sysv/linux/x86_64/64/tst-map-32bit-mod.c | 33 ++++++++++++++ .../unix/sysv/linux/x86_64/64/tst-map-32bit.c | 34 +++++++++++++++ sysdeps/x86/cpu-tunables.c | 7 +++ ...cpu-features-preferred_feature_index_1.def | 1 + 10 files changed, 226 insertions(+), 2 deletions(-) create mode 100644 sysdeps/generic/dl-librecon.h create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/dl-librecon.h create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit.c diff --git a/elf/dl-support.c b/elf/dl-support.c index 4af0b5b2ce..7160bdfcf1 100644 --- a/elf/dl-support.c +++ b/elf/dl-support.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include #include @@ -300,6 +301,9 @@ _dl_non_dynamic_init (void) { static const char unsecure_envvars[] = UNSECURE_ENVVARS +#ifdef EXTRA_UNSECURE_ENVVARS + EXTRA_UNSECURE_ENVVARS +#endif ; const char *cp = unsecure_envvars; diff --git a/elf/rtld.c b/elf/rtld.c index cbbaf4a331..2ca8c128a3 100644 --- a/elf/rtld.c +++ b/elf/rtld.c @@ -32,6 +32,7 @@ #include #include #include +#include #include #include #include @@ -2681,14 +2682,29 @@ process_envvars (struct dl_main_state *state) = _dl_strtoul (&envline[21], NULL) > 1; } break; + + /* We might have some extra environment variable to handle. This + is tricky due to the pre-processing of the length of the name + in the switch statement here. The code here assumes that added + environment variables have a different length. */ +#ifdef EXTRA_LD_ENVVARS + EXTRA_LD_ENVVARS +#endif } } /* Extra security for SUID binaries. Remove all dangerous environment variables. */ - if (__glibc_unlikely (__libc_enable_secure)) + if (__builtin_expect (__libc_enable_secure, 0)) { - const char *nextp = UNSECURE_ENVVARS; + static const char unsecure_envvars[] = +#ifdef EXTRA_UNSECURE_ENVVARS + EXTRA_UNSECURE_ENVVARS +#endif + UNSECURE_ENVVARS; + const char *nextp; + + nextp = unsecure_envvars; do { unsetenv (nextp); diff --git a/sysdeps/generic/dl-librecon.h b/sysdeps/generic/dl-librecon.h new file mode 100644 index 0000000000..19fc70cb29 --- /dev/null +++ b/sysdeps/generic/dl-librecon.h @@ -0,0 +1,24 @@ +/* Optional code to distinguish library flavours. + Copyright (C) 1998-2022 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _DL_LIBRECON_H +#define _DL_LIBRECON_H 1 + +/* In the general case we don't do anything. */ + +#endif /* dl-librecon.h */ diff --git a/sysdeps/unix/sysv/linux/x86_64/64/Makefile b/sysdeps/unix/sysv/linux/x86_64/64/Makefile index a7b6dc5a53..53a401e35e 100644 --- a/sysdeps/unix/sysv/linux/x86_64/64/Makefile +++ b/sysdeps/unix/sysv/linux/x86_64/64/Makefile @@ -1,2 +1,21 @@ # The default ABI is 64. default-abi := 64 + +ifeq ($(subdir),elf) + +tests-map-32bit = \ + tst-map-32bit \ +# tests-map-32bit +tst-map-32bit-no-pie = yes +tests += $(tests-map-32bit) + +modules-map-32bit = \ + tst-map-32bit-mod \ +# modules-map-32bit +modules-names += $(modules-map-32bit) + +tst-map-32bit-ENV = LD_PREFER_MAP_32BIT_EXEC=1 +$(objpfx)tst-map-32bit-mod.so: $(libsupport) +$(objpfx)tst-map-32bit: $(objpfx)tst-map-32bit-mod.so + +endif diff --git a/sysdeps/unix/sysv/linux/x86_64/64/dl-librecon.h b/sysdeps/unix/sysv/linux/x86_64/64/dl-librecon.h new file mode 100644 index 0000000000..246abccd45 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/dl-librecon.h @@ -0,0 +1,43 @@ +/* Optional code to distinguish library flavours. x86-64 version. + Copyright (C) 2015-2022 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _DL_LIBRECON_H + +/* Recognizing extra environment variables. For 64-bit applications, + branch prediction performance may be negatively impacted when the + target of a branch is more than 4GB away from the branch. Add the + Prefer_MAP_32BIT_EXEC bit so that mmap will try to map executable + pages with MAP_32BIT first. NB: MAP_32BIT will map to lower 2GB, + not lower 4GB, address. Prefer_MAP_32BIT_EXEC reduces bits available + for address space layout randomization (ASLR). Prefer_MAP_32BIT_EXEC + is always disabled for SUID programs and can be enabled by setting + environment variable, LD_PREFER_MAP_32BIT_EXEC. */ +#define EXTRA_LD_ENVVARS \ + case 21: \ + if (!__libc_enable_secure \ + && memcmp (envline, "PREFER_MAP_32BIT_EXEC", 21) == 0) \ + GLRO(dl_x86_cpu_features).preferred[index_arch_Prefer_MAP_32BIT_EXEC] \ + |= bit_arch_Prefer_MAP_32BIT_EXEC; \ + break; + +/* Extra unsecure variables. The names are all stuffed in a single + string which means they have to be terminated with a '\0' explicitly. */ +#define EXTRA_UNSECURE_ENVVARS \ + "LD_PREFER_MAP_32BIT_EXEC\0" + +#endif /* dl-librecon.h */ diff --git a/sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h b/sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h new file mode 100644 index 0000000000..7a3c18fb7b --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h @@ -0,0 +1,43 @@ +/* Linux mmap system call. x86-64 version. + Copyright (C) 2015-2022 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef MMAP_X86_64_INTERNAL_H +#define MMAP_X86_64_INTERNAL_H + +#include + +/* If the Prefer_MAP_32BIT_EXEC bit is set, try to map executable or + denywrite pages with MAP_32BIT first. */ +#define MMAP_PREPARE(addr, len, prot, flags, fd, offset) \ + if ((addr) == NULL \ + && (((prot) & PROT_EXEC) != 0 \ + || ((flags) & MAP_DENYWRITE) != 0) \ + && HAS_ARCH_FEATURE (Prefer_MAP_32BIT_EXEC)) \ + { \ + void *ret = (void*) INLINE_SYSCALL_CALL (mmap, (addr), (len), \ + (prot), \ + (flags) | MAP_32BIT, \ + (fd), (offset)); \ + if (ret != MAP_FAILED) \ + return ret; \ + } + +#include_next + +#endif diff --git a/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c new file mode 100644 index 0000000000..0e44d17105 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c @@ -0,0 +1,33 @@ +/* Check that LD_PREFER_MAP_32BIT_EXEC works in shared library. + Copyright (C) 2022 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +static void +dso_do_test (void) +{ +} + +void +dso_check_map_32bit (void) +{ + printf ("dso_do_test: %p\n", dso_do_test); + TEST_VERIFY ((uintptr_t) dso_do_test < 0xffffffffUL); +} diff --git a/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit.c b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit.c new file mode 100644 index 0000000000..ed96d8a4d0 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit.c @@ -0,0 +1,34 @@ +/* Check that LD_PREFER_MAP_32BIT_EXEC works in PIE and shared library. + Copyright (C) 2022 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +extern void dso_check_map_32bit (void); + +static int +do_test (void) +{ + printf ("do_test: %p\n", do_test); + TEST_VERIFY ((uintptr_t) do_test < 0xffffffffUL); + dso_check_map_32bit (); + return 0; +} + +#include diff --git a/sysdeps/x86/cpu-tunables.c b/sysdeps/x86/cpu-tunables.c index 94f4fbf243..8b4dfaa5b3 100644 --- a/sysdeps/x86/cpu-tunables.c +++ b/sysdeps/x86/cpu-tunables.c @@ -259,6 +259,13 @@ TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *valp) 20); } break; + case 21: + { + CHECK_GLIBC_IFUNC_PREFERRED_BOTH (n, cpu_features, + Prefer_MAP_32BIT_EXEC, + disable, 21); + } + break; case 23: { CHECK_GLIBC_IFUNC_PREFERRED_NEED_BOTH diff --git a/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def b/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def index 0e9090e74b..2b2c884be9 100644 --- a/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def +++ b/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def @@ -26,6 +26,7 @@ BIT (I586) BIT (I686) BIT (Slow_SSE4_2) BIT (AVX_Fast_Unaligned_Load) +BIT (Prefer_MAP_32BIT_EXEC) BIT (Prefer_No_VZEROUPPER) BIT (Prefer_ERMS) BIT (Prefer_No_AVX512)