From patchwork Tue Feb 21 21:52:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 65435 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 431A23858CDB for ; Tue, 21 Feb 2023 21:53:27 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 431A23858CDB DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1677016407; bh=dB+Vljg/AQFY8szzgNGdOydhCJqJ+qPdaCEyyXGcDp4=; h=To:Cc:Subject:Date:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From:Reply-To:From; b=se7lGZ/jskxIWbjZoQuWbaTzOd5k/FCwRHaBJw2OoHFQXi8Tl+kbn+M/WoqL6s16u VfPlS2FYetGBc6WWUe5xr6uEyXtv/RetM/ufxfq63TrCA1IExC4v5foBv51INMBh0h 0T8Ll/PksU1I7oNYbkSeK58CTwnhhxeOvvWAvV9U= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by sourceware.org (Postfix) with ESMTPS id 1E13B3858D37 for ; Tue, 21 Feb 2023 21:53:02 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1E13B3858D37 Received: by mail-pj1-x1030.google.com with SMTP id x20-20020a17090a8a9400b00233ba727724so15034pjn.1 for ; Tue, 21 Feb 2023 13:53:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=dB+Vljg/AQFY8szzgNGdOydhCJqJ+qPdaCEyyXGcDp4=; b=AL96OUqFn5+X6DqFJ0mPBzHo9vPlp/pQNrXecnJfrlfLp8olRaTC7c3mB47C9RfqwM u+YVLzGdR/ZcOBHBw5U8O+Q54iqJqpcBEenu2Zz2/QOsXGt/FEaHBLhAUNDYWGsUsmNh DKWJJ5CcOzekW+COV18JzejtDwnsb6TG0uhul8SdjhzNl2qtGo0IqZKvEIDvIjehTJSP Qr6N+JiBg3YM13JQCarOkVFpyBnljQOM96h4YvOWpulcD8D5NC+g+oSAa1wZ/seCYisq objsVeMoOV6CnlzdtABp1+g3yHM9GoSkyVJ+/+CQOgfw7UkikeJ2fgZDvsr8ggrVfn34 98lQ== X-Gm-Message-State: AO0yUKVa38zgC9IFuv2aDTY71bUWmkcZdPJd/6eomem7nc/LA0q2D23b QtUb+DtEzw1YsQrF7rbfrcU83OSDTNWElA== X-Google-Smtp-Source: AK7set9wpFccjpc5nrdqx3OdqIGn1dtkcNt1DTuAWGqfM3ebUTbFAWTmb+mGzZYyKeamUE9JKZwPbw== X-Received: by 2002:a05:6a20:698e:b0:c0:d8e1:82ae with SMTP id t14-20020a056a20698e00b000c0d8e182aemr7457841pzk.36.1677016380784; Tue, 21 Feb 2023 13:53:00 -0800 (PST) Received: from gnu-cfl-3.localdomain ([172.59.161.113]) by smtp.gmail.com with ESMTPSA id q22-20020a170902bd9600b001932a9e4f2csm2998033pls.255.2023.02.21.13.53.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Feb 2023 13:53:00 -0800 (PST) Received: from gnu-cfl-3.. (localhost [IPv6:::1]) by gnu-cfl-3.localdomain (Postfix) with ESMTP id 3F67874015F; Tue, 21 Feb 2023 13:52:59 -0800 (PST) To: libc-alpha@sourceware.org Cc: Carlos O'Donell , Florian Weimer Subject: [PATCH v3] x86-64: Add glibc.cpu.prefer_map_32bit_exec [BZ #28656] Date: Tue, 21 Feb 2023 13:52:59 -0800 Message-Id: <20230221215259.86835-1-hjl.tools@gmail.com> X-Mailer: git-send-email 2.39.1 MIME-Version: 1.0 X-Spam-Status: No, score=-3025.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Libc-alpha" From: "H.J. Lu" Reply-To: "H.J. Lu" Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Crossing 2GB boundaries with indirect calls and jumps can use more branch prediction resources on Intel Golden Cove CPU (see the "Misprediction for Branches >2GB" section in Intel 64 and IA-32 Architectures Optimization Reference Manual.) There is visible performance improvement on workloads with many PLT calls when executable and shared libraries are mmapped below 2GB. Add the Prefer_MAP_32BIT_EXEC bit so that mmap will try to map executable or denywrite pages in shared libraries with MAP_32BIT first. NB: Prefer_MAP_32BIT_EXEC reduces bits available for address space layout randomization (ASLR), which is always disabled for SUID programs and can only be enabled by the tunable, glibc.cpu.prefer_map_32bit_exec, or the environment variable, LD_PREFER_MAP_32BIT_EXEC. This works only between shared libraries or between shared libraries and executables with addresses below 2GB. PIEs are usually loaded at a random address above 4GB by the kernel. Reviewed-by: Carlos O'Donell --- manual/tunables.texi | 33 ++++++++++---- sysdeps/unix/sysv/linux/x86_64/64/Makefile | 25 +++++++++++ .../sysv/linux/x86_64/64/dl-tunables.list | 29 +++++++++++++ .../unix/sysv/linux/x86_64/64/mmap_internal.h | 43 +++++++++++++++++++ .../sysv/linux/x86_64/64/tst-map-32bit-1a.c | 34 +++++++++++++++ .../sysv/linux/x86_64/64/tst-map-32bit-1b.c | 1 + .../sysv/linux/x86_64/64/tst-map-32bit-mod.c | 33 ++++++++++++++ sysdeps/x86/cpu-features.c | 15 +++++++ ...cpu-features-preferred_feature_index_1.def | 1 + 9 files changed, 205 insertions(+), 9 deletions(-) create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/dl-tunables.list create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-1a.c create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-1b.c create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c diff --git a/manual/tunables.texi b/manual/tunables.texi index c2630b83ab..7dd3e9b791 100644 --- a/manual/tunables.texi +++ b/manual/tunables.texi @@ -35,27 +35,32 @@ tunables with minimum and maximum values: @example $ /lib64/ld-linux-x86-64.so.2 --list-tunables glibc.rtld.nns: 0x4 (min: 0x1, max: 0x10) -glibc.elision.skip_lock_after_retries: 3 (min: -2147483648, max: 2147483647) +glibc.elision.skip_lock_after_retries: 3 (min: 0, max: 2147483647) glibc.malloc.trim_threshold: 0x0 (min: 0x0, max: 0xffffffffffffffff) glibc.malloc.perturb: 0 (min: 0, max: 255) glibc.cpu.x86_shared_cache_size: 0x100000 (min: 0x0, max: 0xffffffffffffffff) +glibc.pthread.rseq: 1 (min: 0, max: 1) +glibc.cpu.prefer_map_32bit_exec: 0 (min: 0, max: 1) glibc.mem.tagging: 0 (min: 0, max: 255) -glibc.elision.tries: 3 (min: -2147483648, max: 2147483647) +glibc.elision.tries: 3 (min: 0, max: 2147483647) glibc.elision.enable: 0 (min: 0, max: 1) -glibc.cpu.x86_rep_movsb_threshold: 0x1000 (min: 0x100, max: 0xffffffffffffffff) +glibc.malloc.hugetlb: 0x0 (min: 0x0, max: 0xffffffffffffffff) +glibc.cpu.x86_rep_movsb_threshold: 0x2000 (min: 0x100, max: 0xffffffffffffffff) glibc.malloc.mxfast: 0x0 (min: 0x0, max: 0xffffffffffffffff) -glibc.elision.skip_lock_busy: 3 (min: -2147483648, max: 2147483647) -glibc.malloc.top_pad: 0x0 (min: 0x0, max: 0xffffffffffffffff) +glibc.rtld.dynamic_sort: 2 (min: 1, max: 2) +glibc.elision.skip_lock_busy: 3 (min: 0, max: 2147483647) +glibc.malloc.top_pad: 0x20000 (min: 0x0, max: 0xffffffffffffffff) glibc.cpu.x86_rep_stosb_threshold: 0x800 (min: 0x1, max: 0xffffffffffffffff) -glibc.cpu.x86_non_temporal_threshold: 0xc0000 (min: 0x4040, max: 0x0fffffffffffffff) +glibc.cpu.x86_non_temporal_threshold: 0xc0000 (min: 0x4040, max: 0xfffffffffffffff) glibc.cpu.x86_shstk: +glibc.pthread.stack_cache_size: 0x2800000 (min: 0x0, max: 0xffffffffffffffff) glibc.cpu.hwcap_mask: 0x6 (min: 0x0, max: 0xffffffffffffffff) -glibc.malloc.mmap_max: 0 (min: -2147483648, max: 2147483647) -glibc.elision.skip_trylock_internal_abort: 3 (min: -2147483648, max: 2147483647) +glibc.malloc.mmap_max: 0 (min: 0, max: 2147483647) +glibc.elision.skip_trylock_internal_abort: 3 (min: 0, max: 2147483647) glibc.malloc.tcache_unsorted_limit: 0x0 (min: 0x0, max: 0xffffffffffffffff) glibc.cpu.x86_ibt: glibc.cpu.hwcaps: -glibc.elision.skip_lock_internal_abort: 3 (min: -2147483648, max: 2147483647) +glibc.elision.skip_lock_internal_abort: 3 (min: 0, max: 2147483647) glibc.malloc.arena_max: 0x0 (min: 0x1, max: 0xffffffffffffffff) glibc.malloc.mmap_threshold: 0x0 (min: 0x0, max: 0xffffffffffffffff) glibc.cpu.x86_data_cache_size: 0x8000 (min: 0x0, max: 0xffffffffffffffff) @@ -580,6 +585,16 @@ instead. This tunable is specific to i386 and x86-64. @end deftp +@deftp Tunable glibc.cpu.prefer_map_32bit_exec +When this tunable is set to \code{1}, shared libraries of non-setuid +programs will be loaded below 2GB with MAP_32BIT. + +Note that the @env{LD_PREFER_MAP_32BIT_EXEC} environment is an alias of +this tunable. + +This tunable is specific to 64-bit x86-64. +@end deftp + @node Memory Related Tunables @section Memory Related Tunables @cindex memory related tunables diff --git a/sysdeps/unix/sysv/linux/x86_64/64/Makefile b/sysdeps/unix/sysv/linux/x86_64/64/Makefile index a7b6dc5a53..8ff4f27786 100644 --- a/sysdeps/unix/sysv/linux/x86_64/64/Makefile +++ b/sysdeps/unix/sysv/linux/x86_64/64/Makefile @@ -1,2 +1,27 @@ # The default ABI is 64. default-abi := 64 + +ifeq ($(subdir),elf) +ifneq ($(have-tunables),no) + +tests-map-32bit = \ + tst-map-32bit-1a \ + tst-map-32bit-1b \ +# tests-map-32bit +tst-map-32bit-1a-no-pie = yes +tst-map-32bit-1b-no-pie = yes +tests += $(tests-map-32bit) + +modules-map-32bit = \ + tst-map-32bit-mod \ +# modules-map-32bit +modules-names += $(modules-map-32bit) + +$(objpfx)tst-map-32bit-mod.so: $(libsupport) +tst-map-32bit-1a-ENV = LD_PREFER_MAP_32BIT_EXEC=1 +$(objpfx)tst-map-32bit-1a: $(objpfx)tst-map-32bit-mod.so +tst-map-32bit-1b-ENV = GLIBC_TUNABLES=glibc.cpu.prefer_map_32bit_exec=1 +$(objpfx)tst-map-32bit-1b: $(objpfx)tst-map-32bit-mod.so + +endif +endif diff --git a/sysdeps/unix/sysv/linux/x86_64/64/dl-tunables.list b/sysdeps/unix/sysv/linux/x86_64/64/dl-tunables.list new file mode 100644 index 0000000000..0aab52e662 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/dl-tunables.list @@ -0,0 +1,29 @@ +# x86-64 specific tunables. +# Copyright (C) 2023 Free Software Foundation, Inc. +# This file is part of the GNU C Library. + +# The GNU C Library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. + +# The GNU C Library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. + +# You should have received a copy of the GNU Lesser General Public +# License along with the GNU C Library; if not, see +# . + +glibc { + cpu { + prefer_map_32bit_exec { + type: INT_32 + minval: 0 + maxval: 1 + env_alias: LD_PREFER_MAP_32BIT_EXEC + security_level: SXID_IGNORE + } + } +} diff --git a/sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h b/sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h new file mode 100644 index 0000000000..33dec3f805 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h @@ -0,0 +1,43 @@ +/* Linux mmap system call. x86-64 version. + Copyright (C) 2015-2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef MMAP_X86_64_INTERNAL_H +#define MMAP_X86_64_INTERNAL_H + +#include + +/* If the Prefer_MAP_32BIT_EXEC bit is set, try to map executable or + denywrite pages with MAP_32BIT first. */ +#define MMAP_PREPARE(addr, len, prot, flags, fd, offset) \ + if ((addr) == NULL \ + && (((prot) & PROT_EXEC) != 0 \ + || ((flags) & MAP_DENYWRITE) != 0) \ + && HAS_ARCH_FEATURE (Prefer_MAP_32BIT_EXEC)) \ + { \ + void *ret = (void*) INLINE_SYSCALL_CALL (mmap, (addr), (len), \ + (prot), \ + (flags) | MAP_32BIT, \ + (fd), (offset)); \ + if (ret != MAP_FAILED) \ + return ret; \ + } + +#include_next + +#endif diff --git a/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-1a.c b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-1a.c new file mode 100644 index 0000000000..abc396589e --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-1a.c @@ -0,0 +1,34 @@ +/* Check that LD_PREFER_MAP_32BIT_EXEC works in PDE and shared library. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +extern void dso_check_map_32bit (void); + +static int +do_test (void) +{ + printf ("do_test: %p\n", do_test); + TEST_VERIFY ((uintptr_t) do_test < 0xffffffffUL); + dso_check_map_32bit (); + return 0; +} + +#include diff --git a/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-1b.c b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-1b.c new file mode 100644 index 0000000000..34ab01c773 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-1b.c @@ -0,0 +1 @@ +#include "tst-map-32bit-1a.c" diff --git a/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c new file mode 100644 index 0000000000..78d4b6133c --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c @@ -0,0 +1,33 @@ +/* Check that LD_PREFER_MAP_32BIT_EXEC works in shared library. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +static void +dso_do_test (void) +{ +} + +void +dso_check_map_32bit (void) +{ + printf ("dso_do_test: %p\n", dso_do_test); + TEST_VERIFY ((uintptr_t) dso_do_test < 0xffffffffUL); +} diff --git a/sysdeps/x86/cpu-features.c b/sysdeps/x86/cpu-features.c index a2197ed211..822688e21f 100644 --- a/sysdeps/x86/cpu-features.c +++ b/sysdeps/x86/cpu-features.c @@ -27,6 +27,16 @@ extern void TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *) attribute_hidden; +# ifdef __LP64__ +static void +TUNABLE_CALLBACK (set_prefer_map_32bit_exec) (tunable_val_t *valp) +{ + if (valp->numval) + GLRO(dl_x86_cpu_features).preferred[index_arch_Prefer_MAP_32BIT_EXEC] + |= bit_arch_Prefer_MAP_32BIT_EXEC; +} +# endif + # if CET_ENABLED extern void TUNABLE_CALLBACK (set_x86_ibt) (tunable_val_t *) attribute_hidden; @@ -705,6 +715,11 @@ no_cpuid: #if HAVE_TUNABLES TUNABLE_GET (hwcaps, tunable_val_t *, TUNABLE_CALLBACK (set_hwcaps)); +# ifdef __LP64__ + TUNABLE_GET (prefer_map_32bit_exec, tunable_val_t *, + TUNABLE_CALLBACK (set_prefer_map_32bit_exec)); +# endif + bool disable_xsave_features = false; if (!CPU_FEATURE_USABLE_P (cpu_features, OSXSAVE)) diff --git a/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def b/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def index e45f9cb159..d20c5b3196 100644 --- a/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def +++ b/sysdeps/x86/include/cpu-features-preferred_feature_index_1.def @@ -26,6 +26,7 @@ BIT (I586) BIT (I686) BIT (Slow_SSE4_2) BIT (AVX_Fast_Unaligned_Load) +BIT (Prefer_MAP_32BIT_EXEC) BIT (Prefer_No_VZEROUPPER) BIT (Prefer_ERMS) BIT (Prefer_No_AVX512)