x86-64: Restore LD_PREFER_MAP_32BIT_EXEC support [BZ #28656]
Checks
Context |
Check |
Description |
dj/TryBot-apply_patch |
success
|
Patch applied to master at the time it was sent
|
dj/TryBot-32bit |
success
|
Build for i686
|
Commit Message
Crossing 2GB boundaries with indirect calls and jumps can use more
branch prediction resources on Intel Golden Cove CPU (see the
"Misprediction for Branches >2GB" section in Intel 64 and IA-32
Architectures Optimization Reference Manual.) There is visible
performance improvement on workloads with many PLT calls when executable
and shared libraries are mmapped below 2GB. Add the Prefer_MAP_32BIT_EXEC
bit so that mmap will try to map executable or denywrite pages in shared
libraries with MAP_32BIT first.
NB: Prefer_MAP_32BIT_EXEC reduces bits available for address space
layout randomization (ASLR), which is always disabled for SUID programs
and can only be enabled by setting environment variable,
LD_PREFER_MAP_32BIT_EXEC. LD_PREFER_MAP_32BIT_EXEC works only between
shared libraries or between shared libraries and executables with
addresses below 2GB. PIEs are usually mapped above 4GB by the kernel.
---
elf/dl-support.c | 4 ++
elf/rtld.c | 18 +++++++-
sysdeps/generic/dl-librecon.h | 24 +++++++++++
sysdeps/unix/sysv/linux/x86_64/64/Makefile | 19 ++++++++
.../unix/sysv/linux/x86_64/64/dl-librecon.h | 43 +++++++++++++++++++
.../unix/sysv/linux/x86_64/64/mmap_internal.h | 43 +++++++++++++++++++
.../sysv/linux/x86_64/64/tst-map-32bit-mod.c | 33 ++++++++++++++
.../unix/sysv/linux/x86_64/64/tst-map-32bit.c | 34 +++++++++++++++
sysdeps/x86/cpu-tunables.c | 7 +++
...cpu-features-preferred_feature_index_1.def | 1 +
10 files changed, 225 insertions(+), 1 deletion(-)
create mode 100644 sysdeps/generic/dl-librecon.h
create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/dl-librecon.h
create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/mmap_internal.h
create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit-mod.c
create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/tst-map-32bit.c
Comments
* H. J. Lu via Libc-alpha:
> Crossing 2GB boundaries with indirect calls and jumps can use more
> branch prediction resources on Intel Golden Cove CPU (see the
> "Misprediction for Branches >2GB" section in Intel 64 and IA-32
> Architectures Optimization Reference Manual.) There is visible
> performance improvement on workloads with many PLT calls when executable
> and shared libraries are mmapped below 2GB. Add the Prefer_MAP_32BIT_EXEC
> bit so that mmap will try to map executable or denywrite pages in shared
> libraries with MAP_32BIT first.
>
> NB: Prefer_MAP_32BIT_EXEC reduces bits available for address space
> layout randomization (ASLR), which is always disabled for SUID programs
> and can only be enabled by setting environment variable,
> LD_PREFER_MAP_32BIT_EXEC. LD_PREFER_MAP_32BIT_EXEC works only between
> shared libraries or between shared libraries and executables with
> addresses below 2GB. PIEs are usually mapped above 4GB by the kernel.
I still think we should fix this in the kernel, using MAP_DENYWRITE as a
hint for placement. This way, it's easier to turn it on unconditionally
for the whole system because the lower 4 GiB will not be polluted by
code mappings.
Thanks,
Florian
On Thu, Feb 9, 2023 at 6:35 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> * H. J. Lu via Libc-alpha:
>
> > Crossing 2GB boundaries with indirect calls and jumps can use more
> > branch prediction resources on Intel Golden Cove CPU (see the
> > "Misprediction for Branches >2GB" section in Intel 64 and IA-32
> > Architectures Optimization Reference Manual.) There is visible
> > performance improvement on workloads with many PLT calls when executable
> > and shared libraries are mmapped below 2GB. Add the Prefer_MAP_32BIT_EXEC
> > bit so that mmap will try to map executable or denywrite pages in shared
> > libraries with MAP_32BIT first.
> >
> > NB: Prefer_MAP_32BIT_EXEC reduces bits available for address space
> > layout randomization (ASLR), which is always disabled for SUID programs
> > and can only be enabled by setting environment variable,
> > LD_PREFER_MAP_32BIT_EXEC. LD_PREFER_MAP_32BIT_EXEC works only between
> > shared libraries or between shared libraries and executables with
> > addresses below 2GB. PIEs are usually mapped above 4GB by the kernel.
>
> I still think we should fix this in the kernel, using MAP_DENYWRITE as a
> hint for placement. This way, it's easier to turn it on unconditionally
> for the whole system because the lower 4 GiB will not be polluted by
> code mappings.
>
Kernel MM change may take a long time and this mitigation may only be needed
for a few specific applications. On the other hand, it is simpler to
use tunable
instead.
Thanks.
On 2/10/23 11:50, H.J. Lu wrote:
> On Thu, Feb 9, 2023 at 6:35 AM Florian Weimer <fweimer@redhat.com> wrote:
>>
>> * H. J. Lu via Libc-alpha:
>>
>>> Crossing 2GB boundaries with indirect calls and jumps can use more
>>> branch prediction resources on Intel Golden Cove CPU (see the
>>> "Misprediction for Branches >2GB" section in Intel 64 and IA-32
>>> Architectures Optimization Reference Manual.) There is visible
>>> performance improvement on workloads with many PLT calls when executable
>>> and shared libraries are mmapped below 2GB. Add the Prefer_MAP_32BIT_EXEC
>>> bit so that mmap will try to map executable or denywrite pages in shared
>>> libraries with MAP_32BIT first.
>>>
>>> NB: Prefer_MAP_32BIT_EXEC reduces bits available for address space
>>> layout randomization (ASLR), which is always disabled for SUID programs
>>> and can only be enabled by setting environment variable,
>>> LD_PREFER_MAP_32BIT_EXEC. LD_PREFER_MAP_32BIT_EXEC works only between
>>> shared libraries or between shared libraries and executables with
>>> addresses below 2GB. PIEs are usually mapped above 4GB by the kernel.
>>
>> I still think we should fix this in the kernel, using MAP_DENYWRITE as a
>> hint for placement. This way, it's easier to turn it on unconditionally
>> for the whole system because the lower 4 GiB will not be polluted by
>> code mappings.
>>
>
> Kernel MM change may take a long time and this mitigation may only be needed
> for a few specific applications. On the other hand, it is simpler to
> use tunable
> instead.
I tend to agree with you here that a Kernel MM change is going to take too long and may
be possibly fragile.
I would like to see this use a tunable with an alias.
Note that the Linux man pages continue to document LD_PREFER_MAP_32BIT_EXEC :-)
On Fri, Feb 10, 2023 at 1:54 PM Carlos O'Donell <carlos@redhat.com> wrote:
>
> On 2/10/23 11:50, H.J. Lu wrote:
> > On Thu, Feb 9, 2023 at 6:35 AM Florian Weimer <fweimer@redhat.com> wrote:
> >>
> >> * H. J. Lu via Libc-alpha:
> >>
> >>> Crossing 2GB boundaries with indirect calls and jumps can use more
> >>> branch prediction resources on Intel Golden Cove CPU (see the
> >>> "Misprediction for Branches >2GB" section in Intel 64 and IA-32
> >>> Architectures Optimization Reference Manual.) There is visible
> >>> performance improvement on workloads with many PLT calls when executable
> >>> and shared libraries are mmapped below 2GB. Add the Prefer_MAP_32BIT_EXEC
> >>> bit so that mmap will try to map executable or denywrite pages in shared
> >>> libraries with MAP_32BIT first.
> >>>
> >>> NB: Prefer_MAP_32BIT_EXEC reduces bits available for address space
> >>> layout randomization (ASLR), which is always disabled for SUID programs
> >>> and can only be enabled by setting environment variable,
> >>> LD_PREFER_MAP_32BIT_EXEC. LD_PREFER_MAP_32BIT_EXEC works only between
> >>> shared libraries or between shared libraries and executables with
> >>> addresses below 2GB. PIEs are usually mapped above 4GB by the kernel.
> >>
> >> I still think we should fix this in the kernel, using MAP_DENYWRITE as a
> >> hint for placement. This way, it's easier to turn it on unconditionally
> >> for the whole system because the lower 4 GiB will not be polluted by
> >> code mappings.
> >>
> >
> > Kernel MM change may take a long time and this mitigation may only be needed
> > for a few specific applications. On the other hand, it is simpler to
> > use tunable
> > instead.
>
> I tend to agree with you here that a Kernel MM change is going to take too long and may
> be possibly fragile.
>
> I would like to see this use a tunable with an alias.
Done:
https://patchwork.sourceware.org/project/glibc/patch/20230210222142.17943-1-hjl.tools@gmail.com/
> Note that the Linux man pages continue to document LD_PREFER_MAP_32BIT_EXEC :-)
>
> --
> Cheers,
> Carlos.
>
Thanks.
@@ -35,6 +35,7 @@
#include <dl-machine.h>
#include <libc-lock.h>
#include <dl-cache.h>
+#include <dl-librecon.h>
#include <dl-procinfo.h>
#include <unsecvars.h>
#include <hp-timing.h>
@@ -320,6 +321,9 @@ _dl_non_dynamic_init (void)
{
static const char unsecure_envvars[] =
UNSECURE_ENVVARS
+#ifdef EXTRA_UNSECURE_ENVVARS
+ EXTRA_UNSECURE_ENVVARS
+#endif
;
const char *cp = unsecure_envvars;
@@ -32,6 +32,7 @@
#include <fpu_control.h>
#include <hp-timing.h>
#include <libc-lock.h>
+#include <dl-librecon.h>
#include <unsecvars.h>
#include <dl-cache.h>
#include <dl-osinfo.h>
@@ -2679,6 +2680,14 @@ process_envvars (struct dl_main_state *state)
= _dl_strtoul (&envline[21], NULL) > 1;
}
break;
+
+ /* We might have some extra environment variable to handle. This
+ is tricky due to the pre-processing of the length of the name
+ in the switch statement here. The code here assumes that added
+ environment variables have a different length. */
+#ifdef EXTRA_LD_ENVVARS
+ EXTRA_LD_ENVVARS
+#endif
}
}
@@ -2686,7 +2695,14 @@ process_envvars (struct dl_main_state *state)
variables. */
if (__glibc_unlikely (__libc_enable_secure))
{
- const char *nextp = UNSECURE_ENVVARS;
+ static const char unsecure_envvars[] =
+#ifdef EXTRA_UNSECURE_ENVVARS
+ EXTRA_UNSECURE_ENVVARS
+#endif
+ UNSECURE_ENVVARS;
+ const char *nextp;
+
+ nextp = unsecure_envvars;
do
{
unsetenv (nextp);
new file mode 100644
@@ -0,0 +1,24 @@
+/* Optional code to distinguish library flavours.
+ Copyright (C) 1998-2023 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#ifndef _DL_LIBRECON_H
+#define _DL_LIBRECON_H 1
+
+/* In the general case we don't do anything. */
+
+#endif /* dl-librecon.h */
@@ -1,2 +1,21 @@
# The default ABI is 64.
default-abi := 64
+
+ifeq ($(subdir),elf)
+
+tests-map-32bit = \
+ tst-map-32bit \
+# tests-map-32bit
+tst-map-32bit-no-pie = yes
+tests += $(tests-map-32bit)
+
+modules-map-32bit = \
+ tst-map-32bit-mod \
+# modules-map-32bit
+modules-names += $(modules-map-32bit)
+
+tst-map-32bit-ENV = LD_PREFER_MAP_32BIT_EXEC=1
+$(objpfx)tst-map-32bit-mod.so: $(libsupport)
+$(objpfx)tst-map-32bit: $(objpfx)tst-map-32bit-mod.so
+
+endif
new file mode 100644
@@ -0,0 +1,43 @@
+/* Optional code to distinguish library flavours. x86-64 version.
+ Copyright (C) 2015-2023 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#ifndef _DL_LIBRECON_H
+
+/* Recognizing extra environment variables. For 64-bit applications,
+ branch prediction performance may be negatively impacted when the
+ target of a branch is more than 4GB away from the branch. Add the
+ Prefer_MAP_32BIT_EXEC bit so that mmap will try to map executable
+ pages with MAP_32BIT first. NB: MAP_32BIT will map to lower 2GB,
+ not lower 4GB, address. Prefer_MAP_32BIT_EXEC reduces bits available
+ for address space layout randomization (ASLR). Prefer_MAP_32BIT_EXEC
+ is always disabled for SUID programs and can be enabled by setting
+ environment variable, LD_PREFER_MAP_32BIT_EXEC. */
+#define EXTRA_LD_ENVVARS \
+ case 21: \
+ if (!__libc_enable_secure \
+ && memcmp (envline, "PREFER_MAP_32BIT_EXEC", 21) == 0) \
+ GLRO(dl_x86_cpu_features).preferred[index_arch_Prefer_MAP_32BIT_EXEC] \
+ |= bit_arch_Prefer_MAP_32BIT_EXEC; \
+ break;
+
+/* Extra unsecure variables. The names are all stuffed in a single
+ string which means they have to be terminated with a '\0' explicitly. */
+#define EXTRA_UNSECURE_ENVVARS \
+ "LD_PREFER_MAP_32BIT_EXEC\0"
+
+#endif /* dl-librecon.h */
new file mode 100644
@@ -0,0 +1,43 @@
+/* Linux mmap system call. x86-64 version.
+ Copyright (C) 2015-2023 Free Software Foundation, Inc.
+
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public License as
+ published by the Free Software Foundation; either version 2.1 of the
+ License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#ifndef MMAP_X86_64_INTERNAL_H
+#define MMAP_X86_64_INTERNAL_H
+
+#include <ldsodefs.h>
+
+/* If the Prefer_MAP_32BIT_EXEC bit is set, try to map executable or
+ denywrite pages with MAP_32BIT first. */
+#define MMAP_PREPARE(addr, len, prot, flags, fd, offset) \
+ if ((addr) == NULL \
+ && (((prot) & PROT_EXEC) != 0 \
+ || ((flags) & MAP_DENYWRITE) != 0) \
+ && HAS_ARCH_FEATURE (Prefer_MAP_32BIT_EXEC)) \
+ { \
+ void *ret = (void*) INLINE_SYSCALL_CALL (mmap, (addr), (len), \
+ (prot), \
+ (flags) | MAP_32BIT, \
+ (fd), (offset)); \
+ if (ret != MAP_FAILED) \
+ return ret; \
+ }
+
+#include_next <mmap_internal.h>
+
+#endif
new file mode 100644
@@ -0,0 +1,33 @@
+/* Check that LD_PREFER_MAP_32BIT_EXEC works in shared library.
+ Copyright (C) 2023 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <support/check.h>
+
+static void
+dso_do_test (void)
+{
+}
+
+void
+dso_check_map_32bit (void)
+{
+ printf ("dso_do_test: %p\n", dso_do_test);
+ TEST_VERIFY ((uintptr_t) dso_do_test < 0xffffffffUL);
+}
new file mode 100644
@@ -0,0 +1,34 @@
+/* Check that LD_PREFER_MAP_32BIT_EXEC works in PDE and shared library.
+ Copyright (C) 2023 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <https://www.gnu.org/licenses/>. */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <support/check.h>
+
+extern void dso_check_map_32bit (void);
+
+static int
+do_test (void)
+{
+ printf ("do_test: %p\n", do_test);
+ TEST_VERIFY ((uintptr_t) do_test < 0xffffffffUL);
+ dso_check_map_32bit ();
+ return 0;
+}
+
+#include <support/test-driver.c>
@@ -259,6 +259,13 @@ TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *valp)
20);
}
break;
+ case 21:
+ {
+ CHECK_GLIBC_IFUNC_PREFERRED_BOTH (n, cpu_features,
+ Prefer_MAP_32BIT_EXEC,
+ disable, 21);
+ }
+ break;
case 23:
{
CHECK_GLIBC_IFUNC_PREFERRED_NEED_BOTH
@@ -26,6 +26,7 @@ BIT (I586)
BIT (I686)
BIT (Slow_SSE4_2)
BIT (AVX_Fast_Unaligned_Load)
+BIT (Prefer_MAP_32BIT_EXEC)
BIT (Prefer_No_VZEROUPPER)
BIT (Prefer_ERMS)
BIT (Prefer_No_AVX512)