[v8,0/4] arm64: Enable BTI for the executable as well as the interpreter

Deployments of BTI on arm64 have run into issues interacting with
systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
linked executables the kernel will only handle architecture specific
properties like BTI for the interpreter, the expectation is that the
interpreter will then handle any properties on the main executable.
For BTI this means remapping the executable segments PROT_EXEC |
PROT_BTI.

This interacts poorly with MemoryDenyWriteExecute since that is
implemented using a seccomp filter which prevents setting PROT_EXEC on
already mapped memory and lacks the context to be able to detect that
memory is already mapped with PROT_EXEC.  This series resolves this by
handling the BTI property for both the interpreter and the main
executable.

This does mean that we may get more code with BTI enabled if running on
a system without BTI support in the dynamic linker, this is expected to
be a safe configuration and testing seems to confirm that. It also
reduces the flexibility userspace has to disable BTI but it is expected
that for cases where there are problems which require BTI to be disabled
it is more likely that it will need to be disabled on a system level.

v8:
 - Rebase onto v5.17-rc1.
v7:
 - Rebase onto v5.16-rc1.
v6:
 - Rebase onto v5.15-rc1.
v5:
 - Rebase onto v5.14-rc2.
 - Tweak changelog on patch 1.
 - Use the helper for interpreter/executable flag in elf.h as well.
v4:
 - Rebase onto v5.14-rc1.
v3:
 - Fix passing of properties for parsing by the main executable.
 - Drop has_interp from arch_parse_elf_property().
 - Coding style tweaks.
v2:
 - Add a patch dropping has_interp from arch_adjust_elf_prot()
 - Fix bisection issue with static executables on arm64 in the first
   patch.

Mark Brown (4):
  elf: Allow architectures to parse properties on the main executable
  arm64: Enable BTI for main executable as well as the interpreter
  elf: Remove has_interp property from arch_adjust_elf_prot()
  elf: Remove has_interp property from arch_parse_elf_property()

 arch/arm64/include/asm/elf.h | 14 ++++++++++++--
 arch/arm64/kernel/process.c  | 16 +++-------------
 fs/binfmt_elf.c              | 29 ++++++++++++++++++++---------
 include/linux/elf.h          |  8 +++++---
 4 files changed, 40 insertions(+), 27 deletions(-)


base-commit: e783362eb54cd99b2cac8b3a9aeac942e6f6ac07

On Mon, Jan 24, 2022 at 03:07:00PM +0000, Mark Brown wrote:
> Deployments of BTI on arm64 have run into issues interacting with
> systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> linked executables the kernel will only handle architecture specific
> properties like BTI for the interpreter, the expectation is that the
> interpreter will then handle any properties on the main executable.
> For BTI this means remapping the executable segments PROT_EXEC |
> PROT_BTI.
> 
> This interacts poorly with MemoryDenyWriteExecute since that is
> implemented using a seccomp filter which prevents setting PROT_EXEC on
> already mapped memory and lacks the context to be able to detect that
> memory is already mapped with PROT_EXEC.  This series resolves this by
> handling the BTI property for both the interpreter and the main
> executable.

This appears to be a user-visible change which cannot be detected or
disabled from userspace. If there is code out there which does not work
when BTI is enabled, won't that now explode when the kernel enables it?
How are we supposed to handle such a regression?

Will

On Tue, Feb 15, 2022 at 06:34:56PM +0000, Will Deacon wrote:
> On Mon, Jan 24, 2022 at 03:07:00PM +0000, Mark Brown wrote:
> > Deployments of BTI on arm64 have run into issues interacting with
> > systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> > linked executables the kernel will only handle architecture specific
> > properties like BTI for the interpreter, the expectation is that the
> > interpreter will then handle any properties on the main executable.
> > For BTI this means remapping the executable segments PROT_EXEC |
> > PROT_BTI.
> > 
> > This interacts poorly with MemoryDenyWriteExecute since that is
> > implemented using a seccomp filter which prevents setting PROT_EXEC on
> > already mapped memory and lacks the context to be able to detect that
> > memory is already mapped with PROT_EXEC.  This series resolves this by
> > handling the BTI property for both the interpreter and the main
> > executable.
> 
> This appears to be a user-visible change which cannot be detected or
> disabled from userspace. If there is code out there which does not work
> when BTI is enabled, won't that now explode when the kernel enables it?
> How are we supposed to handle such a regression?

If this ever happens, the only workaround is to disable BTI on the
kernel command line. If we need a knob closer to user, we could add a
sysctl option (as we did for the tagged address ABI, though I doubt
people are even aware that exists). The dynamic loader doesn't do
anything smart when deciding to map objects with PROT_BTI (like env
variables), it simply relies on the ELF information.

I think that's very unlikely and feedback from Szabolcs in the past and
additional testing by Mark and Jeremy was that it should be fine. The
architecture allows interworking between BTI and non-BTI objects and on
distros with both BTI and MDWE enabled, this is already the case: the
main executable is mapped without PROT_BTI while the libraries will be
mapped with PROT_BTI. The new behaviour allows both to be mapped with
PROT_BTI, just as if MDWE was disabled.

I think the only difference would be with a BTI-unware dynamic loader
(e.g. older distro). Here the main executable, if compiled with BTI,
would be mapped as executable while the rest of the libraries are
non-BTI. The interworking should be fine but we can't test everything
since such BTI binaries would not normally be part of the distro.

If there are dodgy libraries out there that do tricks and branch into
the middle of a function in the main executable, they will fail with
this series but also fail if MDWE is disabled and the dynamic linker is
BTI-aware. So this hardly counts as a use-case.

For consistency, I think whoever does the initial mapping should also
set the correct attributes as we do for static binaries. If you think
another knob is needed other than the cmdline, I'm fine with it.

The 02/16/2022 13:34, Catalin Marinas via Libc-alpha wrote:
> On Tue, Feb 15, 2022 at 06:34:56PM +0000, Will Deacon wrote:
> > This appears to be a user-visible change which cannot be detected or
> > disabled from userspace. If there is code out there which does not work
> > when BTI is enabled, won't that now explode when the kernel enables it?
> > How are we supposed to handle such a regression?
> 
> If this ever happens, the only workaround is to disable BTI on the
> kernel command line. If we need a knob closer to user, we could add a
> sysctl option (as we did for the tagged address ABI, though I doubt
> people are even aware that exists). The dynamic loader doesn't do
> anything smart when deciding to map objects with PROT_BTI (like env
> variables), it simply relies on the ELF information.
> 
> I think that's very unlikely and feedback from Szabolcs in the past and
> additional testing by Mark and Jeremy was that it should be fine. The
> architecture allows interworking between BTI and non-BTI objects and on
> distros with both BTI and MDWE enabled, this is already the case: the
> main executable is mapped without PROT_BTI while the libraries will be
> mapped with PROT_BTI. The new behaviour allows both to be mapped with
> PROT_BTI, just as if MDWE was disabled.
> 
> I think the only difference would be with a BTI-unware dynamic loader
> (e.g. older distro). Here the main executable, if compiled with BTI,
> would be mapped as executable while the rest of the libraries are
> non-BTI. The interworking should be fine but we can't test everything
> since such BTI binaries would not normally be part of the distro.

note: a bti marked exe has to be built against a bti enabled
runtime (so crt files etc are bti compatible) and any system
that executes such an exe must be newer and thus bti aware
(according to glibc abi compatibility rules).

so while it's possible that ld.so is not bti marked when an
exe is, the ld.so will be at least bti aware, which means it
will map all bti binaries (including the exe) with PROT_BTI.

so this doesn't really affect glibc based systems. and there
are not many other systems that can produce bti marked exes.
(i don't think this can cause problems on android either)

if we ever wanted to map bti marked binaries without PROT_BTI
and introduced a knob to do that in ld.so, then this change
would be problematic (we cannot easily remove PROT_BTI from
the exe), but we don't have such plans.

> 
> If there are dodgy libraries out there that do tricks and branch into
> the middle of a function in the main executable, they will fail with
> this series but also fail if MDWE is disabled and the dynamic linker is
> BTI-aware. So this hardly counts as a use-case.
> 
> For consistency, I think whoever does the initial mapping should also
> set the correct attributes as we do for static binaries. If you think
> another knob is needed other than the cmdline, I'm fine with it.
> 
> -- 
> Catalin

On Wed, Feb 16, 2022 at 04:49:54PM +0000, Szabolcs Nagy wrote:

> if we ever wanted to map bti marked binaries without PROT_BTI
> and introduced a knob to do that in ld.so, then this change
> would be problematic (we cannot easily remove PROT_BTI from
> the exe), but we don't have such plans.

In general or only in the case where MWDE is enabled (in which case it's
the same problem as exists today trying to enable BTI in the presence of
MWDE)?

The 02/16/2022 17:01, Mark Brown wrote:
> On Wed, Feb 16, 2022 at 04:49:54PM +0000, Szabolcs Nagy wrote:
> 
> > if we ever wanted to map bti marked binaries without PROT_BTI
> > and introduced a knob to do that in ld.so, then this change
> > would be problematic (we cannot easily remove PROT_BTI from
> > the exe), but we don't have such plans.
> 
> In general or only in the case where MWDE is enabled (in which case it's
> the same problem as exists today trying to enable BTI in the presence of
> MWDE)?

ah yes with mwde only.

On Wed, Feb 16, 2022 at 01:34:25PM +0000, Catalin Marinas wrote:
> On Tue, Feb 15, 2022 at 06:34:56PM +0000, Will Deacon wrote:

> > This appears to be a user-visible change which cannot be detected or
> > disabled from userspace. If there is code out there which does not work
> > when BTI is enabled, won't that now explode when the kernel enables it?
> > How are we supposed to handle such a regression?

> If this ever happens, the only workaround is to disable BTI on the
> kernel command line. If we need a knob closer to user, we could add a
> sysctl option (as we did for the tagged address ABI, though I doubt
> people are even aware that exists). The dynamic loader doesn't do
> anything smart when deciding to map objects with PROT_BTI (like env
> variables), it simply relies on the ELF information.

The dynamic loader is the place where I'd expect to do any
per-executable workarounds, but currently that's not actually
implemented anywhere.  Someone could also make a tool to strip BTI
annotations from executables.

> I think the only difference would be with a BTI-unware dynamic loader
> (e.g. older distro). Here the main executable, if compiled with BTI,
> would be mapped as executable while the rest of the libraries are
> non-BTI. The interworking should be fine but we can't test everything
> since such BTI binaries would not normally be part of the distro.

> If there are dodgy libraries out there that do tricks and branch into
> the middle of a function in the main executable, they will fail with
> this series but also fail if MDWE is disabled and the dynamic linker is
> BTI-aware. So this hardly counts as a use-case.

I'm not aware of any issues we've run into which are due to interworking
between binaries rather than within a binary due to either miscompilation
or doing something in hand coded assembler that needs updating for BTI.
It doesn't mean it can't happen but it's hard to see what people might
be doing.

> For consistency, I think whoever does the initial mapping should also
> set the correct attributes as we do for static binaries. If you think
> another knob is needed other than the cmdline, I'm fine with it.

Might also be worth pointing out that we already map the vDSO with BTI
enabled if it's built with BTI.

On Wed, Feb 16, 2022 at 01:34:25PM +0000, Catalin Marinas wrote:
> On Tue, Feb 15, 2022 at 06:34:56PM +0000, Will Deacon wrote:
> > On Mon, Jan 24, 2022 at 03:07:00PM +0000, Mark Brown wrote:
> > > Deployments of BTI on arm64 have run into issues interacting with
> > > systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> > > linked executables the kernel will only handle architecture specific
> > > properties like BTI for the interpreter, the expectation is that the
> > > interpreter will then handle any properties on the main executable.
> > > For BTI this means remapping the executable segments PROT_EXEC |
> > > PROT_BTI.
> > > 
> > > This interacts poorly with MemoryDenyWriteExecute since that is
> > > implemented using a seccomp filter which prevents setting PROT_EXEC on
> > > already mapped memory and lacks the context to be able to detect that
> > > memory is already mapped with PROT_EXEC.  This series resolves this by
> > > handling the BTI property for both the interpreter and the main
> > > executable.
> > 
> > This appears to be a user-visible change which cannot be detected or
> > disabled from userspace. If there is code out there which does not work
> > when BTI is enabled, won't that now explode when the kernel enables it?
> > How are we supposed to handle such a regression?
> 
> If this ever happens, the only workaround is to disable BTI on the
> kernel command line. If we need a knob closer to user, we could add a
> sysctl option (as we did for the tagged address ABI, though I doubt
> people are even aware that exists). The dynamic loader doesn't do
> anything smart when deciding to map objects with PROT_BTI (like env
> variables), it simply relies on the ELF information.
> 
> I think that's very unlikely and feedback from Szabolcs in the past and
> additional testing by Mark and Jeremy was that it should be fine. The
> architecture allows interworking between BTI and non-BTI objects and on
> distros with both BTI and MDWE enabled, this is already the case: the
> main executable is mapped without PROT_BTI while the libraries will be
> mapped with PROT_BTI. The new behaviour allows both to be mapped with
> PROT_BTI, just as if MDWE was disabled.
> 
> I think the only difference would be with a BTI-unware dynamic loader
> (e.g. older distro). Here the main executable, if compiled with BTI,
> would be mapped as executable while the rest of the libraries are
> non-BTI. The interworking should be fine but we can't test everything
> since such BTI binaries would not normally be part of the distro.
> 
> If there are dodgy libraries out there that do tricks and branch into
> the middle of a function in the main executable, they will fail with
> this series but also fail if MDWE is disabled and the dynamic linker is
> BTI-aware. So this hardly counts as a use-case.
> 
> For consistency, I think whoever does the initial mapping should also
> set the correct attributes as we do for static binaries. If you think
> another knob is needed other than the cmdline, I'm fine with it.

I still think this new behaviour should be opt-in, so adding a sysctl for
that would be my preference if we proceed with this approach.

Will

On Fri, Feb 25, 2022 at 01:53:51PM +0000, Will Deacon wrote:

> I still think this new behaviour should be opt-in, so adding a sysctl for
> that would be my preference if we proceed with this approach.

I'm happy to have a sysctl but I'd rather it be opt out rather than opt
in since it seems better to default to enabling the security feature
when there is a strong expectation that it would seem better to enable
it by default sine it's not expected to be disruptive and the sysctl is
more of a "what if there's a problem" thing.

On Fri, Feb 25, 2022 at 03:11:43PM +0000, Mark Brown wrote:
> On Fri, Feb 25, 2022 at 01:53:51PM +0000, Will Deacon wrote:
> 
> > I still think this new behaviour should be opt-in, so adding a sysctl for
> > that would be my preference if we proceed with this approach.
> 
> I'm happy to have a sysctl but I'd rather it be opt out rather than opt
> in since it seems better to default to enabling the security feature
> when there is a strong expectation that it would seem better to enable
> it by default sine it's not expected to be disruptive and the sysctl is
> more of a "what if there's a problem" thing.

I think new behaviour has to be opt-in, so that if somebody takes a new
kernel then we can guarantee it's not going to break them. Systemd can
enable this unconditionally if it wants to.

Will