[v13,0/2] arm64: Enable BTI for the executable as well as the interpreter

  Deployments of BTI on arm64 have run into issues interacting with
systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
linked executables the kernel will only handle architecture specific
properties like BTI for the interpreter, the expectation is that the
interpreter will then handle any properties on the main executable.
For BTI this means remapping the executable segments PROT_EXEC |
PROT_BTI.

This interacts poorly with MemoryDenyWriteExecute since that is
implemented using a seccomp filter which prevents setting PROT_EXEC on
already mapped memory and lacks the context to be able to detect that
memory is already mapped with PROT_EXEC.  This series resolves this by
providing a sysctl which when enabled will cause the kernel to handle
the BTI property for both the interpreter and the main executable.

v13:
 - Rebase onto v5.18-rc3.
v12:
 - Rebase onto v5.18-rc1.
v11:
 - Ignore extra PT_INTERPs.
v10:
 - Add a sysctl abi.bti_main controlling the new behaviour.
v9:
 - Rebase onto v5.17-rc3.
v8:
 - Rebase onto v5.17-rc1.
v7:
 - Rebase onto v5.16-rc1.
v6:
 - Rebase onto v5.15-rc1.
v5:
 - Rebase onto v5.14-rc2.
 - Tweak changelog on patch 1.
 - Use the helper for interpreter/executable flag in elf.h as well.
v4:
 - Rebase onto v5.14-rc1.
v3:
 - Fix passing of properties for parsing by the main executable.
 - Drop has_interp from arch_parse_elf_property().
 - Coding style tweaks.
v2:
 - Add a patch dropping has_interp from arch_adjust_elf_prot()
 - Fix bisection issue with static executables on arm64 in the first
   patch.

Mark Brown (2):
  elf: Allow architectures to parse properties on the main executable
  arm64: Enable BTI for main executable as well as the interpreter

 arch/arm64/include/asm/elf.h | 14 ++++++++--
 arch/arm64/kernel/process.c  | 52 +++++++++++++++++++++++++++---------
 fs/binfmt_elf.c              | 34 ++++++++++++++++-------
 include/linux/elf.h          |  4 ++-
 4 files changed, 78 insertions(+), 26 deletions(-)


base-commit: b2d229d4ddb17db541098b83524d901257e93845
  

On Tue, 19 Apr 2022 11:51:54 +0100, Mark Brown wrote:
> Deployments of BTI on arm64 have run into issues interacting with
> systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> linked executables the kernel will only handle architecture specific
> properties like BTI for the interpreter, the expectation is that the
> interpreter will then handle any properties on the main executable.
> For BTI this means remapping the executable segments PROT_EXEC |
> PROT_BTI.
> 
> [...]

Applied to for-next/execve, thanks!

[1/2] elf: Allow architectures to parse properties on the main executable
      https://git.kernel.org/kees/c/b2f2553c8e89
[2/2] arm64: Enable BTI for main executable as well as the interpreter
      https://git.kernel.org/kees/c/b65c760600e2
  

On Tue, Apr 19, 2022 at 10:33:06PM -0700, Kees Cook wrote:
> On Tue, 19 Apr 2022 11:51:54 +0100, Mark Brown wrote:
> > Deployments of BTI on arm64 have run into issues interacting with
> > systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> > linked executables the kernel will only handle architecture specific
> > properties like BTI for the interpreter, the expectation is that the
> > interpreter will then handle any properties on the main executable.
> > For BTI this means remapping the executable segments PROT_EXEC |
> > PROT_BTI.
> > 
> > [...]
> 
> Applied to for-next/execve, thanks!
> 
> [1/2] elf: Allow architectures to parse properties on the main executable
>       https://git.kernel.org/kees/c/b2f2553c8e89
> [2/2] arm64: Enable BTI for main executable as well as the interpreter
>       https://git.kernel.org/kees/c/b65c760600e2

Kees, please can you drop this series while Catalin's alternative solution
is under discussion (his Reviewed-by preceded the other patches)?

https://lore.kernel.org/r/20220413134946.2732468-1-catalin.marinas@arm.com

Both series expose new behaviours to userspace and we don't need both.

Thanks,

Will
  

On Wed, Apr 20, 2022 at 10:36:13AM +0100, Will Deacon wrote:
> On Tue, Apr 19, 2022 at 10:33:06PM -0700, Kees Cook wrote:
> > On Tue, 19 Apr 2022 11:51:54 +0100, Mark Brown wrote:
> > > Deployments of BTI on arm64 have run into issues interacting with
> > > systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> > > linked executables the kernel will only handle architecture specific
> > > properties like BTI for the interpreter, the expectation is that the
> > > interpreter will then handle any properties on the main executable.
> > > For BTI this means remapping the executable segments PROT_EXEC |
> > > PROT_BTI.
> > > 
> > > [...]
> > 
> > Applied to for-next/execve, thanks!
> > 
> > [1/2] elf: Allow architectures to parse properties on the main executable
> >       https://git.kernel.org/kees/c/b2f2553c8e89
> > [2/2] arm64: Enable BTI for main executable as well as the interpreter
> >       https://git.kernel.org/kees/c/b65c760600e2
> 
> Kees, please can you drop this series while Catalin's alternative solution
> is under discussion (his Reviewed-by preceded the other patches)?
> 
> https://lore.kernel.org/r/20220413134946.2732468-1-catalin.marinas@arm.com
> 
> Both series expose new behaviours to userspace and we don't need both.

I agree. Even though the patches have my reviewed-by, I think we should
postpone them until we figure out a better W^X solution that does not
affect BTI (and if we can't, we revisit these patches).

Arguably, the two approaches are complementary but the way this series
turned out is for the BTI on main executable to be default off. I have a
worry that the feature won't get used, so we just carry unnecessary code
in the kernel. Jeremy also found this approach less than ideal:

https://lore.kernel.org/r/59fc8a58-5013-606b-f544-8277cda18e50@arm.com
  

On Wed, Apr 20, 2022 at 10:57:30AM +0100, Catalin Marinas wrote:
> On Wed, Apr 20, 2022 at 10:36:13AM +0100, Will Deacon wrote:

> > Kees, please can you drop this series while Catalin's alternative solution
> > is under discussion (his Reviewed-by preceded the other patches)?

> > https://lore.kernel.org/r/20220413134946.2732468-1-catalin.marinas@arm.com

> > Both series expose new behaviours to userspace and we don't need both.

> I agree. Even though the patches have my reviewed-by, I think we should
> postpone them until we figure out a better W^X solution that does not
> affect BTI (and if we can't, we revisit these patches).

Indeed.  I had been expecting this to follow the pattern of the previous
nine months or so and be mostly ignored for the time being while
Catalin's new series goes forward.  Now that it's applied it might be
worth keeping the first patch still in case someone else needs it but
the second patch can probably wait.

> Arguably, the two approaches are complementary but the way this series
> turned out is for the BTI on main executable to be default off. I have a
> worry that the feature won't get used, so we just carry unnecessary code
> in the kernel. Jeremy also found this approach less than ideal:

> https://lore.kernel.org/r/59fc8a58-5013-606b-f544-8277cda18e50@arm.com

I'm not sure there was a fundamental concern with the approach there but
rather some pushback on the instance on turning it off by default.
  

Hi,

On 4/20/22 06:57, Mark Brown wrote:
> On Wed, Apr 20, 2022 at 10:57:30AM +0100, Catalin Marinas wrote:
>> On Wed, Apr 20, 2022 at 10:36:13AM +0100, Will Deacon wrote:
> 
>>> Kees, please can you drop this series while Catalin's alternative solution
>>> is under discussion (his Reviewed-by preceded the other patches)?
> 
>>> https://lore.kernel.org/r/20220413134946.2732468-1-catalin.marinas@arm.com
> 
>>> Both series expose new behaviours to userspace and we don't need both.
> 
>> I agree. Even though the patches have my reviewed-by, I think we should
>> postpone them until we figure out a better W^X solution that does not
>> affect BTI (and if we can't, we revisit these patches).
> 
> Indeed.  I had been expecting this to follow the pattern of the previous
> nine months or so and be mostly ignored for the time being while
> Catalin's new series goes forward.  Now that it's applied it might be
> worth keeping the first patch still in case someone else needs it but
> the second patch can probably wait.
> 
>> Arguably, the two approaches are complementary but the way this series
>> turned out is for the BTI on main executable to be default off. I have a
>> worry that the feature won't get used, so we just carry unnecessary code
>> in the kernel. Jeremy also found this approach less than ideal:
> 
>> https://lore.kernel.org/r/59fc8a58-5013-606b-f544-8277cda18e50@arm.com
> 
> I'm not sure there was a fundamental concern with the approach there but
> rather some pushback on the instance on turning it off by default.

Right, this one seems to have the smallest impact on systemd as it 
exists today. I would have expected the default to be on, because IMHO 
this set corrects what at first glance just looks like a small 
oversight. I find the ABI questions a bit theoretical, given that this 
should only affect environments that don't exist outside of 
labs/development orgs at this point (aka systemd services on HW that 
implements BTI).


The other approach works, and if the systemd folks are on board with it 
also should solve the underlying problem, but it creates a bit of a 
compatibility problem with existing containers/etc that might exist 
today (although running systemd/services in a container is itself a 
discussion).

So, frankly, I don't see why they aren't complementary. This fixes a bug 
we have today, the other set creates a generic mechanism for the future.

Thanks,
  

On Wed, Apr 20, 2022 at 10:36:13AM +0100, Will Deacon wrote:
> On Tue, Apr 19, 2022 at 10:33:06PM -0700, Kees Cook wrote:
> > On Tue, 19 Apr 2022 11:51:54 +0100, Mark Brown wrote:
> > > Deployments of BTI on arm64 have run into issues interacting with
> > > systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> > > linked executables the kernel will only handle architecture specific
> > > properties like BTI for the interpreter, the expectation is that the
> > > interpreter will then handle any properties on the main executable.
> > > For BTI this means remapping the executable segments PROT_EXEC |
> > > PROT_BTI.
> > > 
> > > [...]
> > 
> > Applied to for-next/execve, thanks!
> > 
> > [1/2] elf: Allow architectures to parse properties on the main executable
> >       https://git.kernel.org/kees/c/b2f2553c8e89
> > [2/2] arm64: Enable BTI for main executable as well as the interpreter
> >       https://git.kernel.org/kees/c/b65c760600e2
> 
> Kees, please can you drop this series while Catalin's alternative solution
> is under discussion (his Reviewed-by preceded the other patches)?
> 
> https://lore.kernel.org/r/20220413134946.2732468-1-catalin.marinas@arm.com
> 
> Both series expose new behaviours to userspace and we don't need both.

Ah-ha! I wasn't sure if they were solving the same problem or not.
  

On Wed, Apr 20, 2022 at 08:39:14AM -0500, Jeremy Linton wrote:
> On 4/20/22 06:57, Mark Brown wrote:
> > On Wed, Apr 20, 2022 at 10:57:30AM +0100, Catalin Marinas wrote:
> > > On Wed, Apr 20, 2022 at 10:36:13AM +0100, Will Deacon wrote:
> > 
> > > > Kees, please can you drop this series while Catalin's alternative solution
> > > > is under discussion (his Reviewed-by preceded the other patches)?
> > 
> > > > https://lore.kernel.org/r/20220413134946.2732468-1-catalin.marinas@arm.com
> > 
> > > > Both series expose new behaviours to userspace and we don't need both.
> > 
> > > I agree. Even though the patches have my reviewed-by, I think we should
> > > postpone them until we figure out a better W^X solution that does not
> > > affect BTI (and if we can't, we revisit these patches).
> > 
> > Indeed.  I had been expecting this to follow the pattern of the previous
> > nine months or so and be mostly ignored for the time being while
> > Catalin's new series goes forward.  Now that it's applied it might be
> > worth keeping the first patch still in case someone else needs it but
> > the second patch can probably wait.
> > 
> > > Arguably, the two approaches are complementary but the way this series
> > > turned out is for the BTI on main executable to be default off. I have a
> > > worry that the feature won't get used, so we just carry unnecessary code
> > > in the kernel. Jeremy also found this approach less than ideal:
> > 
> > > https://lore.kernel.org/r/59fc8a58-5013-606b-f544-8277cda18e50@arm.com
> > 
> > I'm not sure there was a fundamental concern with the approach there but
> > rather some pushback on the instance on turning it off by default.
> 
> Right, this one seems to have the smallest impact on systemd as it exists
> today. I would have expected the default to be on, because IMHO this set
> corrects what at first glance just looks like a small oversight. I find the
> ABI questions a bit theoretical, given that this should only affect
> environments that don't exist outside of labs/development orgs at this point
> (aka systemd services on HW that implements BTI).
> 
> 
> The other approach works, and if the systemd folks are on board with it also
> should solve the underlying problem, but it creates a bit of a compatibility
> problem with existing containers/etc that might exist today (although
> running systemd/services in a container is itself a discussion).
> 
> So, frankly, I don't see why they aren't complementary. This fixes a bug we
> have today, the other set creates a generic mechanism for the future.

Okay, well, how about I drop this for now, and I'll Ack the ELF loader
changes so this can go through the arm64 tree if there is consensus.
  

On Tue, Apr 19, 2022 at 10:33:06PM -0700, Kees Cook wrote:
> On Tue, 19 Apr 2022 11:51:54 +0100, Mark Brown wrote:
> > Deployments of BTI on arm64 have run into issues interacting with
> > systemd's MemoryDenyWriteExecute feature.  Currently for dynamically
> > linked executables the kernel will only handle architecture specific
> > properties like BTI for the interpreter, the expectation is that the
> > interpreter will then handle any properties on the main executable.
> > For BTI this means remapping the executable segments PROT_EXEC |
> > PROT_BTI.
> > 
> > [...]
> 
> Applied to for-next/execve, thanks!

Now un-applied! :)
  

On Wed, Apr 20, 2022 at 08:39:14AM -0500, Jeremy Linton wrote:
> On 4/20/22 06:57, Mark Brown wrote:
> > On Wed, Apr 20, 2022 at 10:57:30AM +0100, Catalin Marinas wrote:
> > > On Wed, Apr 20, 2022 at 10:36:13AM +0100, Will Deacon wrote:
> > > > Kees, please can you drop this series while Catalin's alternative solution
> > > > is under discussion (his Reviewed-by preceded the other patches)?
> > 
> > > > https://lore.kernel.org/r/20220413134946.2732468-1-catalin.marinas@arm.com
> > 
> > > > Both series expose new behaviours to userspace and we don't need both.
[...]
> > > Arguably, the two approaches are complementary but the way this series
> > > turned out is for the BTI on main executable to be default off. I have a
> > > worry that the feature won't get used, so we just carry unnecessary code
> > > in the kernel. Jeremy also found this approach less than ideal:
> > 
> > > https://lore.kernel.org/r/59fc8a58-5013-606b-f544-8277cda18e50@arm.com
> > 
> > I'm not sure there was a fundamental concern with the approach there but
> > rather some pushback on the instance on turning it off by default.
> 
> Right, this one seems to have the smallest impact on systemd as it exists
> today.

It had a bigger impact on glibc which had to rework the dynamic library
mapping to use munmap/mmap() instead of an mprotect() (though that's
already done). I think glibc still prefers the mprotect() approach for
dynamic libraries.

> I would have expected the default to be on, because IMHO this set
> corrects what at first glance just looks like a small oversight.

This was a design decision at the time, maybe not the best but it gives
us some flexibility (and we haven't thought of MDWE).

> I find the ABI questions a bit theoretical, given that this should
> only affect environments that don't exist outside of labs/development
> orgs at this point (aka systemd services on HW that implements BTI).

The worry is not what breaks now but rather what happens when today's
distros will eventually be deployed on large-scale BTI-capable hardware.
It's a very small risk but non-zero. The idea is that if we come across
some weird problem, a fixed-up dynamic loader could avoid enabling BTI
on a per-process basis without the need to do this at the system level.

Personally I'm fine with this risk. Will is not and I respect his
position, hence I started the other thread to come up with a MDWE
alternative.

> The other approach works, and if the systemd folks are on board with it also
> should solve the underlying problem, but it creates a bit of a compatibility
> problem with existing containers/etc that might exist today (although
> running systemd/services in a container is itself a discussion).
> 
> So, frankly, I don't see why they aren't complementary.

They are complementary, though if we change the MDWE approach, there's
less of a need for this patchset.
  

Hi,

On 4/21/22 04:34, Catalin Marinas wrote:
> On Wed, Apr 20, 2022 at 08:39:14AM -0500, Jeremy Linton wrote:
>> On 4/20/22 06:57, Mark Brown wrote:
>>> On Wed, Apr 20, 2022 at 10:57:30AM +0100, Catalin Marinas wrote:
>>>> On Wed, Apr 20, 2022 at 10:36:13AM +0100, Will Deacon wrote:
>>>>> Kees, please can you drop this series while Catalin's alternative solution
>>>>> is under discussion (his Reviewed-by preceded the other patches)?
>>>
>>>>> https://lore.kernel.org/r/20220413134946.2732468-1-catalin.marinas@arm.com
>>>
>>>>> Both series expose new behaviours to userspace and we don't need both.
> [...]
>>>> Arguably, the two approaches are complementary but the way this series
>>>> turned out is for the BTI on main executable to be default off. I have a
>>>> worry that the feature won't get used, so we just carry unnecessary code
>>>> in the kernel. Jeremy also found this approach less than ideal:
>>>
>>>> https://lore.kernel.org/r/59fc8a58-5013-606b-f544-8277cda18e50@arm.com
>>>
>>> I'm not sure there was a fundamental concern with the approach there but
>>> rather some pushback on the instance on turning it off by default.
>>
>> Right, this one seems to have the smallest impact on systemd as it exists
>> today.
> 
> It had a bigger impact on glibc which had to rework the dynamic library
> mapping to use munmap/mmap() instead of an mprotect() (though that's
> already done). I think glibc still prefers the mprotect() approach for
> dynamic libraries.
> 
>> I would have expected the default to be on, because IMHO this set
>> corrects what at first glance just looks like a small oversight.
> 
> This was a design decision at the time, maybe not the best but it gives
> us some flexibility (and we haven't thought of MDWE).
> 
>> I find the ABI questions a bit theoretical, given that this should
>> only affect environments that don't exist outside of labs/development
>> orgs at this point (aka systemd services on HW that implements BTI).
> 
> The worry is not what breaks now but rather what happens when today's
> distros will eventually be deployed on large-scale BTI-capable hardware.
> It's a very small risk but non-zero. The idea is that if we come across
> some weird problem, a fixed-up dynamic loader could avoid enabling BTI
> on a per-process basis without the need to do this at the system level.
> 
> Personally I'm fine with this risk. Will is not and I respect his
> position, hence I started the other thread to come up with a MDWE
> alternative.

To clarify though, there is already a way to restore process 
functionality to the small subset of services with the MDWE enabled, 
which is simply to flip off MDWE on the service and let some future 
loader flag clear PROT_BTI in the code path it would normally be setting 
PROT_EXE|PROT_BIT on.

Or maybe simpler yet, we provide a tool which wipes out the gnu BTI note 
on binaries that are found to have BTI bugs, thereby (correctly) fixing 
the problem at its source. This is at least presumably doable if we are 
also assuming we can update glibc/etc in any environment with the problem.

But again, systemd MDWE + BTI are less than a dozen processes, and if we 
are worried about the big hammer of disabling BTI system wide, we 
probably should have the ability to disable it per-process rather than 
worrying about this obscure edge case.





> 
>> The other approach works, and if the systemd folks are on board with it also
>> should solve the underlying problem, but it creates a bit of a compatibility
>> problem with existing containers/etc that might exist today (although
>> running systemd/services in a container is itself a discussion).
>>
>> So, frankly, I don't see why they aren't complementary.
> 
> They are complementary, though if we change the MDWE approach, there's
> less of a need for this patchset.
>
  

On Thu, Apr 21, 2022 at 10:52:52AM -0500, Jeremy Linton wrote:

> Or maybe simpler yet, we provide a tool which wipes out the gnu BTI note on
> binaries that are found to have BTI bugs, thereby (correctly) fixing the
> problem at its source. This is at least presumably doable if we are also
> assuming we can update glibc/etc in any environment with the problem.

This seems like the most sensible thing if we do find we're running into
BTI executables that are incorrectly annotated and difficult to fix - it
avoids having to manage any new permissions for bypassing BTI.