[v2,0/4] Improved ASLR

  Problem with using sbrk() for allocations is that the location of the
memory is relatively predicatable since it's always located next to
data segment. This series makes the tunables system, malloc() and TCB
use mmap() instead, except when instructed by tunable
glibc.malloc.use_sbrk.

In this version, mmap() is also used for temporary storage for
tunables environment variable. Since the tunable to select using
sbrk() is unavailable at that point of time, mmap() is always
used. mmap() and mmap_noerrno() (other functions use this suffix) have
been refactored (Adhemerval Zanella), there's also a version for Hurd.

Topi Miettinen (4):
  csu: randomize location of TCB
  malloc: use mmap() to improve ASLR
  dl-sysdep: disable remaining calls to sbrk()
  tunables: use mmap() instead of sbrk()

 csu/libc-tls.c                               | 40 ++++++++++++++++----
 elf/dl-sysdep.c                              | 11 +++++-
 elf/dl-tunables.c                            |  9 +++--
 elf/dl-tunables.list                         |  7 ++++
 include/sys/mman.h                           |  5 +++
 malloc/arena.c                               | 11 +++++-
 malloc/morecore.c                            | 10 +++++
 manual/tunables.texi                         |  5 +++
 sysdeps/mach/hurd/dl-sysdep.c                | 18 +++++++--
 sysdeps/unix/sysv/linux/dl-sysdep.c          | 10 +++++
 sysdeps/unix/sysv/linux/mmap.c               | 30 ++++++++++++---
 sysdeps/unix/sysv/linux/mmap64.c             | 23 ++++++++---
 sysdeps/unix/sysv/linux/mmap_internal.h      |  2 +-
 sysdeps/unix/sysv/linux/s390/mmap_internal.h |  2 +-
 14 files changed, 154 insertions(+), 29 deletions(-)


base-commit: aa69f19a937b679816ef10e8620ea1141bb1734b
  

On Sat, Nov 28, 2020 at 01:59:41PM +0200, Topi Miettinen via Libc-alpha wrote:
> Problem with using sbrk() for allocations is that the location of the
> memory is relatively predicatable since it's always located next to
> data segment. This series makes the tunables system, malloc() and TCB
> use mmap() instead, except when instructed by tunable
> glibc.malloc.use_sbrk.

The above description is contrary to present reality on Linux. With
kernel.randomize_va_space=2 (default), the brk area starts at a
randomize gap above end of data/bss. This is *stronger* ASLR than
mmap, which aside from the initial gap, generally appears just below
the previous map and thereby at a predictable offset from an anchor in
a shared library.

Rich


> In this version, mmap() is also used for temporary storage for
> tunables environment variable. Since the tunable to select using
> sbrk() is unavailable at that point of time, mmap() is always
> used. mmap() and mmap_noerrno() (other functions use this suffix) have
> been refactored (Adhemerval Zanella), there's also a version for Hurd.
> 
> Topi Miettinen (4):
>   csu: randomize location of TCB
>   malloc: use mmap() to improve ASLR
>   dl-sysdep: disable remaining calls to sbrk()
>   tunables: use mmap() instead of sbrk()
> 
>  csu/libc-tls.c                               | 40 ++++++++++++++++----
>  elf/dl-sysdep.c                              | 11 +++++-
>  elf/dl-tunables.c                            |  9 +++--
>  elf/dl-tunables.list                         |  7 ++++
>  include/sys/mman.h                           |  5 +++
>  malloc/arena.c                               | 11 +++++-
>  malloc/morecore.c                            | 10 +++++
>  manual/tunables.texi                         |  5 +++
>  sysdeps/mach/hurd/dl-sysdep.c                | 18 +++++++--
>  sysdeps/unix/sysv/linux/dl-sysdep.c          | 10 +++++
>  sysdeps/unix/sysv/linux/mmap.c               | 30 ++++++++++++---
>  sysdeps/unix/sysv/linux/mmap64.c             | 23 ++++++++---
>  sysdeps/unix/sysv/linux/mmap_internal.h      |  2 +-
>  sysdeps/unix/sysv/linux/s390/mmap_internal.h |  2 +-
>  14 files changed, 154 insertions(+), 29 deletions(-)
> 
> 
> base-commit: aa69f19a937b679816ef10e8620ea1141bb1734b
> -- 
> 2.29.2
  

On 3.12.2020 1.09, Rich Felker wrote:
> On Sat, Nov 28, 2020 at 01:59:41PM +0200, Topi Miettinen via Libc-alpha wrote:
>> Problem with using sbrk() for allocations is that the location of the
>> memory is relatively predicatable since it's always located next to
>> data segment. This series makes the tunables system, malloc() and TCB
>> use mmap() instead, except when instructed by tunable
>> glibc.malloc.use_sbrk.
> 
> The above description is contrary to present reality on Linux. With
> kernel.randomize_va_space=2 (default), the brk area starts at a
> randomize gap above end of data/bss. This is *stronger* ASLR than
> mmap, which aside from the initial gap, generally appears just below
> the previous map and thereby at a predictable offset from an anchor in
> a shared library.

Thanks for the review. I'd note that the randomization for brk is only 
12 bits, so it's still relatively predictable. With 
randomize_va_space=3, the randomization provided by mmap() will be much 
better, the maximum allowed by the kernel design. The best in all cases 
would be if randomize_va_space=3 could be detected by libc and then it 
would change the default choice between brk() and mmap() automatically, 
or libc could generate a random address by itself and use 
mmap(random_addr,, MAP_FIXED_NOREPLACE) instead of brk(). For example, 
kernel could use aux vectors to pass the information.

-Topi

> 
> Rich
> 
> 
>> In this version, mmap() is also used for temporary storage for
>> tunables environment variable. Since the tunable to select using
>> sbrk() is unavailable at that point of time, mmap() is always
>> used. mmap() and mmap_noerrno() (other functions use this suffix) have
>> been refactored (Adhemerval Zanella), there's also a version for Hurd.
>>
>> Topi Miettinen (4):
>>    csu: randomize location of TCB
>>    malloc: use mmap() to improve ASLR
>>    dl-sysdep: disable remaining calls to sbrk()
>>    tunables: use mmap() instead of sbrk()
>>
>>   csu/libc-tls.c                               | 40 ++++++++++++++++----
>>   elf/dl-sysdep.c                              | 11 +++++-
>>   elf/dl-tunables.c                            |  9 +++--
>>   elf/dl-tunables.list                         |  7 ++++
>>   include/sys/mman.h                           |  5 +++
>>   malloc/arena.c                               | 11 +++++-
>>   malloc/morecore.c                            | 10 +++++
>>   manual/tunables.texi                         |  5 +++
>>   sysdeps/mach/hurd/dl-sysdep.c                | 18 +++++++--
>>   sysdeps/unix/sysv/linux/dl-sysdep.c          | 10 +++++
>>   sysdeps/unix/sysv/linux/mmap.c               | 30 ++++++++++++---
>>   sysdeps/unix/sysv/linux/mmap64.c             | 23 ++++++++---
>>   sysdeps/unix/sysv/linux/mmap_internal.h      |  2 +-
>>   sysdeps/unix/sysv/linux/s390/mmap_internal.h |  2 +-
>>   14 files changed, 154 insertions(+), 29 deletions(-)
>>
>>
>> base-commit: aa69f19a937b679816ef10e8620ea1141bb1734b
>> -- 
>> 2.29.2
  

On Thu, Dec 03, 2020 at 10:43:29AM +0200, Topi Miettinen wrote:
> On 3.12.2020 1.09, Rich Felker wrote:
> >On Sat, Nov 28, 2020 at 01:59:41PM +0200, Topi Miettinen via Libc-alpha wrote:
> >>Problem with using sbrk() for allocations is that the location of the
> >>memory is relatively predicatable since it's always located next to
> >>data segment. This series makes the tunables system, malloc() and TCB
> >>use mmap() instead, except when instructed by tunable
> >>glibc.malloc.use_sbrk.
> >
> >The above description is contrary to present reality on Linux. With
> >kernel.randomize_va_space=2 (default), the brk area starts at a
> >randomize gap above end of data/bss. This is *stronger* ASLR than
> >mmap, which aside from the initial gap, generally appears just below
> >the previous map and thereby at a predictable offset from an anchor in
> >a shared library.
> 
> Thanks for the review. I'd note that the randomization for brk is
> only 12 bits, so it's still relatively predictable. With
> randomize_va_space=3, the randomization provided by mmap() will be
> much better, the maximum allowed by the kernel design. The best in
> all cases would be if randomize_va_space=3 could be detected by libc
> and then it would change the default choice between brk() and mmap()
> automatically, or libc could generate a random address by itself and
> use mmap(random_addr,, MAP_FIXED_NOREPLACE) instead of brk(). For
> example, kernel could use aux vectors to pass the information.

Why isn't the proposed randomize_va_space=3 kernel patch just
using a completely random base for brk region (and reserving at least
a few GB for it to grow into)? Then all existing programs would get
the benefit.

Rich
  

On 3.12.2020 17.26, Rich Felker wrote:
> On Thu, Dec 03, 2020 at 10:43:29AM +0200, Topi Miettinen wrote:
>> On 3.12.2020 1.09, Rich Felker wrote:
>>> On Sat, Nov 28, 2020 at 01:59:41PM +0200, Topi Miettinen via Libc-alpha wrote:
>>>> Problem with using sbrk() for allocations is that the location of the
>>>> memory is relatively predicatable since it's always located next to
>>>> data segment. This series makes the tunables system, malloc() and TCB
>>>> use mmap() instead, except when instructed by tunable
>>>> glibc.malloc.use_sbrk.
>>>
>>> The above description is contrary to present reality on Linux. With
>>> kernel.randomize_va_space=2 (default), the brk area starts at a
>>> randomize gap above end of data/bss. This is *stronger* ASLR than
>>> mmap, which aside from the initial gap, generally appears just below
>>> the previous map and thereby at a predictable offset from an anchor in
>>> a shared library.
>>
>> Thanks for the review. I'd note that the randomization for brk is
>> only 12 bits, so it's still relatively predictable. With
>> randomize_va_space=3, the randomization provided by mmap() will be
>> much better, the maximum allowed by the kernel design. The best in
>> all cases would be if randomize_va_space=3 could be detected by libc
>> and then it would change the default choice between brk() and mmap()
>> automatically, or libc could generate a random address by itself and
>> use mmap(random_addr,, MAP_FIXED_NOREPLACE) instead of brk(). For
>> example, kernel could use aux vectors to pass the information.
> 
> Why isn't the proposed randomize_va_space=3 kernel patch just
> using a completely random base for brk region (and reserving at least
> a few GB for it to grow into)? Then all existing programs would get
> the benefit.

Great idea! I'll try to add that to the patch set.

-Topi
  

On Thu, Dec 03, 2020 at 06:05:56PM +0200, Topi Miettinen wrote:
> On 3.12.2020 17.26, Rich Felker wrote:
> >On Thu, Dec 03, 2020 at 10:43:29AM +0200, Topi Miettinen wrote:
> >>On 3.12.2020 1.09, Rich Felker wrote:
> >>>On Sat, Nov 28, 2020 at 01:59:41PM +0200, Topi Miettinen via Libc-alpha wrote:
> >>>>Problem with using sbrk() for allocations is that the location of the
> >>>>memory is relatively predicatable since it's always located next to
> >>>>data segment. This series makes the tunables system, malloc() and TCB
> >>>>use mmap() instead, except when instructed by tunable
> >>>>glibc.malloc.use_sbrk.
> >>>
> >>>The above description is contrary to present reality on Linux. With
> >>>kernel.randomize_va_space=2 (default), the brk area starts at a
> >>>randomize gap above end of data/bss. This is *stronger* ASLR than
> >>>mmap, which aside from the initial gap, generally appears just below
> >>>the previous map and thereby at a predictable offset from an anchor in
> >>>a shared library.
> >>
> >>Thanks for the review. I'd note that the randomization for brk is
> >>only 12 bits, so it's still relatively predictable. With
> >>randomize_va_space=3, the randomization provided by mmap() will be
> >>much better, the maximum allowed by the kernel design. The best in
> >>all cases would be if randomize_va_space=3 could be detected by libc
> >>and then it would change the default choice between brk() and mmap()
> >>automatically, or libc could generate a random address by itself and
> >>use mmap(random_addr,, MAP_FIXED_NOREPLACE) instead of brk(). For
> >>example, kernel could use aux vectors to pass the information.
> >
> >Why isn't the proposed randomize_va_space=3 kernel patch just
> >using a completely random base for brk region (and reserving at least
> >a few GB for it to grow into)? Then all existing programs would get
> >the benefit.
> 
> Great idea! I'll try to add that to the patch set.

Great! FYI we also use brk in musl libc's mallocng as a source of
memory for metadata areas that are expected not the be at predictable
offset from any module's text or data segment or from heap allocations
(which all come from mmap), so increased randomization for its
location will help a lot.

As a related aside, you probably need to recommend increasing
vm.max_map_count with kernel.randomize_va_space==3. Otherwise it
becomes pretty easy to hit the default VMA limit of 64k.

Rich
  

On 03/12/2020 13:05, Topi Miettinen via Libc-alpha wrote:
> On 3.12.2020 17.26, Rich Felker wrote:
>> On Thu, Dec 03, 2020 at 10:43:29AM +0200, Topi Miettinen wrote:
>>> On 3.12.2020 1.09, Rich Felker wrote:
>>>> On Sat, Nov 28, 2020 at 01:59:41PM +0200, Topi Miettinen via Libc-alpha wrote:
>>>>> Problem with using sbrk() for allocations is that the location of the
>>>>> memory is relatively predicatable since it's always located next to
>>>>> data segment. This series makes the tunables system, malloc() and TCB
>>>>> use mmap() instead, except when instructed by tunable
>>>>> glibc.malloc.use_sbrk.
>>>>
>>>> The above description is contrary to present reality on Linux. With
>>>> kernel.randomize_va_space=2 (default), the brk area starts at a
>>>> randomize gap above end of data/bss. This is *stronger* ASLR than
>>>> mmap, which aside from the initial gap, generally appears just below
>>>> the previous map and thereby at a predictable offset from an anchor in
>>>> a shared library.
>>>
>>> Thanks for the review. I'd note that the randomization for brk is
>>> only 12 bits, so it's still relatively predictable. With
>>> randomize_va_space=3, the randomization provided by mmap() will be
>>> much better, the maximum allowed by the kernel design. The best in
>>> all cases would be if randomize_va_space=3 could be detected by libc
>>> and then it would change the default choice between brk() and mmap()
>>> automatically, or libc could generate a random address by itself and
>>> use mmap(random_addr,, MAP_FIXED_NOREPLACE) instead of brk(). For
>>> example, kernel could use aux vectors to pass the information.
>>
>> Why isn't the proposed randomize_va_space=3 kernel patch just
>> using a completely random base for brk region (and reserving at least
>> a few GB for it to grow into)? Then all existing programs would get
>> the benefit.
> 
> Great idea! I'll try to add that to the patch set.

Which the sbrk entropy increase, will use mmap on TCB allocation add 
any security gain?
  

On 04/12/2020 07:03, Topi Miettinen wrote:
> On 3.12.2020 22.33, Adhemerval Zanella wrote:
>>
>>
>> On 03/12/2020 13:05, Topi Miettinen via Libc-alpha wrote:
>>> On 3.12.2020 17.26, Rich Felker wrote:
>>>> On Thu, Dec 03, 2020 at 10:43:29AM +0200, Topi Miettinen wrote:
>>>>> On 3.12.2020 1.09, Rich Felker wrote:
>>>>>> On Sat, Nov 28, 2020 at 01:59:41PM +0200, Topi Miettinen via Libc-alpha wrote:
>>>>>>> Problem with using sbrk() for allocations is that the location of the
>>>>>>> memory is relatively predicatable since it's always located next to
>>>>>>> data segment. This series makes the tunables system, malloc() and TCB
>>>>>>> use mmap() instead, except when instructed by tunable
>>>>>>> glibc.malloc.use_sbrk.
>>>>>>
>>>>>> The above description is contrary to present reality on Linux. With
>>>>>> kernel.randomize_va_space=2 (default), the brk area starts at a
>>>>>> randomize gap above end of data/bss. This is *stronger* ASLR than
>>>>>> mmap, which aside from the initial gap, generally appears just below
>>>>>> the previous map and thereby at a predictable offset from an anchor in
>>>>>> a shared library.
>>>>>
>>>>> Thanks for the review. I'd note that the randomization for brk is
>>>>> only 12 bits, so it's still relatively predictable. With
>>>>> randomize_va_space=3, the randomization provided by mmap() will be
>>>>> much better, the maximum allowed by the kernel design. The best in
>>>>> all cases would be if randomize_va_space=3 could be detected by libc
>>>>> and then it would change the default choice between brk() and mmap()
>>>>> automatically, or libc could generate a random address by itself and
>>>>> use mmap(random_addr,, MAP_FIXED_NOREPLACE) instead of brk(). For
>>>>> example, kernel could use aux vectors to pass the information.
>>>>
>>>> Why isn't the proposed randomize_va_space=3 kernel patch just
>>>> using a completely random base for brk region (and reserving at least
>>>> a few GB for it to grow into)? Then all existing programs would get
>>>> the benefit.
>>>
>>> Great idea! I'll try to add that to the patch set.
>>
>> Which the sbrk entropy increase, will mmap use on TCB allocation add
>> any security gain?
> 
> I think the biggest improvement comes from randomizing the location either by improving sbrk() or mmap(). Using mmap() for TCB gives some additional improvement that the area will be more or less separate from the other allocations. Analysis of the effect is complicated by the actual usage of randomize_va_space, either by kernel default setting, more importantly what the distros will actually use and finally the end users may be enlightened enough to change the setting. Also maximum randomization may be too aggressive for 32 bit architectures because of fragmentation of the virtual memory.

If it could be mitigated by the kernel with a better sbrk entropy I 
would prefer than to add the tunable machinery on glibc.  Also, tunables
is not meant to be deployment mechanism on a security feature where
the idea is to set if globally (and tunables also does not have a 
future-proof ABI).

We might try to use a different strategy and replace sbrk call with
mmap without adding a tunable, but it would add underlying mmap overhead
so I am not sure if it should be a default option.  The initial 
assessment seems that the overhead should be ok, but synthetic benchmark
can be misleading. We would need more data of the expected extra overhead
it might imposes (increase in mmap segments, performance on different
workloads, multithread costs, etc).

One option might a configure switch, which I am not very found as well
(it increases maintainability costs).
  

On 4.12.2020 15.04, Adhemerval Zanella wrote:
> 
> 
> On 04/12/2020 07:03, Topi Miettinen wrote:
>> On 3.12.2020 22.33, Adhemerval Zanella wrote:
>>>
>>>
>>> On 03/12/2020 13:05, Topi Miettinen via Libc-alpha wrote:
>>>> On 3.12.2020 17.26, Rich Felker wrote:
>>>>> On Thu, Dec 03, 2020 at 10:43:29AM +0200, Topi Miettinen wrote:
>>>>>> On 3.12.2020 1.09, Rich Felker wrote:
>>>>>>> On Sat, Nov 28, 2020 at 01:59:41PM +0200, Topi Miettinen via Libc-alpha wrote:
>>>>>>>> Problem with using sbrk() for allocations is that the location of the
>>>>>>>> memory is relatively predicatable since it's always located next to
>>>>>>>> data segment. This series makes the tunables system, malloc() and TCB
>>>>>>>> use mmap() instead, except when instructed by tunable
>>>>>>>> glibc.malloc.use_sbrk.
>>>>>>>
>>>>>>> The above description is contrary to present reality on Linux. With
>>>>>>> kernel.randomize_va_space=2 (default), the brk area starts at a
>>>>>>> randomize gap above end of data/bss. This is *stronger* ASLR than
>>>>>>> mmap, which aside from the initial gap, generally appears just below
>>>>>>> the previous map and thereby at a predictable offset from an anchor in
>>>>>>> a shared library.
>>>>>>
>>>>>> Thanks for the review. I'd note that the randomization for brk is
>>>>>> only 12 bits, so it's still relatively predictable. With
>>>>>> randomize_va_space=3, the randomization provided by mmap() will be
>>>>>> much better, the maximum allowed by the kernel design. The best in
>>>>>> all cases would be if randomize_va_space=3 could be detected by libc
>>>>>> and then it would change the default choice between brk() and mmap()
>>>>>> automatically, or libc could generate a random address by itself and
>>>>>> use mmap(random_addr,, MAP_FIXED_NOREPLACE) instead of brk(). For
>>>>>> example, kernel could use aux vectors to pass the information.
>>>>>
>>>>> Why isn't the proposed randomize_va_space=3 kernel patch just
>>>>> using a completely random base for brk region (and reserving at least
>>>>> a few GB for it to grow into)? Then all existing programs would get
>>>>> the benefit.
>>>>
>>>> Great idea! I'll try to add that to the patch set.
>>>
>>> Which the sbrk entropy increase, will mmap use on TCB allocation add
>>> any security gain?
>>
>> I think the biggest improvement comes from randomizing the location either by improving sbrk() or mmap(). Using mmap() for TCB gives some additional improvement that the area will be more or less separate from the other allocations. Analysis of the effect is complicated by the actual usage of randomize_va_space, either by kernel default setting, more importantly what the distros will actually use and finally the end users may be enlightened enough to change the setting. Also maximum randomization may be too aggressive for 32 bit architectures because of fragmentation of the virtual memory.
> 
> If it could be mitigated by the kernel with a better sbrk entropy I
> would prefer than to add the tunable machinery on glibc.  Also, tunables
> is not meant to be deployment mechanism on a security feature where
> the idea is to set if globally (and tunables also does not have a
> future-proof ABI).
> 
> We might try to use a different strategy and replace sbrk call with
> mmap without adding a tunable, but it would add underlying mmap overhead
> so I am not sure if it should be a default option.  The initial
> assessment seems that the overhead should be ok, but synthetic benchmark
> can be misleading. We would need more data of the expected extra overhead
> it might imposes (increase in mmap segments, performance on different
> workloads, multithread costs, etc).
> 
> One option might a configure switch, which I am not very found as well
> (it increases maintainability costs).

I've sent a new version (v8) of the kernel patch:
https://lkml.org/lkml/2020/12/20/132

In that version, also heap (memory allocated with brk()) is randomized, 
including also lowest bits 11 to 4. Then malloc() on unmodified glibc 
and musl would be randomized on systems which enable 
randomize_va_space=3. For other systems, it would be good if the low 
bits of address of the heap would be randomized by the dynamic loader by 
calling sbrk() with a small (up to page size, aligned to 16 bytes) 
increment.

Assuming that randomize_va_space=3 becomes mainstream (and the patch is 
accepted...), I think the new tunable for glibc and other changes to 
avoid sbrk() for TCB, dynamic loader and tunables aren't so interesting. 
It would be still good to use mmap() for malloc() since then all arenas 
would be separate from heap. Perhaps this could be selectable with 
mallopt() flag instead of a tunable? This could be also complemented by 
adding a small (aligned) random offset to the address given by mmap() 
for malloc() arenas.

-Topi