malloc: Do not trim when sysmalloc frees the old top chunk [BZ #34882]

Message ID CAD=yzkGxDL0b8HFW0T0rxJcd69R_gRzbo=-5_udJFmsOua4YUw@mail.gmail.com (mailing list archive)
State New
Headers
Series malloc: Do not trim when sysmalloc frees the old top chunk [BZ #34882] |

Checks

Context Check Description
redhat-pt-bot/TryBot-apply_patch success Patch applied to master at the time it was sent
linaro-tcwg-bot/tcwg_glibc_build--master-aarch64 success Build passed
linaro-tcwg-bot/tcwg_glibc_check--master-aarch64 success Test passed
linaro-tcwg-bot/tcwg_glibc_build--master-arm success Build passed
redhat-pt-bot/TryBot-32bit success Build for i686
linaro-tcwg-bot/tcwg_glibc_check--master-arm success Test passed

Commit Message

Danner Stodolsky Oct. 24, 2026, 2:05 a.m. UTC
  When sysmalloc cannot extend the top chunk in place, it obtains a new
top chunk and frees what is left of the old one. That free went through
_int_free_merge_chunk, which ends in _int_free_maybe_trim. If the old
top was at least ATTEMPT_TRIMMING_THRESHOLD bytes, the arena was trimmed
before the new top had been split for the pending request:

* Main arena, non-contiguous: if an earlier sbrk failed (for example
because a foreign mapping briefly sat at the program break) the arena
became non-contiguous via the mmap fallback. A later extension whose
sbrk succeeds frees the old top; systrim then shrinks the new top back
to about M_TOP_PAD bytes, the final size check fails, and malloc
returns NULL with errno ENOMEM. An immediate retry succeeds.

* Thread arena: when the current heap cannot grow, sysmalloc creates a
new heap and frees the old top. heap_trim then unmaps the new heap
again, _int_malloc fails, and the request is satisfied only because
arena_get_retry falls back to another arena.

With the default dynamic thresholds, M_TRIM_THRESHOLD is twice the mmap
threshold, so the new top is normally too small to be trimmed. Setting
M_MMAP_THRESHOLD (or MALLOC_MMAP_THRESHOLD_) leaves M_TRIM_THRESHOLD at
128 KiB, and any heap request between about 132 KiB and the mmap
threshold can then fail in the main arena.

dlmalloc 2.7.1 already suppressed trimming around this free ("Fix
freeing of old top non-contiguous chunk im sysmalloc"). Restore that
behavior by splitting _int_free_merge_chunk_notrim out of
_int_free_merge_chunk and using it for both old-top frees in sysmalloc.
The memory is still trimmed by the next free that reaches the top.

Add tst-malloc-sysmalloc-trim, which reproduces the main-arena case by
mapping a page range at the program break. It fails without this
change.

Tested on x86_64 (master fea6c3582ae1): the new test fails without the
fix and passes with it; make subdirs=malloc check shows no failures.


Signed-off-by: Danner Stodolsky <dstodolsky@bostondynamics.com>
  

Comments

Carlos O'Donell Oct. 26, 2026, 1:59 p.m. UTC | #1
On 10/23/26 10:05 PM, Danner Stodolsky wrote:
> When sysmalloc cannot extend the top chunk in place, it obtains a new
> top chunk and frees what is left of the old one. That free went through
> _int_free_merge_chunk, which ends in _int_free_maybe_trim. If the old
> top was at least ATTEMPT_TRIMMING_THRESHOLD bytes, the arena was trimmed
> before the new top had been split for the pending request:

Danner,

Thank you so much for reaching out to the developers regarding this issue.

This looks like a real problem that you found in production and we want
to address this. We reviewed this during our weekly patch queue review call
and Wilco Dijkstra will be looking into verifying the issue.

(1) LLM-derived contributions.

Currently glibc is working to review adoption of gcc's own AI policy, which
means we could accept test cases written by LLMs, but not for the core
implementation of malloc which can have an impact on the license of the
library itself. However, since your changes is roughly only 3 lines of code
and fairly straight forward, if I asked someone else to rewrite it by hand
we'd end up with pretty much the same solution.

In the future please consider that we're working to keep the implementation
of glibc free from LLM-derived contributions until we have clearer guidance
from the FSF or the GNU Project.

Please do stay engaged here since this looks like a real issue! :-)

(2) Backports.

Are you running an older version of glibc? How far back would you like to
backport this? If this issue is real we'd want to backport this into the
past as much as we can. 
> * Main arena, non-contiguous: if an earlier sbrk failed (for example
> because a foreign mapping briefly sat at the program break) the arena
> became non-contiguous via the mmap fallback. A later extension whose
> sbrk succeeds frees the old top; systrim then shrinks the new top back
> to about M_TOP_PAD bytes, the final size check fails, and malloc
> returns NULL with errno ENOMEM. An immediate retry succeeds.
> 
> * Thread arena: when the current heap cannot grow, sysmalloc creates a
> new heap and frees the old top. heap_trim then unmaps the new heap
> again, _int_malloc fails, and the request is satisfied only because
> arena_get_retry falls back to another arena.
> 
> With the default dynamic thresholds, M_TRIM_THRESHOLD is twice the mmap
> threshold, so the new top is normally too small to be trimmed. Setting
> M_MMAP_THRESHOLD (or MALLOC_MMAP_THRESHOLD_) leaves M_TRIM_THRESHOLD at
> 128 KiB, and any heap request between about 132 KiB and the mmap
> threshold can then fail in the main arena.
> 
> dlmalloc 2.7.1 already suppressed trimming around this free ("Fix
> freeing of old top non-contiguous chunk im sysmalloc"). Restore that
> behavior by splitting _int_free_merge_chunk_notrim out of
> _int_free_merge_chunk and using it for both old-top frees in sysmalloc.
> The memory is still trimmed by the next free that reaches the top.
> 
> Add tst-malloc-sysmalloc-trim, which reproduces the main-arena case by
> mapping a page range at the program break. It fails without this
> change.
> 
> Tested on x86_64 (master fea6c3582ae1): the new test fails without the
> fix and passes with it; make subdirs=malloc check shows no failures.
> 
> 
> Signed-off-by: Danner Stodolsky <dstodolsky@bostondynamics.com <mailto:dstodolsky@bostondynamics.com>>
  
Danner Stodolsky Oct. 26, 2026, 2:10 p.m. UTC | #2
Carlos -

Thanks for the quick response and clarification of the AI policy and having
someone rewrite those 3 lines.

We hit this in glibc 2.35 (Ubtuntu 22.04 LTS, almost out of support window)
but are moving to glibc 2.39 (Ubtuntu 24.04 LTS). We have not seen this in
production, and setting trim to 2x threshold is a straight-forward
workaround, so the backport isn't urgent for us. 2.39 would be appreciated.


On Mon, Oct 26, 2026 at 9:59 AM Carlos O'Donell <carlos@redhat.com> wrote:

> On 10/23/26 10:05 PM, Danner Stodolsky wrote:
> > When sysmalloc cannot extend the top chunk in place, it obtains a new
> > top chunk and frees what is left of the old one. That free went through
> > _int_free_merge_chunk, which ends in _int_free_maybe_trim. If the old
> > top was at least ATTEMPT_TRIMMING_THRESHOLD bytes, the arena was trimmed
> > before the new top had been split for the pending request:
>
> Danner,
>
> Thank you so much for reaching out to the developers regarding this issue.
>
> This looks like a real problem that you found in production and we want
> to address this. We reviewed this during our weekly patch queue review call
> and Wilco Dijkstra will be looking into verifying the issue.
>
> (1) LLM-derived contributions.
>
> Currently glibc is working to review adoption of gcc's own AI policy, which
> means we could accept test cases written by LLMs, but not for the core
> implementation of malloc which can have an impact on the license of the
> library itself. However, since your changes is roughly only 3 lines of code
> and fairly straight forward, if I asked someone else to rewrite it by hand
> we'd end up with pretty much the same solution.
>
> In the future please consider that we're working to keep the implementation
> of glibc free from LLM-derived contributions until we have clearer guidance
> from the FSF or the GNU Project.
>
> Please do stay engaged here since this looks like a real issue! :-)
>
> (2) Backports.
>
> Are you running an older version of glibc? How far back would you like to
> backport this? If this issue is real we'd want to backport this into the
> past as much as we can.
> > * Main arena, non-contiguous: if an earlier sbrk failed (for example
> > because a foreign mapping briefly sat at the program break) the arena
> > became non-contiguous via the mmap fallback. A later extension whose
> > sbrk succeeds frees the old top; systrim then shrinks the new top back
> > to about M_TOP_PAD bytes, the final size check fails, and malloc
> > returns NULL with errno ENOMEM. An immediate retry succeeds.
> >
> > * Thread arena: when the current heap cannot grow, sysmalloc creates a
> > new heap and frees the old top. heap_trim then unmaps the new heap
> > again, _int_malloc fails, and the request is satisfied only because
> > arena_get_retry falls back to another arena.
> >
> > With the default dynamic thresholds, M_TRIM_THRESHOLD is twice the mmap
> > threshold, so the new top is normally too small to be trimmed. Setting
> > M_MMAP_THRESHOLD (or MALLOC_MMAP_THRESHOLD_) leaves M_TRIM_THRESHOLD at
> > 128 KiB, and any heap request between about 132 KiB and the mmap
> > threshold can then fail in the main arena.
> >
> > dlmalloc 2.7.1 already suppressed trimming around this free ("Fix
> > freeing of old top non-contiguous chunk im sysmalloc"). Restore that
> > behavior by splitting _int_free_merge_chunk_notrim out of
> > _int_free_merge_chunk and using it for both old-top frees in sysmalloc.
> > The memory is still trimmed by the next free that reaches the top.
> >
> > Add tst-malloc-sysmalloc-trim, which reproduces the main-arena case by
> > mapping a page range at the program break. It fails without this
> > change.
> >
> > Tested on x86_64 (master fea6c3582ae1): the new test fails without the
> > fix and passes with it; make subdirs=malloc check shows no failures.
> >
> >
> > Signed-off-by: Danner Stodolsky <dstodolsky@bostondynamics.com <mailto:
> dstodolsky@bostondynamics.com>>
>
>
> --
> Cheers,
> Carlos.
>
>
  

Patch

From 26302fb7758b0baaf1c2f4b7c822ab3ba82dfd39 Mon Sep 17 00:00:00 2001
From: Danner Stodolsky <dstodolsky@bostondynamics.com>
Date: Fri, 23 Oct 2026 21:48:34 -0400
Subject: [PATCH] malloc: Do not trim when sysmalloc frees the old top chunk
 [BZ #34882]

When sysmalloc cannot extend the top chunk in place, it obtains a new
top chunk and frees what is left of the old one.  That free went through
_int_free_merge_chunk, which ends in _int_free_maybe_trim.  If the old
top was at least ATTEMPT_TRIMMING_THRESHOLD bytes, the arena was trimmed
before the new top had been split for the pending request:

* Main arena, non-contiguous: if an earlier sbrk failed (for example
  because a foreign mapping briefly sat at the program break) the arena
  became non-contiguous via the mmap fallback.  A later extension whose
  sbrk succeeds frees the old top; systrim then shrinks the new top back
  to about M_TOP_PAD bytes, the final size check fails, and malloc
  returns NULL with errno ENOMEM.  An immediate retry succeeds.

* Thread arena: when the current heap cannot grow, sysmalloc creates a
  new heap and frees the old top.  heap_trim then unmaps the new heap
  again, _int_malloc fails, and the request is satisfied only because
  arena_get_retry falls back to another arena.

With the default dynamic thresholds, M_TRIM_THRESHOLD is twice the mmap
threshold, so the new top is normally too small to be trimmed.  Setting
M_MMAP_THRESHOLD (or MALLOC_MMAP_THRESHOLD_) leaves M_TRIM_THRESHOLD at
128 KiB, and any heap request between about 132 KiB and the mmap
threshold can then fail in the main arena.  We hit this in a non-PIE
Python process whose brk heap grew into the range where the NVIDIA EGL
driver places short-lived MAP_32BIT mappings.

dlmalloc 2.7.1 already suppressed trimming around this free ("Fix
freeing of old top non-contiguous chunk im sysmalloc").  Restore that
behavior by splitting _int_free_merge_chunk_notrim out of
_int_free_merge_chunk and using it for both old-top frees in sysmalloc.
The memory is still trimmed by the next free that reaches the top.

Add tst-malloc-sysmalloc-trim, which reproduces the main-arena case by
mapping a page range at the program break.  It fails without this
change.

Written with Claude Code

Signed-off-by: Danner Stodolsky <dstodolsky@bostondynamics.com>
---
 malloc/Makefile                    |  1 +
 malloc/malloc.c                    | 29 +++++++++---
 malloc/tst-malloc-sysmalloc-trim.c | 73 ++++++++++++++++++++++++++++++
 3 files changed, 97 insertions(+), 6 deletions(-)
 create mode 100644 malloc/tst-malloc-sysmalloc-trim.c

diff --git a/malloc/Makefile b/malloc/Makefile
index 7b704f51..cc15ed87 100644
--- a/malloc/Makefile
+++ b/malloc/Makefile
@@ -47,6 +47,7 @@  tests := \
   tst-malloc-fork-deadlock \
   tst-malloc-random \
   tst-malloc-stats-cancellation \
+  tst-malloc-sysmalloc-trim \
   tst-malloc-tcache-leak \
   tst-malloc-thread-exit \
   tst-malloc-thread-fail \
diff --git a/malloc/malloc.c b/malloc/malloc.c
index 94769d26..333f32dd 100644
--- a/malloc/malloc.c
+++ b/malloc/malloc.c
@@ -822,6 +822,8 @@  typedef struct malloc_chunk* mchunkptr;
 static void*  _int_malloc(mstate, size_t);
 static void _int_free_chunk (mstate, mchunkptr, INTERNAL_SIZE_T);
 static void _int_free_merge_chunk (mstate, mchunkptr, INTERNAL_SIZE_T);
+static INTERNAL_SIZE_T _int_free_merge_chunk_notrim (mstate, mchunkptr,
+						     INTERNAL_SIZE_T);
 static INTERNAL_SIZE_T _int_free_create_chunk (mstate,
 					       mchunkptr, INTERNAL_SIZE_T,
 					       mchunkptr, INTERNAL_SIZE_T);
@@ -2137,7 +2139,9 @@  sysmalloc (INTERNAL_SIZE_T nb, mstate av)
 			CHUNK_HDR_SZ | PREV_INUSE);
               set_foot (chunk_at_offset (old_top, old_size), CHUNK_HDR_SZ);
               set_head (old_top, old_size | PREV_INUSE | NON_MAIN_ARENA);
-              _int_free_merge_chunk (av, old_top, chunksize (old_top));
+              /* Do not trim: the new top has not been split for NB yet,
+                 so trimming it could release the memory just obtained.  */
+              _int_free_merge_chunk_notrim (av, old_top, chunksize (old_top));
             }
           else
             {
@@ -2405,10 +2409,14 @@  sysmalloc (INTERNAL_SIZE_T nb, mstate av)
 						 old_size + CHUNK_HDR_SZ),
 				CHUNK_HDR_SZ | PREV_INUSE);
 
-                      /* If possible, release the rest. */
+                      /* If possible, release the rest, suppressing
+                         trimming: the new top has not been split for NB
+                         yet, so systrim could shrink it below NB and make
+                         this allocation fail.  */
                       if (old_size >= MINSIZE)
                         {
-                          _int_free_merge_chunk (av, old_top, chunksize (old_top));
+                          _int_free_merge_chunk_notrim (av, old_top,
+                                                        chunksize (old_top));
                         }
                     }
                 }
@@ -3997,9 +4005,19 @@  _int_free_chunk (mstate av, mchunkptr p, INTERNAL_SIZE_T size)
 
 /* Try to merge chunk P of SIZE bytes from locked arena AV with its neighbors.
    Put the resulting chunk on the appropriate bin list.  P must not be on a
-   bin list yet, and it can be in use.  */
+   bin list yet, and it can be in use.  Then trim the arena if its top is
+   large enough.  */
 static void
 _int_free_merge_chunk (mstate av, mchunkptr p, INTERNAL_SIZE_T size)
+{
+  size = _int_free_merge_chunk_notrim (av, p, size);
+  _int_free_maybe_trim (av, size);
+}
+
+/* Like _int_free_merge_chunk, but never trims the arena.  Returns the size
+   of the resulting chunk.  */
+static INTERNAL_SIZE_T
+_int_free_merge_chunk_notrim (mstate av, mchunkptr p, INTERNAL_SIZE_T size)
 {
   mchunkptr nextchunk = chunk_at_offset(p, size);
 
@@ -4037,8 +4055,7 @@  _int_free_merge_chunk (mstate av, mchunkptr p, INTERNAL_SIZE_T size)
     }
 
   /* Write the chunk header, maybe after merging with the following chunk.  */
-  size = _int_free_create_chunk (av, p, size, nextchunk, nextsize);
-  _int_free_maybe_trim (av, size);
+  return _int_free_create_chunk (av, p, size, nextchunk, nextsize);
 }
 
 /* Create a chunk at P of SIZE bytes, with SIZE potentially increased
diff --git a/malloc/tst-malloc-sysmalloc-trim.c b/malloc/tst-malloc-sysmalloc-trim.c
new file mode 100644
index 00000000..1e3e3433
--- /dev/null
+++ b/malloc/tst-malloc-sysmalloc-trim.c
@@ -0,0 +1,73 @@ 
+/* Test that sysmalloc does not trim the memory it just obtained.
+   Copyright (C) 2026 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <https://www.gnu.org/licenses/>.  */
+
+/* When the main arena is non-contiguous and sysmalloc extends it with
+   sbrk, it frees the remainder of the old top chunk.  That free used to
+   check for trimming, and the new top chunk, not yet split for the pending
+   request, was trimmed back to about M_TOP_PAD bytes.  malloc then failed
+   with ENOMEM although the memory was available.  A fixed M_MMAP_THRESHOLD
+   keeps M_TRIM_THRESHOLD at its 128 KiB default, which opens the window
+   for any request between that and the mmap threshold.
+
+   The test blocks the program break with a mapping so that the next heap
+   extension falls back to mmap and the arena becomes non-contiguous, then
+   removes the mapping so that the following extension gets sbrk memory
+   again.  */
+
+#include <errno.h>
+#include <malloc.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mman.h>
+#include <unistd.h>
+#include <support/check.h>
+#include <support/xunistd.h>
+
+static int
+do_test (void)
+{
+  TEST_VERIFY_EXIT (mallopt (M_MMAP_THRESHOLD, 4 * 1024 * 1024) == 1);
+
+  void *brk0 = sbrk (0);
+  size_t blocker_size = 704320;
+  void *blocker = mmap (brk0, blocker_size, PROT_READ | PROT_WRITE,
+			MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED_NOREPLACE,
+			-1, 0);
+  if (blocker != brk0)
+    {
+      if (blocker != MAP_FAILED)
+	xmunmap (blocker, blocker_size);
+      FAIL_UNSUPPORTED ("cannot map at the program break %p", brk0);
+    }
+
+  /* sbrk is refused, so this comes from the mmap fallback.  */
+  void *a = malloc (700000);
+  TEST_VERIFY_EXIT (a != NULL);
+  xmunmap (blocker, blocker_size);
+
+  /* Larger than what is left of the old top: sbrk succeeds again.  */
+  void *b = malloc (900000);
+  if (b == NULL)
+    FAIL_EXIT1 ("malloc (900000) failed: %m");
+
+  free (b);
+  free (a);
+  return 0;
+}
+
+#include <support/test-driver.c>
-- 
2.34.1