From patchwork Wed Jul 1 13:39:21 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 138212 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id BA91F4BA2E22 for ; Wed, 1 Jul 2026 13:41:13 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BA91F4BA2E22 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=T28WEnPl X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id 7BE4C4BA2E10 for ; Wed, 1 Jul 2026 13:39:37 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 7BE4C4BA2E10 Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 7BE4C4BA2E10 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913177; cv=none; b=AFQ0IVLLOAJxNzP6zNjuIbVZRhaM05RRMfuIm7i65Yff81THxoYR8wcCoFsJDYrXkPHMmx4m8gNQeyA1unlHEyYZzmxJ5NazjawIMJhPaTTXL0EPQ9spkKKVFPwUP2gB62yyHSFi2wPPLyXE9H7XRuim21OrV9qTISe9qerAnr4= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913177; c=relaxed/simple; bh=B1FaMh35mAF8Jpiwm+qd9Hk63YVz1y4+O9pYgdMnKrA=; h=DKIM-Signature:From:To:Subject:Message-ID:Date:MIME-Version; b=ZwqLllplEr2WeNCFqarA8VTNE1Meh9NoARk9MQXdof0KmVIhGBnmKos636ceX7amuzb1lz1sp1RLyeVtESSXPC9UWpqVMYa2x/K6eh4w+wF87PfuXzouHdUF11zPCEjf1zwE8DiDqAeeYwxOmIFR/KRrVDQeFmzTg5g/vSdsmTI= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=T28WEnPl DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7BE4C4BA2E10 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1782913177; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=HAvi9qSF+kbmS052mwQyhZRRwAlDLZO9LmMgzbGZ2EQ=; b=T28WEnPlQmi69nuZYFh7SQdNqYFuIcrnrLGiuIStb65u/3tUhRZP9/29njwK9Q+kRDcnpy w+F3vz806DaATmDVoQTX5LRoaPfomhx9G7bDlaIYuFZb6uyjepciKTgSblcomaj3PAcycJ H/eY/GIKaqEs5gRyU55NZasjHgT6lDk= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-615-w1EU-RGhOwqdSbMaxtwxMQ-1; Wed, 01 Jul 2026 09:39:35 -0400 X-MC-Unique: w1EU-RGhOwqdSbMaxtwxMQ-1 X-Mimecast-MFC-AGG-ID: w1EU-RGhOwqdSbMaxtwxMQ_1782913175 Received: from mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.95]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 21C291944B24 for ; Wed, 1 Jul 2026 13:39:35 +0000 (UTC) Received: from fweimer-oldenburg.csb.redhat.com (unknown [10.44.32.220]) by mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 68B3E3691C for ; Wed, 1 Jul 2026 13:39:34 +0000 (UTC) From: Florian Weimer To: libc-alpha@sourceware.org Subject: [PATCH 4/4] malloc: Implement merging of tcache chunks during batched frees In-Reply-To: Message-ID: References: X-From-Line: d02dc1493477fca96d0bec166ef01fbb5602b82c Mon Sep 17 00:00:00 2001 Date: Wed, 01 Jul 2026 15:39:21 +0200 User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.6 on 10.30.177.95 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: j_a3jdZ9wFKywEM2KHQU9cHBV4I2TtMIajPbQ17Bsy0_1782913175 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, RCVD_IN_SBL_CSS, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org This avoids significant overhead due to batched frees, as chunks are repeated linked and unlinked from bins. --- malloc/malloc.c | 85 +++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 69 insertions(+), 16 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index 61be8fe6bb..a4d70084c9 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -3007,41 +3007,93 @@ tcache_free_init (void *mem) __libc_free (mem); } -/* If the arena does not change between chunks, keep the lock. */ -static inline void -__libc_free_batched_loop (bool do_lock, mstate av, mchunkptr p, INTERNAL_SIZE_T size, +#if USE_TCACHE +/* Free the chunk at EXPECTED_BEFORE + SIZE. Before that, if AV is + NULL, obtain the arena from the chunk and lock it. Otherwise, + assume that AV matches the chunk. Return the AV value. */ +static __always_inline mstate +__libc_free_batched_do_free (bool do_lock, mstate av, void *expected_before, + void *expected_after, INTERNAL_SIZE_T size) +{ + mchunkptr chunk = expected_before + size; + if (do_lock && av == NULL) + { + av = arena_for_chunk (chunk); + __libc_lock_lock (av->mutex); + } + _int_free_merge_chunk (av, chunk, expected_after - (void *) chunk); + return av; +} + +/* Free part of a tcache chain into the lower-level allocator. */ +static __always_inline void +__libc_free_batched_loop (bool do_lock, mchunkptr p, INTERNAL_SIZE_T size, tcache_perthread_struct *tc, size_t tc_idx) { /* Empty half of the tcache, for a hysteresis effect. */ unsigned int to_free = mp_.tcache_count / 2; + /* The arena lock is acquired lazily before changes are made to the + heap structure. Merging of adjacent tcache chunks does not need + the lock. */ + mstate av; if (do_lock) - __libc_lock_lock (av->mutex); + av = NULL; + else + av = &main_arena; + + /* To avoid repeated linking and unlink of bins (or cluttering the + unsorted bin) as tcache chunks are freed, adjacent chunks are + merged at the tcache level. A larger chunk is created and freed + directly. + + These variables track the expected merge locations of the current + chunk that has not yet been committed to the lower-level + allocator. Start with the merge locations of p. */ + void *expected_before = (void *) p - size; + void *expected_after = (void *) p + size; - _int_free_merge_chunk (av, p, size); while (tc->entries[tc_idx] != NULL && to_free > 0) { void *mem = tcache_get_n (tc, tc_idx, &tc->entries[tc_idx], false); p = mem2chunk (mem); - size = chunksize (p); - - /* Lock a different arena if necessary. */ - if (do_lock) + if (size != chunksize (p)) + malloc_printerr ("free(): corrupted tcache size"); + + /* Check if the chunk can be merged. This does not require the + lock because there is no structural change of the heap yet. */ + if (p == expected_before) + expected_before -= size; + else if (p == expected_after) + expected_after += size; + else { - mstate chunk_av = arena_for_chunk (p); - if (chunk_av != av) + /* Deallocate the previous chunk that could not be merged. */ + av = __libc_free_batched_do_free (do_lock, av, expected_before, + expected_after, size); + + /* If locking and the arena changes, release the lock now. + It will be reacquired once there is no more tcache-level + merging. */ + if (do_lock && arena_for_chunk (p) != av) { __libc_lock_unlock (av->mutex); - av = chunk_av; - __libc_lock_lock (av->mutex); + av = NULL; } + + /* Continue processing with the chunk that came from tcache. */ + expected_before = (void *) p - size; + expected_after = (void *) p + size; } - _int_free_merge_chunk (av, p, size); to_free--; } + /* Free the last chunk. */ + av = __libc_free_batched_do_free (do_lock, av, expected_before, + expected_after, size); + if (do_lock) __libc_lock_unlock (av->mutex); } @@ -3058,10 +3110,11 @@ __libc_free_batched (mchunkptr p, INTERNAL_SIZE_T size, return malloc_printerr_tail ("free(): invalid size (batch)"); if (SINGLE_THREAD_P) - __libc_free_batched_loop (false, &main_arena, p, size, tc, tc_idx); + __libc_free_batched_loop (false, p, size, tc, tc_idx); else - __libc_free_batched_loop (true, arena_for_chunk (p), p, size, tc, tc_idx); + __libc_free_batched_loop (true, p, size, tc, tc_idx); } +#endif void __libc_free (void *mem)