From patchwork Wed Jul 1 13:38:51 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 138210 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B8BB14BA2E31 for ; Wed, 1 Jul 2026 13:40:02 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B8BB14BA2E31 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YA/2sYG5 X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTP id CE0884BA2E05 for ; Wed, 1 Jul 2026 13:39:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org CE0884BA2E05 Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org CE0884BA2E05 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913147; cv=none; b=AT7RZT/fJR53pqXiW8+Pog5CK7y+uBzDK41J3F/KhsVhthAczOBuqxR6wycbfjzB9zmE7chIyP7Wm9j+lCPvreRFN6p8RnepgTDWlhWD5zYS2Wq/dhwgCW1/PEJfAjwOGUX6ARY7XFJgTxFao/9YS+TvbeQyqQmdk2KTg/xTIHg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913147; c=relaxed/simple; bh=sbcDO831ujo/8xX2rl0rlSofC4DeDu8cNQAHAv+2M5c=; h=DKIM-Signature:From:To:Subject:Message-ID:Date:MIME-Version; b=HH59VssSChf+KzupOZBmAuL2APK0vCJdcye45Pycgp6H8YJbJ9IUVUqLPwLtghAiPdyCA93fI4A/LUgXbbfIYckdFd9SL75QYXg4RVLYw1rSm7z5v3xDBpkX6nLKFd8RFdELVv/mCLZRKdFNKWuGUd201c8RZGax9lSTAxafLrU= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YA/2sYG5 DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org CE0884BA2E05 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1782913147; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=A+BUh3IpBJZ/+EJnwKEJk8qUmhSfcelDkrnF9LruZK8=; b=YA/2sYG5qlCW30zVPlIzV/VY15JT6UpPaMw0oP4QMPB6ttyARazsNBzJXwd8TxHTfDQ698 z2CIIv8+QsaI6jXgwbV6cel0jyco/qTTo0LIchVNfwqRMKsV3YaXUMWR6IuzoYdyHJ/nI0 TVGLE1UNxxcZo8NynvpkU5qOJ5DDnXU= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-522-EzCc7YpAN86mcSTn0Fj5Jg-1; Wed, 01 Jul 2026 09:39:05 -0400 X-MC-Unique: EzCc7YpAN86mcSTn0Fj5Jg-1 X-Mimecast-MFC-AGG-ID: EzCc7YpAN86mcSTn0Fj5Jg_1782913144 Received: from mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.95]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 39F571944B14 for ; Wed, 1 Jul 2026 13:39:04 +0000 (UTC) Received: from fweimer-oldenburg.csb.redhat.com (unknown [10.44.32.220]) by mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 832513189 for ; Wed, 1 Jul 2026 13:39:02 +0000 (UTC) From: Florian Weimer To: libc-alpha@sourceware.org Subject: [PATCH 1/4] malloc: Perform batched frees if tcache is full In-Reply-To: Message-ID: <1dfb2c27aaad8882da29e7f0fe5e5109ef7ac224.1782911956.git.fweimer@redhat.com> References: X-From-Line: 1dfb2c27aaad8882da29e7f0fe5e5109ef7ac224 Mon Sep 17 00:00:00 2001 Date: Wed, 01 Jul 2026 15:38:51 +0200 User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.6 on 10.30.177.95 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: cNmdQm5AJ1yUt8jBNf27FE4yfMArJ4-NZQfcTiu_lCM_1782913144 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, RCVD_IN_SBL_CSS, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Batched frees amortize the locking overhead once the tcache is full. Previously, once the tcache slot was full, each free acquired the arena lock. With this change, neighboring tcache allocations in the same arena re-use an arena lock that already exists. Pass the tcache pointer to __libc_free_batch so that it is not necessary to reload the tcache pointer after the _int_free_chunk calls. --- malloc/malloc.c | 79 ++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 71 insertions(+), 8 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index c39d60b509..99ba94454e 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -2719,7 +2719,8 @@ tcache_put_n (mchunkptr chunk, size_t tc_idx, tcache_entry **ep, bool mangled) available chunks to remove. Removes chunk from the middle of the list. */ static __always_inline void * -tcache_get_n (size_t tc_idx, tcache_entry **ep, bool mangled) +tcache_get_n (tcache_perthread_struct *tc, size_t tc_idx, tcache_entry **ep, + bool mangled) { tcache_entry *e; if (!mangled) @@ -2735,7 +2736,7 @@ tcache_get_n (size_t tc_idx, tcache_entry **ep, bool mangled) else *ep = PROTECT_PTR (ep, REVEAL_PTR (e->next)); - ++(tcache->num_slots[tc_idx]); + ++(tc->num_slots[tc_idx]); e->key = 0; return (void *) e; } @@ -2750,7 +2751,7 @@ tcache_put (mchunkptr chunk, size_t tc_idx) static __always_inline void * tcache_get (size_t tc_idx) { - return tcache_get_n (tc_idx, &tcache->entries[tc_idx], false); + return tcache_get_n (tcache, tc_idx, &tcache->entries[tc_idx], false); } static __always_inline tcache_entry ** @@ -2793,7 +2794,7 @@ tcache_get_large (size_t tc_idx, size_t nb) if (te == NULL || nb != chunksize (mem2chunk (te))) return NULL; - return tcache_get_n (tc_idx, entry, mangled); + return tcache_get_n (tcache, tc_idx, entry, mangled); } static void tcache_init (mstate av); @@ -2826,7 +2827,7 @@ tcache_get_align (size_t nb, size_t alignment) if (te != NULL && csize == nb && PTR_IS_ALIGNED (te, alignment)) - return tcache_get_n (tc_idx, tep, mangled); + return tcache_get_n (tcache, tc_idx, tep, mangled); } return NULL; } @@ -3006,6 +3007,62 @@ tcache_free_init (void *mem) __libc_free (mem); } +/* If the arena does not change between chunks, keep the lock. */ +static inline void +__libc_free_batched_loop (bool do_lock, mstate av, mchunkptr p, INTERNAL_SIZE_T size, + tcache_perthread_struct *tc, size_t tc_idx) +{ + /* Empty half of the tcache, for a hysteresis effect. */ + unsigned int to_free = mp_.tcache_count / 2; + + if (do_lock) + __libc_lock_lock (av->mutex); + + _int_free_merge_chunk (av, p, size); + + while (tc->entries[tc_idx] != NULL && to_free > 0) + { + void *mem = tcache_get_n (tc, tc_idx, &tc->entries[tc_idx], false); + p = mem2chunk (mem); + size = chunksize (p); + + /* Lock a different arena if necessary. */ + if (do_lock) + { + mstate chunk_av = arena_for_chunk (p); + if (chunk_av != av) + { + __libc_lock_unlock (av->mutex); + av = chunk_av; + __libc_lock_lock (av->mutex); + } + } + + _int_free_merge_chunk (av, p, size); + to_free--; + } + + if (do_lock) + __libc_lock_unlock (av->mutex); +} + +/* Deallocate half of the tcache entries into arenas, to amortize the + locking overhead. */ +static __attribute_noinline__ void +__libc_free_batched (mchunkptr p, INTERNAL_SIZE_T size, + tcache_perthread_struct *tc, size_t tc_idx) +{ + /* Check size >= MINSIZE and p + size does not overflow. */ + if (__glibc_unlikely (INT_ADD_OVERFLOW ((uintptr_t) p, + size - MINSIZE))) + return malloc_printerr_tail ("free(): invalid size (batch)"); + + if (SINGLE_THREAD_P) + __libc_free_batched_loop (false, &main_arena, p, size, tc, tc_idx); + else + __libc_free_batched_loop (true, arena_for_chunk (p), p, size, tc, tc_idx); +} + void __libc_free (void *mem) { @@ -3036,6 +3093,13 @@ __libc_free (void *mem) { if (__glibc_likely (tcache->num_slots[tc_idx] != 0)) return tcache_put (p, tc_idx); + else + { + /* Perform batched freeing of tcache entries. */ + if (__glibc_unlikely (tcache_inactive ())) + return tcache_free_init (mem); + return __libc_free_batched (p, size, tcache, tc_idx); + } } else { @@ -3043,10 +3107,9 @@ __libc_free (void *mem) if (size >= MINSIZE && __glibc_likely (tcache->num_slots[tc_idx] != 0)) return tcache_put_large (p, tc_idx); + if (__glibc_unlikely (tcache_inactive ())) + return tcache_free_init (mem); } - - if (__glibc_unlikely (tcache_inactive ())) - return tcache_free_init (mem); } #endif