From patchwork Wed Jul 1 13:38:51 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 138210 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B8BB14BA2E31 for ; Wed, 1 Jul 2026 13:40:02 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B8BB14BA2E31 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YA/2sYG5 X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTP id CE0884BA2E05 for ; Wed, 1 Jul 2026 13:39:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org CE0884BA2E05 Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org CE0884BA2E05 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913147; cv=none; b=AT7RZT/fJR53pqXiW8+Pog5CK7y+uBzDK41J3F/KhsVhthAczOBuqxR6wycbfjzB9zmE7chIyP7Wm9j+lCPvreRFN6p8RnepgTDWlhWD5zYS2Wq/dhwgCW1/PEJfAjwOGUX6ARY7XFJgTxFao/9YS+TvbeQyqQmdk2KTg/xTIHg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913147; c=relaxed/simple; bh=sbcDO831ujo/8xX2rl0rlSofC4DeDu8cNQAHAv+2M5c=; h=DKIM-Signature:From:To:Subject:Message-ID:Date:MIME-Version; b=HH59VssSChf+KzupOZBmAuL2APK0vCJdcye45Pycgp6H8YJbJ9IUVUqLPwLtghAiPdyCA93fI4A/LUgXbbfIYckdFd9SL75QYXg4RVLYw1rSm7z5v3xDBpkX6nLKFd8RFdELVv/mCLZRKdFNKWuGUd201c8RZGax9lSTAxafLrU= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YA/2sYG5 DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org CE0884BA2E05 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1782913147; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=A+BUh3IpBJZ/+EJnwKEJk8qUmhSfcelDkrnF9LruZK8=; b=YA/2sYG5qlCW30zVPlIzV/VY15JT6UpPaMw0oP4QMPB6ttyARazsNBzJXwd8TxHTfDQ698 z2CIIv8+QsaI6jXgwbV6cel0jyco/qTTo0LIchVNfwqRMKsV3YaXUMWR6IuzoYdyHJ/nI0 TVGLE1UNxxcZo8NynvpkU5qOJ5DDnXU= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-522-EzCc7YpAN86mcSTn0Fj5Jg-1; Wed, 01 Jul 2026 09:39:05 -0400 X-MC-Unique: EzCc7YpAN86mcSTn0Fj5Jg-1 X-Mimecast-MFC-AGG-ID: EzCc7YpAN86mcSTn0Fj5Jg_1782913144 Received: from mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.95]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 39F571944B14 for ; Wed, 1 Jul 2026 13:39:04 +0000 (UTC) Received: from fweimer-oldenburg.csb.redhat.com (unknown [10.44.32.220]) by mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 832513189 for ; Wed, 1 Jul 2026 13:39:02 +0000 (UTC) From: Florian Weimer To: libc-alpha@sourceware.org Subject: [PATCH 1/4] malloc: Perform batched frees if tcache is full In-Reply-To: Message-ID: <1dfb2c27aaad8882da29e7f0fe5e5109ef7ac224.1782911956.git.fweimer@redhat.com> References: X-From-Line: 1dfb2c27aaad8882da29e7f0fe5e5109ef7ac224 Mon Sep 17 00:00:00 2001 Date: Wed, 01 Jul 2026 15:38:51 +0200 User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.6 on 10.30.177.95 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: cNmdQm5AJ1yUt8jBNf27FE4yfMArJ4-NZQfcTiu_lCM_1782913144 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, RCVD_IN_SBL_CSS, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Batched frees amortize the locking overhead once the tcache is full. Previously, once the tcache slot was full, each free acquired the arena lock. With this change, neighboring tcache allocations in the same arena re-use an arena lock that already exists. Pass the tcache pointer to __libc_free_batch so that it is not necessary to reload the tcache pointer after the _int_free_chunk calls. --- malloc/malloc.c | 79 ++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 71 insertions(+), 8 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index c39d60b509..99ba94454e 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -2719,7 +2719,8 @@ tcache_put_n (mchunkptr chunk, size_t tc_idx, tcache_entry **ep, bool mangled) available chunks to remove. Removes chunk from the middle of the list. */ static __always_inline void * -tcache_get_n (size_t tc_idx, tcache_entry **ep, bool mangled) +tcache_get_n (tcache_perthread_struct *tc, size_t tc_idx, tcache_entry **ep, + bool mangled) { tcache_entry *e; if (!mangled) @@ -2735,7 +2736,7 @@ tcache_get_n (size_t tc_idx, tcache_entry **ep, bool mangled) else *ep = PROTECT_PTR (ep, REVEAL_PTR (e->next)); - ++(tcache->num_slots[tc_idx]); + ++(tc->num_slots[tc_idx]); e->key = 0; return (void *) e; } @@ -2750,7 +2751,7 @@ tcache_put (mchunkptr chunk, size_t tc_idx) static __always_inline void * tcache_get (size_t tc_idx) { - return tcache_get_n (tc_idx, &tcache->entries[tc_idx], false); + return tcache_get_n (tcache, tc_idx, &tcache->entries[tc_idx], false); } static __always_inline tcache_entry ** @@ -2793,7 +2794,7 @@ tcache_get_large (size_t tc_idx, size_t nb) if (te == NULL || nb != chunksize (mem2chunk (te))) return NULL; - return tcache_get_n (tc_idx, entry, mangled); + return tcache_get_n (tcache, tc_idx, entry, mangled); } static void tcache_init (mstate av); @@ -2826,7 +2827,7 @@ tcache_get_align (size_t nb, size_t alignment) if (te != NULL && csize == nb && PTR_IS_ALIGNED (te, alignment)) - return tcache_get_n (tc_idx, tep, mangled); + return tcache_get_n (tcache, tc_idx, tep, mangled); } return NULL; } @@ -3006,6 +3007,62 @@ tcache_free_init (void *mem) __libc_free (mem); } +/* If the arena does not change between chunks, keep the lock. */ +static inline void +__libc_free_batched_loop (bool do_lock, mstate av, mchunkptr p, INTERNAL_SIZE_T size, + tcache_perthread_struct *tc, size_t tc_idx) +{ + /* Empty half of the tcache, for a hysteresis effect. */ + unsigned int to_free = mp_.tcache_count / 2; + + if (do_lock) + __libc_lock_lock (av->mutex); + + _int_free_merge_chunk (av, p, size); + + while (tc->entries[tc_idx] != NULL && to_free > 0) + { + void *mem = tcache_get_n (tc, tc_idx, &tc->entries[tc_idx], false); + p = mem2chunk (mem); + size = chunksize (p); + + /* Lock a different arena if necessary. */ + if (do_lock) + { + mstate chunk_av = arena_for_chunk (p); + if (chunk_av != av) + { + __libc_lock_unlock (av->mutex); + av = chunk_av; + __libc_lock_lock (av->mutex); + } + } + + _int_free_merge_chunk (av, p, size); + to_free--; + } + + if (do_lock) + __libc_lock_unlock (av->mutex); +} + +/* Deallocate half of the tcache entries into arenas, to amortize the + locking overhead. */ +static __attribute_noinline__ void +__libc_free_batched (mchunkptr p, INTERNAL_SIZE_T size, + tcache_perthread_struct *tc, size_t tc_idx) +{ + /* Check size >= MINSIZE and p + size does not overflow. */ + if (__glibc_unlikely (INT_ADD_OVERFLOW ((uintptr_t) p, + size - MINSIZE))) + return malloc_printerr_tail ("free(): invalid size (batch)"); + + if (SINGLE_THREAD_P) + __libc_free_batched_loop (false, &main_arena, p, size, tc, tc_idx); + else + __libc_free_batched_loop (true, arena_for_chunk (p), p, size, tc, tc_idx); +} + void __libc_free (void *mem) { @@ -3036,6 +3093,13 @@ __libc_free (void *mem) { if (__glibc_likely (tcache->num_slots[tc_idx] != 0)) return tcache_put (p, tc_idx); + else + { + /* Perform batched freeing of tcache entries. */ + if (__glibc_unlikely (tcache_inactive ())) + return tcache_free_init (mem); + return __libc_free_batched (p, size, tcache, tc_idx); + } } else { @@ -3043,10 +3107,9 @@ __libc_free (void *mem) if (size >= MINSIZE && __glibc_likely (tcache->num_slots[tc_idx] != 0)) return tcache_put_large (p, tc_idx); + if (__glibc_unlikely (tcache_inactive ())) + return tcache_free_init (mem); } - - if (__glibc_unlikely (tcache_inactive ())) - return tcache_free_init (mem); } #endif From patchwork Wed Jul 1 13:38:59 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 138211 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4BDB64BA2E3C for ; Wed, 1 Jul 2026 13:41:00 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 4BDB64BA2E3C Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=hFbtOuEy X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id 954B94BA543C for ; Wed, 1 Jul 2026 13:39:15 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 954B94BA543C Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 954B94BA543C Authentication-Results: sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913155; cv=none; b=uvr4jra5v3KCmD1kKpnDo0W2MQapXkGfMO6X7thMNiyl6LZG71O9aTDSccLWEtMITqFyrOPck9vH1VJDe23zeTSinpBFIj8BjKZp/kyD6ZAWjRHgYDcbLONhkMoFdY3+RCZkhu0BuESbpiiWFh8FMPtUTpFB7rjWVoVf0l5eXL8= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913155; c=relaxed/simple; bh=PTJU7W77cTOzz30SpXT6B56WOdsNsSE8MCyZtm6Yw4c=; h=DKIM-Signature:From:To:Subject:Message-ID:Date:MIME-Version; b=d/9r8PodU2AnBp6TtfDU6y0XJ9AEwr/wAKHx+Cqso4l3xi0vifyVOasgaGB486UcwwIqguoXcx9lQM2BR/kGhuRPurA5aQa5HLaAmVPzvVWXMa7HK5qsPCZ0Kus7g+UCD/ErpfRb5lYl9TXOJMnzw14oa3rn9RwbwXU7mRnhlh8= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=hFbtOuEy DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 954B94BA543C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1782913155; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=buo887RN8Y+JB2EhVbQX+AOsTK9Uj8EV6P9Z7v6Qd8g=; b=hFbtOuEyO9eT8ML7WpRGECTKjurMHBRrtPJtqklPrjKaaaWq0TxCFf/XQwTSVyK/t9Rbn1 3MRiWmwuixtpynX4zpaj3o1chD1G0//VwrY/5vlP8dDL6EBYKDOvWyNraRFyH0nsPfFz8H wAJzRRRi+YysslMQngabe+fKqOq5vYk= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-1-RGeZKrdFN8Kv5f7Ksne-DQ-1; Wed, 01 Jul 2026 09:39:12 -0400 X-MC-Unique: RGeZKrdFN8Kv5f7Ksne-DQ-1 X-Mimecast-MFC-AGG-ID: RGeZKrdFN8Kv5f7Ksne-DQ_1782913152 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id F40D0184B09D for ; Wed, 1 Jul 2026 13:39:11 +0000 (UTC) Received: from fweimer-oldenburg.csb.redhat.com (unknown [10.44.32.220]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 478E0195E486 for ; Wed, 1 Jul 2026 13:39:11 +0000 (UTC) From: Florian Weimer To: libc-alpha@sourceware.org Subject: [PATCH 2/4] malloc: Only fill half of the tcache during batch allocation In-Reply-To: Message-ID: References: X-From-Line: daf953123dfc904910820bb4266c8ccc21ca1ecc Mon Sep 17 00:00:00 2001 Date: Wed, 01 Jul 2026 15:38:59 +0200 User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: s7EGkb-9_EZje2KHtgIQHjeWpgAHXJFJVG2Hytfas90_1782913152 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org This leaves room for subsequent frees. --- malloc/malloc.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index 99ba94454e..3505158ac0 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -3570,8 +3570,11 @@ _int_malloc (mstate av, size_t bytes) if (__glibc_unlikely (tcache_inactive ())) tcache_init (av); - /* While bin not empty and tcache not full, copy chunks over. */ - while (tcache->num_slots[tc_idx] != 0 + /* While bin not empty and tcache not full, copy chunks over. + Only fill half of the tcache, so that subsequent frees + do not immediately flush the tcache. */ + unsigned int tcache_target_count = mp_.tcache_count / 2; + while (tcache->num_slots[tc_idx] > tcache_target_count && (tc_victim = last (bin)) != bin) { if (tc_victim != NULL) From patchwork Wed Jul 1 13:39:07 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 138209 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C729E4BA2E16 for ; Wed, 1 Jul 2026 13:39:58 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org C729E4BA2E16 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bnbIbWPZ X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTP id 3F40A4BA2E16 for ; Wed, 1 Jul 2026 13:39:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 3F40A4BA2E16 Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 3F40A4BA2E16 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913162; cv=none; b=u6Aqfr7TQ85R5qX6Lab8yARp1nKe8t7RspPXDqxYaZAnI+QpjlZr6rF+gRpPlpofte2dVCgDEhpS6OteH3/jPDJ7hRnUQ1RYk1pP20eshMpHvWKn6POxI+sYwMibxttdNDVmjjIUUpVquOk157trXaxFQgaRYBo+cl3+mXoUZ0I= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913162; c=relaxed/simple; bh=4F+cost7fnfm9z9SZf4ZbZi5eGfULcexjpH8qalj7sA=; h=DKIM-Signature:From:To:Subject:Message-ID:Date:MIME-Version; b=S0UE5NJrgMAFWEYMkzS7RQfjM+F70GhWnJz/zdF1tHwrNyBE3ZghliNVwYGLSFdF6qjD45VU3jElntTtge8Vrl3KmO3nl2I2x9azxgOEvjVyyFvTUeaqlu4BOJfWuotTHAZxbAieP4vn3XW55IjrbfcsYc32Ei0+3cM1EDnOUpU= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bnbIbWPZ DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 3F40A4BA2E16 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1782913162; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=A0Val5tMWd0ov3M6Uxn1RNEaOizactnNuR9+BjCGadQ=; b=bnbIbWPZK6NLePc0g81xDczWrkCCy11LYUxlap314I2qTGXG3W+sKDVs/TKgwpTbabEVI4 LxHqMw9cmMQ1bjRtpSjmh/CpJAMq4M0NeZtv4SJmXxsjsHkcZ34ESc4Ba4Z1xbiab6a3RD sTPBBRSMoJGq/uMrVvQr1tssCCmqH7Q= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-500--Nl43_d3MHC6at5_DyB_gQ-1; Wed, 01 Jul 2026 09:39:20 -0400 X-MC-Unique: -Nl43_d3MHC6at5_DyB_gQ-1 X-Mimecast-MFC-AGG-ID: -Nl43_d3MHC6at5_DyB_gQ_1782913160 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 179FD184B0AF for ; Wed, 1 Jul 2026 13:39:20 +0000 (UTC) Received: from fweimer-oldenburg.csb.redhat.com (unknown [10.44.32.220]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 600413000C2B for ; Wed, 1 Jul 2026 13:39:19 +0000 (UTC) From: Florian Weimer To: libc-alpha@sourceware.org Subject: [PATCH 3/4] malloc: Increase tcache fill count from 16 to 32 In-Reply-To: Message-ID: References: X-From-Line: caf71948871bae183861116341705e25e808b335 Mon Sep 17 00:00:00 2001 Date: Wed, 01 Jul 2026 15:39:07 +0200 User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 4B7XAc1Srk2rL2olwBNCacWdEtXVc3sNF8mBtX5AefA_1782913160 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-10.7 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_NUMSUBJECT, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org This avoids a regression in the omnetpp and xalancbmk benchmarks of SPEC. Apparently, these benchmarks are very sensitive to the fill rate in malloc. Suggested-by: Wilco Dijkstra Tested-by: Wilco Dijkstra --- malloc/malloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index 3505158ac0..61be8fe6bb 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -306,7 +306,7 @@ verify (PTRDIFF_MAX <= SIZE_MAX / 2); /* This is another arbitrary limit, which tunables can change. Each tcache bin will hold at most this number of chunks. */ -# define TCACHE_FILL_COUNT 16 +# define TCACHE_FILL_COUNT 32 /* Maximum chunks in tcache bins for tunables. This value must fit the range of tcache->num_slots[] entries, else they may overflow. */ From patchwork Wed Jul 1 13:39:21 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 138212 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id BA91F4BA2E22 for ; Wed, 1 Jul 2026 13:41:13 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BA91F4BA2E22 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=T28WEnPl X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id 7BE4C4BA2E10 for ; Wed, 1 Jul 2026 13:39:37 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 7BE4C4BA2E10 Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 7BE4C4BA2E10 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913177; cv=none; b=AFQ0IVLLOAJxNzP6zNjuIbVZRhaM05RRMfuIm7i65Yff81THxoYR8wcCoFsJDYrXkPHMmx4m8gNQeyA1unlHEyYZzmxJ5NazjawIMJhPaTTXL0EPQ9spkKKVFPwUP2gB62yyHSFi2wPPLyXE9H7XRuim21OrV9qTISe9qerAnr4= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1782913177; c=relaxed/simple; bh=B1FaMh35mAF8Jpiwm+qd9Hk63YVz1y4+O9pYgdMnKrA=; h=DKIM-Signature:From:To:Subject:Message-ID:Date:MIME-Version; b=ZwqLllplEr2WeNCFqarA8VTNE1Meh9NoARk9MQXdof0KmVIhGBnmKos636ceX7amuzb1lz1sp1RLyeVtESSXPC9UWpqVMYa2x/K6eh4w+wF87PfuXzouHdUF11zPCEjf1zwE8DiDqAeeYwxOmIFR/KRrVDQeFmzTg5g/vSdsmTI= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=T28WEnPl DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7BE4C4BA2E10 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1782913177; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=HAvi9qSF+kbmS052mwQyhZRRwAlDLZO9LmMgzbGZ2EQ=; b=T28WEnPlQmi69nuZYFh7SQdNqYFuIcrnrLGiuIStb65u/3tUhRZP9/29njwK9Q+kRDcnpy w+F3vz806DaATmDVoQTX5LRoaPfomhx9G7bDlaIYuFZb6uyjepciKTgSblcomaj3PAcycJ H/eY/GIKaqEs5gRyU55NZasjHgT6lDk= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-615-w1EU-RGhOwqdSbMaxtwxMQ-1; Wed, 01 Jul 2026 09:39:35 -0400 X-MC-Unique: w1EU-RGhOwqdSbMaxtwxMQ-1 X-Mimecast-MFC-AGG-ID: w1EU-RGhOwqdSbMaxtwxMQ_1782913175 Received: from mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.95]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 21C291944B24 for ; Wed, 1 Jul 2026 13:39:35 +0000 (UTC) Received: from fweimer-oldenburg.csb.redhat.com (unknown [10.44.32.220]) by mx-prod-int-10.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 68B3E3691C for ; Wed, 1 Jul 2026 13:39:34 +0000 (UTC) From: Florian Weimer To: libc-alpha@sourceware.org Subject: [PATCH 4/4] malloc: Implement merging of tcache chunks during batched frees In-Reply-To: Message-ID: References: X-From-Line: d02dc1493477fca96d0bec166ef01fbb5602b82c Mon Sep 17 00:00:00 2001 Date: Wed, 01 Jul 2026 15:39:21 +0200 User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.6 on 10.30.177.95 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: j_a3jdZ9wFKywEM2KHQU9cHBV4I2TtMIajPbQ17Bsy0_1782913175 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, RCVD_IN_SBL_CSS, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org This avoids significant overhead due to batched frees, as chunks are repeated linked and unlinked from bins. --- malloc/malloc.c | 85 +++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 69 insertions(+), 16 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index 61be8fe6bb..a4d70084c9 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -3007,41 +3007,93 @@ tcache_free_init (void *mem) __libc_free (mem); } -/* If the arena does not change between chunks, keep the lock. */ -static inline void -__libc_free_batched_loop (bool do_lock, mstate av, mchunkptr p, INTERNAL_SIZE_T size, +#if USE_TCACHE +/* Free the chunk at EXPECTED_BEFORE + SIZE. Before that, if AV is + NULL, obtain the arena from the chunk and lock it. Otherwise, + assume that AV matches the chunk. Return the AV value. */ +static __always_inline mstate +__libc_free_batched_do_free (bool do_lock, mstate av, void *expected_before, + void *expected_after, INTERNAL_SIZE_T size) +{ + mchunkptr chunk = expected_before + size; + if (do_lock && av == NULL) + { + av = arena_for_chunk (chunk); + __libc_lock_lock (av->mutex); + } + _int_free_merge_chunk (av, chunk, expected_after - (void *) chunk); + return av; +} + +/* Free part of a tcache chain into the lower-level allocator. */ +static __always_inline void +__libc_free_batched_loop (bool do_lock, mchunkptr p, INTERNAL_SIZE_T size, tcache_perthread_struct *tc, size_t tc_idx) { /* Empty half of the tcache, for a hysteresis effect. */ unsigned int to_free = mp_.tcache_count / 2; + /* The arena lock is acquired lazily before changes are made to the + heap structure. Merging of adjacent tcache chunks does not need + the lock. */ + mstate av; if (do_lock) - __libc_lock_lock (av->mutex); + av = NULL; + else + av = &main_arena; + + /* To avoid repeated linking and unlink of bins (or cluttering the + unsorted bin) as tcache chunks are freed, adjacent chunks are + merged at the tcache level. A larger chunk is created and freed + directly. + + These variables track the expected merge locations of the current + chunk that has not yet been committed to the lower-level + allocator. Start with the merge locations of p. */ + void *expected_before = (void *) p - size; + void *expected_after = (void *) p + size; - _int_free_merge_chunk (av, p, size); while (tc->entries[tc_idx] != NULL && to_free > 0) { void *mem = tcache_get_n (tc, tc_idx, &tc->entries[tc_idx], false); p = mem2chunk (mem); - size = chunksize (p); - - /* Lock a different arena if necessary. */ - if (do_lock) + if (size != chunksize (p)) + malloc_printerr ("free(): corrupted tcache size"); + + /* Check if the chunk can be merged. This does not require the + lock because there is no structural change of the heap yet. */ + if (p == expected_before) + expected_before -= size; + else if (p == expected_after) + expected_after += size; + else { - mstate chunk_av = arena_for_chunk (p); - if (chunk_av != av) + /* Deallocate the previous chunk that could not be merged. */ + av = __libc_free_batched_do_free (do_lock, av, expected_before, + expected_after, size); + + /* If locking and the arena changes, release the lock now. + It will be reacquired once there is no more tcache-level + merging. */ + if (do_lock && arena_for_chunk (p) != av) { __libc_lock_unlock (av->mutex); - av = chunk_av; - __libc_lock_lock (av->mutex); + av = NULL; } + + /* Continue processing with the chunk that came from tcache. */ + expected_before = (void *) p - size; + expected_after = (void *) p + size; } - _int_free_merge_chunk (av, p, size); to_free--; } + /* Free the last chunk. */ + av = __libc_free_batched_do_free (do_lock, av, expected_before, + expected_after, size); + if (do_lock) __libc_lock_unlock (av->mutex); } @@ -3058,10 +3110,11 @@ __libc_free_batched (mchunkptr p, INTERNAL_SIZE_T size, return malloc_printerr_tail ("free(): invalid size (batch)"); if (SINGLE_THREAD_P) - __libc_free_batched_loop (false, &main_arena, p, size, tc, tc_idx); + __libc_free_batched_loop (false, p, size, tc, tc_idx); else - __libc_free_batched_loop (true, arena_for_chunk (p), p, size, tc, tc_idx); + __libc_free_batched_loop (true, p, size, tc, tc_idx); } +#endif void __libc_free (void *mem)