From patchwork Thu Aug 29 06:27:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Wangyang" X-Patchwork-Id: 96648 X-Patchwork-Delegate: fweimer@redhat.com Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id BFC92385EC29 for ; Thu, 29 Aug 2024 06:30:32 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.18]) by sourceware.org (Postfix) with ESMTPS id 1BA2E3858283 for ; Thu, 29 Aug 2024 06:30:03 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1BA2E3858283 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=intel.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1BA2E3858283 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=192.198.163.18 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1724913005; cv=none; b=oajXRPQ3HIKnLlyFowsQcalFGhYDDUcCq4JXZCSZGeW6P7ewDab8bskcPvyP4Bxaen8QxT75bji+hNPX8/J5mRd47zGHP0weYHJ4Y2aaQkFAkuHms8/8kcnNTM2CsefwzLlANUTwdKlmtzRvlrvoV9f6VqfrxxGp0WEld6ElwT4= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1724913005; c=relaxed/simple; bh=V8C/kbEeVY3N4TTX+9TykhPqRYKGOKsGOOiofF+1ss4=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=EaqI0wBHCcMnjRzduiSaTaU9FQeSB/AdbAe3oyoyCnSczgzPr25qBphOhJL5tCrlCm6kpGRrWsJFbdaJQlFKkBqOUouO4p+yWrAYWIssTlPCuu2seS8bz+lFyGBHf3n5Tye9l8dIA9QR9QxBxka3xHBgpmfexwUqbI4oDzKnc1k= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724913003; x=1756449003; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=V8C/kbEeVY3N4TTX+9TykhPqRYKGOKsGOOiofF+1ss4=; b=YihGe1elwny2YYLA5gxT2mlkyCKE2SnKoowbRFrqp7W6t+JIKHfv5xb0 CIqWaGh3sROE6k3vhDhqjHhXLbKGZ5AjgV5iyMAsIPQkRkT83FQzEBDx3 ghm+isglNsXNeTI0TPmVdc8lH6Qfxsu4lUo23odnF8hP8cqSUX8d3b3UY INQIe5Wgw+tBJp2fCe6emoNDqQcmv8SbTP/XR8oyuoS5OGBuJsyDcvRD8 0xh0nGEd4o7UhLiUt2cF8r01EoqVXqmUOYT4pjw12jKpwEagSJp9VrrJR RkILh2DpIGwjmS2ML29NLuEhMpAKirnV7Jdmsj3CHcU2zr+MTv6zsABMl w==; X-CSE-ConnectionGUID: yfQ1UED7Qw6o/H7wi9dGMA== X-CSE-MsgGUID: Vh3LZmFUR7qHjRQC6XN0FQ== X-IronPort-AV: E=McAfee;i="6700,10204,11178"; a="22998338" X-IronPort-AV: E=Sophos;i="6.10,185,1719903600"; d="scan'208";a="22998338" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by fmvoesa112.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Aug 2024 23:30:02 -0700 X-CSE-ConnectionGUID: i4CfxXItSLGsTNlEbNhEMA== X-CSE-MsgGUID: HcVTFtlOSB6LJ7fLXc2ysg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,185,1719903600"; d="scan'208";a="63539126" Received: from linux-pnp-server-11.sh.intel.com ([10.239.176.178]) by fmviesa009.fm.intel.com with ESMTP; 28 Aug 2024 23:30:00 -0700 From: Wangyang Guo To: libc-alpha@sourceware.org Cc: fweimer@redhat.com, goldstein.w.n@gmail.com, tianyou.li@intel.com, Wangyang Guo Subject: [PATCH v3 1/5] malloc: Split _int_free() into 3 sub functions Date: Thu, 29 Aug 2024 14:27:28 +0800 Message-ID: <20240829062732.1663342-2-wangyang.guo@intel.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240829062732.1663342-1-wangyang.guo@intel.com> References: <20240829062732.1663342-1-wangyang.guo@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Split _int_free() into 3 smaller functions for flexible combination: * _int_free_check -- sanity check for free * tcache_free -- free memory to tcache (quick path) * _int_free_chunk -- free memory chunk (slow path) --- Changes in v3: - Add comments to the split functions. - Wrap out seldom executed tcache_double_free_verify() as noinline function. - Link to v2: https://sourceware.org/pipermail/libc-alpha/2024-August/159426.html No changes in v2 --- malloc/malloc.c | 133 ++++++++++++++++++++++++++++++------------------ 1 file changed, 84 insertions(+), 49 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index bcb6e5b83c..ef49a13ea7 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -1086,7 +1086,9 @@ typedef struct malloc_chunk* mchunkptr; /* Internal routines. */ static void* _int_malloc(mstate, size_t); -static void _int_free(mstate, mchunkptr, int); +static void _int_free (mstate, mchunkptr, int); +static void _int_free_check (mstate, mchunkptr, INTERNAL_SIZE_T); +static void _int_free_chunk (mstate, mchunkptr, INTERNAL_SIZE_T, int); static void _int_free_merge_chunk (mstate, mchunkptr, INTERNAL_SIZE_T); static INTERNAL_SIZE_T _int_free_create_chunk (mstate, mchunkptr, INTERNAL_SIZE_T, @@ -3206,6 +3208,57 @@ tcache_next (tcache_entry *e) return (tcache_entry *) REVEAL_PTR (e->next); } +/* Verify if the suspicious tcache_entry is double free. + It's not expected to execute very often, mark it as noinline. */ +static __attribute__ ((noinline)) void +tcache_double_free_verify (tcache_entry *e, size_t tc_idx) +{ + tcache_entry *tmp; + size_t cnt = 0; + LIBC_PROBE (memory_tcache_double_free, 2, e, tc_idx); + for (tmp = tcache->entries[tc_idx]; + tmp; + tmp = REVEAL_PTR (tmp->next), ++cnt) + { + if (cnt >= mp_.tcache_count) + malloc_printerr ("free(): too many chunks detected in tcache"); + if (__glibc_unlikely (!aligned_OK (tmp))) + malloc_printerr ("free(): unaligned chunk detected in tcache 2"); + if (tmp == e) + malloc_printerr ("free(): double free detected in tcache 2"); + /* If we get here, it was a coincidence. We've wasted a + few cycles, but don't abort. */ + } +} + +/* Try to free chunk to the tcache, if success return true. + Caller must ensure that chunk and size are valid. */ +static inline bool +tcache_free (mchunkptr p, INTERNAL_SIZE_T size) +{ + bool done = false; + size_t tc_idx = csize2tidx (size); + if (tcache != NULL && tc_idx < mp_.tcache_bins) + { + /* Check to see if it's already in the tcache. */ + tcache_entry *e = (tcache_entry *) chunk2mem (p); + + /* This test succeeds on double free. However, we don't 100% + trust it (it also matches random payload data at a 1 in + 2^ chance), so verify it's not an unlikely + coincidence before aborting. */ + if (__glibc_unlikely (e->key == tcache_key)) + tcache_double_free_verify (e, tc_idx); + + if (tcache->counts[tc_idx] < mp_.tcache_count) + { + tcache_put (p, tc_idx); + done = true; + } + } + return done; +} + static void tcache_thread_shutdown (void) { @@ -4490,14 +4543,9 @@ _int_malloc (mstate av, size_t bytes) ------------------------------ free ------------------------------ */ -static void -_int_free (mstate av, mchunkptr p, int have_lock) +static inline void +_int_free_check (mstate av, mchunkptr p, INTERNAL_SIZE_T size) { - INTERNAL_SIZE_T size; /* its size */ - mfastbinptr *fb; /* associated fastbin */ - - size = chunksize (p); - /* Little security check which won't hurt performance: the allocator never wraps around at the end of the address space. Therefore we can exclude some size values which might appear @@ -4510,48 +4558,15 @@ _int_free (mstate av, mchunkptr p, int have_lock) if (__glibc_unlikely (size < MINSIZE || !aligned_OK (size))) malloc_printerr ("free(): invalid size"); - check_inuse_chunk(av, p); - -#if USE_TCACHE - { - size_t tc_idx = csize2tidx (size); - if (tcache != NULL && tc_idx < mp_.tcache_bins) - { - /* Check to see if it's already in the tcache. */ - tcache_entry *e = (tcache_entry *) chunk2mem (p); - - /* This test succeeds on double free. However, we don't 100% - trust it (it also matches random payload data at a 1 in - 2^ chance), so verify it's not an unlikely - coincidence before aborting. */ - if (__glibc_unlikely (e->key == tcache_key)) - { - tcache_entry *tmp; - size_t cnt = 0; - LIBC_PROBE (memory_tcache_double_free, 2, e, tc_idx); - for (tmp = tcache->entries[tc_idx]; - tmp; - tmp = REVEAL_PTR (tmp->next), ++cnt) - { - if (cnt >= mp_.tcache_count) - malloc_printerr ("free(): too many chunks detected in tcache"); - if (__glibc_unlikely (!aligned_OK (tmp))) - malloc_printerr ("free(): unaligned chunk detected in tcache 2"); - if (tmp == e) - malloc_printerr ("free(): double free detected in tcache 2"); - /* If we get here, it was a coincidence. We've wasted a - few cycles, but don't abort. */ - } - } + check_inuse_chunk (av, p); +} - if (tcache->counts[tc_idx] < mp_.tcache_count) - { - tcache_put (p, tc_idx); - return; - } - } - } -#endif +/* Free chunk P of SIZE bytes to the arena, if arena lock is held, + set have_lock to 1. Caller must ensure chunk and size are valid. */ +static void +_int_free_chunk (mstate av, mchunkptr p, INTERNAL_SIZE_T size, int have_lock) +{ + mfastbinptr *fb; /* associated fastbin */ /* If eligible, place chunk on a fastbin so it can be found @@ -4657,6 +4672,26 @@ _int_free (mstate av, mchunkptr p, int have_lock) } } +/* Free chunk P to its arena AV, if arena lock held, set have_lock to 1. + It will perform sanity check, then try the fast path to free into + tcache. If the attempt not success, free the chunk to arena. */ +static void +_int_free (mstate av, mchunkptr p, int have_lock) +{ + INTERNAL_SIZE_T size; /* its size */ + + size = chunksize (p); + + _int_free_check (av, p, size); + +#if USE_TCACHE + if (tcache_free (p, size)) + return; +#endif + + _int_free_chunk (av, p, size, have_lock); +} + /* Try to merge chunk P of SIZE bytes with its neighbors. Put the resulting chunk on the appropriate bin list. P must not be on a bin list yet, and it can be in use. */