diff mbox series

[v3,1/5] malloc: Split _int_free() into 3 sub functions

Message ID 20240829062732.1663342-2-wangyang.guo@intel.com (mailing list archive)
State Superseded
Delegated to: Florian Weimer
Headers
Series malloc: TCACHE improvement for free and calloc |

Checks

Context Check Description
redhat-pt-bot/TryBot-apply_patch success Patch applied to master at the time it was sent
linaro-tcwg-bot/tcwg_glibc_build--master-arm success Build passed
linaro-tcwg-bot/tcwg_glibc_build--master-aarch64 success Build passed
linaro-tcwg-bot/tcwg_glibc_check--master-arm success Test passed
linaro-tcwg-bot/tcwg_glibc_check--master-aarch64 success Test passed

Commit Message

Wangyang Guo Aug. 29, 2024, 6:27 a.m. UTC 
  Split _int_free() into 3 smaller functions for flexible combination:
* _int_free_check -- sanity check for free
* tcache_free -- free memory to tcache (quick path)
* _int_free_chunk -- free memory chunk (slow path)

---
Changes in v3:
- Add comments to the split functions.
- Wrap out seldom executed tcache_double_free_verify() as noinline function.
- Link to v2: https://sourceware.org/pipermail/libc-alpha/2024-August/159426.html
No changes in v2
---
 malloc/malloc.c | 133 ++++++++++++++++++++++++++++++------------------
 1 file changed, 84 insertions(+), 49 deletions(-)

Comments

Florian Weimer Sept. 30, 2024, 12:27 p.m. UTC | #1 
* Wangyang Guo:

> Split _int_free() into 3 smaller functions for flexible combination:
> * _int_free_check -- sanity check for free
> * tcache_free -- free memory to tcache (quick path)
> * _int_free_chunk -- free memory chunk (slow path)
>
> ---
> Changes in v3:
> - Add comments to the split functions.
> - Wrap out seldom executed tcache_double_free_verify() as noinline function.
> - Link to v2: https://sourceware.org/pipermail/libc-alpha/2024-August/159426.html
> No changes in v2
> ---
>  malloc/malloc.c | 133 ++++++++++++++++++++++++++++++------------------
>  1 file changed, 84 insertions(+), 49 deletions(-)

Overall this looks good.  Do you have commit access?

> +/* Free chunk P of SIZE bytes to the arena, if arena lock is held,
> +   set have_lock to 1. Caller must ensure chunk and size are valid.  */

Suggest: “HAVE_LOCK indicates where the arena for P has already been locked.”
(There's another instance further down below.)

Is it okay if I make these adjustments for you?

Thanks,
Florian
Wangyang Guo Oct. 2, 2024, 1:22 a.m. UTC | #2 
On 9/30/2024 8:27 PM, Florian Weimer wrote:
> * Wangyang Guo:
> 
>> Split _int_free() into 3 smaller functions for flexible combination:
>> * _int_free_check -- sanity check for free
>> * tcache_free -- free memory to tcache (quick path)
>> * _int_free_chunk -- free memory chunk (slow path)
>>
>> ---
>> Changes in v3:
>> - Add comments to the split functions.
>> - Wrap out seldom executed tcache_double_free_verify() as noinline function.
>> - Link to v2: https://sourceware.org/pipermail/libc-alpha/2024-August/159426.html
>> No changes in v2
>> ---
>>   malloc/malloc.c | 133 ++++++++++++++++++++++++++++++------------------
>>   1 file changed, 84 insertions(+), 49 deletions(-)
> 
> Overall this looks good.  Do you have commit access?

I don't have commit access. It would be great if you can help to commit.
> 
>> +/* Free chunk P of SIZE bytes to the arena, if arena lock is held,
>> +   set have_lock to 1. Caller must ensure chunk and size are valid.  */
> 
> Suggest: “HAVE_LOCK indicates where the arena for P has already been locked.”
> (There's another instance further down below.)
> 
> Is it okay if I make these adjustments for you?

That's OK for me.

BR
Wangyang
Wangyang Guo Oct. 14, 2024, 2:03 a.m. UTC | #3 
On 10/2/2024 9:22 AM, Guo, Wangyang wrote:
> On 9/30/2024 8:27 PM, Florian Weimer wrote:
>> * Wangyang Guo:
>>
>>> Split _int_free() into 3 smaller functions for flexible combination:
>>> * _int_free_check -- sanity check for free
>>> * tcache_free -- free memory to tcache (quick path)
>>> * _int_free_chunk -- free memory chunk (slow path)
>>>
>>> ---
>>> Changes in v3:
>>> - Add comments to the split functions.
>>> - Wrap out seldom executed tcache_double_free_verify() as noinline 
>>> function.
>>> - Link to v2: 
>>> https://sourceware.org/pipermail/libc-alpha/2024-August/159426.html
>>> No changes in v2
>>> ---
>>>   malloc/malloc.c | 133 ++++++++++++++++++++++++++++++------------------
>>>   1 file changed, 84 insertions(+), 49 deletions(-)
>>
>> Overall this looks good.  Do you have commit access?
> 
> I don't have commit access. It would be great if you can help to commit.
>>
>>> +/* Free chunk P of SIZE bytes to the arena, if arena lock is held,
>>> +   set have_lock to 1. Caller must ensure chunk and size are valid.  */
>>
>> Suggest: “HAVE_LOCK indicates where the arena for P has already been 
>> locked.”
>> (There's another instance further down below.)
>>
>> Is it okay if I make these adjustments for you?
> 
> That's OK for me.

Hi Florian,

Is there anything else I can help for this patch series?

BR
Wangyang
Wangyang Guo Nov. 5, 2024, 1:26 a.m. UTC | #4 
On 10/14/2024 10:03 AM, Guo, Wangyang wrote:
> On 10/2/2024 9:22 AM, Guo, Wangyang wrote:
>> On 9/30/2024 8:27 PM, Florian Weimer wrote:
>>> * Wangyang Guo:
>>>
>>>> Split _int_free() into 3 smaller functions for flexible combination:
>>>> * _int_free_check -- sanity check for free
>>>> * tcache_free -- free memory to tcache (quick path)
>>>> * _int_free_chunk -- free memory chunk (slow path)
>>>>
>>>> ---
>>>> Changes in v3:
>>>> - Add comments to the split functions.
>>>> - Wrap out seldom executed tcache_double_free_verify() as noinline 
>>>> function.
>>>> - Link to v2: https://sourceware.org/pipermail/libc-alpha/2024- 
>>>> August/159426.html
>>>> No changes in v2
>>>> ---
>>>>   malloc/malloc.c | 133 +++++++++++++++++++++++++++++ 
>>>> +------------------
>>>>   1 file changed, 84 insertions(+), 49 deletions(-)
>>>
>>> Overall this looks good.  Do you have commit access?
>>
>> I don't have commit access. It would be great if you can help to commit.
>>>
>>>> +/* Free chunk P of SIZE bytes to the arena, if arena lock is held,
>>>> +   set have_lock to 1. Caller must ensure chunk and size are 
>>>> valid.  */
>>>
>>> Suggest: “HAVE_LOCK indicates where the arena for P has already been 
>>> locked.”
>>> (There's another instance further down below.)
>>>
>>> Is it okay if I make these adjustments for you?
>>
>> That's OK for me.
> 
> Hi Florian,
> 
> Is there anything else I can help for this patch series?
> 
> BR
> Wangyang
> 

PING.

BR
Wangyang
Wangyang Guo Nov. 11, 2024, 1:44 a.m. UTC | #5 
On 11/5/2024 9:26 AM, Guo, Wangyang wrote:

> On 10/14/2024 10:03 AM, Guo, Wangyang wrote:
>> On 10/2/2024 9:22 AM, Guo, Wangyang wrote:
>>> On 9/30/2024 8:27 PM, Florian Weimer wrote:
>>>> * Wangyang Guo:
>>>>
>>>>> Split _int_free() into 3 smaller functions for flexible combination:
>>>>> * _int_free_check -- sanity check for free
>>>>> * tcache_free -- free memory to tcache (quick path)
>>>>> * _int_free_chunk -- free memory chunk (slow path)
>>>>>
>>>>> ---
>>>>> Changes in v3:
>>>>> - Add comments to the split functions.
>>>>> - Wrap out seldom executed tcache_double_free_verify() as noinline 
>>>>> function.
>>>>> - Link to v2: https://sourceware.org/pipermail/libc-alpha/2024- 
>>>>> August/159426.html
>>>>> No changes in v2
>>>>> ---
>>>>>   malloc/malloc.c | 133 +++++++++++++++++++++++++++++ 
>>>>> +------------------
>>>>>   1 file changed, 84 insertions(+), 49 deletions(-)
>>>>
>>>> Overall this looks good.  Do you have commit access?
>>>
>>> I don't have commit access. It would be great if you can help to 
>>> commit.
>>>>
>>>>> +/* Free chunk P of SIZE bytes to the arena, if arena lock is held,
>>>>> +   set have_lock to 1. Caller must ensure chunk and size are 
>>>>> valid.  */
>>>>
>>>> Suggest: “HAVE_LOCK indicates where the arena for P has already 
>>>> been locked.”
>>>> (There's another instance further down below.)
>>>>
>>>> Is it okay if I make these adjustments for you?
>>>
>>> That's OK for me.
>>
>> Hi Florian,
>>
>> Is there anything else I can help for this patch series?
>>
>> BR
>> Wangyang
>>
>
> PING.
>
> BR
> Wangyang

PING.

BR
Wangyang
Wangyang Guo Nov. 18, 2024, 3:44 a.m. UTC | #6 
On 11/11/2024 9:44 AM, Guo, Wangyang wrote:

> On 11/5/2024 9:26 AM, Guo, Wangyang wrote:
>> On 10/14/2024 10:03 AM, Guo, Wangyang wrote:
>>> On 10/2/2024 9:22 AM, Guo, Wangyang wrote:
>>>> On 9/30/2024 8:27 PM, Florian Weimer wrote:
>>>>> * Wangyang Guo:
>>>>>
>>>>>> Split _int_free() into 3 smaller functions for flexible combination:
>>>>>> * _int_free_check -- sanity check for free
>>>>>> * tcache_free -- free memory to tcache (quick path)
>>>>>> * _int_free_chunk -- free memory chunk (slow path)
>>>>>>
>>>>>> ---
>>>>>> Changes in v3:
>>>>>> - Add comments to the split functions.
>>>>>> - Wrap out seldom executed tcache_double_free_verify() as 
>>>>>> noinline function.
>>>>>> - Link to v2: https://sourceware.org/pipermail/libc-alpha/2024- 
>>>>>> August/159426.html
>>>>>> No changes in v2
>>>>>> ---
>>>>>>   malloc/malloc.c | 133 +++++++++++++++++++++++++++++ 
>>>>>> +------------------
>>>>>>   1 file changed, 84 insertions(+), 49 deletions(-)
>>>>>
>>>>> Overall this looks good.  Do you have commit access?
>>>>
>>>> I don't have commit access. It would be great if you can help to 
>>>> commit.
>>>>>
>>>>>> +/* Free chunk P of SIZE bytes to the arena, if arena lock is held,
>>>>>> +   set have_lock to 1. Caller must ensure chunk and size are 
>>>>>> valid.  */
>>>>>
>>>>> Suggest: “HAVE_LOCK indicates where the arena for P has already 
>>>>> been locked.”
>>>>> (There's another instance further down below.)
>>>>>
>>>>> Is it okay if I make these adjustments for you?
>>>>
>>>> That's OK for me.
>>>
>>> Hi Florian,
>>>
>>> Is there anything else I can help for this patch series?
>>>
>> PING.
> PING.

PING.

BR
Wangyang
Wangyang Guo Nov. 25, 2024, 3:11 a.m. UTC | #7 
On 11/18/2024 11:44 AM, Guo, Wangyang wrote:

> On 11/11/2024 9:44 AM, Guo, Wangyang wrote:
>> On 11/5/2024 9:26 AM, Guo, Wangyang wrote:
>>> On 10/14/2024 10:03 AM, Guo, Wangyang wrote:
>>>> On 10/2/2024 9:22 AM, Guo, Wangyang wrote:
>>>>> On 9/30/2024 8:27 PM, Florian Weimer wrote:
>>>>>> * Wangyang Guo:
>>>>>>
>>>>>>> Split _int_free() into 3 smaller functions for flexible 
>>>>>>> combination:
>>>>>>> * _int_free_check -- sanity check for free
>>>>>>> * tcache_free -- free memory to tcache (quick path)
>>>>>>> * _int_free_chunk -- free memory chunk (slow path)
>>>>>>>
>>>>>>> ---
>>>>>>> Changes in v3:
>>>>>>> - Add comments to the split functions.
>>>>>>> - Wrap out seldom executed tcache_double_free_verify() as 
>>>>>>> noinline function.
>>>>>>> - Link to v2: https://sourceware.org/pipermail/libc-alpha/2024- 
>>>>>>> August/159426.html
>>>>>>> No changes in v2
>>>>>>> ---
>>>>>>>   malloc/malloc.c | 133 +++++++++++++++++++++++++++++ 
>>>>>>> +------------------
>>>>>>>   1 file changed, 84 insertions(+), 49 deletions(-)
>>>>>>
>>>>>> Overall this looks good.  Do you have commit access?
>>>>>
>>>>> I don't have commit access. It would be great if you can help to 
>>>>> commit.
>>>>>>
>>>>>>> +/* Free chunk P of SIZE bytes to the arena, if arena lock is held,
>>>>>>> +   set have_lock to 1. Caller must ensure chunk and size are 
>>>>>>> valid.  */
>>>>>>
>>>>>> Suggest: “HAVE_LOCK indicates where the arena for P has already 
>>>>>> been locked.”
>>>>>> (There's another instance further down below.)
>>>>>>
>>>>>> Is it okay if I make these adjustments for you?
>>>>>
>>>>> That's OK for me.
>>>>
>>>> Hi Florian,
>>>>
>>>> Is there anything else I can help for this patch series?
>>>>
>>> PING.
>> PING.
> PING.

PING.

BR
Wangyang
H.J. Lu Nov. 25, 2024, 4:12 a.m. UTC | #8 
On Mon, Nov 25, 2024 at 11:12 AM Guo, Wangyang <wangyang.guo@intel.com> wrote:
>
> On 11/18/2024 11:44 AM, Guo, Wangyang wrote:
>
> > On 11/11/2024 9:44 AM, Guo, Wangyang wrote:
> >> On 11/5/2024 9:26 AM, Guo, Wangyang wrote:
> >>> On 10/14/2024 10:03 AM, Guo, Wangyang wrote:
> >>>> On 10/2/2024 9:22 AM, Guo, Wangyang wrote:
> >>>>> On 9/30/2024 8:27 PM, Florian Weimer wrote:
> >>>>>> * Wangyang Guo:
> >>>>>>
> >>>>>>> Split _int_free() into 3 smaller functions for flexible
> >>>>>>> combination:
> >>>>>>> * _int_free_check -- sanity check for free
> >>>>>>> * tcache_free -- free memory to tcache (quick path)
> >>>>>>> * _int_free_chunk -- free memory chunk (slow path)
> >>>>>>>
> >>>>>>> ---
> >>>>>>> Changes in v3:
> >>>>>>> - Add comments to the split functions.
> >>>>>>> - Wrap out seldom executed tcache_double_free_verify() as
> >>>>>>> noinline function.
> >>>>>>> - Link to v2: https://sourceware.org/pipermail/libc-alpha/2024-
> >>>>>>> August/159426.html
> >>>>>>> No changes in v2
> >>>>>>> ---
> >>>>>>>   malloc/malloc.c | 133 +++++++++++++++++++++++++++++
> >>>>>>> +------------------
> >>>>>>>   1 file changed, 84 insertions(+), 49 deletions(-)
> >>>>>>
> >>>>>> Overall this looks good.  Do you have commit access?
> >>>>>
> >>>>> I don't have commit access. It would be great if you can help to
> >>>>> commit.
> >>>>>>
> >>>>>>> +/* Free chunk P of SIZE bytes to the arena, if arena lock is held,
> >>>>>>> +   set have_lock to 1. Caller must ensure chunk and size are
> >>>>>>> valid.  */
> >>>>>>
> >>>>>> Suggest: “HAVE_LOCK indicates where the arena for P has already
> >>>>>> been locked.”
> >>>>>> (There's another instance further down below.)
> >>>>>>
> >>>>>> Is it okay if I make these adjustments for you?

This is the patch I am checking with the adjustments suggested
by Florian.

> >>>>> That's OK for me.
> >>>>
> >>>> Hi Florian,
> >>>>
> >>>> Is there anything else I can help for this patch series?
> >>>>
> >>> PING.
> >> PING.
> > PING.
>
> PING.
>
> BR
> Wangyang
>
Wangyang Guo Nov. 25, 2024, 5:54 a.m. UTC | #9 
On 11/25/2024 12:12 PM, H.J. Lu wrote:

> On Mon, Nov 25, 2024 at 11:12 AM Guo, Wangyang<wangyang.guo@intel.com> wrote:
>
> On 11/18/2024 11:44 AM, Guo, Wangyang wrote:
>>> On 11/11/2024 9:44 AM, Guo, Wangyang wrote:
>>>> On 11/5/2024 9:26 AM, Guo, Wangyang wrote:
>>>>> On 10/14/2024 10:03 AM, Guo, Wangyang wrote:
>>>>>> On 10/2/2024 9:22 AM, Guo, Wangyang wrote:
>>>>>>> On 9/30/2024 8:27 PM, Florian Weimer wrote:
>>>>>>>> * Wangyang Guo:
>>>>>>>>
>>>>>>>>> Split _int_free() into 3 smaller functions for flexible
>>>>>>>>> combination:
>>>>>>>>> * _int_free_check -- sanity check for free
>>>>>>>>> * tcache_free -- free memory to tcache (quick path)
>>>>>>>>> * _int_free_chunk -- free memory chunk (slow path)
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Changes in v3:
>>>>>>>>> - Add comments to the split functions.
>>>>>>>>> - Wrap out seldom executed tcache_double_free_verify() as
>>>>>>>>> noinline function.
>>>>>>>>> - Link to v2:https://sourceware.org/pipermail/libc-alpha/2024-
>>>>>>>>> August/159426.html
>>>>>>>>> No changes in v2
>>>>>>>>> ---
>>>>>>>>>    malloc/malloc.c | 133 +++++++++++++++++++++++++++++
>>>>>>>>> +------------------
>>>>>>>>>    1 file changed, 84 insertions(+), 49 deletions(-)
>>>>>>>> Overall this looks good.  Do you have commit access?
>>>>>>> I don't have commit access. It would be great if you can help to
>>>>>>> commit.
>>>>>>>>> +/* Free chunk P of SIZE bytes to the arena, if arena lock is held,
>>>>>>>>> +   set have_lock to 1. Caller must ensure chunk and size are
>>>>>>>>> valid.  */
>>>>>>>> Suggest: “HAVE_LOCK indicates where the arena for P has already
>>>>>>>> been locked.”
>>>>>>>> (There's another instance further down below.)
>>>>>>>>
>>>>>>>> Is it okay if I make these adjustments for you?
> This is the patch I am checking with the adjustments suggested
> by Florian.
Thanks for the help. I will wrap it up as v4 patch.
diff mbox series

Patch

diff --git a/malloc/malloc.c b/malloc/malloc.c
index bcb6e5b83c..ef49a13ea7 100644
--- a/malloc/malloc.c
+++ b/malloc/malloc.c
@@ -1086,7 +1086,9 @@  typedef struct malloc_chunk* mchunkptr;
 /* Internal routines.  */
 
 static void*  _int_malloc(mstate, size_t);
-static void     _int_free(mstate, mchunkptr, int);
+static void _int_free (mstate, mchunkptr, int);
+static void _int_free_check (mstate, mchunkptr, INTERNAL_SIZE_T);
+static void _int_free_chunk (mstate, mchunkptr, INTERNAL_SIZE_T, int);
 static void _int_free_merge_chunk (mstate, mchunkptr, INTERNAL_SIZE_T);
 static INTERNAL_SIZE_T _int_free_create_chunk (mstate,
 					       mchunkptr, INTERNAL_SIZE_T,
@@ -3206,6 +3208,57 @@  tcache_next (tcache_entry *e)
   return (tcache_entry *) REVEAL_PTR (e->next);
 }
 
+/* Verify if the suspicious tcache_entry is double free.
+   It's not expected to execute very often, mark it as noinline.  */
+static __attribute__ ((noinline)) void
+tcache_double_free_verify (tcache_entry *e, size_t tc_idx)
+{
+  tcache_entry *tmp;
+  size_t cnt = 0;
+  LIBC_PROBE (memory_tcache_double_free, 2, e, tc_idx);
+  for (tmp = tcache->entries[tc_idx];
+       tmp;
+       tmp = REVEAL_PTR (tmp->next), ++cnt)
+    {
+      if (cnt >= mp_.tcache_count)
+	malloc_printerr ("free(): too many chunks detected in tcache");
+      if (__glibc_unlikely (!aligned_OK (tmp)))
+	malloc_printerr ("free(): unaligned chunk detected in tcache 2");
+      if (tmp == e)
+	malloc_printerr ("free(): double free detected in tcache 2");
+      /* If we get here, it was a coincidence.  We've wasted a
+	 few cycles, but don't abort.  */
+    }
+}
+
+/* Try to free chunk to the tcache, if success return true.
+   Caller must ensure that chunk and size are valid.  */
+static inline bool
+tcache_free (mchunkptr p, INTERNAL_SIZE_T size)
+{
+  bool done = false;
+  size_t tc_idx = csize2tidx (size);
+  if (tcache != NULL && tc_idx < mp_.tcache_bins)
+    {
+      /* Check to see if it's already in the tcache.  */
+      tcache_entry *e = (tcache_entry *) chunk2mem (p);
+
+      /* This test succeeds on double free.  However, we don't 100%
+	 trust it (it also matches random payload data at a 1 in
+	 2^<size_t> chance), so verify it's not an unlikely
+	 coincidence before aborting.  */
+      if (__glibc_unlikely (e->key == tcache_key))
+	tcache_double_free_verify (e, tc_idx);
+
+      if (tcache->counts[tc_idx] < mp_.tcache_count)
+	{
+	  tcache_put (p, tc_idx);
+	  done = true;
+	}
+    }
+  return done;
+}
+
 static void
 tcache_thread_shutdown (void)
 {
@@ -4490,14 +4543,9 @@  _int_malloc (mstate av, size_t bytes)
    ------------------------------ free ------------------------------
  */
 
-static void
-_int_free (mstate av, mchunkptr p, int have_lock)
+static inline void
+_int_free_check (mstate av, mchunkptr p, INTERNAL_SIZE_T size)
 {
-  INTERNAL_SIZE_T size;        /* its size */
-  mfastbinptr *fb;             /* associated fastbin */
-
-  size = chunksize (p);
-
   /* Little security check which won't hurt performance: the
      allocator never wraps around at the end of the address space.
      Therefore we can exclude some size values which might appear
@@ -4510,48 +4558,15 @@  _int_free (mstate av, mchunkptr p, int have_lock)
   if (__glibc_unlikely (size < MINSIZE || !aligned_OK (size)))
     malloc_printerr ("free(): invalid size");
 
-  check_inuse_chunk(av, p);
-
-#if USE_TCACHE
-  {
-    size_t tc_idx = csize2tidx (size);
-    if (tcache != NULL && tc_idx < mp_.tcache_bins)
-      {
-	/* Check to see if it's already in the tcache.  */
-	tcache_entry *e = (tcache_entry *) chunk2mem (p);
-
-	/* This test succeeds on double free.  However, we don't 100%
-	   trust it (it also matches random payload data at a 1 in
-	   2^<size_t> chance), so verify it's not an unlikely
-	   coincidence before aborting.  */
-	if (__glibc_unlikely (e->key == tcache_key))
-	  {
-	    tcache_entry *tmp;
-	    size_t cnt = 0;
-	    LIBC_PROBE (memory_tcache_double_free, 2, e, tc_idx);
-	    for (tmp = tcache->entries[tc_idx];
-		 tmp;
-		 tmp = REVEAL_PTR (tmp->next), ++cnt)
-	      {
-		if (cnt >= mp_.tcache_count)
-		  malloc_printerr ("free(): too many chunks detected in tcache");
-		if (__glibc_unlikely (!aligned_OK (tmp)))
-		  malloc_printerr ("free(): unaligned chunk detected in tcache 2");
-		if (tmp == e)
-		  malloc_printerr ("free(): double free detected in tcache 2");
-		/* If we get here, it was a coincidence.  We've wasted a
-		   few cycles, but don't abort.  */
-	      }
-	  }
+  check_inuse_chunk (av, p);
+}
 
-	if (tcache->counts[tc_idx] < mp_.tcache_count)
-	  {
-	    tcache_put (p, tc_idx);
-	    return;
-	  }
-      }
-  }
-#endif
+/* Free chunk P of SIZE bytes to the arena, if arena lock is held,
+   set have_lock to 1. Caller must ensure chunk and size are valid.  */
+static void
+_int_free_chunk (mstate av, mchunkptr p, INTERNAL_SIZE_T size, int have_lock)
+{
+  mfastbinptr *fb;             /* associated fastbin */
 
   /*
     If eligible, place chunk on a fastbin so it can be found
@@ -4657,6 +4672,26 @@  _int_free (mstate av, mchunkptr p, int have_lock)
   }
 }
 
+/* Free chunk P to its arena AV, if arena lock held, set have_lock to 1.
+   It will perform sanity check, then try the fast path to free into
+   tcache. If the attempt not success, free the chunk to arena.  */
+static void
+_int_free (mstate av, mchunkptr p, int have_lock)
+{
+  INTERNAL_SIZE_T size;        /* its size */
+
+  size = chunksize (p);
+
+  _int_free_check (av, p, size);
+
+#if USE_TCACHE
+  if (tcache_free (p, size))
+    return;
+#endif
+
+  _int_free_chunk (av, p, size, have_lock);
+}
+
 /* Try to merge chunk P of SIZE bytes with its neighbors.  Put the
    resulting chunk on the appropriate bin list.  P must not be on a
    bin list yet, and it can be in use.  */