From patchwork Tue Apr 15 13:49:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cupertino Miranda X-Patchwork-Id: 110500 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id F28BC3857BA0 for ; Tue, 15 Apr 2025 13:51:48 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org F28BC3857BA0 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2023-11-20 header.b=RZjhzi07; dkim=pass (1024-bit key, unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com header.b=dYu1KqXA X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by sourceware.org (Postfix) with ESMTPS id 4EA023858290 for ; Tue, 15 Apr 2025 13:49:54 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4EA023858290 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=oracle.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=oracle.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 4EA023858290 Authentication-Results: server2.sourceware.org; arc=pass smtp.remote-ip=205.220.165.32 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1744724994; cv=pass; b=b/qNr8dZvHFVBGq9q4g3CNlpd6761HG9O1XAjUOgk6tMPoMoXfJ1wBH2Y4/dveRcj9UxUEkRn2ic3ntbgzSNb53t0B9IMQ3uhmXdFaxHM5YT0730Ci/xCkKPhmxdNxo8lSUbrsbe+PNrIqy1dWUSw7QQc0ywlvX51IjMG3+Blsg= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1744724994; c=relaxed/simple; bh=9yvT9KMSxZhW5+xZjm1sVZHgUKkLK3iyBpHPumGdZPY=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-Id: MIME-Version; b=Zfw1uynemAXx7Hzwo8OwKDEvds0S2cDxzwWpTsc+nFYx3IMp5/Fl94ht1wa63OdSJVMDMhgq9OQnRKAEozlR/+427yp0KFoMchHLXRZ5FlWrovM/PvbQ5kSKm7lArVWlrUf1PHl+ZA7xPPrDcLxfG+zKOd1aJOMiK5lDKjEcvnM= ARC-Authentication-Results: i=2; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 4EA023858290 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 53F6fwqx021684; Tue, 15 Apr 2025 13:49:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= corp-2023-11-20; bh=r/TgdVQceR9V7ss1Ig/6eOXB7epwHJyMTkyKGjNAgTg=; b= RZjhzi07k5cGIIpIW2shIUmkry/h1MzxjnSRTS8OdNcUFGiKFqA6rMhycoLxj7qf iPJB8DK5aTj01NnUHOKSRcvYhm/3rV7Yka/hbFHMMl0Qf2LeJ8HBg1lhzs+Ug7lA IMALZTBDTAI/A3DiUNyWzfJKDDvy1IPt9rvW5rPvwxdJATHjwrjFkJ0SK6hXhrPp x6I+KHn+XpJzHSv/aIrwcDbchIzstIJd6RcNLBnkTiDIJlHqiRHuLrNEhRBkb1NS DGG2if76rpJng6FpWcpwislwQGiKl5hDEzIfEN3Q3dHYgw8VnP/mULmzAJgsNW34 YapNc0cXmKBc4G000qsAww== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 4618rd1m46-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Apr 2025 13:49:49 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.18.1.2/8.18.1.2) with ESMTP id 53FCiC1w038842; Tue, 15 Apr 2025 13:49:47 GMT Received: from ch1pr05cu001.outbound.protection.outlook.com (mail-northcentralusazlp17010000.outbound.protection.outlook.com [40.93.20.0]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 460d4rbt7n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Apr 2025 13:49:47 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tHvXD+Zw8x1psb7bu8DWk031NrrB881P3CeRe5imFRK0taIJbWvEMf9z9XvSk1LTQvwxtkhH3iq139omO6ylsI1Y6O1KcQ8VHA6LNdkWwrRlxChq4a9Nri0tk8WrbksYnpjIo7T4ZnVsPInwCdbn+dUal5Z8c0bGGmNfrQ78beAMs0zWiPi+j85cjk0BZXE3rH2bJBgWN2BSqmo7P1prAK7ysVWjLQWOt9ncRwrh6DSKyhshQ8Zxgdb1ZgJzE11yTe7Jx9ZmlQD5a4acv6mQbcVhs04MnJyvkw0ZRtrHKXGdzc2xMqZM2CCR2SzshAxkbI1AG5dCrAXYETq9spLvOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=r/TgdVQceR9V7ss1Ig/6eOXB7epwHJyMTkyKGjNAgTg=; b=jjNI+qfixLjEMyITTIKyByxuhwkyYHWdDh0+w1SwZI3wsHtiqjNBwSBBkaVIc6DbomsCLh9Fa2f+83WtbSI2xScgPPF3NNTnpPcrFbYLy/A6mbzQQCljZ+Wzh1DzJ0aV2YiSoRvBAiq2h5DzdS9/kp2rkFozmdqrZMumYQeEOvkvWhXY9S2wVgOwJMLU0v1suNiILIRBNB/hzlD/GsgKIbPDsTV385O7BsqBin6IUUVU5kK3Yo2i4N47N8AQ/7EwoJqi5CiBd/JCHvMI1CoECvL65lvCpMz3VQBLkKkH1GPVf4991dLqkYVGPpPpSZBCBQocehz/W/Y4jeA60dOGZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=r/TgdVQceR9V7ss1Ig/6eOXB7epwHJyMTkyKGjNAgTg=; b=dYu1KqXA+NEIDDYUA1tOLM69Fiodp0R6nn7S9eXSSZ9Iqn5meoYYbqHhnJ8DtBmOw/HCmcLdWuUYqg2s8KPr1SZHyjLKxCZFCruZ6FrnjmDqlDYn3marx8fboYcFXgz1r8BjDyfk6yj1IdsGGeuRCjOttum7uiFaLNauvEtOHQs= Received: from MN2PR10MB4382.namprd10.prod.outlook.com (2603:10b6:208:1d7::13) by CY8PR10MB6705.namprd10.prod.outlook.com (2603:10b6:930:91::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8632.36; Tue, 15 Apr 2025 13:49:43 +0000 Received: from MN2PR10MB4382.namprd10.prod.outlook.com ([fe80::5033:84a3:f348:fefb]) by MN2PR10MB4382.namprd10.prod.outlook.com ([fe80::5033:84a3:f348:fefb%4]) with mapi id 15.20.8632.030; Tue, 15 Apr 2025 13:49:43 +0000 From: Cupertino Miranda To: libc-alpha@sourceware.org Cc: jose.marchesi@oracle.com, elena.zannoni@oracle.com, Wilco.Dijkstra@arm.com, fweimer@redhat.com, dj@redhat.com, Cupertino Miranda Subject: [PATCH v6 2/3] malloc: add tcache support for large chunk caching Date: Tue, 15 Apr 2025 14:49:19 +0100 Message-Id: <20250415134920.287189-3-cupertino.miranda@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20250415134920.287189-1-cupertino.miranda@oracle.com> References: <20250415134920.287189-1-cupertino.miranda@oracle.com> X-ClientProxiedBy: LO4P123CA0245.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a7::16) To MN2PR10MB4382.namprd10.prod.outlook.com (2603:10b6:208:1d7::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN2PR10MB4382:EE_|CY8PR10MB6705:EE_ X-MS-Office365-Filtering-Correlation-Id: c960ee55-edbd-44d0-6727-08dd7c24606a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|366016; X-Microsoft-Antispam-Message-Info: vH1LBi8V6uiIJe+PaVyWFjN78RWIpNFLczfDaS+t/4di3KNVpq2pvmg39q2+770M5Ll6YSVlu5BIZqN63c6NXWsFB5XJX9suTSrQtHMRpDHRpQivpiRIO0u5ZVfMM/Y9+lmiFP2ctR5CCyOFw1KLQ5CD/sf8eUzw8FFUEcO55qKVqJwW7mbhgHa4E0LWj6ydcwo4VgeEzAINsm/RPmQVGKWr0bFf9SmXu9K8YHgvbSwqFluxa+o+l5TDYt5bLh0Oiqyn82vMNwyEpHYhei3D/mVqfmC/RCYbQtt9n/2wxGZIcttnzU6/bFWZpZV4/kJKA6jIbVlu6GEP/8f7aMSLhxyOvN8jm/QtL+YX2S0+GcocGO0gB6eUZDZlaGXXEd9D6ONwtlLEALMetLTC1ZICHwB7NRSaFcn7M7wc4oWTTNCayg1dZLw7b1rPjcmo+XfUMnlXSVLoHi5HLSww5ejB68xIh4GNVpJdsNV7O2FeJfPlX7a8gYLJB7WDI6M/+KYwKl8YDE84rY8s+RmaU0X2QkydqpcfKKEMI6a+ig8HTn1euLRjirZ+2vdIGNKnaEF2NNq5HQ1AXzPmFwWsAh9z6+GAtV259o1qGNt8AkwmQ2jXG10JV3pZfuZmcTxazWYnT8eKA+O6UQFxD3NkfptgJQU/AHt1j6wG8GgB2S2sz0BuzXK6BXHpXrIQbD/O3Z7CnywGK9eRSgEIAmg8hCHd5AT/wM61K5Ctiz82t+4/V3V3qKlgfGRE+VzNubJjevRy4JciRpt6rUiVez/PWNJ3mnHQtKiwk+7os9gVzLpnxqZrf/EHAes3XvHppbI3MzRPH2Ki9r1HW1cuL/tRnhVrCLZyaOi3QUpRSr+NVxxtCIHL4/jPDPxaYW93WvQ3yfIxOcVfwZb7Yep/avLRUbLZFqX9/qA3TzrP65L+YT9Z98HLbj4WkkEUJYRRSPbk1OnUaqa+UpDq2eIjolGOpbDgiCJ9vwvHPG9gJ/R0qW+SNq79y/5F0C825XN4DHrc6uKaa7Zx8lR0Loo0yg6j+cHrBwkCc221TiLYWlIuU1ac7ChJPOUJ0e4oYvB5FS/e9tRSWrvj3gfD6tevOi1rta0cLjRzkNGQ7f8gqNy0lZT9UtnbbqhmE+BxBBOOOlkTEFafL2naYGE7+5qXHKUKFEsXCcLfnlV1jqVBD7HOdfp1AVOg7dBzPw+Lr3igJSYo9l1j5fJ+VJI5KtB+Byvprf39QzRq6NfoH/bF1cEoPcfr9Lmt/O1huQHeg7BwHYYnHEyZX5+rhs7ncMXTDBnhGcZc2BPQNU1u3DVmklzHw4lPgMZCr3UsUJGW9izhywCHL5ovEwJHnVGc4/AqamhyweTHRAyHB3pt3lh63zdMoL9L/4s89TglBbsNvb8Hrd11Nk4TP55pKpBD6TadITodgLdDB1paF+rOp/0xyRrbJrDshYM= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MN2PR10MB4382.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(376014)(366016); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sXOIzUv2JlNtBRx6xkL07a88nU+pVUBL1n1Tl27N0dK69IoLw7bbDOQNmTO2maLjP7XB6RZBefQpRDzkbn2O1o6YRcZo/aUAOURVBy0dzs8dPY4i5XBrJGImCCkIG3lsBdYfL7RniW6oCt7s1al7dyiiPbONWXrMB4cdiNMWI+eWfA82ptAe5CGAcfnI2L2dl95Q3ZJdx9nzeag6Sv/7AXOBB5o3zT5/0wcu8DEzgfQPWQ8DdvtXdqyOO2XiR1dG+cf3rulvfSMF9igy3scYKfp13ag61SLaxNwCQWgD60bEtef+sMEGGnRNLUfI/Ho3W8LvyGhchSm0AYxkOXJPDdkBu8htLOt7hGDY1/0hxWCHQ1T8TUnpeg+hdO65W45AfBLguoBJu4Wj97phUEfwcOUQFxxspHDs11PmTytUx5LOC+c/jVVrGEzGFGCZfoloaUMpitMDHRmTYFxRbQ40958puPl859E1aoYcYI3S4h7LBOWOZk+lriiwRgIMwhuiHFl6zTrVUOx+YJeoXmLkiX/Ha2GnX3hN+pl/m9JJzMncv6BI/U8ttgPPcyv6NKOkVKU+0AF67znapntIMWZMGidTFHX4QuFWm9mUVFapkE6Lcx5fD85Hfby3JZu8DaJeQDn7Sftv9toAMgAj3OrDzIxe+XZbAkJ54c1Wm0H/GQRq/tOWBStbL10T/UyOs9PtHunXw7BAhDO/9MbuJF+j9SoVRLerF0cNf8F++maiFvMN3q0InbKVqO8oVRgcWuUULJaR7RqtcRalkLU6y5nQ1JQImphj5VUfsTgtlautlX2ICmhhdYTA2XMuTiASBhya1/G/IdVpMfPJF44GQHC4CdVYdm+ChTHDNOcDZjZmABkj6Y251F0rECV2a3ZnpoCg2vfqf4QSWcwfvu77DlB1SYlauGWaSPxM5jU7WaV9jiumBLfi05YzB9DvXO24Sy1CdZTr8SivbcD5OtWSpAxD79kAeNCpkTbOJXml6wFzg3/PgUtk47oYK191eJeZNJcd1uAogqdoC7vcPPDU7wXMu4l4Ax4aWK0nDlkpBeis4c0neBk8zbhX40J2gystlJF7eE8hYsV1Uk9QOBKDGAxLwj++V5SPEKZKShIj9kAGQsc+NKZQy5Ot7rQSMeyGY30W2oYsIZm+PqTnf4Q6PsrVyFMX+fyYjF8NBzxZKWXd5erNtE/VJIGLUUmlpvkYLHxUoVMeLtB/j5NzR2HDNBXKdGEf05omtyYCoVKlgP2RsSXckpulPDSbpby36oVJLUD/jPwAaBh3gY+4P3GyvCKN0cgtMjEx+HV6sUMalIN3FGi8E6HOLMQmLsi1d6DAxWnJP8f0xdbIn9bzMPI2uPygeWSvY3UbtIUzPZKZqqBfs5Ac/UqmNfXsZLqNcFEMvAHHfNEJ4IHF6P8sYxgdfxJZBljt+Wub5sNlCNOuGfD24EtWgL5CVsj5GZhzrYnmpPmxb/pHKzy3/wy7lDcEG8oBlqkT4iMLAIHIbpKgk+GUZCFNeL1Rjw0hAY6Daw9eP50fNlYaLYBTI6vzcNKPfQ3dfJuKFVjZ3mLifntnwPCTNHjdSWXJXdFuIUBxLbzrN59+9mfl4RXMDlMCIxc7UodYDA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: rCBM5DVlm5E/fYtz7bt/u6UTZ3En6Ryhm3Bzfo5j/Au/jGB8IODAo1T+wC0kEKlyQMFbdxSR1Y+MIGTIO5ZZWqsa0FqXFUj01R00C2bXelRiSX8le6dXvZsaUEWkm1UA+CNC+/N1dDGWCNwLkAIik2EIyjpvWwxOf00lkYXSXeNGXXJfRL0soPNMP0v1BJkBCTL69IvGImxkb3wZVV6eQjOCBAbn8PAIIReaJes9vppcfcfuCdLFAXI3R/pa2l6FAXf1OU68Isl5q7hAdnNla6UH7vi0zdMcXjg3fuVYVgy8Wmi2jRgF8gfI0vs8pvPuFvmkqG3LAorDiMrumAgy0odnoy2wxdirvrNega1L+Mug9ijSiNC9QwcQ1SNr8LSFTm4Al7wCCCXcwguGIb43h+hFvdRgrXBs6ArFjtAXVd8ALxn+wr8wIh84rRPPPth5PG3o26vbJAbn41nXlpYnIr/ChZyogXXlBw38Qib0LDHiuar7Bt+CsbZC9EeETFsINhlWvvA+84YqdA24HReKKLPFpKEwghgKk5Pgmdfy8inMdXqFDYi+ZsiJ1FHC8Gb09O6v5I/2kvqbZH6cbGPVD0ZlKw7ni87Qe6TUly6V07Q= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c960ee55-edbd-44d0-6727-08dd7c24606a X-MS-Exchange-CrossTenant-AuthSource: MN2PR10MB4382.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Apr 2025 13:49:43.7048 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rXbPM3DGK5HTXViPdS0eDDFrWiVpZZbAN3+Hmj/2AzJzdhnAznqMLoP+lUV8yHvXwIYfKLy/rLbvBLkQRvWCMzSBjLJn4jVPKuFLWUh7/6M= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6705 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-15_06,2025-04-15_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2502280000 definitions=main-2504150098 X-Proofpoint-ORIG-GUID: HHhuzL24v9Uh0INcojGbP2LFPOIRNx-8 X-Proofpoint-GUID: HHhuzL24v9Uh0INcojGbP2LFPOIRNx-8 X-Spam-Status: No, score=-12.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H5, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Existing tcache implementation in glibc seems to focus in caching smaller data size allocations, limiting the size of the allocation to 1KB. This patch changes tcache implementation to allow to cache any chunk size allocations. The implementation adds extra bins (linked-lists) which store chunks with different ranges of allocation sizes. Bin selection is done in multiples in powers of 2 and chunks are inserted in growing size ordering within the bin. The last bin contains all other sizes of allocations. This patch although by default preserves the same implementation, limitting caches to 1KB chunks, it now allows to increase the max size for the cached chunks with the tunable glibc.malloc.tcache_max. --- malloc/malloc.c | 159 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 115 insertions(+), 44 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index c0423a2e10..54769cbbd6 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -292,13 +292,19 @@ #if USE_TCACHE /* We want 64 entries. This is an arbitrary limit, which tunables can reduce. */ # define TCACHE_MAX_BINS 64 +/* Number of bins beyond the TCACHE_MAX_BINS reserved for ranges of chunksizes + to allow tcaches to cache chunks beyond ~1k size. */ +# define TCACHE_UNBOUND_SIZE_BINS 10 +# define TCACHE_ALL_BINS (TCACHE_MAX_BINS + TCACHE_UNBOUND_SIZE_BINS) # define MAX_TCACHE_SIZE tidx2usize (TCACHE_MAX_BINS-1) +# define TCACHE_FIXED_SIZE_BINS \ + (mp_.tcache_bins < TCACHE_MAX_BINS ? mp_.tcache_bins : TCACHE_MAX_BINS) /* Only used to pre-fill the tunables. */ -# define tidx2usize(idx) (((size_t) idx) * MALLOC_ALIGNMENT + MINSIZE - SIZE_SZ) +# define tidx2usize(idx) (((size_t) idx) * MALLOC_ALIGNMENT + MINSIZE - MALLOC_ALIGNMENT + 1) -/* When "x" is from chunksize(). */ -# define csize2tidx(x) (((x) - MINSIZE) / MALLOC_ALIGNMENT) +/* When "x" is from chunksize(). Only usable for single sized tcache bins. */ +# define fast_csize2tidx(x) (((x) - MINSIZE) / MALLOC_ALIGNMENT) /* When "x" is a user-provided size. */ # define usize2tidx(x) csize2tidx (request2size (x)) @@ -1929,7 +1935,7 @@ static struct malloc_par mp_ = , .tcache_count = TCACHE_FILL_COUNT, .tcache_bins = TCACHE_MAX_BINS, - .tcache_max_bytes = tidx2usize (TCACHE_MAX_BINS-1), + .tcache_max_bytes = MAX_TCACHE_SIZE, .tcache_unsorted_limit = 0 /* No limit. */ #endif }; @@ -3122,8 +3128,8 @@ typedef struct tcache_entry time), this is for performance reasons. */ typedef struct tcache_perthread_struct { - uint16_t counts[TCACHE_MAX_BINS]; - tcache_entry *entries[TCACHE_MAX_BINS]; + uint16_t counts[TCACHE_MAX_BINS + TCACHE_UNBOUND_SIZE_BINS]; + tcache_entry *entries[TCACHE_MAX_BINS + TCACHE_UNBOUND_SIZE_BINS]; } tcache_perthread_struct; static __thread bool tcache_shutting_down = false; @@ -3156,6 +3162,45 @@ tcache_key_initialize (void) } } +static __always_inline size_t +csize2tidx(size_t nb) +{ + if (__glibc_likely (nb < tidx2usize (TCACHE_MAX_BINS))) + return fast_csize2tidx(nb); + else if (__glibc_likely (mp_.tcache_bins < TCACHE_MAX_BINS)) + return TCACHE_MAX_BINS; + else + { + size_t idx = TCACHE_MAX_BINS + + __builtin_clz (tidx2usize (TCACHE_MAX_BINS-1)) + - __builtin_clz (nb); + return idx < TCACHE_ALL_BINS ? idx : TCACHE_ALL_BINS - 1; + } +} + +/* Returns the position in the tcache bin where the specified size is first found in. + This function is used by both tcache_get and tcache_put to locate the proper + position in the tcache bin linked-list. */ +static __always_inline tcache_entry ** +tcache_location_for_size (size_t nb, size_t tc_idx) +{ + if (__glibc_likely (nb <= tidx2usize (TCACHE_MAX_BINS-1))) + return &(tcache->entries[tc_idx]); + else + { + tcache_entry **tep = &(tcache->entries[tc_idx]); + tcache_entry *te = REVEAL_PTR (*tep); + while (te != NULL + && __glibc_unlikely (chunksize (mem2chunk (te)) < nb)) + { + tep = & (te->next); + te = REVEAL_PTR (te->next); + } + + return tep; + } +} + /* Caller must ensure that we know tc_idx is valid and there's room for more chunks. */ static __always_inline void @@ -3167,8 +3212,9 @@ tcache_put (mchunkptr chunk, size_t tc_idx) detect a double free. */ e->key = tcache_key; - e->next = PROTECT_PTR (&e->next, REVEAL_PTR (tcache->entries[tc_idx])); - tcache->entries[tc_idx] = PROTECT_PTR (&(tcache->entries[tc_idx]), e); + tcache_entry **entry = tcache_location_for_size (chunksize (chunk), tc_idx); + e->next = PROTECT_PTR (&e->next, REVEAL_PTR (*entry)); + *entry = PROTECT_PTR (entry, e); ++(tcache->counts[tc_idx]); } @@ -3176,7 +3222,7 @@ tcache_put (mchunkptr chunk, size_t tc_idx) available chunks to remove. Removes chunk from the middle of the list. */ static __always_inline void * -tcache_get_n (size_t tc_idx, tcache_entry **ep) +tcache_remove_entry (size_t tc_idx, tcache_entry **ep) { tcache_entry *e = REVEAL_PTR (*ep); @@ -3190,18 +3236,18 @@ tcache_get_n (size_t tc_idx, tcache_entry **ep) return (void *) e; } -/* Like the above, but removes from the head of the list. */ +/* Like the above, but traverses the list to find proper location to get the + entry within the respective tcache bin. */ static __always_inline void * -tcache_get (size_t tc_idx) +tcache_get (size_t nb, size_t tc_idx) { - return tcache_get_n (tc_idx, & tcache->entries[tc_idx]); -} + tcache_entry **entry = tcache_location_for_size (nb, tc_idx); + tcache_entry *e = REVEAL_PTR (*entry); + if (__glibc_unlikely (tc_idx >= TCACHE_MAX_BINS) + && (e == NULL || chunksize (mem2chunk (e)) != nb)) + return NULL; -/* Iterates through the tcache linked list. */ -static __always_inline tcache_entry * -tcache_next (tcache_entry *e) -{ - return (tcache_entry *) REVEAL_PTR (e->next); + return tcache_remove_entry (tc_idx, entry); } /* Check if tcache is available for alloc by corresponding tc_idx. */ @@ -3258,7 +3304,7 @@ tcache_thread_shutdown (void) /* Free all of the entries and the tcache itself back to the arena heap for coalescing. */ - for (i = 0; i < TCACHE_MAX_BINS; ++i) + for (i = 0; i < TCACHE_MAX_BINS + TCACHE_UNBOUND_SIZE_BINS; ++i) { while (REVEAL_PTR (tcache_tmp->entries[i])) { @@ -3317,7 +3363,7 @@ tcache_init(void) memset (tcache, 0, sizeof (tcache_perthread_struct)); } - for (int i = 0; i < mp_.tcache_bins; i++) + for (int i = 0; i < TCACHE_MAX_BINS + TCACHE_UNBOUND_SIZE_BINS; i++) tcache->entries[i] = PROTECT_PTR (&(tcache->entries[i]), NULL); } @@ -3344,7 +3390,7 @@ tcache_try_malloc (size_t bytes, void **memptr) MAYBE_INIT_TCACHE (); if (tcache_available (tc_idx)) - *memptr = tcache_get (tc_idx); + *memptr = tcache_get (tbytes, tc_idx); else *memptr = NULL; @@ -3373,8 +3419,20 @@ __libc_malloc2 (size_t bytes) if (!__malloc_initialized) ptmalloc_init (); +#if USE_TCACHE MAYBE_INIT_TCACHE (); + size_t nb = checked_request2size (bytes); + size_t tc_idx = csize2tidx (nb); + + if (tcache_available (tc_idx)) + { + void *tmp = tcache_get (nb, tc_idx); + if (tmp != NULL) + return tag_new_usable (tmp); + } +#endif + if (SINGLE_THREAD_P) { victim = tag_new_usable (_int_malloc (&main_arena, bytes)); @@ -3409,10 +3467,17 @@ void * __libc_malloc (size_t bytes) { #if USE_TCACHE - size_t tc_idx = csize2tidx (checked_request2size (bytes)); + size_t nb = checked_request2size (bytes); + size_t tc_idx = fast_csize2tidx (nb); - if (tcache_available (tc_idx)) - return tag_new_usable (tcache_get (tc_idx)); + /* Limit this call to chunks that fit in same dimention tcache bins. + Large chunk sizes are cached inside __libc_malloc2. */ + if (tc_idx < TCACHE_MAX_BINS + && tcache_available (tc_idx)) + { + tcache_entry **entry = &tcache->entries[tc_idx]; + return tag_new_usable (tcache_remove_entry (tc_idx, entry)); + } #endif return __libc_malloc2 (bytes); @@ -3683,20 +3748,28 @@ _mid_memalign (size_t alignment, size_t bytes, void *address) if (tcache_available (tc_idx)) { - /* The tcache itself isn't encoded, but the chain is. */ - tcache_entry **tep = & tcache->entries[tc_idx]; + tcache_entry **tep = tcache_location_for_size (tbytes, tc_idx); tcache_entry *te = REVEAL_PTR (*tep); + + /* Make sure returned chunk is of expected size. */ + if (te == NULL + || (tc_idx >= TCACHE_MAX_BINS + && chunksize (mem2chunk (te)) != tbytes)) + goto tcache_memalign_abort; + + /* The tcache itself isn't encoded, but the chain is. */ while (te != NULL && !PTR_IS_ALIGNED (te, alignment)) { tep = & (te->next); - te = tcache_next (te); + te = REVEAL_PTR (te->next); } if (te != NULL) { - void *victim = tcache_get_n (tc_idx, tep); + void *victim = tcache_remove_entry (tc_idx, tep); return tag_new_usable (victim); } } +tcache_memalign_abort: } #endif @@ -3983,8 +4056,8 @@ _int_malloc (mstate av, size_t bytes) #if USE_TCACHE /* While we're here, if we see other chunks of the same size, stash them in the tcache. */ - size_t tc_idx = csize2tidx (nb); - if (tcache != NULL && tc_idx < mp_.tcache_bins) + size_t tc_idx = fast_csize2tidx (nb); + if (tcache != NULL && tc_idx < TCACHE_FIXED_SIZE_BINS) { mchunkptr tc_victim; @@ -4044,8 +4117,8 @@ _int_malloc (mstate av, size_t bytes) #if USE_TCACHE /* While we're here, if we see other chunks of the same size, stash them in the tcache. */ - size_t tc_idx = csize2tidx (nb); - if (tcache != NULL && tc_idx < mp_.tcache_bins) + size_t tc_idx = fast_csize2tidx (nb); + if (__glibc_likely (tcache != NULL) && tc_idx < TCACHE_FIXED_SIZE_BINS) { mchunkptr tc_victim; @@ -4106,8 +4179,8 @@ _int_malloc (mstate av, size_t bytes) #if USE_TCACHE INTERNAL_SIZE_T tcache_nb = 0; - size_t tc_idx = csize2tidx (nb); - if (tcache != NULL && tc_idx < mp_.tcache_bins) + size_t tc_idx = fast_csize2tidx (nb); + if (tcache != NULL && tc_idx < TCACHE_FIXED_SIZE_BINS) tcache_nb = nb; int return_cached = 0; @@ -4285,7 +4358,7 @@ _int_malloc (mstate av, size_t bytes) && mp_.tcache_unsorted_limit > 0 && tcache_unsorted_count > mp_.tcache_unsorted_limit) { - return tcache_get (tc_idx); + return tcache_get (nb, tc_idx); } #endif @@ -4298,7 +4371,7 @@ _int_malloc (mstate av, size_t bytes) /* If all the small chunks we found ended up cached, return one now. */ if (return_cached) { - return tcache_get (tc_idx); + return tcache_get (nb, tc_idx); } #endif @@ -5530,14 +5603,12 @@ do_set_arena_max (size_t value) static __always_inline int do_set_tcache_max (size_t value) { - if (value <= MAX_TCACHE_SIZE) - { - LIBC_PROBE (memory_tunable_tcache_max_bytes, 2, value, mp_.tcache_max_bytes); - mp_.tcache_max_bytes = value; - mp_.tcache_bins = csize2tidx (request2size(value)) + 1; - return 1; - } - return 0; + LIBC_PROBE (memory_tunable_tcache_max_bytes, 2, value, mp_.tcache_max_bytes); + mp_.tcache_max_bytes = value; + mp_.tcache_bins = TCACHE_MAX_BINS + TCACHE_UNBOUND_SIZE_BINS; + mp_.tcache_bins = csize2tidx (request2size(value)) + 1; + + return 1; } static __always_inline int