From patchwork Thu Mar 4 16:30:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szabolcs Nagy X-Patchwork-Id: 42242 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B171A3892448; Thu, 4 Mar 2021 16:30:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B171A3892448 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1614875430; bh=h6wQf/Z4Tv9L2hHJ82WSBq583UHze61qcf3iO6fwWEU=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=r7FWOaT4PFidtRc6NEUYR4B0lFmqoUNQLQk9mPY9oZSuaFQT7uHrv7o4hT94FXgT8 09ODFf4MUu2MghtAYpcdQkMdW9WWXcHn8xgEA+SVsANR6D73ihy/PkhpomBkxvyLGb vfZmeoJayYF97ZaZIvp3Lpiefb0eP+3JTy1A3n58= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2088.outbound.protection.outlook.com [40.107.20.88]) by sourceware.org (Postfix) with ESMTPS id 484603858039 for ; Thu, 4 Mar 2021 16:30:26 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 484603858039 Received: from AM5PR0202CA0006.eurprd02.prod.outlook.com (2603:10a6:203:69::16) by PA4PR08MB6271.eurprd08.prod.outlook.com (2603:10a6:102:eb::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17; Thu, 4 Mar 2021 16:30:24 +0000 Received: from AM5EUR03FT003.eop-EUR03.prod.protection.outlook.com (2603:10a6:203:69:cafe::d3) by AM5PR0202CA0006.outlook.office365.com (2603:10a6:203:69::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Thu, 4 Mar 2021 16:30:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; sourceware.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;sourceware.org; dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT003.mail.protection.outlook.com (10.152.16.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Thu, 4 Mar 2021 16:30:23 +0000 Received: ("Tessian outbound 8f3114660466:v71"); Thu, 04 Mar 2021 16:30:23 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: cb0d4612812a2232 X-CR-MTA-TID: 64aa7808 Received: from 474bac6424a7.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id F0775ED0-7511-4BC3-996F-CAC5B917FE0E.1; Thu, 04 Mar 2021 16:30:17 +0000 Received: from FRA01-PR2-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 474bac6424a7.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 04 Mar 2021 16:30:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Typ429zkE1C2VKnnsgpL99XzUdnyF9DcLrpw+RRjhvgw1pzckP+9h6mxzDcTTs1X/w/oYYCDOLmG2v4lAI8hGXyTBBjAfNL+nQPvYpZzp0P4UzUlJ7azapFqyOTKZMF8bqZun9zOjosdmw143cxO9BAXMLMqBBknH+MOm0aRZuaP6636PNxV/lIsBgqicnqUReYx29QuZvhvOc+ahY2muFkdwxFMundr7cnAOXTFaLRoeF1exfTUXNMLg06LmSYG/eo/gTmoUh0T2u3tyXoEAy6Tmv4C1NM1vEKPuCtcUISnffJXTEJUzpb6/M4STIDykzyVV9RXQOgiqIsBCPHQ2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h6wQf/Z4Tv9L2hHJ82WSBq583UHze61qcf3iO6fwWEU=; b=MpBHfDhQLmjE6L1jvUZ+wCL68QOGZ7XgiU9zrPN/dv4Z6K2N/Ix+9xRS14aMAy57iWQx77MiU0GQgsjOR9keAAGbmqJoqKOH6Mtpq0CDHwPDCqV2EG+BAJikTf5AIVeXp9Y0GLtKxwPxgHrre2yNzaiQWs68o3MlJonOxSk1rI0LHjz/GTw2rzqc2/cZ0BvXEWwGi9Iufro8ij+X1pVR7Wh5lOoArdKcc7/Lr5lTpanbjWceDeN7JRW97HqS5SqpHQVyV01J0hxUM80SFjyHCr82MtkvA17ubS4jQCZQ4Ax223sm+uL96cDC9ArOI6Y94sXn6+1LGEmwVyukqykdqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none Authentication-Results-Original: sourceware.org; dkim=none (message not signed) header.d=none;sourceware.org; dmarc=none action=none header.from=arm.com; Received: from PA4PR08MB6320.eurprd08.prod.outlook.com (2603:10a6:102:e5::9) by PR2PR08MB4683.eurprd08.prod.outlook.com (2603:10a6:101:27::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.19; Thu, 4 Mar 2021 16:30:15 +0000 Received: from PA4PR08MB6320.eurprd08.prod.outlook.com ([fe80::60f0:3773:69b8:e336]) by PA4PR08MB6320.eurprd08.prod.outlook.com ([fe80::60f0:3773:69b8:e336%2]) with mapi id 15.20.3912.021; Thu, 4 Mar 2021 16:30:15 +0000 To: libc-alpha@sourceware.org, Richard.Earnshaw@arm.com, DJ Delorie Subject: [PATCH 00/16] memory tagging improvements Date: Thu, 4 Mar 2021 16:30:02 +0000 Message-Id: X-Mailer: git-send-email 2.17.1 X-Originating-IP: [217.140.106.55] X-ClientProxiedBy: SN4PR0601CA0001.namprd06.prod.outlook.com (2603:10b6:803:2f::11) To PA4PR08MB6320.eurprd08.prod.outlook.com (2603:10a6:102:e5::9) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from localhost.localdomain (217.140.106.55) by SN4PR0601CA0001.namprd06.prod.outlook.com (2603:10b6:803:2f::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Thu, 4 Mar 2021 16:30:14 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 6f23dedc-2a96-4ada-31e9-08d8df2acfb2 X-MS-TrafficTypeDiagnostic: PR2PR08MB4683:|PA4PR08MB6271: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true NoDisclaimer: true X-MS-Oob-TLC-OOBClassifiers: OLM:10000;OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: NBeAbPBrdu4dykLWkpbIpSAjh0ONcwPaM0DFHELXUXLKklQmFYOBdN5YbIYNgXv/udRftf2d7jshoz0eR3ZZI3P4Uth/RVny/eqzIcDMk+Xae7QhzlHNGwj0zSdVdwN5miGW25F2VHYToPj3ZkbAIC2qhup1FNai4ENIyRwkYS7ZTpbNy4tRuC5w2bsTc2sHzsbXagrp1ioJBZ1b6Bzq/gElsCzE/WS8bQwcD6jij8oRcGlVM4DQs8MzXljH0fhUQkAc/qwL7HcO3g9DE5hsprytYj+c95kLfp1vodmluCjWWEH1dOlCDy4vTzi98AVIGJJcKDlJAwBlzYanPmgnLTSx/nwfA+0tb0cgwBw55jQSAw2bxwGm+Ypb3Rnr2mmxQz5/AdUdq1vxHoiwdPHYAA5zsCOJ4UIZZsz0G9nu/pXeqzOGA3WeBBCZrvP1emr4JVk7ZhyXjVJzZv+Xj30+AbyFfDUHYPrYcds+vGn/xXfHk+DSp2r1dJE4z2Z3Po4qI4ktczQVSH/29btQFVp/uUe7GN/DkUjbBGa+WJImgO7AvNYAMBvLyzrjz+tzEzpw2w2i5iTERWaZyOXk9nYrDg== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PA4PR08MB6320.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(136003)(39850400004)(346002)(376002)(316002)(26005)(186003)(6916009)(16526019)(2616005)(52116002)(86362001)(956004)(6506007)(6512007)(6666004)(44832011)(478600001)(36756003)(2906002)(83380400001)(66556008)(6486002)(66476007)(8676002)(8936002)(5660300002)(66946007)(69590400012); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: gIkRYowbiwSbCLoAhUlJjdMOn82R/YUeBT+E9mGqOOAX1U8XvoCnqAAQiTkmozZah008C8isyl8/OVMPWL0xcT9p4nkSorBveKnCki+cXX3uoMIN73Ft4F8FKX5QvC+OUXSGcf59iSI0mIjyR0Bmh9INY5IqehTSAw80ts2yEa9u4W6WSMCMpaYEdeRuIeXxu5DVrlfNXdoKNdIq4A2spTclk7UWomd9jvtCELVb/aaVKRG8ukCUA6KCQQR862ANpoyTZs+ImCHvqXKYIbVLfXHno7zaCwWTr72Os9lP5RHFFWXVMdyFJpl9+L06149JHZr2d5TDpfYYDsRId1osRzzRZOviQ00za/4wpH6JUE0B13LjqkfI+G1rPVadVuBXRJbc6k/pHEtq8KVgsW9v2l1acm8xCAr+4pW0GEqkQskFY9d6GW2TH++2+JnZfsFIsFoTQBUTPNDxvzKjHqOE2jcdgUVZwzE6315TTe/ng1vSIaKefvy3NPSERqfygEd+r0WRAVu6eUsLWpXIFBe5h9BVdJfjlhOeF1fBNtpnJdvLr8iZAf2lysDG3JB1sHnQT7qF+VlMJGc0ZhlTXdkskoqVNFJ6KbrxYlG+NxQWUP2mf1kwYSk+St+uF6jr4ORvpzJMYtAcZ+afrjCt6+Vzy38mdmypriDROYgDczBQhZ29Uh8mtjjzDrPyXCmNZVIIrxlNq2LMBXERlMUWWXmDU8jsVasg4kaySHRkwkokaaXRe46x5ZZ6ypGZq8lAyFAltfDDIPE4P/iagRFhffKhRje/aklwfWTFbg0ldxiOkZFokWjuqsIc/WOPKGG9surQ4oa5yMroeqnhQXlLxuosFtiRMccIX+QM2BsIbAmu7h5+GXf0xndk47+sxDM4ogz8e9ei63wy9+uc8b2+GhzAJ3FhlLE7pjSYaAj9c+ynlmQd73b5S1MbdKlLL+XRdFhepfEHAlJ9oxcZFMkm2SiyKZNUik0mwMXx0fjGy/icO18UL/ZRzHO3+BkELvilBrejNY/LEok30fMmHJj/j/hhWAtpa+Gi98vG4LrtGWY1MBM0Cu0Tkvcrw1UyXN3kqxcV3RjU2HJiTgWjrrj5rwwde/vNdq+o1xmECvP4BRygmKqNv2UiIO5RSXhEpL475O/pxUISHoplzyCZtyKsC/QaFffJRJmORQCPPDlThQpxMQNf2qfpyxP44hfxoaJvOKtOaIxw1II1YJC2mkc92R20hzK974Y08jbNZZd5H9tRp7DCgKBS4GUoizLtttB47y63xtWO6mXShF0XlNwfN/PBIN2Owzxcgww6XzybjumKhRpnLjQuBJ4EgrU+k8B/Pord X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR2PR08MB4683 Original-Authentication-Results: sourceware.org; dkim=none (message not signed) header.d=none; sourceware.org; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT003.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: 97d36d39-a20d-4ff3-80aa-08d8df2aca82 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: E4buK5281UPDwdfl1pYFoqMfAfbc7muC6IHRSM+XVCkIfaaej/g3XbMuQD4yyXZbvYu7dAAj8gteiPlQNUcgLiH5+yt7C1qRdjZtwj0CSdmbQHnP6kTly8sD7IMX3WbKAgE79upcBcITlR7wsGpUjx6CqxNGKORf/gAsM9S8UABRtlOnLd3N72dxHJkzpZcQ64olcwmZVmWQNQUD7NbQln6DLiT2nLJm6VUVZ6Q0qRxCoY92lx15plNF+Yid+De6HqmhpBgXTCUm3kR7x1bn+35CocGHxcssZYISrnsZWco0z0XqyHRTRVaD2XfMISqwyNK6cyJCdpz5OZj7cg1kWlo4JgIYYNIvOv3A137XTaMUeey687uSsKuS3presCk30Ni+Kv00cJe8NhaUrGY3cMJ6yISwLAz6358yNpqoTgCg2tuCumU58lt6HkKOYG5S4ziCK/JCq901Xp/hROf6yBiv2Q9bjbpYv5GJbICe1Q02ia+rcHo2ctMKzyjEM5FP8rfe+ZLDf/nL9TTdlm00SoMJZzwHQAX5E/B/4XVlNam3fJcTDVh4vCg258zJQIs1xT6grUP7R+F2ok8OgSf2qJEgz+jM9ptHPc36e0bRriRFfLute8M8GowYT+FeRB+Ii6SeoQnMapHyXcAEKZHaPbOifBSNDB4JEXs2PlJsgLZdkOCULS5tGaCeHBtjkEU8 X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(39850400004)(136003)(346002)(396003)(376002)(36840700001)(46966006)(70586007)(5660300002)(69590400012)(70206006)(82310400003)(478600001)(356005)(956004)(2906002)(6862004)(6486002)(186003)(16526019)(36860700001)(44832011)(83380400001)(8936002)(6506007)(8676002)(47076005)(6666004)(6512007)(81166007)(336012)(36756003)(26005)(2616005)(86362001)(316002)(82740400003); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2021 16:30:23.8417 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6f23dedc-2a96-4ada-31e9-08d8df2acfb2 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM5EUR03FT003.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR08MB6271 X-Spam-Status: No, score=-7.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, MSGID_FROM_MTA_HEADER, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Szabolcs Nagy via Libc-alpha From: Szabolcs Nagy Reply-To: Szabolcs Nagy Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" This set tries to reduce the overhead of memory tagging support: when glibc is built with --enable-memory-tagging there are runtime checks that affect performance even if heap tagging is not enabled at runtime. I refactored the code to reduce internal abstractions and ifdefs. And a number of patches optimize heap tagging on aarch64 with MTE. Also available in the nsz/mtag branch. Issues found and fixed: - realloc failed to untag the old memory in some cases. - incorrect but useless macro definition in public sys/prctl.h. - incorrect but harmless internal MTAG_MMAP_FLAGS declaration. Issues found but not fixed: - large granule is not supported: both chunk header and user allocation must be tag granule aligned, but the code assumes that the chunk header is 2*SIZE_SZ so larger granule is not supported. - request2size and checked_request2size are not consistent which looks wrong (likely works in practice other than some sizes don't end up in fastbin or tcache that one might expected to, but i think this can break easily). Fixing it is not obvious because of assumptions about the chunk layout that is no longer true with mtag enabled. Other possible optimizations: - ifunc malloc apis good: near zero overhead for runtime dispatch. bad: malloc has to be built twice or larger refactoring is needed. - unconditional top byte zero internal data is tagged 0, only user allocation is tagged non-zero (we rely on this e.g. where MADV_DONTNEED is used: then the kernel zeros the tags so with non-zero tags internal pointers would not be able to access recycled memory), so tag_at(p) for internal data could be unconditional top byte clear. good: faster than the branch in tag_at, may improve security (?). bad: design change. - unconditional layout padding up to tag granule can be done unconditionally good: no runtime check, same layout with and without heap tagging. bad: wasting space with small allocations, design change. This patchset tried to avoid intrusive changes that affect behaviour. Benchmarked using bench-malloc-thread glibc ubenchmark on Neoverse N1. For size=16 the runtime check overhead is ~ 32% before the patch set ~ 16% after the patch set. On a practical workload or for large sizes the overhead should be tiny. Szabolcs Nagy (16): malloc: Fix a realloc crash with heap tagging [BZ 27468] Remove PR_TAGGED_ADDR_ENABLE from sys/prctl.h malloc: Move MTAG_MMAP_FLAGS definition malloc: Simplify __mtag_tag_new_usable malloc: Avoid taggig mmaped memory on free malloc: Ensure the generic mtag hooks are not used malloc: Refactor TAG_ macros to avoid indirection malloc: Use global flag instead of function pointer dispatch for mtag malloc: Only support zeroing and not arbitrary memset with mtag malloc: Change calloc when tagging is disabled malloc: Use branches instead of mtag_granule_mask malloc: Use mtag_enabled instead of USE_MTAG aarch64: inline __libc_mtag_address_get_tag aarch64: inline __libc_mtag_new_tag aarch64: Optimize __libc_mtag_tag_region aarch64: Optimize __libc_mtag_tag_zero_region include/malloc.h | 7 - malloc/arena.c | 45 +----- malloc/hooks.c | 20 ++- malloc/malloc.c | 177 ++++++++++++----------- sysdeps/aarch64/Makefile | 4 +- sysdeps/aarch64/__mtag_address_get_tag.S | 32 ---- sysdeps/aarch64/__mtag_memset_tag.S | 53 ------- sysdeps/aarch64/__mtag_new_tag.S | 37 ----- sysdeps/aarch64/__mtag_tag_region.S | 98 ++++++++++--- sysdeps/aarch64/__mtag_tag_zero_region.S | 113 +++++++++++++++ sysdeps/aarch64/libc-mtag.h | 32 ++-- sysdeps/generic/libc-mtag.h | 43 ++++-- sysdeps/unix/sysv/linux/sys/prctl.h | 4 - 13 files changed, 353 insertions(+), 312 deletions(-) delete mode 100644 sysdeps/aarch64/__mtag_address_get_tag.S delete mode 100644 sysdeps/aarch64/__mtag_memset_tag.S delete mode 100644 sysdeps/aarch64/__mtag_new_tag.S create mode 100644 sysdeps/aarch64/__mtag_tag_zero_region.S