From patchwork Thu Mar 4 16:34:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szabolcs Nagy X-Patchwork-Id: 42257 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 2D9183AAA0C5; Thu, 4 Mar 2021 16:34:58 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2D9183AAA0C5 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1614875698; bh=x63UrQJbnN7ggisiq1fBWiYUb5wWDCQ9XiL1E7RWdZ0=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=dXCmirWVBWyvVD/O1FLnTWwPXsrvF4/en8zdATUadZ6TqBfxiugh/cIySTLPbdUzs WXHs3F2rkg+uB+s+yCIrw2PxI+Hh/54HzMQ1cJc6CwtcvdP8zbLU84wXwCKP5pixRc ipxRYXjlz75V48GRhUiQrlrVLjNd6htbHZt6i++s= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-eopbgr80089.outbound.protection.outlook.com [40.107.8.89]) by sourceware.org (Postfix) with ESMTPS id 0366B3AAA0B4 for ; Thu, 4 Mar 2021 16:34:54 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 0366B3AAA0B4 Received: from MRXP264CA0015.FRAP264.PROD.OUTLOOK.COM (2603:10a6:500:15::27) by DB9PR08MB6714.eurprd08.prod.outlook.com (2603:10a6:10:2a4::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17; Thu, 4 Mar 2021 16:34:52 +0000 Received: from VE1EUR03FT062.eop-EUR03.prod.protection.outlook.com (2603:10a6:500:15:cafe::3a) by MRXP264CA0015.outlook.office365.com (2603:10a6:500:15::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.23 via Frontend Transport; Thu, 4 Mar 2021 16:34:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; sourceware.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;sourceware.org; dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT062.mail.protection.outlook.com (10.152.18.252) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Thu, 4 Mar 2021 16:34:51 +0000 Received: ("Tessian outbound efd554c08f3f:v71"); Thu, 04 Mar 2021 16:34:51 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 0e2284d6beaa5322 X-CR-MTA-TID: 64aa7808 Received: from c00f180731ce.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 67CC4DC0-2DFF-4E26-8CB4-71C84BB25575.1; Thu, 04 Mar 2021 16:34:46 +0000 Received: from EUR05-DB8-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id c00f180731ce.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 04 Mar 2021 16:34:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OLdqvxSwgri+N1mAC4bXQc2FhQV/2NwBsLolhe+dVWc9oPvc1Bb268TVA5E223xerTvHN0r74SyDwIOSxDZ+GZoY2e4Fb9OEbKrbI2kh7TqaC1LSmPokMd/ZsIRKKbhVmQWyBQloRppfZUlMBYRc4IUX6S4mzUibl48nqfW+5tni2zgTKpVUZqFGrgOAm9L+gyuLtl+OwKdD6vR6RzgvLjnnhDJcFxtBKUiRBaanNiVVeyAmuUE2ZzaVOiPfFTkRH2YSfpVA8oF47hSGtAu+ELw/45ViW6vyXRUNjySVkr0MnL1/MDB+inJnBd/djRfCYwZyZsXQmE+iuZ7p5slKOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=x63UrQJbnN7ggisiq1fBWiYUb5wWDCQ9XiL1E7RWdZ0=; b=SX8oAWgYY95xop4BjINd3pItMQQd81z9+Lg3wH3CaT3lF8FaiO42Z4bDp0BRB+AQqZQ/K+lY2dNUCqiws4TkBSxSryLMkv43FFRJ5NlHlLtPEOfkQZy4ZCa1iZYgvplJwmI6QztwWOuS2wJXGxwgqeuo94EQuFI3o9pz6YeIs2F7hE/Zs7S2gpKd42FXuGPNMPqTa722iaTk2wBzJLDZ7lg3f1zgQ1GFSbCXQJs0Kca6oEni4Gfqdyfw7EqVFBLkNRRi4+YSCDyMYArn98GloQuelxNvJGDOnwqAwe/UDZ7SsXrxeG1cQshcZ7UL3KPEsTtcJIY35Fodrm2PotTb+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none Authentication-Results-Original: sourceware.org; dkim=none (message not signed) header.d=none;sourceware.org; dmarc=none action=none header.from=arm.com; Received: from PA4PR08MB6320.eurprd08.prod.outlook.com (2603:10a6:102:e5::9) by PAXPR08MB6480.eurprd08.prod.outlook.com (2603:10a6:102:155::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.19; Thu, 4 Mar 2021 16:34:45 +0000 Received: from PA4PR08MB6320.eurprd08.prod.outlook.com ([fe80::60f0:3773:69b8:e336]) by PA4PR08MB6320.eurprd08.prod.outlook.com ([fe80::60f0:3773:69b8:e336%2]) with mapi id 15.20.3912.021; Thu, 4 Mar 2021 16:34:45 +0000 To: libc-alpha@sourceware.org, Richard.Earnshaw@arm.com, DJ Delorie Subject: [PATCH 15/16] aarch64: Optimize __libc_mtag_tag_region Date: Thu, 4 Mar 2021 16:34:39 +0000 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: X-Originating-IP: [217.140.106.55] X-ClientProxiedBy: LO4P123CA0225.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a6::14) To PA4PR08MB6320.eurprd08.prod.outlook.com (2603:10a6:102:e5::9) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from localhost.localdomain (217.140.106.55) by LO4P123CA0225.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a6::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.25 via Frontend Transport; Thu, 4 Mar 2021 16:34:45 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 468d0d95-c654-4f9a-b2b9-08d8df2b6f8c X-MS-TrafficTypeDiagnostic: PAXPR08MB6480:|DB9PR08MB6714: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true NoDisclaimer: true X-MS-Oob-TLC-OOBClassifiers: OLM:10000;OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: XC4YotCNAmXqM8yZ54wioMCjmD/405nAe81oPo86bqNSD8nP1I67CTZeH3BntEiVUC7KUXXR2r83o/HIwH+ZkwC6KHTRRCjcmkP3yInd4+soy9QWvBJUVjD4epYvQHkPa3qIC5+d2Vt0y4PZY8J3zF43ZDVkYoet09sZh801GXdHV2zO61linoLh8Hyo/PmJPczvMmSqgIcY0roUhHSuOz9VZzCtrI6NIEUy8dMYZvZAMYMd26q72W9c2khtVa6I1yQOiOwxD1tT9Vq95sOGVcvgi0BS3zi3foEFEyuKANxPIUohAlGQRCS/av43rbM+zziiSjMhmBe2oVe1YOp3Wd7aRFzC/edozBlxNRGONwL6JqRI5y43+YTT4kL9a0hyMnU0UBVNAApzQFVQofJc+Y9A80QnUadEWPU+A8865ij3zyD/8ARaQ8ejACKfDK3Lcn586SezzfKvLUg2h+NOGQ5YYu5Y36Vu87YCht23nXvUOSsglztMG2hLt/UNykLdf/GiLsVAYLSHdAl/xXGOH/VLPk32jPeio1K+T/jrnlBWHtdmfxrROjwvQpNJXauJbyZGjKdkAqjhf4TfNC34og== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PA4PR08MB6320.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(396003)(39850400004)(346002)(376002)(366004)(6512007)(2906002)(2616005)(956004)(6916009)(316002)(83380400001)(69590400012)(16526019)(66946007)(6506007)(8676002)(36756003)(5660300002)(6486002)(86362001)(44832011)(52116002)(6666004)(8936002)(26005)(186003)(478600001)(66556008)(66476007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: P5FQaNlE74L4+U5nItXiwiT6MVoAgK6oMKf9i9qEDljeLy38DgwzgmMyzsPrr5htiw/538v2Jn54JsTkmRtREcY8uJ/zxCf7xjN02VNyx8jmcaofYq4Y+e1O6SkvyKhQYyhg5JfFrFo6rngc9c+BJFZMokWHH0A4XroxnSJ9EnAyJA4EG7PqrCYjelwsT0+4osX9YTL5U1UjZNJfAjsbzDAihLdNDaMKwuET30RhhOEZqS9mYfd30HO8bFCfj624rzpOvDBmudJdPzlLIaQsQlP4GAjIrnFB/gWnr/uhT1+qeKfRoZq5Jls/0zMJLIIKQS3yOKBbWfoj3ZnFX+dBGOqdY0C0D4d+KkQJMUDJijN7jybKI9PBdivhwNhSr7wp+uhrpesWOz9fJ7a76sdM4kHIra6rax4Jw4SJQfMNWiCh5OH2ByBIG49gu/UT2yoTkAm2pe3j+QXau7U+gpxZYVUmB4VIzQvOOJ4SY8fi6c9ymKLqbG0rxmFfm8gFjvqXNqTXGIf1QIaiBi4+mXtRexXlo0Xphbtb4eAsOlpICgGfGdpwO16qo0sSjSHn2LAwwWjo9wkA36Twqc5ElESznZwuWchdwCRTBIprYN1cwNzj5COGZLmoCKskyDkelEImQUg2p/8K7vMmIm+6+KdON+NuT7kGZ14qEu1F12S3OVBUYtLz8pu8KEUCj3P1VdRlfuqrMwAKwGOJMbKTDpQqu0GryE84mspTqSGhfeeJ6knE6I5iz9RFkI/cjOmBnpqucQZlc1T8Vh6V4i6Rne3OTheI9YKAUPCu+kG+3625iL+B/nM3lLMu8Bvyt5ny/w4WlVkqukVQpPQaLcI3XCvLddIuDClqCehDal71w7XSImQjlwFBNusi5YdHwr2VPZX1Vu6kVTD7vOFgQnY9ur3WnsF7p7vNhc0lgCxIuj68tXObmh9AfJ1zZZho/vECCgCLcYoCnYwMg1/M6o+fPm65m0X4tkYqravGryMnSp8aLCfRV7B2e09FvYZV5wVrpotp4I14yDjgXA1kgjP/mokZCoo9GqjHMnzPK249++mo87m+Bttmibb6vKKQtCfBRm6X9fDhaOLSRRy9TC6BnjsItanpRn8H5F0zUZFauNY7OW4ZXN8/1XGJCA5gRg+72jZVDrjWsI4DWRB5GlRW6jZ8iMuyvBBDz+S+VEgGTbBzNvGuqOpaE3rLT67/zj8ny4Vr3gldHGaonRAHyBLNb0IA34iurGS7DqI+xj/LHsaITQxaWLBBMOASPKs5nP3l64apvlkv9oy6DBeqJZcUk9MApGMBhSToFEVB6p8pZ6IufkJD8NsjZhvVc7IAwgOPvxu2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAXPR08MB6480 Original-Authentication-Results: sourceware.org; dkim=none (message not signed) header.d=none; sourceware.org; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT062.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: d0f001d3-778c-48fd-ffe6-08d8df2b6b73 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: P/pRRO2eG9QpQvDfvi3p2ptFBeCyxS1bbrkMwv4PzFAsn+o7RirJ8+w/WrZaC1FasQW07HEIxG5I0pH/49HRRq5aQ/PEaVGa71R0/KBkLR7BD/kOfjLoAPkmVSO0V0Re5yKn7m4a+JMeohydc4KZ6tctWxFBw3BEibEGanvKuUbpcPp98ACz+YklCOH+WYKCg8v74Vw5ZQsEJhTw5oMokUrtcy0Qc568G2QbUX9ogJv7oV6s5y038uUrzmw5dkqDZOLYGFknTistXOdDTGxOF7m3jJSbXaiqQn5lXhZqtrLX9+ORy2XcRuThQlDLoXQgpxKiJSyiJoDsWhKGidlOI0XCayV1uChlW0cyEl9r7svQygpb/4O9zA9ecslxwiUhjYF5H1iyj/KZJTs/nrOOSWnt6ic7c7qhWU9BKMNpenKhK0Bxd3qOC4PCFHvVlf708Gme9RvrPos1Lge3ejZdHJPVn71u7tna05kK9sBx7KN9u6ROEH8b2K6ctljQY8U0K3hLYTlfAto2OdC1/V46+CwWRFJGQ6S3BE8+QQQXdM7TAEh9SqSf23jWplTndTIOdaeBlBN0PBP3BGUUkbOhxMOwSMg3jDwg8jguTHaZLi9RxQkdM1KbdkIfL/Q5a+SJfopqdEjoYgi1ke/gIFlRZRQifeshKUgk2ZSyvaGcfTSN5rZJmSvDIvRV5qyL3NhZ X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(39850400004)(346002)(136003)(396003)(376002)(46966006)(36840700001)(82740400003)(2906002)(6512007)(956004)(86362001)(8676002)(2616005)(70206006)(26005)(83380400001)(356005)(82310400003)(8936002)(336012)(70586007)(6666004)(6506007)(36860700001)(186003)(47076005)(69590400012)(316002)(36756003)(44832011)(6862004)(6486002)(81166007)(478600001)(16526019)(5660300002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2021 16:34:51.9695 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 468d0d95-c654-4f9a-b2b9-08d8df2b6f8c X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR03FT062.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB6714 X-Spam-Status: No, score=-13.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, MSGID_FROM_MTA_HEADER, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Szabolcs Nagy via Libc-alpha From: Szabolcs Nagy Reply-To: Szabolcs Nagy Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" This is a target hook for memory tagging, the original was a naive implementation. The optimized version relies on "dc gva" to tag 64 bytes at a time for large allocations and optimizes small cases without adding too many branches. This was not benchmarked on real cpu, but expected to be faster than the naive implementation. --- sysdeps/aarch64/__mtag_tag_region.S | 98 +++++++++++++++++++++++------ 1 file changed, 80 insertions(+), 18 deletions(-) diff --git a/sysdeps/aarch64/__mtag_tag_region.S b/sysdeps/aarch64/__mtag_tag_region.S index 9a8a3ffb60..cae0c8f121 100644 --- a/sysdeps/aarch64/__mtag_tag_region.S +++ b/sysdeps/aarch64/__mtag_tag_region.S @@ -20,32 +20,94 @@ #ifdef USE_MTAG -/* Use the same register names and assignments as memset. */ - +/* Assumptions: + * + * ARMv8-a, AArch64, MTE, LP64 ABI. + * + * Interface contract: + * Address is 16 byte aligned and size is multiple of 16. + * Returns the passed pointer. + * The memory region may remain untagged if tagging is not enabled. + */ .arch armv8.5-a .arch_extension memtag -/* NB, only supported on variants with 64-bit pointers. */ +#define dstin x0 +#define count x1 +#define dst x2 +#define dstend x3 +#define tmp x4 +#define zva_val x4 + +ENTRY (__libc_mtag_tag_region) + PTR_ARG (0) + SIZE_ARG (1) + + add dstend, dstin, count -/* FIXME: This is a minimal implementation. We could do better than - this for larger values of COUNT. */ + cmp count, 96 + b.hi L(set_long) -#define dstin x0 -#define count x1 -#define dst x2 + tbnz count, 6, L(set96) -ENTRY_ALIGN(__libc_mtag_tag_region, 6) + /* Set 0, 16, 32, or 48 bytes. */ + lsr tmp, count, 5 + add tmp, dstin, tmp, lsl 4 + cbz count, L(end) + stg dstin, [dstin] + stg dstin, [tmp] + stg dstin, [dstend, -16] +L(end): + ret + + .p2align 4 + /* Set 64..96 bytes. Write 64 bytes from the start and + 32 bytes from the end. */ +L(set96): + st2g dstin, [dstin] + st2g dstin, [dstin, 32] + st2g dstin, [dstend, -32] + ret - mov dst, dstin -L(loop): - stg dst, [dst], #16 - subs count, count, 16 - bne L(loop) -#if 0 - /* This is not currently needed, since for now we are only called - to tag memory that is taggable. */ - ldg dstin, [dstin] // Recover the tag created (might be untagged). + .p2align 4 + /* Size is > 96 bytes. */ +L(set_long): + cmp count, 160 + b.lo L(no_zva) + +#ifndef SKIP_ZVA_CHECK + mrs zva_val, dczid_el0 + and zva_val, zva_val, 31 + cmp zva_val, 4 /* ZVA size is 64 bytes. */ + b.ne L(no_zva) #endif + st2g dstin, [dstin] + st2g dstin, [dstin, 32] + bic dst, dstin, 63 + sub count, dstend, dst /* Count is now 64 too large. */ + sub count, count, 128 /* Adjust count and bias for loop. */ + + .p2align 4 +L(zva_loop): + add dst, dst, 64 + dc gva, dst + subs count, count, 64 + b.hi L(zva_loop) + st2g dstin, [dstend, -64] + st2g dstin, [dstend, -32] ret + +L(no_zva): + sub dst, dstin, 32 /* Dst is biased by -32. */ + sub count, count, 64 /* Adjust count for loop. */ +L(no_zva_loop): + st2g dstin, [dst, 32] + st2g dstin, [dst, 64]! + subs count, count, 64 + b.hi L(no_zva_loop) + st2g dstin, [dstend, -64] + st2g dstin, [dstend, -32] + ret + END (__libc_mtag_tag_region) #endif /* USE_MTAG */