From patchwork Mon Oct 23 12:38:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wilco Dijkstra X-Patchwork-Id: 78358 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3ADA43858433 for ; Mon, 23 Oct 2023 12:39:23 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from EUR02-VI1-obe.outbound.protection.outlook.com (mail-vi1eur02on2077.outbound.protection.outlook.com [40.107.241.77]) by sourceware.org (Postfix) with ESMTPS id 4CA5F3858D39 for ; Mon, 23 Oct 2023 12:39:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4CA5F3858D39 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 4CA5F3858D39 Authentication-Results: server2.sourceware.org; arc=fail smtp.remote-ip=40.107.241.77 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1698064749; cv=fail; b=aVXjosEhDRU5ExpDDoN5n2NeEe/N6aZ75ctmwZl9uSSEMQRnqrirt+yd5B2TGkYgg3Af1PZckMSGfAQWdwySpZ/N8t4UO9us1OQs2sg5EAYs5207ShoaHi4NZt4RI/Ad56dNxn3+Dhaf6FE/LrM3gRRgQymbREFymXr43VPMQOo= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1698064749; c=relaxed/simple; bh=NhnTLZiC9FySTJemNNmhWWXGx5rW/GLMq/qM5ymKp4I=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=mWZ1TPGejmrNVGbMPwZwxEHoeWwZqR8iL6Tt3EpXkjhaVLewLhBuCkMWpN1sT2toatI+3o1oMIGVbAkyo4IMo7pfmUFMkbvVuk8UQI1X6xmN50byiLYWawU3lCcqONwUCE36khz8pWHh5ONJzvLPlApKmECmx0nJ6zUSEIN0igI= ARC-Authentication-Results: i=2; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W2agRj59RjTpbTOM1P4+0ML+nNeapTF8SSebi0DFX3c=; b=5hpfWcvAtXRLs4FuQjjaL1HxVkxErutY+jqs0L03aVmehFmUUANKjyeG2HEvNKaEsCibzy9R4y5aImigXiVO7T5eEQfF+flcKtfBlmHH9Ab2CWrQryUd2mosHj9hypezvSK+NgJu73V702YsQKU8dU+3hH3GL2kcnBBgJduvlz4= Received: from AS9PR05CA0269.eurprd05.prod.outlook.com (2603:10a6:20b:493::15) by AS8PR08MB9219.eurprd08.prod.outlook.com (2603:10a6:20b:5a2::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.29; Mon, 23 Oct 2023 12:39:04 +0000 Received: from AMS0EPF000001AA.eurprd05.prod.outlook.com (2603:10a6:20b:493:cafe::ca) by AS9PR05CA0269.outlook.office365.com (2603:10a6:20b:493::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.33 via Frontend Transport; Mon, 23 Oct 2023 12:39:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AMS0EPF000001AA.mail.protection.outlook.com (10.167.16.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6933.15 via Frontend Transport; Mon, 23 Oct 2023 12:39:03 +0000 Received: ("Tessian outbound 36e2b13fe8d0:v215"); Mon, 23 Oct 2023 12:39:03 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: c69a0cac05338607 X-CR-MTA-TID: 64aa7808 Received: from dca0e6e88741.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 8B9928A6-7DFB-47BA-8F90-EE8F38D5EC95.1; Mon, 23 Oct 2023 12:38:57 +0000 Received: from EUR05-AM6-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id dca0e6e88741.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Mon, 23 Oct 2023 12:38:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ki+dq2BxsHiGIizrfPD1fdf7Cjk1RdUgNnslOydQysp5FFOI4Vys+molb0jOKTjLXfVWGiIrF1dS7F8qBlyz80mhglCluFiw3Lg1y4vdAmJOLk+dQP8NFRR92gO6W/2pGhHKM3bvWBHsfenS1JP2HO9+6a4ZWO+R6HZDHpmh2gcwzlitjvdSeZ2Z6VjsaJhLgHJ0D+XFd2APy6jtZi8n9FK1r5SZf08SgMK3E/28Qa8KYbwqxcLnFEvBK+dkm95kPax7rtTOE2Ucib1xAKLC3strhwcZt3STYqOxBUUGivTKQqlNpkURcNY07ldri0aRFyVwGkGUMf0NvaALxQ2s8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=W2agRj59RjTpbTOM1P4+0ML+nNeapTF8SSebi0DFX3c=; b=KmqzB1uhB+rkhEtcxhbM0gmRqwPgx+Z0OifNF9yQUm2927YbuUka3xYTXcT4RQw4wA+dpXnin1xtQ1JrNE9UFnNdFvkxbqEdDRIDqS7Cunt85b+4mCoaXJH6UkPMsdPymcrny8ngkPqkWCz+ko3paLlbc6tCEy/GvyYroCwloS1FEKk5XdQPjJ+Peno3orkkq7WpbHTr62s/SMjCZ8RWYH/VO1k1ujWfvcGIZhDkln5XPVrtHX+DFI91bpvIfaNob9At6Sd+ASC1TXjpSzKCOcl2qaemiygnIZLNQwQRLPW3nPSnxDIlibfVwyJetr1mYW/IHO1iaBgfd7mir0O2PQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W2agRj59RjTpbTOM1P4+0ML+nNeapTF8SSebi0DFX3c=; b=5hpfWcvAtXRLs4FuQjjaL1HxVkxErutY+jqs0L03aVmehFmUUANKjyeG2HEvNKaEsCibzy9R4y5aImigXiVO7T5eEQfF+flcKtfBlmHH9Ab2CWrQryUd2mosHj9hypezvSK+NgJu73V702YsQKU8dU+3hH3GL2kcnBBgJduvlz4= Received: from PAWPR08MB8982.eurprd08.prod.outlook.com (2603:10a6:102:33f::20) by AM8PR08MB5684.eurprd08.prod.outlook.com (2603:10a6:20b:1dc::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.33; Mon, 23 Oct 2023 12:38:53 +0000 Received: from PAWPR08MB8982.eurprd08.prod.outlook.com ([fe80::2939:a00c:af3:2551]) by PAWPR08MB8982.eurprd08.prod.outlook.com ([fe80::2939:a00c:af3:2551%4]) with mapi id 15.20.6907.032; Mon, 23 Oct 2023 12:38:53 +0000 From: Wilco Dijkstra To: 'GNU C Library' CC: Szabolcs Nagy , Adhemerval Zanella Subject: [PATCH v2] AArch64: Add support for MOPS memcpy/memmove/memset Thread-Topic: [PATCH v2] AArch64: Add support for MOPS memcpy/memmove/memset Thread-Index: AQHaBa10rMs0dgp/zEaw/FR4BB907Q== Date: Mon, 23 Oct 2023 12:38:53 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: PAWPR08MB8982:EE_|AM8PR08MB5684:EE_|AMS0EPF000001AA:EE_|AS8PR08MB9219:EE_ X-MS-Office365-Filtering-Correlation-Id: 9757d22f-7737-4638-946c-08dbd3c50a56 x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: cu3JotDrKZJQTyG1pGmFqgqg6JDJWrllVENKKO+dAHqYLzMyN0hkLvlQyq1toY6VhWwf5Vpd32fagFj9eQPTN2i57VwAi0+oqHis4VuCfVG4jHccQWeZSY/HBzN2dbvmhywvJZE64vc3rfV32YQ7gYLBH2lvtSRqqCAER6GXFDg/nhGpHSKFvcyh2OW0gdXKYFXUi69wmKSgXRyDw4f261c3RsTiqVhOFd+VCcfsXbrs7Wxr8N2jrmWc1kosbHTmw0yrnq33LCldiKmPvFOKcE/LvtarC+j7tDytG1TwDowhpWyPp5ZntUN6Z10KN2K3Vwrmku6OcokUa/jiBUpcNM2UwivaeW+EbvgQc497dWxpwUtkwfXS3iRYcfOK/zgIMK2r8iFQkpDc+W1oor/zttmHzJP2MX6QPKKBHYiXH67UrB/NTsbL8kS57XnvmfSnrQuhcNoIed/cMDm4CtpDhWYevV0c91MKiLTIahi8mabCp1JbcHSpaTm3HFJx1p/KhK+AUZ9tRSB/S/PZmZjiW0JIlxTpTj5ezK4vprsuVpmMEvESFQkxazfMac4zZ2JMOyDbv30sOD+1PV4WXl+v2BGop5u9R0A5Cb3/3UqjcaKYlrEI/QwrK6uDXz0NYF+E X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAWPR08MB8982.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(39850400004)(376002)(136003)(396003)(346002)(366004)(230922051799003)(186009)(451199024)(64100799003)(1800799009)(38070700009)(38100700002)(2906002)(30864003)(41300700001)(55016003)(5660300002)(52536014)(86362001)(33656002)(8676002)(4326008)(8936002)(7696005)(71200400001)(478600001)(6506007)(26005)(6916009)(91956017)(122000001)(316002)(66946007)(76116006)(64756008)(66446008)(66476007)(54906003)(66556008)(9686003)(2004002); DIR:OUT; SFP:1101; MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM8PR08MB5684 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AMS0EPF000001AA.eurprd05.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 7a9b0bff-6606-42c3-bf70-08dbd3c50406 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oCB4q+Ioa2+q9WAhxHsOiImzU0Cvuw4tJ4le7mlRf0216HeAUZ/Objv+K3a3YTZ/5S+DRZk6+957R4StEqcjmqYoMAQcYTsY3nantV8QMVuAocc2jFueai6z5VIyIaad55GyydFQLLzDwtUzaNCZr1rU+N7AwX41i+1awperLcxvnPzuSQY6qHWDLz27mBzQEB4UitO1limTUry38cFMnrBQJt9zJ5BKvgRG2ip5CCgS/Pz9LZc7YuR2YnEp4AkKrFrrRkt/72hbzg/MB3ixJqvagRXkiW1/x3ZJePW31dXUJCsIQXz6vuduAEVDSRpmN3+30hkXbxkyMSCVMBenMk/5omWaAbxY+NRgiuXP6q6m1Cu1BETNCIZ+cmCxpKJkE92+fKLCvE9CceE+bqaxMv05+Af1nNJNWkDVwSNMrWd0gdE48LeTaSgnPHvg6Xu8d/DuiESXrO0jqRND7G4a4pNm9dlewvHU82StR3mgyfkFFOpMIL5k6r9ogrAaeS8AEcAyhTyyiOcax20qvJvpTi3q5KicN58JIjQbPVSrD1tCMoUDHS7YcyQV27yZpjaU1x19tFRTP1kcJvVxF6041Dsxnq+Dmp3MFx2/3UTI7RJfc9CU36Jp742TjmaHdHg8o4LHCkKH1Dz+tGoGaSLv8A5/7tVkaRi90nl/01C9fyEAn6NmtZtwdshki2hdTDbFX5bNQRaWqiaPgqQ1g+qg6VEvaVHmPhfYksx0B9y65RUUdsg39odABInBC21vVgUr X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230031)(4636009)(396003)(39860400002)(376002)(136003)(346002)(230922051799003)(186009)(64100799003)(451199024)(1800799009)(82310400011)(36840700001)(46966006)(40470700004)(30864003)(55016003)(2906002)(36860700001)(316002)(6916009)(70206006)(70586007)(54906003)(81166007)(82740400003)(107886003)(356005)(6506007)(478600001)(7696005)(9686003)(40480700001)(47076005)(336012)(41300700001)(4326008)(40460700003)(52536014)(86362001)(5660300002)(33656002)(8936002)(8676002)(26005)(2004002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2023 12:39:03.7896 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9757d22f-7737-4638-946c-08dbd3c50a56 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AMS0EPF000001AA.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB9219 X-Spam-Status: No, score=-10.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, FORGED_SPF_HELO, GIT_PATCH_0, KAM_DMARC_NONE, KAM_LOTSOFHASH, KAM_SHORT, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org v2: remove hidden defines, split into separate files Add support for MOPS in cpu_features and INIT_ARCH. Add ifuncs using MOPS for memcpy, memmove and memset (use .inst for now so it works with all binutils versions without needing complex configure and conditional compilation). OK for commit? Reviewed-by: Szabolcs Nagy diff --git a/sysdeps/aarch64/multiarch/Makefile b/sysdeps/aarch64/multiarch/Makefile index 223777d94e350fdfd1bb82a4b38eea4653d63057..e6099548b96b80b656f3f22cd8d5598634647360 100644 --- a/sysdeps/aarch64/multiarch/Makefile +++ b/sysdeps/aarch64/multiarch/Makefile @@ -5,14 +5,17 @@ sysdep_routines += \ memcpy_a64fx \ memcpy_falkor \ memcpy_generic \ + memcpy_mops \ memcpy_sve \ memcpy_thunderx \ memcpy_thunderx2 \ + memmove_mops \ memset_a64fx \ memset_emag \ memset_falkor \ memset_generic \ memset_kunpeng \ + memset_mops \ strlen_asimd \ strlen_mte \ # sysdep_routines diff --git a/sysdeps/aarch64/multiarch/ifunc-impl-list.c b/sysdeps/aarch64/multiarch/ifunc-impl-list.c index d274f01fdbd1c99d7f872bbdcdc7b2a1cbc7b042..da7f1153778efd41a15ef5aa900252f9e492b18d 100644 --- a/sysdeps/aarch64/multiarch/ifunc-impl-list.c +++ b/sysdeps/aarch64/multiarch/ifunc-impl-list.c @@ -41,6 +41,7 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, memcpy, sve, __memcpy_a64fx) IFUNC_IMPL_ADD (array, i, memcpy, sve, __memcpy_sve) #endif + IFUNC_IMPL_ADD (array, i, memcpy, mops, __memcpy_mops) IFUNC_IMPL_ADD (array, i, memcpy, 1, __memcpy_generic)) IFUNC_IMPL (i, name, memmove, IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_thunderx) @@ -50,6 +51,7 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, memmove, sve, __memmove_a64fx) IFUNC_IMPL_ADD (array, i, memmove, sve, __memmove_sve) #endif + IFUNC_IMPL_ADD (array, i, memmove, mops, __memmove_mops) IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_generic)) IFUNC_IMPL (i, name, memset, /* Enable this on non-falkor processors too so that other cores @@ -60,6 +62,7 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, #if HAVE_AARCH64_SVE_ASM IFUNC_IMPL_ADD (array, i, memset, sve && zva_size == 256, __memset_a64fx) #endif + IFUNC_IMPL_ADD (array, i, memset, mops, __memset_mops) IFUNC_IMPL_ADD (array, i, memset, 1, __memset_generic)) IFUNC_IMPL (i, name, memchr, IFUNC_IMPL_ADD (array, i, memchr, !mte, __memchr_nosimd) diff --git a/sysdeps/aarch64/multiarch/init-arch.h b/sysdeps/aarch64/multiarch/init-arch.h index 6de081e3814d55812572c889fb8749e0e4e6222a..e23e6ff29042a68ce5499eddcc35b4f7f2ffa14d 100644 --- a/sysdeps/aarch64/multiarch/init-arch.h +++ b/sysdeps/aarch64/multiarch/init-arch.h @@ -35,4 +35,6 @@ bool __attribute__((unused)) mte = \ MTE_ENABLED (); \ bool __attribute__((unused)) sve = \ - GLRO(dl_aarch64_cpu_features).sve; + GLRO(dl_aarch64_cpu_features).sve; \ + bool __attribute__((unused)) mops = \ + GLRO(dl_aarch64_cpu_features).mops; diff --git a/sysdeps/aarch64/multiarch/memcpy.c b/sysdeps/aarch64/multiarch/memcpy.c index 3aae915c5f479fe5426399fbaa2f951d5771a997..9aace954cbfd1eb3e2b35e570e4eb31bbb3c6cfe 100644 --- a/sysdeps/aarch64/multiarch/memcpy.c +++ b/sysdeps/aarch64/multiarch/memcpy.c @@ -34,12 +34,16 @@ extern __typeof (__redirect_memcpy) __memcpy_thunderx2 attribute_hidden; extern __typeof (__redirect_memcpy) __memcpy_falkor attribute_hidden; extern __typeof (__redirect_memcpy) __memcpy_a64fx attribute_hidden; extern __typeof (__redirect_memcpy) __memcpy_sve attribute_hidden; +extern __typeof (__redirect_memcpy) __memcpy_mops attribute_hidden; static inline __typeof (__redirect_memcpy) * select_memcpy_ifunc (void) { INIT_ARCH (); + if (mops) + return __memcpy_mops; + if (sve && HAVE_AARCH64_SVE_ASM) { if (IS_A64FX (midr)) diff --git a/sysdeps/aarch64/multiarch/memcpy_mops.S b/sysdeps/aarch64/multiarch/memcpy_mops.S new file mode 100644 index 0000000000000000000000000000000000000000..46856296640267c7b8d9d7d8002296fe9404849d --- /dev/null +++ b/sysdeps/aarch64/multiarch/memcpy_mops.S @@ -0,0 +1,39 @@ +/* Optimized memcpy for MOPS. + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include + +/* Assumptions: + * + * AArch64, MOPS. + * + */ + +ENTRY (__memcpy_mops) + PTR_ARG (0) + PTR_ARG (1) + SIZE_ARG (2) + + mov x3, x0 + .inst 0x19010443 /* cpyfp [x3]!, [x1]!, x2! */ + .inst 0x19410443 /* cpyfm [x3]!, [x1]!, x2! */ + .inst 0x19810443 /* cpyfe [x3]!, [x1]!, x2! */ + ret + +END (__memcpy_mops) diff --git a/sysdeps/aarch64/multiarch/memmove.c b/sysdeps/aarch64/multiarch/memmove.c index 312f90f111868c71dd1f32f4a175a28e93c948ee..fd346e7b73a86a076ba8e1cdd7fd588098333f48 100644 --- a/sysdeps/aarch64/multiarch/memmove.c +++ b/sysdeps/aarch64/multiarch/memmove.c @@ -34,12 +34,16 @@ extern __typeof (__redirect_memmove) __memmove_thunderx2 attribute_hidden; extern __typeof (__redirect_memmove) __memmove_falkor attribute_hidden; extern __typeof (__redirect_memmove) __memmove_a64fx attribute_hidden; extern __typeof (__redirect_memmove) __memmove_sve attribute_hidden; +extern __typeof (__redirect_memmove) __memmove_mops attribute_hidden; static inline __typeof (__redirect_memmove) * select_memmove_ifunc (void) { INIT_ARCH (); + if (mops) + return __memmove_mops; + if (sve && HAVE_AARCH64_SVE_ASM) { if (IS_A64FX (midr)) diff --git a/sysdeps/aarch64/multiarch/memmove_mops.S b/sysdeps/aarch64/multiarch/memmove_mops.S new file mode 100644 index 0000000000000000000000000000000000000000..c5ea66be3a081b18013c911acf3d4d493f3f2a8f --- /dev/null +++ b/sysdeps/aarch64/multiarch/memmove_mops.S @@ -0,0 +1,39 @@ +/* Optimized memmove for MOPS. + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include + +/* Assumptions: + * + * AArch64, MOPS. + * + */ + +ENTRY (__memmove_mops) + PTR_ARG (0) + PTR_ARG (1) + SIZE_ARG (2) + + mov x3, x0 + .inst 0x1d010443 /* cpyp [x3]!, [x1]!, x2! */ + .inst 0x1d410443 /* cpym [x3]!, [x1]!, x2! */ + .inst 0x1d810443 /* cpye [x3]!, [x1]!, x2! */ + ret + +END (__memmove_mops) diff --git a/sysdeps/aarch64/multiarch/memset.c b/sysdeps/aarch64/multiarch/memset.c index f9c81d3d8e46ec86d9f100835e1c43210899770a..23fc66e15879847557b0e4f6941f03bc7ac5cab9 100644 --- a/sysdeps/aarch64/multiarch/memset.c +++ b/sysdeps/aarch64/multiarch/memset.c @@ -33,12 +33,16 @@ extern __typeof (__redirect_memset) __memset_emag attribute_hidden; extern __typeof (__redirect_memset) __memset_kunpeng attribute_hidden; extern __typeof (__redirect_memset) __memset_a64fx attribute_hidden; extern __typeof (__redirect_memset) __memset_generic attribute_hidden; +extern __typeof (__redirect_memset) __memset_mops attribute_hidden; static inline __typeof (__redirect_memset) * select_memset_ifunc (void) { INIT_ARCH (); + if (mops) + return __memset_mops; + if (sve && HAVE_AARCH64_SVE_ASM) { if (IS_A64FX (midr) && zva_size == 256) diff --git a/sysdeps/aarch64/multiarch/memset_mops.S b/sysdeps/aarch64/multiarch/memset_mops.S new file mode 100644 index 0000000000000000000000000000000000000000..ca820b8636204e9a2c59f8786d9cf805719a5930 --- /dev/null +++ b/sysdeps/aarch64/multiarch/memset_mops.S @@ -0,0 +1,38 @@ +/* Optimized memset for MOPS. + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include + +/* Assumptions: + * + * AArch64, MOPS. + * + */ + +ENTRY (__memset_mops) + PTR_ARG (0) + SIZE_ARG (2) + + mov x3, x0 + .inst 0x19c10443 /* setp [x3]!, x2!, x1 */ + .inst 0x19c14443 /* setm [x3]!, x2!, x1 */ + .inst 0x19c18443 /* sete [x3]!, x2!, x1 */ + ret + +END (__memset_mops) diff --git a/sysdeps/unix/sysv/linux/aarch64/cpu-features.h b/sysdeps/unix/sysv/linux/aarch64/cpu-features.h index d67d286b536ea599dcd8873fa2b882c5c27e8b05..40b709677d86f040c653315199f62677425abc58 100644 --- a/sysdeps/unix/sysv/linux/aarch64/cpu-features.h +++ b/sysdeps/unix/sysv/linux/aarch64/cpu-features.h @@ -76,6 +76,7 @@ struct cpu_features /* Currently, the GLIBC memory tagging tunable only defines 8 bits. */ uint8_t mte_state; bool sve; + bool mops; }; #endif /* _CPU_FEATURES_AARCH64_H */ diff --git a/sysdeps/unix/sysv/linux/aarch64/cpu-features.c b/sysdeps/unix/sysv/linux/aarch64/cpu-features.c index dc09c1c8274855c8215c2c6d57af9a2b5f8a7e2f..233d5b2407e2b792805b7fa661852f59fca0cb71 100644 --- a/sysdeps/unix/sysv/linux/aarch64/cpu-features.c +++ b/sysdeps/unix/sysv/linux/aarch64/cpu-features.c @@ -120,4 +120,7 @@ init_cpu_features (struct cpu_features *cpu_features) /* Check if SVE is supported. */ cpu_features->sve = GLRO (dl_hwcap) & HWCAP_SVE; + + /* Check if MOPS is supported. */ + cpu_features->mops = GLRO (dl_hwcap2) & HWCAP2_MOPS; }