From patchwork Wed Jun 3 15:19:20 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136413 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 705B74BA2E23 for ; Wed, 3 Jun 2026 15:34:28 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 705B74BA2E23 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=DBhi8Ibj; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=DBhi8Ibj X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from DU2PR03CU002.outbound.protection.outlook.com (mail-northeuropeazon11011053.outbound.protection.outlook.com [52.101.65.53]) by sourceware.org (Postfix) with ESMTPS id 6BDC24BA2E38 for ; Wed, 3 Jun 2026 15:21:10 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6BDC24BA2E38 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 6BDC24BA2E38 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.65.53 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500070; cv=pass; b=gb0eTeTT3m7PaOmljo6z4lR4PRPnrRRrjl7ZIa+iNendCndwvAGD/9bFa9/3HuiO/YLmH8NQQoeyAQ2GiwPyvpwJCAASSC+ms4u7LyLbpLOp4BuJugKkUf6j6WKOlrbWe/+if3AA/eSLzYRWfya2nh1+3XLN1miOIzIierd/NHM= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500070; c=relaxed/simple; bh=XQCGLoi1qKntUT8OR6JY3n8WF6ZZ7iIhi8fVJ28dgMg=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=jP0of94eB9HZC4c1oKIJFoYoe6x+XcFonkWkIF+5Z1k5hS/JmeFiGtlhyVd8KRapXzmcm0/HW/74iJBh4GR9CnLvKPY4SKkx0QlH6jSJ4XZVaQ9V6KGLvUDRzzlOWry882xXk5zXs6qY0ObG1gk3b9i2a6H/1pKps+ZOEToo3RI= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=DBhi8Ibj; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=DBhi8Ibj DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 6BDC24BA2E38 ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=N+hsy53OmrV+iUl7b2MSk5q5kzJqsMifqPInTu3tgXWaq4aX25M1SmUaksuMmYix54R72extm3AEGmdwHnQFNkG8YlM17SxtAARpQPldjuB3Ng+oMBpFsmB5biMlvmkmBcyAGDhMkAvDNY/JZcoch+mEa/iNnvbjk3UUIgRZeJvuLP0XpCn+/rfzwPZ4qnhYo9nTH9MdLIRaaTkmDCB9tOMXbDWH6dlVd8P3BzxM23EGctOBHVViuV0nz6DeJG/iDyBnt1B1LKFRZWdHelMTFYRXDCwsdausu+83hJc86UHRbhaH8sho+s+KXfGRQqQNMOVHDV7fPD1+vTBxY65g7A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eu1MhsBAJeIwooclwULT1d9fe+DvUQ82K0cetPK6nrs=; b=wQDh828INh1vIWG5jEPS+nBeYGWgzD0YwuzhA1k8bKRyjbQeTLK0FWR0N2BOzu8pLblPanO+2KtmChBlC3Lo/CGW7EDmz3NgaUrA/WFIw0PBi5ENFsxN0ZR4sgQ+PPkbeKaaCxfcQq6iSzdT9emxulXesDKevWBDCk8nSdaSmYxCO8hij2jrbYzOOW+ji/jojmRECFlMT47b1sQ7krhu/Vb/amTdhrtiXpmIURKMKz2zOZpb157oP4Daar/fTjGUQYuui8hnp//BIS+mt2mxEHK8en2pyR3QX1VVVewzwO4LKd1UN1S5hWjAHCqor2APEeNVaLWjg1wB81ttR/GKrQ== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eu1MhsBAJeIwooclwULT1d9fe+DvUQ82K0cetPK6nrs=; b=DBhi8Ibj0DAG9jv+AYPCAAXD2XxkHO9QRKGMSmm3DxY6SnZRV/ZAjw5pa8SVl8G7fLaMshEJDHTsmgHIZLvu1I0EhZaMDQjVf9JF7d0M5s+LW0nbcZrUZiD6Rm29q2Aw6UWOOj2nZ5vf7KhWwF3TL9FG04TWFgdFjH8gFDql6BY= Received: from AS4P250CA0026.EURP250.PROD.OUTLOOK.COM (2603:10a6:20b:5e3::14) by GVXPR08MB11227.eurprd08.prod.outlook.com (2603:10a6:150:289::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.17; Wed, 3 Jun 2026 15:21:02 +0000 Received: from AM1PEPF000252DA.eurprd07.prod.outlook.com (2603:10a6:20b:5e3:cafe::89) by AS4P250CA0026.outlook.office365.com (2603:10a6:20b:5e3::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:21:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by AM1PEPF000252DA.mail.protection.outlook.com (10.167.16.52) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:21:02 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=JkUjwY2CJcwoTsgeXId55Z7i+xLPgyBV10PCc6/jzLdn0HSCtTxLrOKP5m9eiguxI6hwlbXWzgcUey5jgEnUDPrsPwQjpD7/MmdULn4rXHGQybHXQOiul7Qs0/uTTYda75jQVu5XIi7+0Y/gH6KZqULTADpUM1bkU8vvRLu2RgqwpacXd6xH3FP/rEqYsL3ornhqq1awfrGcxA3XnQY8n7ejd7Acyycmfj6Mpp2xNuqxJRWf2n2SSTItm7gyOuDxXLwzlIoPGckqgYVwUYewjk8mnkstqvqcpc28KzN4fXOajfSBInA2PMHXLC5AhooyxsHIItICVyzpSJvOEwQ66g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eu1MhsBAJeIwooclwULT1d9fe+DvUQ82K0cetPK6nrs=; b=CpwfaOoumUWcR1zWM5yMxzZNA45axwPOfuN6yHFKqClpHCtnlISXt91Zn/Bes8GDdi27mzL0z3wtPZclaeTmT6VwEJzp0AXGHVdmrExc0LKE4B8+mZT9pnb1bkqoGWE6YDdbBNy/lh2ekg89/8Za6yw2SDXUqOdu61xf87epLxi1m3F1UnZc7MkxjNuZvfq4Nsgy/O4MAI8l2r3TwbaSB2z2RKAHb5x6GooTZljKThtpsUuKPyo6WFOZmaPLRcQGAsUpHVlzXI1kQD/+LmjOwI0MLTwa0esnefbewILBpszJHa/hubO9tHjqP06WXLTLQ7XQqUwVpnadAs8Jlvu58g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eu1MhsBAJeIwooclwULT1d9fe+DvUQ82K0cetPK6nrs=; b=DBhi8Ibj0DAG9jv+AYPCAAXD2XxkHO9QRKGMSmm3DxY6SnZRV/ZAjw5pa8SVl8G7fLaMshEJDHTsmgHIZLvu1I0EhZaMDQjVf9JF7d0M5s+LW0nbcZrUZiD6Rm29q2Aw6UWOOj2nZ5vf7KhWwF3TL9FG04TWFgdFjH8gFDql6BY= Received: from DU2PR04CA0151.eurprd04.prod.outlook.com (2603:10a6:10:2b0::6) by PAWPR08MB10135.eurprd08.prod.outlook.com (2603:10a6:102:35a::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.15; Wed, 3 Jun 2026 15:19:58 +0000 Received: from DU6PEPF00009528.eurprd02.prod.outlook.com (2603:10a6:10:2b0:cafe::50) by DU2PR04CA0151.outlook.office365.com (2603:10a6:10:2b0::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009528.mail.protection.outlook.com (10.167.8.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:58 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:35 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:35 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 08/12] AArch64/SVE: Optimize vec_init for partial SVE vector modes Date: Wed, 3 Jun 2026 15:19:20 +0000 Message-ID: <20260603151924.53706-9-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009528:EE_|PAWPR08MB10135:EE_|AM1PEPF000252DA:EE_|GVXPR08MB11227:EE_ X-MS-Office365-Filtering-Correlation-Id: 67f6184e-ec34-48e1-7df0-08dec183b920 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|82310400026|1800799024|376014|36860700016|56012099006|22082099003|18002099003|6133799003|11063799006; X-Microsoft-Antispam-Message-Info-Original: gHJlGpXMbhca8nRs9ij42E6zPQxmGknOuohplE8ti72qFNlGdOfh4Ozy/jsK7ST3xebjqk4Q0Iu66EKBCotCfuTS/mclQWCgSX+Hd7KC7pt0Iwqh3cjMwe1A3OazPza2+HY3hmXpfeAzpMsTKAgYsQoceiW1pzZYn37cr8/ZfNNaxgstO+/XJ/dN9m8Ou+iVFEqn6D+0pJ/ktlDTQO9sydUB2/lqf34UvWWiJGgE3OwbnpcdRMmlRLyjK4r9p8IMbgmLLOy8kS5UG3bx8h/9lpP0mi/ZkpgefeCC3lyxoMNU5EOblasDkpAkTR/QV32EpfdPNVjokzXWvi19D60oIOkx3rXiR5tuOTGdyRBw24gCaqc9dY7Z7K78BRogykbx3WhT3OK3T4tcOgyO+LDBQxQ2h9i0XRelgnG2ktnXFcKfVlgSMaM/W46OtFhRFCj8TK4Qb+TpnZScazssfoBLCe9hqru89D9LhRmvDqvaKEgMKl4DRveJ7rQ4qVpJqxTtZ3fOD4oje8OfKLhFhS4DjloneclYy6S4UcUXFRHJFQI/Zs9xe/GM4gFdBA/6rzZUl/DSM/Ej0C89QsydIPuSOhn1VMS4Y/h+t1/NHGEuLyLRkHkZ/wZX/YyOOh/tEosChBiQoBrhSstaDIQG1upY+RgZ3/Q8LO77ufGslR+xqCl8EcyVTqlldFwRgQZLtrXJ X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(82310400026)(1800799024)(376014)(36860700016)(56012099006)(22082099003)(18002099003)(6133799003)(11063799006); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: Rm5J+kkwDOzoDVqEb4Av1520i5neKjOIMUZJUESNHGWONhkldSoRxvAo7lqn5Q2tGys55nPWi0ofnKl6qRdmDFbdTS70o0JUXUK+F1n/NKtpZwbWKZP55tvUACqKxadEgYhE/FSlUcXruVznPPFewRTYGuzuYwyjJowJ2EIoVs/suANoHRhtChiuwKLCz+Ug5lGZRU1RDoL+cBm4lzkK6Uhz6ErBCChrRuxCdaJoP2KBj2qYcE1TTBgm/3+f+cPDSkerbQNKWksSkqGgy2VESF9n72b6Id6zRUW6b61bZF6LUmQda6sb7sO4rQEL2PPq+jYoDyzxkwEoUfWXnxzvsw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB10135 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM1PEPF000252DA.eurprd07.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: ea277d63-387e-403f-32ee-08dec18392cc X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|36860700016|82310400026|14060799003|1800799024|35042699022|56012099006|11063799006|18002099003|6133799003|22082099003|13003099007; X-Microsoft-Antispam-Message-Info: K3tA6eQnkFBIk0b55ymcBOsuaoA80QAXchEeKWkdbOEgBQffpU+L36RgqM3764OTePPe0lKz+PZkIFSZ1CkCyO5+IR/xBTvMHUdta/xnqk6aSKgf+ygwqB5479WWOQWk+zr8uYEsgU9Gb5XDcl6kjptqNmVkRkXiBQBkLch32/jXEHIa5aw64jsSgchdNwtfi37fiLf5d6tI5eBj0YcAQUI1u6ZYcsB0kErR6CoEOvQ8xVLb+qI8MUh7ZqNa+F2cdi3BQUP6NNZUIbZHJyoqtZdDpp+JVzSwsJB6GpkxQr44EHdiCmYKd7qH0Oc5bxnLErA/Z4LVym+ND2F0LjB8zvLCE3XMvyOhsKoQo0GpzDr/HSfPuhfbalI56vHtWyL5/1TYgSelJ+Gvqg+QbyCAqn/NVKTPmIjcuIdGApd/yhd5GhLKHwp1/bElpuos+ma4uQoHUW+jcWDKab8F3KbcZVbrrGXTufT3f/V8IHe0kjtPcDmtgDQVPruUsBfF70y0+CnRhkqRTHh2d8yyiBjpJWcpx1lyE0B/XkYYmuv9xZ8ndGhLh7EJRSMCojb8fCpD9nYwwKzLEAEScinLsfa3gWCkCfosTkZdRjUizWMa13KNsRhv9SmVRtW8YrRm0x0EHUc6Ly+NJ43ZNh5O6CWO9mE1vhFdqU/NXdhct0Vi6r9I+/Na0cZdykSTWVggO8KA X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(376014)(36860700016)(82310400026)(14060799003)(1800799024)(35042699022)(56012099006)(11063799006)(18002099003)(6133799003)(22082099003)(13003099007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ac7Tjm9ZUdN1wAgaLjMmDCuuVrLdEzGzPtV3cfcBUh8bKnu9O9tW+oR7FqJRamP27rtL2xv2s9FhUF7ImCquAR9uJ/gLr5L12S8KTRdt0VGfkF/XK9Z/y7x0BZKHQwAE7mzJhx6XeRzh+ftvJdbHCt4tVGrObyWK1UwprWfYiOefmWrsCSx/cqGUNt4i7+tytq38ZtAgACLGPkweV6BqLB3iNU/gtCmzthA8ChdkioMlMI4gZ8oo2gEopKFwNyS3xis6OzizFQ/fziSnNoAkxFae53wB7wzuEQTAceA/vr8HVZKOlT/fUhcj0j3/j31SIg6fAN6OkLGYM8XxTOBblhDxr6bxJycwmH7CWyGUFcI5vutDtgIijKvRgWwPn2NkpGkrKF8Wp0goaOUrK/DQxpHnWyxLV26yZxEugTkf3BmiOmnjxARJ+24ZkYjzDjCR X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:21:02.2923 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 67f6184e-ec34-48e1-7df0-08dec183b920 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: AM1PEPF000252DA.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GVXPR08MB11227 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_BLOCKED, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org When basic block vectorization is extended to support predicated vector tails, it attempts to vectorize more stores. This is only done if the cost model deems it profitable, but the cost model assumes that vec_init is cheap; in practice, that was not always true. For example, uint8_t * vectp.2689; vector([4,4]) unsigned char _846; vector([4,4]) slp_mask_848; ... _846 = {_20, _30, _40, _50}; vectp.2689_847 = src_61(D) + 64; slp_mask_848 = .WHILE_ULT (0, 4, { 0, ... }); .MASK_STORE (vectp.2689_847, 8B, slp_mask_848, _846); was expected to have a vector cost of 4 (the same as the scalar cost) but the code actually generated for _846 = {_20, _30, _40, _50}; was a repetitive series of write-modify-read operations using four stack locations for temporary storage: (set (reg:VNx16BI Y) (const_vector:VNx16BI repeat [ (const_int 1 [0x1]) ])) (set (mem/c:VNx4QI (plus:DI (reg/f:DI 96 virtual-stack-vars) (const_poly_int:DI [-O, -O])) [0 S[O, O] A8]) (unspec:VNx4QI [ (subreg:VNx4BI (reg:VNx16BI Y) 0) (reg:VNx4QI 205 [ _845 ]) ] UNSPEC_PRED_X)) (set (reg:QI Z) (subreg:QI (reg:SI X [ _W ]) 0)) (set (mem/c:QI (plus:DI (reg/f:DI 96 virtual-stack-vars) (const_poly_int:DI [-O, -O])) [0 S1 A8]) (reg:QI Z)) (set (reg:VNx4QI 205 [ _845 ]) (unspec:VNx4QI [ (subreg:VNx4BI (reg:VNx16BI V) 0) (mem/c:VNx4QI (plus:DI (reg/f:DI 96 virtual-stack-vars) (const_poly_int:DI [-O, -O])) [0 S[O, O] A8]) ] UNSPEC_PRED_X)) (repeated four times) which compiled to something like: addpl x5, sp, #6 st1b {z27.s}, p7, sp, #3, mul vl strb w10, [x5] ld1b {z28.s}, p7/z, sp, #3, mul vl (repeated four times) With these changes, the compiled code is instead: mov z31.b, w0 insr z31.s, s28 insr z31.s, s29 insr z31.s, s30 which is not yet optimal but is a great improvement. To achieve that, "vec_init" was modified to accept all SVE vector modes, which means that the associated function aarch64_sve_expand_vector_init must now handle all partial modes (namely, VNx8QI, VNx4QI, VNx2QI, VNx4HI, VNx2HI, VNx2SI, VNx2HF, VNx4HF, VNx2SF, VNx2BF, VNx4BF). I verified that the following dependencies already handle partial vector modes: - "@aarch64_sve_" (for ZIP1) - "*vec_duplicate_reg" - maybe_code_for_aarch64_sve_rev I did not verify that emit_move_insn (which is a dependency of aarch64_sve_expand_vector_init_handle_trailing_constants) handles partial vector modes, but it seems highly likely. "vec_shl_insert_" has been modified to accept SVE_ALL instead of only SVE_FULL and operate on container instead of element types. gcc/ChangeLog: * config/aarch64/aarch64-sve.md: Update vec_init and vec_shl_insert_ to accept all SVE vector modes. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sve/slp_stack.c: New test. --- gcc/config/aarch64/aarch64-sve.md | 16 +++++------ .../gcc.target/aarch64/sve/slp_stack.c | 27 +++++++++++++++++++ 2 files changed, 35 insertions(+), 8 deletions(-) create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_stack.c diff --git a/gcc/config/aarch64/aarch64-sve.md b/gcc/config/aarch64/aarch64-sve.md index 585a587d8cf..6750428255c 100644 --- a/gcc/config/aarch64/aarch64-sve.md +++ b/gcc/config/aarch64/aarch64-sve.md @@ -2959,7 +2959,7 @@ (define_insn "@aarch64_sve_ld1ro" ;; ------------------------------------------------------------------------- (define_expand "vec_init" - [(match_operand:SVE_FULL 0 "register_operand") + [(match_operand:SVE_ALL 0 "register_operand") (match_operand 1 "")] "TARGET_SVE" { @@ -3003,17 +3003,17 @@ (define_expand "vec_initvnx16qivnx2qi" ;; Shift an SVE vector left and insert a scalar into element 0. (define_insn "vec_shl_insert_" - [(set (match_operand:SVE_FULL 0 "register_operand") - (unspec:SVE_FULL - [(match_operand:SVE_FULL 1 "register_operand") + [(set (match_operand:SVE_ALL 0 "register_operand") + (unspec:SVE_ALL + [(match_operand:SVE_ALL 1 "register_operand") (match_operand: 2 "aarch64_reg_or_zero")] UNSPEC_INSR))] "TARGET_SVE" {@ [ cons: =0 , 1 , 2 ; attrs: movprfx ] - [ ?w , 0 , rZ ; * ] insr\t%0., %2 - [ w , 0 , w ; * ] insr\t%0., %2 - [ ??&w , w , rZ ; yes ] movprfx\t%0, %1\;insr\t%0., %2 - [ ?&w , w , w ; yes ] movprfx\t%0, %1\;insr\t%0., %2 + [ ?w , 0 , rZ ; * ] insr\t%0., %2 + [ w , 0 , w ; * ] insr\t%0., %2 + [ ??&w , w , rZ ; yes ] movprfx\t%0, %1\;insr\t%0., %2 + [ ?&w , w , w ; yes ] movprfx\t%0, %1\;insr\t%0., %2 } [(set_attr "sve_type" "sve_int_general")] ) diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_stack.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_stack.c new file mode 100644 index 00000000000..76be816e0d6 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_stack.c @@ -0,0 +1,27 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize -mcpu=neoverse-n2 --param=aarch64-autovec-preference=sve-only -msve-vector-bits=scalable" } */ + +#include + +/* Without an efficient implementation of vec_init for partial SVE types, a + decision to vectorize a group in the basic block vectorizer can result in + code that repeatedly stores a whole vector on the stack, overwrites one + element, reloads the whole vector, stores it to another location, + overwrites another element, etc. This is a fairly minimal reproducer. */ +void +vec_slp_pathological_stack (uint8_t *src) +{ + int lt = src[-33]; + int l0 = src[-1]; + int l1 = src[31]; + int t0 = src[-32]; + int t1 = src[-31]; + int t2 = src[-30]; + src[64] = (l1 + (2 * l0) + lt + 2) >> 2; + src[65] = (lt + t0 + 1) >> 1; + src[66] = (t0 + t1 + 1) >> 1; + src[67] = (t1 + t2 + 1) >> 1; +} + +/* { dg-final { scan-assembler-not {sp} } } + */