From patchwork Wed Jun 3 15:19:13 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136410 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D6BBE4BA2E1F for ; Wed, 3 Jun 2026 15:29:05 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D6BBE4BA2E1F Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=expiWOit; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=expiWOit X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from DB3PR0202CU003.outbound.protection.outlook.com (mail-northeuropeazon11010039.outbound.protection.outlook.com [52.101.84.39]) by sourceware.org (Postfix) with ESMTPS id E17344BA2E1C for ; Wed, 3 Jun 2026 15:20:51 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org E17344BA2E1C Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org E17344BA2E1C Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.84.39 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500052; cv=pass; b=tKPxZeqp9b3M2K5fPCeWWtzINSuptRzORCPzUHlAp1pmm9FwrUwS3rX5c76gQ3JHTVbkfl3OxnXA0+IJhKRxGD6Roq71MYGZLSCJuwtIk2m95TdlYMP0thY9xePUDTAKCGbZw45fpKr3oDqktqPKtfo7Y8r4s/VX5eyVAUgldgg= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500052; c=relaxed/simple; bh=8WzJIX576bntqXRSQdxWkzLcHjRGze7rS7I1ncNv6Ik=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=vqhoA2v8WnNodPintOVqHga3Lq9sW+OTXrutBQGaT7tx4GeHMlnFXeKnTLBPsUjDUbIsRZhveQkSKlndGTpsJpq9BuqFaTYYScLst689JifQVuruPUvVuaCsuXVC9xJWSzRv7H1FF+mu8XnkxISqpR9LRq6Kk8pSiipAlJMRnu0= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=expiWOit; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=expiWOit DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E17344BA2E1C ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=LPyH+mBkqyMw+tpixrm5vOOlQqeIHYoqOFlWTmq5SCNJrleGMYdSQPJaxYRGLBddPh2BgmE4v6d22a8b0ovlF9wXyqIqxNmlyzNoyKt+Tghsl0xGNTfwkuZhCCzZvIcwAlvwm8TzjM+S+H7o1BDQk9muomqdlYDfh7dWjd2Kw8FfWca5pgDNda9w0tX7RDsBbmVCSgvpzPYEgWffXD56yc3x5rwPGCqFiz5q8bOpf3F0hnFis84w6bPOelHYWoAXzwifpZml9Ynl++UjkCzghmtxa6JbNgalQZQpJpAFvQYrptxrXUnDlpEVCp+vvPofaLS6lBbbYxlMtV7VoA+AUA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QwhtYpY6Smy1YT2xnbZc8/AH1XX2Tpe0K3m4FJg22Qw=; b=e3uUCkX9HHsf9PE177z7E3pJa1jFg6qU6dhtnav9/kYMgRb0uu6+8eIPZPf3DaSUxVz16qouOwf/wkWl24slel8tk0XL94sU5fjEbhsSJsYKK47coFC57QVFMh/A6vxmXB+v/FAhwkdn2sWvLVrYBGZ3mOd3Ygl+scP3K6z/7SzSuFa6JGfjgxsFJwicqqcJhPv7vgzVaT6EXbjzUvsbeOFXxhyooj3LOSg9nVUl+XkxReXWCrkOeHhhQi+HaKfLtOkFlkNqsZ9dGjlF2ZkqA47/4hiqd7chaRwFn7rkRd6CIa6HpqEXKU5yqwOdakq/5LRJZgClvaWTtC9f0t7FlA== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QwhtYpY6Smy1YT2xnbZc8/AH1XX2Tpe0K3m4FJg22Qw=; b=expiWOitfTxxJknvNrNIlPj8eu6mNrm64ZbPraeAiYUmqcLBbiJCDrFhuGirW4Ez43BPiHHPnV4J9N1h8dUTirvb1HMWUz2mkyWSZh24rxuds0imMLHDgkwTXQjtu30axL7oQJWrLy2f9UNxZmTNevEEgOeWY47A7R8w+vBwZeo= Received: from DU2PR04CA0245.eurprd04.prod.outlook.com (2603:10a6:10:28e::10) by DU4PR08MB11077.eurprd08.prod.outlook.com (2603:10a6:10:576::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:20:44 +0000 Received: from DB5PEPF00014B9E.eurprd02.prod.outlook.com (2603:10a6:10:28e:cafe::90) by DU2PR04CA0245.outlook.office365.com (2603:10a6:10:28e::10) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:20:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by DB5PEPF00014B9E.mail.protection.outlook.com (10.167.8.171) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:20:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=oZ5nTiKeJ/z9+USIWZBb9JYArPxXUEcBgN41fHGdYnf8F+GRlyGghFNOv7zcStWpfGme+gMZVj9jVhTn4MjuvvDZDIsk/+9D/w/FCiofA85kIxbUZZHF/0phaVZNMV26RXZHrHCCNojlinDCkOut2EzX43W76GlYuXQF5rHvI9FZZWGsG6pRIxuAAPOWhVVXa8DCbGo11D5ajPWCvyq7h3iJiHmqyi0LG+ltb9rlia+YX+4ad9GyH2sdUfa0uS68F6UmwPSfI2NPQNgxD/HEzDI1RRFIaWjByKdlQOLcd8c9dDxGWLtdhBvT4qdugMROQ/ibuWkQlC2QhfJIpBkstA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QwhtYpY6Smy1YT2xnbZc8/AH1XX2Tpe0K3m4FJg22Qw=; b=gDqK4WiIbV/p1ZJrpD3jCzU0vYkHTQnkNkBbT0EU6lgA5Lnbe1HfmmNuMsoKhxuQrmhbBteQiQgN4Qdh56Gh3ejA9GI9X+jV55sfd2aO+Qb9wmjn/5CQ9Fy/0XRMbwalD2yMUhZkuD3iSUxrDI5BZImxdi9d5r6+x95LVv22FaVhAxnd4UluQsTGVBkKclCmRXB9fJ/F/jn2uH9wbbaZq/FCN95XLEUAq/JURfH+0lKnpHXExD0f+PdUamr5JQ/V3zXYzIdz2jU8Wo03wkmXZPlV00oPLzkL/YyHSjk5gewOZU/7QJXq11dd93+dragdfo/3M4bw1G2SMcMOGHvw3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QwhtYpY6Smy1YT2xnbZc8/AH1XX2Tpe0K3m4FJg22Qw=; b=expiWOitfTxxJknvNrNIlPj8eu6mNrm64ZbPraeAiYUmqcLBbiJCDrFhuGirW4Ez43BPiHHPnV4J9N1h8dUTirvb1HMWUz2mkyWSZh24rxuds0imMLHDgkwTXQjtu30axL7oQJWrLy2f9UNxZmTNevEEgOeWY47A7R8w+vBwZeo= Received: from DU2PR04CA0180.eurprd04.prod.outlook.com (2603:10a6:10:2b0::35) by AS2PR08MB10251.eurprd08.prod.outlook.com (2603:10a6:20b:647::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.48.16; Wed, 3 Jun 2026 15:19:36 +0000 Received: from DU6PEPF00009528.eurprd02.prod.outlook.com (2603:10a6:10:2b0:cafe::30) by DU2PR04CA0180.outlook.office365.com (2603:10a6:10:2b0::35) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009528.mail.protection.outlook.com (10.167.8.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:36 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:34 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:34 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 01/12] Preparation to support predicated vector tails for BB SLP Date: Wed, 3 Jun 2026 15:19:13 +0000 Message-ID: <20260603151924.53706-2-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009528:EE_|AS2PR08MB10251:EE_|DB5PEPF00014B9E:EE_|DU4PR08MB11077:EE_ X-MS-Office365-Filtering-Correlation-Id: bda49d04-1714-43df-5d18-08dec183ad74 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|1800799024|376014|82310400026|36860700016|22082099003|18002099003|56012099006|3023799007|11063799006|6133799003|5023799004; X-Microsoft-Antispam-Message-Info-Original: XqWfJ5SWGwAz568ltdc9x5zLx61MGBdRrLr52jdo38nSxJFGJQa7iOJ6Tk9Bhb9FyqhXeEjEeFa+LPiBf5qZFEvALn+mDTZltxImjRGS5dwr0jo9k6P2AAFbJdwqHne/nzXYc4rN89zC4+jHaKGWj1Mc7JWDM4x9bTGeEFc8hqgP5oLUgt8Wb57wvoWUKdmPSzWIuWBPnx92ET1eGNgNY2EfLvLHKOk5M/S4dFwrxwxxWQ/0SFJK13tYTJ/WnjRc3ifZnrOPTbkw63ZZbT9Vs2D9fCAKgycUyhn0Jqc2DqJhqAYaKpwDWXLlk9GZKuq/AAQxgHjZ+1S1JbvpPTt9TrJft+FgYMeSfpmWQ09kyB53+1OCkJV/uksKmBTJYivyoRyegSrB/lQwJUwRvUxr9tTQBo1yl/83cZ8+FaLU7glpUDJ/TjaUwe+0JW8tm06OYRsPVklwYgevJBvCKCilUdpSsLsMhVFeEy0FKgnw5l/7hXaWk0QxxMmp2argDjJsPgE3q9zDqKTPK5FJX5uIAma7x2kNvx/Jqgre5K4Co8NlKWI3Uf0hrWQfCCoJUIO15CXYPbWTE0gm01NgmAXfCXiiuvKp/LY9Edwghjygcmgo9uQIW29RUlaQHMOk8+TT7IdgJCd9nTn6vmhYY0814AIa4Oy3Bvep7QOyPfB0srIA+y+INcurh4Jty1EXrfjdQYB+vKZYEFhZdAx7aaipI2gFH2ukmCsKrmcZKVhqXT0= X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(376014)(82310400026)(36860700016)(22082099003)(18002099003)(56012099006)(3023799007)(11063799006)(6133799003)(5023799004); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: lSiu76CkrPjWiSMy81gSC3a3WE0Z+0Bq/yOyZvppXCwpu7888PwuwSGaarBGhEPWWR87iIwGxBymtQxwq5jZ2MUVj32Yh7l0TmODApKqez+J4nNv3115lrVhSIQqJJ0HhcMNqCPkz9aC0fiSsO3tCfOOiYF7w+wnJqhFaDqv7zzvzJIOzgDbNwifdTSXxX3psfKxBlQW6oQff11mrQdd2imOivIEnqHax2BfBmSeuLFYyamebzvoeYi4rih7GTEg7nLAJEyBSO79Phu/AM/yOctOezguHPoABkji8HWhFrtvmUlE6901ZdE9JHsG/Lq+zFsLMlUZLniTD2ehsyW4pg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR08MB10251 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5PEPF00014B9E.eurprd02.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 538038f2-e38b-4212-0673-08dec1838614 X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|36860700016|82310400026|35042699022|1800799024|14060799003|11063799006|5023799004|56012099006|3023799007|22082099003|18002099003|6133799003; X-Microsoft-Antispam-Message-Info: GBy50+Kd+dBn594d6CkEhlFSpx08VGILJz9zaQZ85YQKnnHF4RaRoUkJd5idhjSUtBSWjo56DAu4lgSi2DEkHNR4wp63W3Is1cuWZcapBDVmaILjYasWVEyiTpcoyPHGNHgvyCg81OwWQ/VoVozNsVhbrMNLauHEhDQkPJrhJHZdmNJtfN2nwT7Buhrof+F1I3oAxR7szllia88OctW0xaeQuoGxjjwTcEy7WX7NIIUGgFtTOQ8XmCotKk/Ax+t66Rq635ukbpYbZN1EBzR9Kbm1uH71X1JibXAk+CTiX95Yuh8eIugsADRBATwiCWGEYVXUPTtpwiYTovGx2F4TLIvVgFgYi88iPvr1fgcim25MG7AAWBtftH4wtuoRn29FltCAENrdLDBg+khNz1KlEp/i2RB5+0FlMCdWUU7DCTQUMEJNtt64etnmHPdxBygabG4oUP2u7xjgpPRD2uGTI+RKbKHxzElU2WNMq82aLK1BXryzese50HlKn4ruKfAWw1aoI8qclVxnMyAXcbjiyHHk8YL8FmLiEMudlQOQAXMrq1dsVucfNU3D5xWXp1Rxo1fT0GJLbgYfEu+qMBkbAH0pGZfylz00FzWmUEAstjwPsIZqdt+ON7j7/jUQtM5p0F3CgX5tagex0BwrZ20M8gmrva6N/eENwNezsr1Abg9Li6YR87Xisc8CwqDzxCfnQCrzzaQLaDniw29KWdHNkSuN5pXfxR+JsL+U1PCJ87Y= X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(376014)(36860700016)(82310400026)(35042699022)(1800799024)(14060799003)(11063799006)(5023799004)(56012099006)(3023799007)(22082099003)(18002099003)(6133799003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Y6kPxzo631zZAcqgnC96Fl1CtJsUC5qfvV7jg8gNwtAdEZ7Ku7Z4EIEJsqvTJnqlX8bkGvpsY3W63Jp3N2oEU1IYiOI7CZJB3orrDG+josoCDJ6eHKzrqlMy+O8RL/JugTjBV4IpT7bfxgQjQyf+TR+V0MWDlZlsOMhWExwpj2lfqufTHt/MsSGErAPdD+JMlu25C2KKx2Ye2YO+EG6JnhCoeyAVX5lAuqpbkdn5Yd2JfbL4pqLZbnmdhb8cVQDhfV60vsTNYf0gvewvnLrtLglB/q8u/SPeHW9sM55ffPV+/bVixCP4pmi3S0CJWL+Ql7A7y6554gCE+b+2T5pCl9FCv5mmv3cSQpUq4yQ1KSOfqIiGPgtDMp3DRzGGApfGabuTuIU/yKrlAjQCeTwnVIqZqgEdaCdRNXGeYRoK2fUyeckiNWD0pwKtZeaQJut2 X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:20:42.7046 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bda49d04-1714-43df-5d18-08dec183ad74 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: DB5PEPF00014B9E.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU4PR08MB11077 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, RCVD_IN_DNSWL_BLOCKED, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Calls to vect_(get|record)_loop_(mask|len) are replaced with calls to new wrappers that have an extra (SLP node) parameter and which can operate on any vec_info, not just a loop_vec_info. These wrappers pass calls through to the original functions (and ignore the SLP node) when invoked with a loop_vec_info; otherwise, vect_record_(len|mask) do nothing (for now). Direct use of LOOP_VINFO_FULLY_(MASKED|WITH_LENGTH)_P, and LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P is replaced with wrappers named vect_(fully_masked|fully_with_length)_p, vect_can_use_partial_vectors_p (for queries) and vect_cannot_use_partial_vectors (for updates). For BB SLP, vect_cannot_use_partial_vectors does nothing and the getters return false (for now). To minimize code churn, the new wrappers are only used in code shared between loop vectorization and BB SLP. gcc/ChangeLog: * tree-vect-stmts.cc (vect_record_mask): New function that wraps calls to vect_record_loop_mask. (vect_get_mask): New function that wraps calls to vect_get_loop_mask. (vect_record_len): New function that wraps calls to vect_record_loop_len. (vect_get_len): New function that wraps calls to vect_get_loop_len. (check_load_store_for_partial_vectors): Substitute a parameter of type vec_info * for loop_vec_info. Use vect_record_len instead of vect_record_loop_len. Use vect_record_mask instead of vect_record_loop_mask. Use vect_cannot_use_partial_vectors instead of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P. Delete local aliases for LOOP_VINFO_MASKS and LOOP_VINFO_LENS. (prepare_vec_mask): Substitute a parameter of type vec_info * for loop_vec_info. Rename the loop_mask parameter. (vectorizable_call): Delete local aliases for LOOP_VINFO_MASKS and LOOP_VINFO_LENS. Rename masked_loop_p as masked_p and remove requirement for non-null loop_vinfo. Rename len_loop_p as len_p and remove requirement for non-null loop_vinfo. Use vect_cannot_use_partial_vectors instead of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P. Use vect_record_len instead of vect_record_loop_len. Use vect_record_mask instead of vect_record_loop_mask. Use vect_fully_masked_p instead of LOOP_VINFO_FULLY_MASKED_P. Use vect_fully_with_length_p instead of LOOP_VINFO_FULLY_WITH_LENGTH_P. Use vect_get_mask instead of vect_get_loop_mask. Use vect_get_len instead of vect_get_loop_len. Pass vec_info * instead of loop_vec_info to prepare_vec_mask. (vectorizable_simd_clone_call): Use vect_can_use_partial_vectors_p and vect_cannot_use_partial_vectors instead of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P. Use vect_record_mask instead of vect_record_loop_mask. Use vect_fully_masked_p instead of LOOP_VINFO_FULLY_MASKED_P. Use vect_get_mask instead of vect_get_loop_mask. Pass vec_info * instead of loop_vec_info to prepare_vec_mask. Delete local aliases for LOOP_VINFO_MASKS. (vectorizable_conversion): Use vect_can_use_partial_vectors_p and vect_cannot_use_partial_vectors instead of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P. (vectorizable_operation): Delete local alias for LOOP_VINFO_LENS. Rename masked_loop_p as masked_p and remove requirement for non-null loop_vinfo. Rename len_loop_p as len_p and remove requirement for non-null loop_vinfo. Use vect_can_use_partial_vectors_p and vect_cannot_use_partial_vectors instead of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P. Use vect_record_len instead of vect_record_loop_len. Use vect_record_mask instead of vect_record_loop_mask. Use vect_fully_masked_p instead of LOOP_VINFO_FULLY_MASKED_P. Use vect_fully_with_length_p instead of LOOP_VINFO_FULLY_WITH_LENGTH_P. Use vect_get_mask instead of vect_get_loop_mask. Use vect_get_len instead of vect_get_loop_len. (vectorizable_store): Use vect_can_use_partial_vectors_p instead of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P. Pass vec_info * instead of loop_vec_info to check_load_store_for_partial_vectors. Delete local alias for LOOP_VINFO_MASKS. Use vect_fully_masked_p instead of LOOP_VINFO_FULLY_MASKED_P. Use vect_fully_with_length_p instead of LOOP_VINFO_FULLY_WITH_LENGTH_P. Pass vec_info * instead of loop_vec_info to prepare_vec_mask. (vectorizable_load): As above. * tree-vectorizer.h (class vec_info): Moved declaration of the vec_cond_masked_set member from class _loop_vec_info to class vec_info. (class _loop_vec_info): As above. (prepare_vec_mask): Substitute a parameter of type vec_info * for loop_vec_info. (vect_can_use_partial_vectors_p): New function that wraps use of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P as a getter. (vect_cannot_use_partial_vectors): New function that wraps use of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P to clear the flag. (vect_fully_with_length_p): New function that wraps use of LOOP_VINFO_FULLY_WITH_LENGTH_P. (vect_fully_masked_p): New function that wraps use of LOOP_VINFO_FULLY_MASKED_P. --- gcc/tree-vect-stmts.cc | 416 ++++++++++++++++++++++------------------- gcc/tree-vectorizer.h | 73 +++++++- 2 files changed, 295 insertions(+), 194 deletions(-) diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 7687c107a6a..15fca17a407 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -1382,12 +1382,88 @@ vectorizable_internal_function (combined_fn cfn, tree fndecl, return IFN_LAST; } +/* Record that a complete set of masks associated with VINFO would need to + contain a sequence of NVECTORS masks that each control a vector of type + VECTYPE. If SCALAR_MASK is nonnull, the fully-masked loop would AND + these vector masks with the vector version of SCALAR_MASK. */ +static void +vect_record_mask (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors, + tree vectype, tree scalar_mask) +{ + loop_vec_info loop_vinfo = dyn_cast (vinfo); + if (loop_vinfo) + vect_record_loop_mask (loop_vinfo, &LOOP_VINFO_MASKS (loop_vinfo), nvectors, + vectype, scalar_mask); + else + (void) slp_node; /* FORNOW */ +} + +/* Given a complete set of masks associated with VINFO, extract mask number + INDEX for an rgroup that operates on NVECTORS vectors of type VECTYPE, where + 0 <= INDEX < NVECTORS. Alternatively, if doing basic block vectorization, + materialize an equivalent mask for SLP_NODE. Insert any set-up statements + before GSI. */ +static tree +vect_get_mask (vec_info *vinfo, slp_tree slp_node, gimple_stmt_iterator *gsi, + unsigned int nvectors, tree vectype, unsigned int index) +{ + gcc_assert (vect_fully_masked_p (vinfo, slp_node)); + loop_vec_info loop_vinfo = dyn_cast (vinfo); + if (loop_vinfo) + return vect_get_loop_mask (loop_vinfo, gsi, &LOOP_VINFO_MASKS (loop_vinfo), + nvectors, vectype, index); + else + { + (void) slp_node; /* FORNOW */ + return NULL_TREE; + } +} + +/* Record that a complete set of lengths associated with VINFO would need to + contain a sequence of NVECTORS lengths for controlling an operation on + VECTYPE. The operation splits each element of VECTYPE into FACTOR separate + subelements, measuring the length as a number of these subelements. */ +static void +vect_record_len (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors, + tree vectype, unsigned int factor) +{ + loop_vec_info loop_vinfo = dyn_cast (vinfo); + if (loop_vinfo) + vect_record_loop_len (loop_vinfo, &LOOP_VINFO_LENS (loop_vinfo), nvectors, + vectype, factor); + else + (void) slp_node; /* FORNOW */ +} + +/* Given a complete set of lengths associated with VINFO, extract length number + INDEX for an rgroup that operates on NVECTORS vectors of type VECTYPE, where + 0 <= INDEX < NVECTORS. Alternatively, if doing basic block vectorization, + materialize an equivalent length for SLP_NODE. Return a value that contains + FACTOR multiplied by the number of elements that should be processed. Insert + any set-up statements before GSI. */ + +static tree +vect_get_len (vec_info *vinfo, slp_tree slp_node, gimple_stmt_iterator *gsi, + unsigned int nvectors, tree vectype, unsigned int index, + unsigned int factor, bool adjusted) +{ + gcc_assert (vect_fully_with_length_p (vinfo, slp_node)); + loop_vec_info loop_vinfo = dyn_cast (vinfo); + if (loop_vinfo) + return vect_get_loop_len (loop_vinfo, gsi, &LOOP_VINFO_LENS (loop_vinfo), + nvectors, vectype, index, factor, adjusted); + else + { + (void) slp_node; /* FORNOW */ + return NULL_TREE; + } +} static tree permute_vec_elements (vec_info *, tree, tree, tree, stmt_vec_info, gimple_stmt_iterator *); -/* Check whether a load or store statement in the loop described by - LOOP_VINFO is possible in a loop using partial vectors. This is +/* Check whether a load or store statement in the loop or SLP group described by + VINFO is possible using partial vectors. This is testing whether the vectorizer pass has the appropriate support, as well as whether the target does. @@ -1402,13 +1478,14 @@ static tree permute_vec_elements (vec_info *, tree, tree, tree, stmt_vec_info, Clear LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P if a loop using partial vectors is not supported, otherwise record the required rgroup control - types. + types. For basic block SLP, simply record which style of partial vectors + is available (if any). If partial vectors can be used and ELSVALS is nonzero the supported else values will be added to the vector ELSVALS points to. */ static void -check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, +check_load_store_for_partial_vectors (vec_info *vinfo, tree vectype, slp_tree slp_node, vec_load_store_type vls_type, int group_size, @@ -1416,6 +1493,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, slp_tree mask_node, vec *elsvals = nullptr) { + loop_vec_info loop_vinfo = dyn_cast (vinfo); vect_memory_access_type memory_access_type = ls->memory_access_type; /* Invariant loads need no special support. */ @@ -1439,9 +1517,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, scalar_mask = def; } - unsigned int nvectors = vect_get_num_copies (loop_vinfo, slp_node); - vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo); - vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo); + unsigned int nvectors = vect_get_num_copies (vinfo, slp_node); machine_mode vecmode = TYPE_MODE (vectype); bool is_load = (vls_type == VLS_LOAD); if (memory_access_type == VMAT_LOAD_STORE_LANES) @@ -1452,10 +1528,10 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, elsvals) : vect_store_lanes_supported (vectype, group_size, true)); if (ifn == IFN_MASK_LEN_LOAD_LANES || ifn == IFN_MASK_LEN_STORE_LANES) - vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, 1); + vect_record_len (vinfo, slp_node, nvectors, vectype, 1); else if (ifn == IFN_MASK_LOAD_LANES || ifn == IFN_MASK_STORE_LANES) - vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype, - scalar_mask); + vect_record_mask (vinfo, slp_node, nvectors, vectype, + scalar_mask); else { if (dump_enabled_p ()) @@ -1463,7 +1539,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, "can't operate on partial vectors because" " the target doesn't have an appropriate" " load/store-lanes instruction.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + vect_cannot_use_partial_vectors (vinfo, slp_node); } return; } @@ -1497,14 +1573,13 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, memory_type, off_vectype, scale, elsvals)) - vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, 1); + vect_record_len (vinfo, slp_node, nvectors, vectype, 1); else if (internal_gather_scatter_fn_supported_p (ifn, vectype, memory_type, off_vectype, scale, elsvals) || memory_access_type == VMAT_GATHER_SCATTER_LEGACY) - vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype, - scalar_mask); + vect_record_mask (vinfo, slp_node, nvectors, vectype, scalar_mask); else { if (dump_enabled_p ()) @@ -1512,7 +1587,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, "can't operate on partial vectors because" " the target doesn't have an appropriate" " gather load or scatter store instruction.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + vect_cannot_use_partial_vectors (vinfo, slp_node); } return; } @@ -1525,7 +1600,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, "can't operate on partial vectors because an" " access isn't contiguous.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + vect_cannot_use_partial_vectors (vinfo, slp_node); return; } @@ -1535,7 +1610,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, "can't operate on partial vectors when emulating" " vector operations.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + vect_cannot_use_partial_vectors (vinfo, slp_node); return; } @@ -1551,7 +1626,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, }; poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); - poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo); + poly_uint64 vf = loop_vinfo ? LOOP_VINFO_VECT_FACTOR (loop_vinfo) : 1; machine_mode mask_mode; machine_mode vmode; bool using_partial_vectors_p = false; @@ -1560,7 +1635,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, { nvectors = group_memory_nvectors (group_size * vf, nunits); unsigned factor = (vecmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vecmode); - vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, factor); + vect_record_len (vinfo, slp_node, nvectors, vectype, factor); using_partial_vectors_p = true; } else if (targetm.vectorize.get_mask_mode (vecmode).exists (&mask_mode) @@ -1568,7 +1643,7 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, elsvals)) { nvectors = group_memory_nvectors (group_size * vf, nunits); - vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype, scalar_mask); + vect_record_mask (vinfo, slp_node, nvectors, vectype, scalar_mask); using_partial_vectors_p = true; } @@ -1579,35 +1654,35 @@ check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype, "can't operate on partial vectors because the" " target doesn't have the appropriate partial" " vectorization load or store.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + vect_cannot_use_partial_vectors (vinfo, slp_node); } } /* Return the mask input to a masked load or store. VEC_MASK is the vectorized - form of the scalar mask condition and LOOP_MASK, if nonnull, is the mask - that needs to be applied to all loads and stores in a vectorized loop. - Return VEC_MASK if LOOP_MASK is null or if VEC_MASK is already masked, - otherwise return VEC_MASK & LOOP_MASK. + form of the scalar mask condition and LOOP_OR_TAIL_MASK, if nonnull, is the + mask that needs to be applied to all loads and stores in a vectorized loop. + Return VEC_MASK if LOOP_OR_TAIL_MASK is null or if VEC_MASK is already + masked, otherwise return VEC_MASK & LOOP_OR_TAIL_MASK. MASK_TYPE is the type of both masks. If new statements are needed, insert them before GSI. */ tree -prepare_vec_mask (loop_vec_info loop_vinfo, tree mask_type, tree loop_mask, +prepare_vec_mask (vec_info *vinfo, tree mask_type, tree loop_or_tail_mask, tree vec_mask, gimple_stmt_iterator *gsi) { gcc_assert (useless_type_conversion_p (mask_type, TREE_TYPE (vec_mask))); - if (!loop_mask) + if (!loop_or_tail_mask) return vec_mask; - gcc_assert (TREE_TYPE (loop_mask) == mask_type); + gcc_assert (TREE_TYPE (loop_or_tail_mask) == mask_type); - if (loop_vinfo->vec_cond_masked_set.contains ({ vec_mask, loop_mask })) + if (vinfo->vec_cond_masked_set.contains ({ vec_mask, loop_or_tail_mask })) return vec_mask; tree and_res = make_temp_ssa_name (mask_type, NULL, "vec_mask_and"); - gimple *and_stmt = gimple_build_assign (and_res, BIT_AND_EXPR, - vec_mask, loop_mask); + gimple *and_stmt + = gimple_build_assign (and_res, BIT_AND_EXPR, vec_mask, loop_or_tail_mask); gsi_insert_before (gsi, and_stmt, GSI_SAME_STMT); return and_res; @@ -3658,7 +3733,7 @@ vectorizable_call (vec_info *vinfo, dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, "can't use a fully-masked loop because no" " conditional operation is available.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + vect_cannot_use_partial_vectors (vinfo, slp_node); } } } @@ -3708,8 +3783,6 @@ vectorizable_call (vec_info *vinfo, internal_fn cond_fn = (internal_fn_mask_index (ifn) != -1 ? ifn : get_conditional_internal_fn (ifn)); internal_fn cond_len_fn = get_len_internal_fn (cond_fn); - vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL); - vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL); unsigned int nvectors = vect_get_num_copies (vinfo, slp_node); if (cost_vec) /* transformation not required. */ { @@ -3727,8 +3800,7 @@ vectorizable_call (vec_info *vinfo, DUMP_VECT_SCOPE ("vectorizable_call"); vect_model_simple_cost (vinfo, 1, slp_node, cost_vec); - if (loop_vinfo - && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) + if (vect_can_use_partial_vectors_p (vinfo, slp_node) && (reduc_idx >= 0 || could_trap || mask_opno >= 0)) { if (reduc_idx >= 0 @@ -3743,7 +3815,7 @@ vectorizable_call (vec_info *vinfo, dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, "can't use a fully-masked loop because no" " conditional operation is available.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + vect_cannot_use_partial_vectors (vinfo, slp_node); } else { @@ -3753,11 +3825,10 @@ vectorizable_call (vec_info *vinfo, if (cond_len_fn != IFN_LAST && direct_internal_fn_supported_p (cond_len_fn, vectype_out, OPTIMIZE_FOR_SPEED)) - vect_record_loop_len (loop_vinfo, lens, nvectors, vectype_out, - 1); + vect_record_len (vinfo, slp_node, nvectors, vectype_out, 1); else - vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype_out, - scalar_mask); + vect_record_mask (vinfo, slp_node, nvectors, vectype_out, + scalar_mask); } } return true; @@ -3772,10 +3843,10 @@ vectorizable_call (vec_info *vinfo, scalar_dest = gimple_call_lhs (stmt); vec_dest = vect_create_destination_var (scalar_dest, vectype_out); - bool masked_loop_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo); - bool len_loop_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo); + bool masked_p = vect_fully_masked_p (vinfo, slp_node); + bool len_p = vect_fully_with_length_p (vinfo, slp_node); unsigned int vect_nargs = nargs; - if (len_loop_p && (reduc_idx >= 0 || could_trap || mask_opno >= 0)) + if (len_p && (reduc_idx >= 0 || could_trap || mask_opno >= 0)) { ifn = cond_len_fn; /* COND_* -> COND_LEN_* takes 2 extra arguments:LEN,BIAS. */ @@ -3785,7 +3856,7 @@ vectorizable_call (vec_info *vinfo, if (mask_opno == -1) vect_nargs += 2; } - else if (masked_loop_p && mask_opno == -1 && (reduc_idx >= 0 || could_trap)) + else if (masked_p && mask_opno == -1 && (reduc_idx >= 0 || could_trap)) { ifn = cond_fn; vect_nargs += 2; @@ -3830,23 +3901,22 @@ vectorizable_call (vec_info *vinfo, { int varg = 0; /* Add the mask if necessary. */ - if ((masked_loop_p || len_loop_p) && mask_opno == -1 + if ((masked_p || len_p) && mask_opno == -1 && internal_fn_mask_index (ifn) != -1) { gcc_assert (internal_fn_mask_index (ifn) == varg); - if (masked_loop_p) + if (masked_p) { unsigned int vec_num = vec_oprnds0.length (); - vargs[varg++] = vect_get_loop_mask (loop_vinfo, gsi, - masks, vec_num, - vectype_out, i); + vargs[varg++] = vect_get_mask (vinfo, slp_node, gsi, + vec_num, vectype_out, i); } else { tree mask_vectype = truth_type_for (vectype_out); - vargs[varg++] = vect_build_all_ones_mask (loop_vinfo, - stmt_info, - mask_vectype); + vargs[varg++] + = vect_build_all_ones_mask (vinfo, stmt_info, + mask_vectype); } } size_t k; @@ -3856,7 +3926,7 @@ vectorizable_call (vec_info *vinfo, vargs[varg++] = vec_oprndsk[i]; } /* Add the else value if necessary. */ - if ((masked_loop_p || len_loop_p) && mask_opno == -1 + if ((masked_p || len_p) && mask_opno == -1 && internal_fn_else_index (ifn) != -1) { gcc_assert (internal_fn_else_index (ifn) == varg); @@ -3895,24 +3965,26 @@ vectorizable_call (vec_info *vinfo, } else { - if (len_opno >= 0 && len_loop_p) + if (len_opno >= 0 && len_p) { unsigned int vec_num = vec_oprnds0.length (); - tree len = vect_get_loop_len (loop_vinfo, gsi, lens, - vec_num, vectype_out, i, 1, true); + tree len = vect_get_len (vinfo, slp_node, gsi, vec_num, + vectype_out, i, 1, true); signed char biasval - = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo); + = loop_vinfo + ? LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) + : 0; tree bias = build_int_cst (intQI_type_node, biasval); vargs[len_opno] = len; vargs[len_opno + 1] = bias; } - else if (mask_opno >= 0 && masked_loop_p) + else if (mask_opno >= 0 && masked_p) { unsigned int vec_num = vec_oprnds0.length (); - tree mask = vect_get_loop_mask (loop_vinfo, gsi, masks, - vec_num, vectype_out, i); + tree mask = vect_get_mask (vinfo, slp_node, gsi, vec_num, + vectype_out, i); vargs[mask_opno] - = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask, + = prepare_vec_mask (vinfo, TREE_TYPE (mask), mask, vargs[mask_opno], gsi); } @@ -4442,14 +4514,14 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info, return false; } - if (!bestn_inbranch && loop_vinfo) + if (!bestn_inbranch) { if (dump_enabled_p () - && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)) + && vect_can_use_partial_vectors_p (vinfo, slp_node)) dump_printf_loc (MSG_NOTE, vect_location, "can't use a fully-masked loop because no" " masked simd clone was available.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + vect_cannot_use_partial_vectors (vinfo, slp_node); } /* When the original call is pure or const but the SIMD ABI dictates @@ -4473,17 +4545,15 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info, for (i = 0; i < (bestn_inbranch ? bestn_inbranch : bestn)->simdclone->nargs; i++) { - if (loop_vinfo - && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) + if (vect_can_use_partial_vectors_p (vinfo, slp_node) && (bestn_inbranch->simdclone->args[i].arg_type == SIMD_CLONE_ARG_TYPE_MASK)) { if (masked_call_offset) /* When there is an explicit mask we require the number of elements to match up. */ - vect_record_loop_mask (loop_vinfo, - &LOOP_VINFO_MASKS (loop_vinfo), - ncopies_in, vectype, NULL_TREE); + vect_record_mask (vinfo, slp_node, ncopies_in, vectype, + NULL_TREE); else { /* When there is no explicit mask on the call we have @@ -4514,9 +4584,8 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info, callee_nelements = TYPE_VECTOR_SUBPARTS (masktype); } auto o = vector_unroll_factor (nunits, callee_nelements); - vect_record_loop_mask (loop_vinfo, - &LOOP_VINFO_MASKS (loop_vinfo), - ncopies * o, masktype, NULL_TREE); + vect_record_mask (vinfo, slp_node, ncopies * o, masktype, + NULL_TREE); } } else if ((bestn->simdclone->args[i].arg_type @@ -4711,23 +4780,16 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info, if (m == 0) vec_oprnds_i[i] = 0; vec_oprnd0 = vec_oprnds[i][vec_oprnds_i[i]++]; - if (loop_vinfo - && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)) + if (vect_fully_masked_p (vinfo, slp_node)) { - vec_loop_masks *loop_masks - = &LOOP_VINFO_MASKS (loop_vinfo); - tree loop_mask - = vect_get_loop_mask (loop_vinfo, gsi, - loop_masks, ncopies_in, - vectype, j); + tree mask + = vect_get_mask (vinfo, slp_node, gsi, + ncopies_in, vectype, j); vec_oprnd0 - = prepare_vec_mask (loop_vinfo, - TREE_TYPE (loop_mask), - loop_mask, vec_oprnd0, - gsi); - loop_vinfo->vec_cond_masked_set.add ({ vec_oprnd0, - loop_mask }); - + = prepare_vec_mask (vinfo, TREE_TYPE (mask), + mask, vec_oprnd0, gsi); + vinfo->vec_cond_masked_set.add ( + {vec_oprnd0, mask}); } vec_oprnd0 = build3 (VEC_COND_EXPR, atype, vec_oprnd0, @@ -4926,12 +4988,9 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info, o = vector_unroll_factor (nunits, callee_nelements); for (m = j * o; m < (j + 1) * o; m++) { - if (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)) - { - vec_loop_masks *loop_masks = &LOOP_VINFO_MASKS (loop_vinfo); - mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks, - ncopies * o, mask_vectype, m); - } + if (vect_fully_masked_p (vinfo, slp_node)) + mask = vect_get_mask (vinfo, slp_node, gsi, ncopies * o, + mask_vectype, m); else mask = vect_build_all_ones_mask (vinfo, stmt_info, mask_argtype); @@ -5338,7 +5397,7 @@ vectorizable_conversion (vec_info *vinfo, vec vec_oprnds1 = vNULL; tree vop0; bb_vec_info bb_vinfo = dyn_cast (vinfo); - loop_vec_info loop_vinfo = dyn_cast (vinfo); + loop_vec_info loop_vinfo = dyn_cast (vinfo); int multi_step_cvt = 0; vec interm_types = vNULL; tree intermediate_type, cvt_type = NULL_TREE; @@ -5700,8 +5759,7 @@ vectorizable_conversion (vec_info *vinfo, } if (modifier == WIDEN - && loop_vinfo - && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) + && vect_can_use_partial_vectors_p (vinfo, slp_node) && (code1 == VEC_WIDEN_MULT_EVEN_EXPR || widening_evenodd_fn_p (code1))) { @@ -5710,7 +5768,7 @@ vectorizable_conversion (vec_info *vinfo, "can't use a fully-masked loop because" " widening operation on even/odd elements" " mixes up lanes.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + vect_cannot_use_partial_vectors (vinfo, slp_node); } if (cost_vec) /* transformation not required. */ @@ -6792,7 +6850,6 @@ vectorizable_operation (vec_info *vinfo, int reduc_idx = SLP_TREE_REDUC_IDX (slp_node); vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL); - vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL); internal_fn cond_fn = get_conditional_internal_fn (code); internal_fn cond_len_fn = get_conditional_len_internal_fn (code); @@ -6809,27 +6866,24 @@ vectorizable_operation (vec_info *vinfo, if (cost_vec) /* transformation not required. */ { - if (loop_vinfo - && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) + if (vect_can_use_partial_vectors_p (vinfo, slp_node) && mask_out_inactive) { if (cond_len_fn != IFN_LAST && direct_internal_fn_supported_p (cond_len_fn, vectype, OPTIMIZE_FOR_SPEED)) - vect_record_loop_len (loop_vinfo, lens, vec_num, vectype, - 1); + vect_record_len (vinfo, slp_node, vec_num, vectype, 1); else if (cond_fn != IFN_LAST && direct_internal_fn_supported_p (cond_fn, vectype, OPTIMIZE_FOR_SPEED)) - vect_record_loop_mask (loop_vinfo, masks, vec_num, - vectype, NULL); + vect_record_mask (vinfo, slp_node, vec_num, vectype, NULL); else { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, "can't use a fully-masked loop because no" " conditional operation is available.\n"); - LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + vect_cannot_use_partial_vectors (vinfo, slp_node); } } @@ -6889,8 +6943,8 @@ vectorizable_operation (vec_info *vinfo, dump_printf_loc (MSG_NOTE, vect_location, "transform binary/unary operation.\n"); - bool masked_loop_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo); - bool len_loop_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo); + bool masked_p = vect_fully_masked_p (vinfo, slp_node); + bool len_p = vect_fully_with_length_p (vinfo, slp_node); /* POINTER_DIFF_EXPR has pointer arguments which are vectorized as vectors with unsigned elements, but the result is signed. So, we @@ -7078,12 +7132,11 @@ vectorizable_operation (vec_info *vinfo, gimple_assign_set_lhs (new_stmt, new_temp); vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi); } - else if ((masked_loop_p || len_loop_p) && mask_out_inactive) + else if ((masked_p || len_p) && mask_out_inactive) { tree mask; - if (masked_loop_p) - mask = vect_get_loop_mask (loop_vinfo, gsi, masks, - vec_num, vectype, i); + if (masked_p) + mask = vect_get_mask (vinfo, slp_node, gsi, vec_num, vectype, i); else /* Dummy mask. */ mask = build_minus_one_cst (truth_type_for (vectype)); @@ -7107,19 +7160,19 @@ vectorizable_operation (vec_info *vinfo, (cond_fn, vectype, vops.length () - 1, &vops[1]); vops.quick_push (else_value); } - if (len_loop_p) + if (len_p) { - tree len = vect_get_loop_len (loop_vinfo, gsi, lens, - vec_num, vectype, i, 1, true); + tree len = vect_get_len (vinfo, slp_node, gsi, vec_num, vectype, + i, 1, true); signed char biasval - = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo); + = loop_vinfo ? LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) + : 0; tree bias = build_int_cst (intQI_type_node, biasval); vops.quick_push (len); vops.quick_push (bias); } gcall *call - = gimple_build_call_internal_vec (masked_loop_p ? cond_fn - : cond_len_fn, + = gimple_build_call_internal_vec (masked_p ? cond_fn : cond_len_fn, vops); new_temp = make_ssa_name (vec_dest, call); gimple_call_set_lhs (call, new_temp); @@ -7133,8 +7186,7 @@ vectorizable_operation (vec_info *vinfo, /* When combining two masks check if either of them is elsewhere combined with a loop mask, if that's the case we can mark that the new combined mask doesn't need to be combined with a loop mask. */ - if (masked_loop_p - && code == BIT_AND_EXPR + if (loop_vinfo && masked_p && code == BIT_AND_EXPR && VECTOR_BOOLEAN_TYPE_P (vectype)) { if (loop_vinfo->scalar_cond_masked_set.contains ({ op0, vec_num })) @@ -7166,7 +7218,7 @@ vectorizable_operation (vec_info *vinfo, /* Enter the combined value into the vector cond hash so we don't AND it with a loop mask again. */ if (mask) - loop_vinfo->vec_cond_masked_set.add ({ new_temp, mask }); + vinfo->vec_cond_masked_set.add ({ new_temp, mask }); } if (vec_cvt_dest) @@ -8307,9 +8359,8 @@ vectorizable_store (vec_info *vinfo, bool costing_p = cost_vec; if (costing_p) /* transformation not required. */ { - if (loop_vinfo - && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)) - check_load_store_for_partial_vectors (loop_vinfo, vectype, slp_node, + if (vect_can_use_partial_vectors_p (vinfo, slp_node)) + check_load_store_for_partial_vectors (vinfo, vectype, slp_node, vls_type, group_size, &ls, mask_node); @@ -8672,10 +8723,6 @@ vectorizable_store (vec_info *vinfo, } gcc_assert (alignment_support_scheme); - vec_loop_masks *loop_masks - = (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo) - ? &LOOP_VINFO_MASKS (loop_vinfo) - : NULL); vec_loop_lens *loop_lens = (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo) ? &LOOP_VINFO_LENS (loop_vinfo) @@ -8687,14 +8734,14 @@ vectorizable_store (vec_info *vinfo, Shouldn't go with length-based approach if fully masked. */ if (cost_vec == NULL) /* The cost_vec is NULL during transform. */ - gcc_assert ((!loop_lens || !loop_masks)); + gcc_assert ((!vect_fully_with_length_p (vinfo, slp_node) + || !vect_fully_masked_p (vinfo, slp_node))); /* Targets with store-lane instructions must not require explicit realignment. vect_supportable_dr_alignment always returns either dr_aligned or dr_unaligned_supported for masked operations. */ - gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES - && !mask_node - && !loop_masks) + gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES && !mask_node + && !vect_fully_masked_p (vinfo, slp_node)) || alignment_support_scheme == dr_aligned || alignment_support_scheme == dr_unaligned_supported); @@ -8824,20 +8871,19 @@ vectorizable_store (vec_info *vinfo, tree final_mask = NULL; tree final_len = NULL; tree bias = NULL; - if (loop_masks) - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks, - ncopies, vectype, j); + if (vect_fully_masked_p (vinfo, slp_node)) + final_mask + = vect_get_mask (vinfo, slp_node, gsi, ncopies, vectype, j); if (vec_mask) - final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask, + final_mask = prepare_vec_mask (vinfo, mask_vectype, final_mask, vec_mask, gsi); if (lanes_ifn == IFN_MASK_LEN_STORE_LANES) { - if (loop_lens) - final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens, - ncopies, vectype, j, 1, true); - else - final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype)); + if (vect_fully_with_length_p (vinfo, slp_node)) + final_len = vect_get_len (vinfo, slp_node, gsi, ncopies, + vectype, j, 1, true); + else final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype)); signed char biasval = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo); bias = build_int_cst (intQI_type_node, biasval); @@ -8966,12 +9012,11 @@ vectorizable_store (vec_info *vinfo, tree bias = NULL_TREE; if (!costing_p) { - if (loop_masks) - final_mask = vect_get_loop_mask (loop_vinfo, gsi, - loop_masks, num_stmts, - vectype, j); + if (vect_fully_masked_p (vinfo, slp_node)) + final_mask + = vect_get_mask (vinfo, slp_node, gsi, num_stmts, vectype, j); if (vec_mask) - final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, + final_mask = prepare_vec_mask (vinfo, mask_vectype, final_mask, vec_mask, gsi); } @@ -9034,9 +9079,8 @@ vectorizable_store (vec_info *vinfo, if (ls.gs.ifn == IFN_MASK_LEN_SCATTER_STORE) { if (loop_lens) - final_len = vect_get_loop_len (loop_vinfo, gsi, - loop_lens, num_stmts, - vectype, j, 1, true); + final_len = vect_get_len (vinfo, slp_node, gsi, num_stmts, + vectype, j, 1, true); else final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype)); @@ -9326,9 +9370,7 @@ vectorizable_store (vec_info *vinfo, || useless_type_conversion_p (vectype, TREE_TYPE (vec_oprnd))); bool simd_lane_access_p = STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) != 0; - if (!costing_p - && simd_lane_access_p - && !loop_masks + if (!costing_p && simd_lane_access_p && !vect_fully_masked_p (vinfo, slp_node) && TREE_CODE (DR_BASE_ADDRESS (first_dr_info->dr)) == ADDR_EXPR && VAR_P (TREE_OPERAND (DR_BASE_ADDRESS (first_dr_info->dr), 0)) && integer_zerop (get_dr_vinfo_offset (vinfo, first_dr_info)) @@ -9382,13 +9424,12 @@ vectorizable_store (vec_info *vinfo, tree final_mask = NULL_TREE; tree final_len = NULL_TREE; tree bias = NULL_TREE; - if (loop_masks) - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks, - vec_num, vectype, i); + if (vect_fully_masked_p (vinfo, slp_node)) + final_mask = vect_get_mask (vinfo, slp_node, gsi, vec_num, vectype, i); if (vec_mask) vec_mask = vec_masks[i]; if (vec_mask) - final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask, + final_mask = prepare_vec_mask (vinfo, mask_vectype, final_mask, vec_mask, gsi); if (i > 0) @@ -9424,8 +9465,8 @@ vectorizable_store (vec_info *vinfo, new_vmode = new_ovmode.require (); unsigned factor = (new_ovmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vmode); - final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens, - vec_num, vectype, i, factor, true); + final_len = vect_get_len (vinfo, slp_node, gsi, vec_num, vectype, i, + factor, true); } else if (final_mask) { @@ -9939,9 +9980,8 @@ vectorizable_load (vec_info *vinfo, if (costing_p) /* transformation not required. */ { - if (loop_vinfo - && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)) - check_load_store_for_partial_vectors (loop_vinfo, vectype, slp_node, + if (vect_can_use_partial_vectors_p (vinfo, slp_node)) + check_load_store_for_partial_vectors (vinfo, vectype, slp_node, VLS_LOAD, group_size, &ls, mask_node, &ls.elsvals); @@ -10563,14 +10603,10 @@ vectorizable_load (vec_info *vinfo, ref_type = reference_alias_ptr_type (DR_REF (first_dr_info->dr)); } - vec_loop_masks *loop_masks - = (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo) - ? &LOOP_VINFO_MASKS (loop_vinfo) - : NULL); vec_loop_lens *loop_lens = (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo) - ? &LOOP_VINFO_LENS (loop_vinfo) - : NULL); + ? &LOOP_VINFO_LENS (loop_vinfo) + : NULL); /* The vect_transform_stmt and vect_analyze_stmt will go here but there are some difference here. We cannot enable both the lens and masks @@ -10578,15 +10614,15 @@ vectorizable_load (vec_info *vinfo, Shouldn't go with length-based approach if fully masked. */ if (cost_vec == NULL) /* The cost_vec is NULL during transform. */ - gcc_assert ((!loop_lens || !loop_masks)); + gcc_assert ((!vect_fully_with_length_p (vinfo, slp_node) + || !vect_fully_masked_p (vinfo, slp_node))); /* Targets with store-lane instructions must not require explicit realignment. vect_supportable_dr_alignment always returns either dr_aligned or dr_unaligned_supported for (non-length) masked operations. */ - gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES - && !mask_node - && !loop_masks) + gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES && !mask_node + && !vect_fully_masked_p (vinfo, slp_node)) || mat_gather_scatter_p (memory_access_type) || alignment_support_scheme == dr_aligned || alignment_support_scheme == dr_unaligned_supported); @@ -10785,18 +10821,18 @@ vectorizable_load (vec_info *vinfo, tree final_mask = NULL_TREE; tree final_len = NULL_TREE; tree bias = NULL_TREE; - if (loop_masks) - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks, - ncopies, vectype, j); + if (vect_fully_masked_p (vinfo, slp_node)) + final_mask + = vect_get_mask (vinfo, slp_node, gsi, ncopies, vectype, j); if (vec_mask) - final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask, + final_mask = prepare_vec_mask (vinfo, mask_vectype, final_mask, vec_mask, gsi); if (lanes_ifn == IFN_MASK_LEN_LOAD_LANES) { - if (loop_lens) - final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens, - ncopies, vectype, j, 1, true); + if (vect_fully_with_length_p (vinfo, slp_node)) + final_len = vect_get_len (vinfo, slp_node, gsi, ncopies, + vectype, j, 1, true); else final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype)); signed char biasval @@ -10934,11 +10970,11 @@ vectorizable_load (vec_info *vinfo, { if (mask_node) vec_mask = vec_masks[i]; - if (loop_masks) - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks, - vec_num, vectype, i); + if (vect_fully_masked_p (vinfo, slp_node)) + final_mask + = vect_get_mask (vinfo, slp_node, gsi, vec_num, vectype, i); if (vec_mask) - final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, + final_mask = prepare_vec_mask (vinfo, mask_vectype, final_mask, vec_mask, gsi); if (i > 0 && !STMT_VINFO_GATHER_SCATTER_P (stmt_info)) @@ -11005,8 +11041,8 @@ vectorizable_load (vec_info *vinfo, if (ls.gs.ifn == IFN_MASK_LEN_GATHER_LOAD) { if (loop_lens) - final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens, - vec_num, vectype, i, 1, true); + final_len = vect_get_len (vinfo, slp_node, gsi, vec_num, + vectype, i, 1, true); else final_len = build_int_cst (sizetype, TYPE_VECTOR_SUBPARTS (vectype)); @@ -11412,11 +11448,11 @@ vectorizable_load (vec_info *vinfo, { if (mask_node) vec_mask = vec_masks[i]; - if (loop_masks) - final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks, - vec_num, vectype, i); + if (vect_fully_masked_p (vinfo, slp_node)) + final_mask + = vect_get_mask (vinfo, slp_node, gsi, vec_num, vectype, i); if (vec_mask) - final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, + final_mask = prepare_vec_mask (vinfo, mask_vectype, final_mask, vec_mask, gsi); if (i > 0) @@ -11462,8 +11498,8 @@ vectorizable_load (vec_info *vinfo, new_vmode = new_ovmode.require (); unsigned factor = (new_ovmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vmode); - final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens, - vec_num, vectype, i, factor, true); + final_len = vect_get_len (vinfo, slp_node, gsi, vec_num, + vectype, i, factor, true); } else if (final_mask) { diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 35e531c11c1..a3855568b09 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -686,6 +686,9 @@ public: and are reset when undoing patterns. */ gimple_seq inv_pattern_def_seq; + /* Set of vector conditions that have loop mask applied. */ + vec_cond_masked_set_type vec_cond_masked_set; + private: stmt_vec_info new_stmt_vec_info (gimple *stmt); void set_vinfo_for_stmt (gimple *, stmt_vec_info, bool = true); @@ -1021,9 +1024,6 @@ public: /* Set of scalar conditions that have loop mask applied. */ scalar_cond_masked_set_type scalar_cond_masked_set; - /* Set of vector conditions that have loop mask applied. */ - vec_cond_masked_set_type vec_cond_masked_set; - /* If we are using a loop mask to align memory addresses, this variable contains the number of vector elements that we should skip in the first iteration of the vector loop (i.e. the number of leading @@ -2789,7 +2789,7 @@ extern void vect_free_slp_tree (slp_tree); extern bool compatible_calls_p (gcall *, gcall *, bool); extern int vect_slp_child_index_for_operand (const stmt_vec_info, int op); -extern tree prepare_vec_mask (loop_vec_info, tree, tree, tree, +extern tree prepare_vec_mask (vec_info *, tree, tree, tree, gimple_stmt_iterator *); extern tree vect_get_mask_load_else (int, tree); extern bool vect_load_perm_consecutive_p (slp_tree, unsigned = UINT_MAX); @@ -2925,6 +2925,71 @@ vect_is_reduction (slp_tree slp_node) return SLP_TREE_REDUC_IDX (slp_node) != -1; } +/* Return true if VINFO is vectorizer state for loop vectorization and we + still have the option of vectorizing the loop using partially-populated + vectors; or, true if VINFO is for basic-block vectorization and we still have + the option of vectorizing the given SLP_NODE using partial vectors; + otherwise, return false. */ +inline bool +vect_can_use_partial_vectors_p (vec_info *vinfo, slp_tree slp_node) +{ + loop_vec_info loop_vinfo = dyn_cast (vinfo); + if (loop_vinfo) + return LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo); + + (void) slp_node; /* FORNOW */ + return false; +} + +/* If VINFO is vectorizer state for loop vectorization then record that we no + longer have the option of vectorizing the loop using partially-populated + vectors; or, if VINFO is for basic-block vectorization then record that we no + longer have the option of vectorizing the given SLP_NODE using partial + vectors. */ +inline void +vect_cannot_use_partial_vectors (vec_info *vinfo, slp_tree slp_node) +{ + loop_vec_info loop_vinfo = dyn_cast (vinfo); + if (loop_vinfo) + LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; + else + (void) slp_node; /* FORNOW */ +} + +/* Return true if VINFO is vectorizer state for loop vectorization, we've + decided to use partially-populated vectors so that the loop can handle fewer + than VF scalars, and at least one length has been recorded; or, if VINFO is + for basic-block vectorization and SLP_NODE will be vectorized by using + lengths to prevent use of inactive scalar lanes. */ +inline bool +vect_fully_with_length_p (vec_info *vinfo, slp_tree slp_node) +{ + if (loop_vec_info loop_vinfo = dyn_cast (vinfo)) + return LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo); + else + { + (void) slp_node; /* FORNOW */ + return false; + } +} + +/* Return true if VINFO is vectorizer state for loop vectorization, we've + decided to use partially-populated vectors so that the loop can handle fewer + than VF scalars, and at least one mask has been recorded; or, if VINFO is + for basic-block vectorization and SLP_NODE will be vectorized by using masks + to prevent use of inactive scalar lanes. */ +inline bool +vect_fully_masked_p (vec_info *vinfo, slp_tree slp_node) +{ + if (loop_vec_info loop_vinfo = dyn_cast (vinfo)) + return LOOP_VINFO_FULLY_MASKED_P (loop_vinfo); + else + { + (void) slp_node; /* FORNOW */ + return false; + } +} + /* If STMT_INFO describes a reduction, return the vect_reduction_type of the reduction it describes, otherwise return -1. */ inline int From patchwork Wed Jun 3 15:19:14 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136408 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 7C9E74BA23C5 for ; Wed, 3 Jun 2026 15:26:20 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7C9E74BA23C5 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=njvfm+X9; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=njvfm+X9 X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from PA4PR04CU001.outbound.protection.outlook.com (mail-francecentralazon11013069.outbound.protection.outlook.com [40.107.162.69]) by sourceware.org (Postfix) with ESMTPS id ECEC64BA2E3A for ; Wed, 3 Jun 2026 15:20:53 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org ECEC64BA2E3A Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org ECEC64BA2E3A Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=40.107.162.69 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500054; cv=pass; b=JRcyqxtDHCuxXeK+sc9Ddu5oSG/NxHDIJIDelA/y5JJtWA5S8mxQHDW/nW3Lg8b9Ius/5ul0JJmFwrEQDix5x2opnmGxGB44o5H7U2X8aPGk96u+LWvQpg0HF3phinerruOhI6/kq20ihNxNqFw1wQzp8PxXxaRqFPspwmxXO6g= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500054; c=relaxed/simple; bh=SBLsui3m28S5W+XW1v0vxH0z2N5/UeIJImDQrXYQSJU=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=LP+9B3BLWGQQh14lCyW/bSfS+cSrPNkqkXMmNaZYqhyGAHwmJuAC139TYWdfKsM45llEf/s7exHwIBeA2YNEOVRI9s/68atIHOckVXcyNgaA1VGCj2bq4TR9uT62IMD9NBgr1UzhmHiDnIYyeVGUbe+2qHlb9pgFgMLCr+WRPEU= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=njvfm+X9; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=njvfm+X9 DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org ECEC64BA2E3A ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=R/vueiSFP0HwlfEEnobi5uWPA6hPEWSBjKEbry7HXoLPPJ3fOrEL7y/Xy+3ff/rGUPLJsDqwZT07IQFJS1v1Bgy17yxyHMVzbHgZcG92reeABV9uYem+UjaP1YAV0W5CtAVbUKYqkgwqUmOqXU+Yvwh/xj4MeIOh8mAwA0ITQ0rT+vVoEvshw/cl+h1qTKOWQ6BkrFO0apXBkI8Iz1W0/XRdWzIS9aiVLskba8GIdPn8W3Pg1pwygbCym7/ISt0a6SQx1TPyWfVblTyfJQlojSE0wPQAkcFWpES+DtV1ytGy+KI1ZCUuNeDWNEKPGraKelqplazNdTh0xVzcwE6icA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qxIuFg4M98kC1dhvVXbYyjuqtkqBmUZN5xb0cQR2byg=; b=pQSZ/XXhDP5y3AfP4t8lqmV6t8T9UyzLOMq2zEqWIr7yOF2Av+pNQKmw2wwhtZIfegc03bvtC4ke7wH1QaRV//fxUQRKMV+IBP+zxqXxjiuhAy2rVaG8ZVHR6p/1Q59rTFPXFY8R49Kpu5fkOzg+rsMbKnW2l9i6Mj72l9AdfCMpsladIwjOiMQYzv0D/SIwGIAeBkm1fHGmpEj0QUhsa6uYN9HspdGzdcpWFvdq5ZOUVll7tbwsyglbh++nCZY+W7vNr0ArePRPYvHer2i9I7piFA+gUFCgotwgyU8i7OEYDDC0CL7AbJ4ls76b0f7X8xEFHUf/buAdSTyEd0/crw== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qxIuFg4M98kC1dhvVXbYyjuqtkqBmUZN5xb0cQR2byg=; b=njvfm+X9d6X8vZgLMHyM6KCwdL1tExDO0Y2WI22NwLmrkLa0JrJr2ECdgaJggRoI5Ktn6kN6/itZxI2/neDXgZPGmsPmvPuIPD9TqXU7EYzB0zGSi45OQDQMWcLwEPCqL379i2FKNnozKAf4pFbX8qwR8vaxMEcPKbv92i3dK1k= Received: from DU7PR01CA0030.eurprd01.prod.exchangelabs.com (2603:10a6:10:50e::19) by AM0PR08MB5524.eurprd08.prod.outlook.com (2603:10a6:208:181::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:20:46 +0000 Received: from DU2PEPF0001E9C4.eurprd03.prod.outlook.com (2603:10a6:10:50e:cafe::28) by DU7PR01CA0030.outlook.office365.com (2603:10a6:10:50e::19) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:20:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by DU2PEPF0001E9C4.mail.protection.outlook.com (10.167.8.73) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:20:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=D8ar+EhSLyE6eNNIcMRT17UcFsoxfGHK7HrWDKDVO4t0tRUOdNkXqo675q1DXyLGmW2cglUFcJTaX+G5aZmzi7j54bKhO6xxUGfNQg1VHINY2onvkNLaNs+WLoju9YqY4Ln8GjkLdatslCTXk9VtKC22y1ef3me5fSI6wqglap/vgGbJffBQoOqCIqbweLzj2uAbU2TXaM6wGyI3UJ1uL5/yHnLxzlokH50DaFlR9n0thHgjUFZOXifOw/v7ujMb/DImYq6ZehxQa352bhue8W94Oeust0n+5cm5WD5Qny3sluQrPMTteVQRI/samE6Cbe/iFUUnAuGZ7MvgJ1tNZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qxIuFg4M98kC1dhvVXbYyjuqtkqBmUZN5xb0cQR2byg=; b=GeBdXqnf1muvQvarZvOlcCvxJz8gn2gRDr+bJ8llMhc0R6iO8kCJb8Ow95fCE9yslKNjGppxX5MwuXR6O2Z3nrhU1hg0FbndAg3Da0dLrXSxR7KYrHeKrqstM3A4YlGSuhb6Yl6LDYSGo+j+l8ORIdSn6lsdBgJSXEB6u5tGaiQzlVbxpqQNCad191BA2mrqKWPIQTNYYxJORVCKkDsYykXXlVcaNLBvTbuNLQnk9F56e3268OFkrnlSD4x5I3CY1MyZqVybcXMBlSj4+i4MMEuV6JqNC3chD8BF+mtlS5QWFVpN2HY2vtDyXuIc3ESWazjqER0k8j6kJWsmZSfCGw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qxIuFg4M98kC1dhvVXbYyjuqtkqBmUZN5xb0cQR2byg=; b=njvfm+X9d6X8vZgLMHyM6KCwdL1tExDO0Y2WI22NwLmrkLa0JrJr2ECdgaJggRoI5Ktn6kN6/itZxI2/neDXgZPGmsPmvPuIPD9TqXU7EYzB0zGSi45OQDQMWcLwEPCqL379i2FKNnozKAf4pFbX8qwR8vaxMEcPKbv92i3dK1k= Received: from DU2PR04CA0165.eurprd04.prod.outlook.com (2603:10a6:10:2b0::20) by PAWPR08MB10090.eurprd08.prod.outlook.com (2603:10a6:102:367::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:19:37 +0000 Received: from DU6PEPF00009528.eurprd02.prod.outlook.com (2603:10a6:10:2b0:cafe::5f) by DU2PR04CA0165.outlook.office365.com (2603:10a6:10:2b0::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009528.mail.protection.outlook.com (10.167.8.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:37 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:34 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:34 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 02/12] Implement recording/getting of mask/length for BB SLP Date: Wed, 3 Jun 2026 15:19:14 +0000 Message-ID: <20260603151924.53706-3-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009528:EE_|PAWPR08MB10090:EE_|DU2PEPF0001E9C4:EE_|AM0PR08MB5524:EE_ X-MS-Office365-Filtering-Correlation-Id: 60c4a14f-cedd-4e29-51a4-08dec183af70 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|1800799024|82310400026|376014|36860700016|6133799003|3023799007|22082099003|18002099003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info-Original: Sjk7QiZEPuW/9SKjrDJn/BENs321jSHITAgsqzOga7uKk/34t5vBs0NU1TPHT3i0KEUoirN7ZyNdv4ZdpRhQB7ORvIMTnBUmy4KD7t5xrD8uC0+2P0O1rrYt0d2o5ZFQ24lbtecQVBOhovDuYZSWjyPYLP9YV8BowPUWVvi6W8Bx++Vk6VxX50UvsV4uCZCt1JA6V74zCK3nTtX9o/tZMwQBjYzo/huC3QLGaqL66yrS+bzaD8gVdaLpR/6XWQgNmi8LECswF1fdIKlvTTtYSFsxybgcpAeebZr7ocDn/kQ5H8W00F3ACxzoGkqTXRXtrKz+tIqhaMlJq0N/Rv0IxFiYYtIYxxItxr45gqwl3azY9NRDZ0WwWbZGtoIeHBjXFrfojjH9fEcP7qH4TfKJof7x9JMZfQzcpGLNnKAe922L9X9m8lctmGq1e6EK0dtXOLb51f6HjJrzYBLu+EKDXjyctG9BUiGPAMiLGqFO630tbZ7RrWpHmig04XzRqgB5XjzUQyMI3f8IoKOQpwYLxL1zdAmweCxWQsDAew6Z8/4dIgB+pq3konwWLE8MjH7Dpea3agzAAiMkpKgzgvHZuTnIzGgIwg+R5UiPlNPcX4H670RTSpKHtZVAyKXADwEi7ctJd8VjD4P451Rjv2nUpYdIeHK8GiID6AqptkzyBiQaDsWklg4koWiagBqkDdg1X3Ai+z2H0s5F5zPzwxoYCs26czEfTxGD3bgwWzTUf5Q= X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(82310400026)(376014)(36860700016)(6133799003)(3023799007)(22082099003)(18002099003)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: S2J2auI+dJnGfoXeIeJpwIeI5CidvUdEhRl0VThUMEXHKa5EsJx7HyYRNcTXf7WWtY8H5PTNmRO3OtVALRXZY0mcI5F4kh3URz3s/2mW9+4/Gmvky8vjjNDh7qYIQrywkJnmGLpyTV9cinAz+JtxNkvVcN525rre/xprb8NzsIt4Ln+qW/DzsBUGaFc/FTCfcwVPcKWjh9GCBSZU3VKkLVcf5mRXLo4Wpvt1pmfpqa6s7Q4OcCc0H05Sg1SO2FP0p17JoIp5Wrx07sW8yt/9qWyg7vfs/RhlDRr7/K8aXN//vNEz4dn3VNnnsJDjxuFj7oD0oJx1um/S/CUxfEr7lg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB10090 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DU2PEPF0001E9C4.eurprd03.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 5987f8ea-7800-4b8c-ce87-08dec183869e X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700016|14060799003|1800799024|82310400026|35042699022|376014|22082099003|6133799003|18002099003|3023799007|56012099006|11063799006; X-Microsoft-Antispam-Message-Info: a5tHvZqlNc5TL0cIFAj/PGKXZ4c3I4pv8xJ4fWAxCSFp3gHgZjHUvOrHXjJqd+6hj2VMSF3xecyDQk6E7EffJ2M5Ig7JWQAbZe2SwAKQS61uzs93FZ9S424N+Co7S6yVU9Ze3a/x5Uyg/hu4kbP2TJm8caX20HpAX9VEwXkXsTTdZrjQifwbDeKgs5f8SEVw0BVQ+Uad2uONcEmL5bDXMB2RqR95o5KOi/fR6EzAXh2OnqGP6rlWnAME4Xta6q9ny+qyPWb0rnkm+A+gndplvBWftCZ/W4fFQ3PBphBDFcaRXbb238+WWh5tw03gB5b/tL0AAD5agtv68DiuWmuntdARyp9jSt/267f0mCklLoy36uSsRu4/nB0jmc2N+z5qqzJzb6FaermxqPnlUZOxxj8RrRKS41psQmWGULJG6XbTf2k5mRjyqWdGivr22fwLKPCmqVgvME8bEQk/eGbogUlZL6tuHMWZhOpLWE+sEuGrBukwefi8QlZHe52PqdRQmqFOnlOBKDTakjPxS1PZABQkMKxjXFbeQfWSQ0JlDavyAkGM0gh+wQh3h1v9X9+5Jca+23aZeDeyhQQL37u4dKGywn4gZnkkr4o9nJ5jXYogpiWd13U5KD8Am7ND2TqY2HfKwoqJaYdhGYoszJI/IqrU6R7oLLSUHWD3GDfBsAJl2ivVU2qLYeN73YoA4pCYy3JLWV3E1wlfYL8zoCukjHC/e5Fgh6ya994SZcyPlzE= X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(36860700016)(14060799003)(1800799024)(82310400026)(35042699022)(376014)(22082099003)(6133799003)(18002099003)(3023799007)(56012099006)(11063799006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ZOixNCKg2qBtXborT6JaQsYqwXPKsh35Qht99neBnoTDO8+1bMKLiVvsDgpeqf0gWKOJnjQ8nsnlhtwkDg/R1g2vppys3lq2jEkww8D10VpplQS2noTB7/nRkQx489HSNZ0X0Nlq96/372g+IMRrdjieq1dLSsdx185v5DLtTe4dK2oDRwTM8NK/qf/QT6Wq7hiUt+G56BAQVOJNwY/f2VOvsm8dHhfx780LVEu2Y6pG2CN5oSRcFOT5PZrtxc2VKx2ZayTjEs5hSiBvrKbP9S/QEo8xKHkjjrosaqmvljKhi/238fOQv6KrCRGFJKldfcC6mcP6SSNoOEs9OGHml6e+UA4jZjTTRtuWwuoGeMhBfN56qSNA3dqMuiDWfKt4v0uNk078dO4zyXS/JI49jzQxjNa2kaqJKrwp80SYeIYtfW2tqM1bA2KHmspuU5wh X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:20:46.0360 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 60c4a14f-cedd-4e29-51a4-08dec183af70 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: DU2PEPF0001E9C4.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB5524 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, RCVD_IN_DNSWL_BLOCKED, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Add two new fields to SLP tree nodes, which are accessed as SLP_TREE_CAN_USE_PARTIAL_VECTORS_P and SLP_TREE_PARTIAL_VECTORS_STYLE. SLP_TREE_CAN_USE_PARTIAL_VECTORS_P is analogous to the existing predicate LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P. It is initialized to true. This flag just records whether the target could vectorize a node using a partial vector; it does not say anything about whether the vector actually is partial, or how the target would support use of a partial vector. Some kinds of node require mask/length for partial vectors; others don't. In the latter case (e.g., for add operations), SLP_TREE_CAN_USE_PARTIAL_VECTORS_P will remain true. SLP_TREE_PARTIAL_VECTORS_STYLE is analogous to the existing field LOOP_VINFO_PARTIAL_VECTORS_STYLE. Both are initialized to 'none'. The vect_partial_vectors_avx512 enumerator is not used for BB SLP. Unlike loop vectorization, a different style of partial vectors can be chosen for each node during analysis of that node. Implement the recently-introduced wrapper functions, vect_record_(len|mask), for BB SLP by setting SLP_TREE_PARTIAL_VECTORS_STYLE to indicate that a mask or length should be used for a given SLP node. The passed-in vec_info is ignored. Implement the vect_fully_(masked|with_length)_p wrapper functions for BB SLP by checking the SLP_TREE_PARTIAL_VECTORS_STYLE. This should be sufficient because at most one of vect_record_(len|mask) and vect_cannot_use_partial_vectors are expected to be called for any given SLP node. SLP_TREE_CAN_USE_PARTIAL_VECTORS_P should be true if the style is not 'none', but its value isn't used beyond the analysis phase. The implementations of vect_get_mask and vect_get_len for BB SLP are non-trivial (albeit simpler than for loop vectorization), therefore they are delegated to SLP-specific functions defined in tree-vect-slp.cc. Implement the vect_cannot_use_partial_vectors wrapper function by setting the SLP_TREE_CAN_USE_PARTIAL_VECTORS_P flag to false. To prevent regressions, vect_can_use_partial_vectors_p still returns false for BB SLP regardless (for now). This prevents vect_record_mask or vect_record_len from being called. gcc/ChangeLog: * tree-vect-slp.cc (_slp_tree::_slp_tree): initialize new partial_vector_style, can_use_partial_vectors and num_partial_vectors members. (vect_slp_analyze_node_operations): Account for worst-case prologue costs of per-node partial-vector mask or length materialisation. (vect_slp_record_bb_style): Set the partial vector style of an SLP node, checking that the style does not flip-flop between mask and length. (vect_slp_record_bb_mask): Use vect_slp_record_bb_style to set the partial vector style of the SLP tree node to vect_partial_vectors_while_ult. (vect_slp_get_bb_mask): New function to materialize a mask for basic block SLP vectorization. (vect_slp_record_bb_len): Use vect_slp_record_bb_style to set the partial vector style of the SLP tree node to vect_partial_vectors_len. (vect_slp_get_bb_len): New function to materialize a length for basic block SLP vectorization. * tree-vect-stmts.cc (vectorizable_internal_function): (vect_record_mask): Handle the basic block SLP use case by delegating to vect_slp_record_bb_mask. (vect_get_mask): Handle the basic block SLP use case by delegating to vect_slp_get_bb_mask. (vect_record_len): Handle the basic block SLP use case by delegating to vect_slp_record_bb_len. (vect_get_len): Handle the basic block SLP use case by delegating to vect_slp_get_bb_len. (vect_gen_while_ssa_name): New function containing code refactored out of vect_gen_while for reuse by vect_slp_get_bb_mask. (vect_gen_while): Use vect_gen_while_ssa_name instead of custom code for some of the implementation. * tree-vectorizer.h (enum vect_partial_vector_style): Move this definition earlier to allow reuse by struct _slp_tree. (struct _slp_tree): Add a partial_vector_style member to record whether to use a length or mask for the SLP tree node, if partial vectors are required and supported. Add a can_use_partial_vectors member to record whether partial vectors are supported for the SLP tree node. Add a num_partial_vectors member for costing. (SLP_TREE_PARTIAL_VECTORS_STYLE): New member accessor macro. (SLP_TREE_CAN_USE_PARTIAL_VECTORS_P): New member accessor macro. (SLP_TREE_NUM_PARTIAL_VECTORS): New member accessor macro. (vect_gen_while_ssa_name): Declaration of a new function. (vect_slp_get_bb_mask): As above. (vect_slp_get_bb_len): As above. (vect_cannot_use_partial_vectors): Handle the basic block SLP use-case by setting SLP_TREE_CAN_USE_PARTIAL_VECTORS_P to false. (vect_fully_with_length_p): Handle the basic block SLP use case by checking whether the SLP_TREE_PARTIAL_VECTORS_STYLE is vect_partial_vectors_len. (vect_fully_masked_p): Handle the basic block SLP use case by checking whether the SLP_TREE_PARTIAL_VECTORS_STYLE is vect_partial_vectors_while_ult. --- gcc/tree-vect-slp.cc | 182 +++++++++++++++++++++++++++++++++++++++++ gcc/tree-vect-stmts.cc | 52 +++++++----- gcc/tree-vectorizer.h | 52 ++++++++---- 3 files changed, 247 insertions(+), 39 deletions(-) diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index 075e93f04a9..4dd7e6e1e21 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -125,6 +125,9 @@ _slp_tree::_slp_tree () SLP_TREE_GS_BASE (this) = NULL_TREE; this->ldst_lanes = false; this->avoid_stlf_fail = false; + SLP_TREE_PARTIAL_VECTORS_STYLE (this) = vect_partial_vectors_none; + SLP_TREE_CAN_USE_PARTIAL_VECTORS_P (this) = true; + SLP_TREE_NUM_PARTIAL_VECTORS (this) = 0; SLP_TREE_VECTYPE (this) = NULL_TREE; SLP_TREE_REPRESENTATIVE (this) = NULL; this->cycle_info.id = -1; @@ -8958,6 +8961,40 @@ vect_slp_analyze_node_operations (vec_info *vinfo, slp_tree node, vect_prologue_cost_for_slp (vinfo, child, cost_vec); } + if (res) + { + /* Take care of special costs for partial vectors. + Costing each partial vector is excessive for many SLP instances, + because it is common to materialise identical masks/lengths for related + operations (e.g., for vector loads and stores of the same length). + Masks/lengths can also be shared between SLP subgraphs or eliminated by + pattern-based lowering during instruction selection. However, it's + simpler and safer to use the worst-case cost; if this ends up being the + tie-breaker between vectorizing or not, then it's probably better not + to vectorize. */ + const int num_partial_vectors = SLP_TREE_NUM_PARTIAL_VECTORS (node); + + if (SLP_TREE_PARTIAL_VECTORS_STYLE (node) + == vect_partial_vectors_while_ult) + { + gcc_assert (num_partial_vectors > 0); + record_stmt_cost (cost_vec, num_partial_vectors, vector_stmt, NULL, + NULL, NULL_TREE, 0, vect_prologue); + } + else if (SLP_TREE_PARTIAL_VECTORS_STYLE (node) + == vect_partial_vectors_len) + { + /* Need to set up a length in the prologue. */ + gcc_assert (num_partial_vectors > 0); + record_stmt_cost (cost_vec, num_partial_vectors, scalar_stmt, NULL, + NULL, NULL_TREE, 0, vect_prologue); + } + else + { + gcc_assert (num_partial_vectors == 0); + } + } + /* If this node or any of its children can't be vectorized, try pruning the tree here rather than felling the whole thing. */ if (!res && vect_slp_convert_to_external (vinfo, node, node_instance)) @@ -12441,3 +12478,148 @@ vect_schedule_slp (vec_info *vinfo, const vec &slp_instances) } } } + +/* Record that a specific partial vector style could be used to vectorize + SLP_NODE if required. */ + +static void +vect_slp_record_bb_style (slp_tree slp_node, vect_partial_vector_style style) +{ + gcc_assert (style != vect_partial_vectors_none); + gcc_assert (style != vect_partial_vectors_avx512); + + if (SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) == vect_partial_vectors_none) + SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) = style; + else + gcc_assert (SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) == style); +} + +/* Record that a complete set of masks associated with SLP_NODE would need to + contain a sequence of NVECTORS masks that each control a vector of type + VECTYPE. If SCALAR_MASK is nonnull, the fully-masked loop would AND + these vector masks with the vector version of SCALAR_MASK. */ +void +vect_slp_record_bb_mask (slp_tree slp_node, unsigned int /* nvectors */, + tree /* vectype */, tree /* scalar_mask */) +{ + vect_slp_record_bb_style (slp_node, vect_partial_vectors_while_ult); + + /* FORNOW: this often overestimates the number of masks for costing purposes + because, after lowering, masks have often been eliminated, shared between + SLP nodes, or even shared between SLP subgraphs. */ + SLP_TREE_NUM_PARTIAL_VECTORS(slp_node) ++; +} + +/* Materialize mask number INDEX for a group of scalar stmts in SLP_NODE that + operate on NVECTORS vectors of type VECTYPE, where 0 <= INDEX < NVECTORS. + Insert any set-up statements before GSI. */ + +tree +vect_slp_get_bb_mask (slp_tree slp_node, gimple_stmt_iterator *gsi, + unsigned int nvectors, tree vectype, unsigned int index) +{ + gcc_assert (SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) + == vect_partial_vectors_while_ult); + gcc_assert (nvectors >= 1); + gcc_assert (index < nvectors); + + const poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); + const unsigned int group_size = SLP_TREE_LANES (slp_node); + unsigned int mask_size = group_size; + const tree masktype = truth_type_for (vectype); + + if (nunits.is_constant ()) + { + /* Only the last vector can be a partial vector. */ + if (index + 1 < nvectors) + return build_minus_one_cst (masktype); + + /* Return a mask for a possibly-partial tail vector. */ + const unsigned int const_nunits = nunits.to_constant (); + const unsigned int head_size = (nvectors - 1) * const_nunits; + gcc_assert (head_size <= group_size); + mask_size = group_size - head_size; + + if (mask_size == const_nunits) + return build_minus_one_cst (masktype); + } + else + { + /* Return a mask for a single variable-length vector. */ + gcc_assert (nvectors == 1); + gcc_assert (known_le (mask_size, nunits)); + } + + /* FORNOW: don't bother maintaining a set of mask constants to allow + sharing between nodes belonging to the same instance of bb_vec_info + or even within the same SLP subgraph. */ + gimple_seq stmts = NULL; + const tree cmp_type = size_type_node; + const tree start_index = build_zero_cst (cmp_type); + const tree end_index = build_int_cst (cmp_type, mask_size); + const tree mask = make_temp_ssa_name (masktype, NULL, "slp_mask"); + vect_gen_while_ssa_name (&stmts, masktype, start_index, end_index, mask); + gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT); + return mask; +} + +/* Record that a complete set of lengths associated with SLP_NODE would need to + contain a sequence of NVECTORS lengths for controlling an operation on + VECTYPE. The operation splits each element of VECTYPE into FACTOR separate + subelements, measuring the length as a number of these subelements. */ + +void +vect_slp_record_bb_len (slp_tree slp_node, unsigned int /* nvectors */, + tree /* vectype */, unsigned int /* factor */) +{ + vect_slp_record_bb_style (slp_node, vect_partial_vectors_len); + + /* FORNOW: this probably overestimates the number of lengths for costing + purposes because, after lowering, lengths might have been eliminated, + shared between SLP nodes, or even shared between SLP subgraphs. */ + SLP_TREE_NUM_PARTIAL_VECTORS (slp_node)++; +} + +/* Materialize length number INDEX for a group of scalar stmts in SLP_NODE that + operate on NVECTORS vectors of type VECTYPE, where 0 <= INDEX < NVECTORS. + Return a value that contains FACTOR multiplied by the number of elements that + should be processed. */ + +tree +vect_slp_get_bb_len (slp_tree slp_node, unsigned int nvectors, tree vectype, + unsigned int index, unsigned int factor, bool adjusted) +{ + gcc_checking_assert (SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) + == vect_partial_vectors_len); + gcc_assert (nvectors >= 1); + gcc_assert (index < nvectors); + (void) adjusted; + + const poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); + const unsigned int group_size = SLP_TREE_LANES (slp_node); + unsigned int len = group_size; + + if (nunits.is_constant ()) + { + const unsigned int const_nunits = nunits.to_constant (); + + /* Only the last vector can be a partial vector. */ + if (index + 1 < nvectors) + len = const_nunits; + else + { + /* Return a length for a possibly-partial tail vector. */ + const unsigned int head_size = (nvectors - 1) * const_nunits; + gcc_assert (head_size <= group_size); + len = group_size - head_size; + } + } + else + { + /* Return a length for a single variable-length vector. */ + gcc_assert (nvectors == 1); + gcc_assert (known_le (len, nunits)); + } + + return size_int (len * factor); +} diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 15fca17a407..ecad74e7cbf 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -1385,7 +1385,9 @@ vectorizable_internal_function (combined_fn cfn, tree fndecl, /* Record that a complete set of masks associated with VINFO would need to contain a sequence of NVECTORS masks that each control a vector of type VECTYPE. If SCALAR_MASK is nonnull, the fully-masked loop would AND - these vector masks with the vector version of SCALAR_MASK. */ + these vector masks with the vector version of SCALAR_MASK. Alternatively, + if doing basic block vectorization, record that a mask could be used to + vectorize SLP_NODE if required. */ static void vect_record_mask (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors, tree vectype, tree scalar_mask) @@ -1395,7 +1397,7 @@ vect_record_mask (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors, vect_record_loop_mask (loop_vinfo, &LOOP_VINFO_MASKS (loop_vinfo), nvectors, vectype, scalar_mask); else - (void) slp_node; /* FORNOW */ + vect_slp_record_bb_mask (slp_node, nvectors, vectype, scalar_mask); } /* Given a complete set of masks associated with VINFO, extract mask number @@ -1413,16 +1415,15 @@ vect_get_mask (vec_info *vinfo, slp_tree slp_node, gimple_stmt_iterator *gsi, return vect_get_loop_mask (loop_vinfo, gsi, &LOOP_VINFO_MASKS (loop_vinfo), nvectors, vectype, index); else - { - (void) slp_node; /* FORNOW */ - return NULL_TREE; - } + return vect_slp_get_bb_mask (slp_node, gsi, nvectors, vectype, index); } /* Record that a complete set of lengths associated with VINFO would need to contain a sequence of NVECTORS lengths for controlling an operation on VECTYPE. The operation splits each element of VECTYPE into FACTOR separate - subelements, measuring the length as a number of these subelements. */ + subelements, measuring the length as a number of these subelements. + Alternatively, if doing basic block vectorization, record that a length limit + could be used to vectorize SLP_NODE if required. */ static void vect_record_len (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors, tree vectype, unsigned int factor) @@ -1432,7 +1433,7 @@ vect_record_len (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors, vect_record_loop_len (loop_vinfo, &LOOP_VINFO_LENS (loop_vinfo), nvectors, vectype, factor); else - (void) slp_node; /* FORNOW */ + vect_slp_record_bb_len (slp_node, nvectors, vectype, factor); } /* Given a complete set of lengths associated with VINFO, extract length number @@ -1453,10 +1454,8 @@ vect_get_len (vec_info *vinfo, slp_tree slp_node, gimple_stmt_iterator *gsi, return vect_get_loop_len (loop_vinfo, gsi, &LOOP_VINFO_LENS (loop_vinfo), nvectors, vectype, index, factor, adjusted); else - { - (void) slp_node; /* FORNOW */ - return NULL_TREE; - } + return vect_slp_get_bb_len (slp_node, nvectors, vectype, index, factor, + adjusted); } static tree permute_vec_elements (vec_info *, tree, tree, tree, stmt_vec_info, @@ -14710,24 +14709,35 @@ supportable_indirect_convert_operation (code_helper code, mask[I] is true iff J + START_INDEX < END_INDEX for all J <= I. Add the statements to SEQ. */ +void +vect_gen_while_ssa_name (gimple_seq *seq, tree mask_type, tree start_index, + tree end_index, tree ssa_name) +{ + tree cmp_type = TREE_TYPE (start_index); + gcc_checking_assert (direct_internal_fn_supported_p (IFN_WHILE_ULT, cmp_type, + mask_type, + OPTIMIZE_FOR_SPEED)); + gcall *call + = gimple_build_call_internal (IFN_WHILE_ULT, 3, start_index, end_index, + build_zero_cst (mask_type)); + gimple_call_set_lhs (call, ssa_name); + gimple_seq_add_stmt (seq, call); +} + +/* Like vect_gen_while_ssa_name except that it creates a new SSA_NAME node + for type MASK_TYPE defined in the created GIMPLE_CALL statement. If NAME + is not a null pointer then it is used for the SSA_NAME in dumps. */ + tree vect_gen_while (gimple_seq *seq, tree mask_type, tree start_index, tree end_index, const char *name) { - tree cmp_type = TREE_TYPE (start_index); - gcc_checking_assert (direct_internal_fn_supported_p (IFN_WHILE_ULT, - cmp_type, mask_type, - OPTIMIZE_FOR_SPEED)); - gcall *call = gimple_build_call_internal (IFN_WHILE_ULT, 3, - start_index, end_index, - build_zero_cst (mask_type)); tree tmp; if (name) tmp = make_temp_ssa_name (mask_type, NULL, name); else tmp = make_ssa_name (mask_type); - gimple_call_set_lhs (call, tmp); - gimple_seq_add_stmt (seq, call); + vect_gen_while_ssa_name (seq, mask_type, start_index, end_index, tmp); return tmp; } diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index a3855568b09..f79f04ff8ac 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -312,6 +312,13 @@ struct vect_load_store_data : vect_data { bool subchain_p; // VMAT_STRIDED_SLP and VMAT_GATHER_SCATTER }; +enum vect_partial_vector_style { + vect_partial_vectors_none, + vect_partial_vectors_while_ult, + vect_partial_vectors_avx512, + vect_partial_vectors_len +}; + /* A computation tree of an SLP instance. Each node corresponds to a group of stmts to be packed in a SIMD stmt. */ struct _slp_tree { @@ -377,7 +384,16 @@ struct _slp_tree { /* For BB vect, flag to indicate this load node should be vectorized as to avoid STLF fails because of related stores. */ bool avoid_stlf_fail; - + /* The style used for implementing partial vectors if LANES is less than + the minimum number of lanes implied by the VECTYPE. */ + vect_partial_vector_style partial_vector_style; + /* Flag to indicate whether we still have the option of vectorizing this node + using partial vectors (i.e. using lengths or masks to prevent use of + inactive scalar lanes). */ + bool can_use_partial_vectors; + /* Number of partial vectors, for costing purposes. Should be 0 unless a + partial vector style has been set. */ + int num_partial_vectors; int vertex; /* The kind of operation as determined by analysis and optional @@ -476,6 +492,9 @@ public: #define SLP_TREE_GS_BASE(S) (S)->gs_base #define SLP_TREE_REDUC_IDX(S) (S)->cycle_info.reduc_idx #define SLP_TREE_PERMUTE_P(S) ((S)->code == VEC_PERM_EXPR) +#define SLP_TREE_PARTIAL_VECTORS_STYLE(S) (S)->partial_vector_style +#define SLP_TREE_CAN_USE_PARTIAL_VECTORS_P(S) (S)->can_use_partial_vectors +#define SLP_TREE_NUM_PARTIAL_VECTORS(S) (S)->num_partial_vectors inline vect_memory_access_type SLP_TREE_MEMORY_ACCESS_TYPE (slp_tree node) @@ -486,13 +505,6 @@ SLP_TREE_MEMORY_ACCESS_TYPE (slp_tree node) return VMAT_UNINITIALIZED; } -enum vect_partial_vector_style { - vect_partial_vectors_none, - vect_partial_vectors_while_ult, - vect_partial_vectors_avx512, - vect_partial_vectors_len -}; - /* Key for map that records association between scalar conditions and corresponding loop mask, and is populated by vect_record_loop_mask. */ @@ -2607,6 +2619,7 @@ extern tree vect_gen_perm_mask_checked (tree, const vec_perm_indices &); extern void optimize_mask_stores (class loop*); extern tree vect_gen_while (gimple_seq *, tree, tree, tree, const char * = nullptr); +extern void vect_gen_while_ssa_name (gimple_seq *, tree, tree, tree, tree); extern tree vect_gen_while_not (gimple_seq *, tree, tree, tree); extern opt_result vect_get_vector_types_for_stmt (vec_info *, stmt_vec_info, tree *, @@ -2788,7 +2801,14 @@ extern slp_tree vect_create_new_slp_node (unsigned, tree_code); extern void vect_free_slp_tree (slp_tree); extern bool compatible_calls_p (gcall *, gcall *, bool); extern int vect_slp_child_index_for_operand (const stmt_vec_info, int op); - +extern void vect_slp_record_bb_mask (slp_tree slp_node, unsigned int nvectors, + tree vectype, tree scalar_mask); +extern tree vect_slp_get_bb_mask (slp_tree, gimple_stmt_iterator *, + unsigned int, tree, unsigned int); +extern void vect_slp_record_bb_len (slp_tree slp_node, unsigned int nvectors, + tree vectype, unsigned int factor); +extern tree vect_slp_get_bb_len (slp_tree, unsigned int, tree, unsigned int, + unsigned int, bool); extern tree prepare_vec_mask (vec_info *, tree, tree, tree, gimple_stmt_iterator *); extern tree vect_get_mask_load_else (int, tree); @@ -2953,7 +2973,7 @@ vect_cannot_use_partial_vectors (vec_info *vinfo, slp_tree slp_node) if (loop_vinfo) LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false; else - (void) slp_node; /* FORNOW */ + SLP_TREE_CAN_USE_PARTIAL_VECTORS_P (slp_node) = false; } /* Return true if VINFO is vectorizer state for loop vectorization, we've @@ -2967,10 +2987,8 @@ vect_fully_with_length_p (vec_info *vinfo, slp_tree slp_node) if (loop_vec_info loop_vinfo = dyn_cast (vinfo)) return LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo); else - { - (void) slp_node; /* FORNOW */ - return false; - } + return SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) + == vect_partial_vectors_len; } /* Return true if VINFO is vectorizer state for loop vectorization, we've @@ -2984,10 +3002,8 @@ vect_fully_masked_p (vec_info *vinfo, slp_tree slp_node) if (loop_vec_info loop_vinfo = dyn_cast (vinfo)) return LOOP_VINFO_FULLY_MASKED_P (loop_vinfo); else - { - (void) slp_node; /* FORNOW */ - return false; - } + return SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) + == vect_partial_vectors_while_ult; } /* If STMT_INFO describes a reduction, return the vect_reduction_type From patchwork Wed Jun 3 15:19:15 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136404 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0AFEA4BA5439 for ; Wed, 3 Jun 2026 15:23:04 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 0AFEA4BA5439 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=BUL87YN0; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=BUL87YN0 X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from DB3PR0202CU003.outbound.protection.outlook.com (mail-northeuropeazon11010043.outbound.protection.outlook.com [52.101.84.43]) by sourceware.org (Postfix) with ESMTPS id 9B9D64BA2E19 for ; Wed, 3 Jun 2026 15:20:50 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 9B9D64BA2E19 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 9B9D64BA2E19 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.84.43 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500050; cv=pass; b=T5lCzsIIwIf5YQsp4GkGynuGpngKBfg8naoEfQYmsh2/R/Wh+2oKWhI6VSWMDN/1OC9hOxPjtethKSAzW42pcxXcOF44M2DcLovPvvfBMQVRMnJpFh96Fl0C2rJzs5IwFHFZNyx3YIQ+0tOeZjT9HDTMcTPXBxxGnVHKQfcAauQ= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500050; c=relaxed/simple; bh=TDlJuctGIBxghcyvVGrI1eU2u4KZ+4gzotJXpmAg4+w=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=fOWdno+VHpPxa5BrQKYFCwFrAxtW5fj1EcDtV1JbCh0ODxItSRpFR45TgccTafKMd2cywaYCuSh7skrx8TuziORBk/AkLDgSsBgKCHtefW0dqXv3wBHHak+HwXRmaYTiYIadlWzHJuEJiOKMgVUl2Wr/LUcw5djYLKp+0/7L8ks= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=BUL87YN0; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=BUL87YN0 DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9B9D64BA2E19 ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=FRkdPbbw7905a4yVsXy52IQzqN2GczWT+4rLiCymHlRrxwz9G8vJRcyEZYdyhHKS46KEKzSfkBbpMQrS+8gNFOwXkqZprLqDCbnAwQSD3/WR9Nh79ejp0YhqziY0hut6E2Ej71gz42TMgB64qft0cw89rBOSY39T/Vys5VxiGcqNHZUpsJkf1pjY6+Q09gHiv6DnIAnnmWtzGsyWFUsVDq0u/hpUW7Ba0RYirRwyY1pGc+Y5bi7J98KO7buqfz4zkxGzwpBmyydy9UvUKnwDcXBs8pOUGmlIHhjabclTqN8aEWmF6xk98A1Cf9sMQfV+7rh4gz5sHqa7HB5JQg2PTg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HFFQlPEUJVE+QJ1pjyS8FVwtoekBjQzXU9rNNQdxpRA=; b=B1wiXSJRSyXCg6FgpO3cksjeKyC3zOLXetUCGBUV9+L8hHRuk+XyeDvuWG0ZrBqNoNGf5nbs3Glk8pbrNbfgOr+NB1BjakefJgAf74JwdaAhzruu5lqucPoybDnTfmhsqj/hw/SGS2sv8WWcJe9S4nVcBS/IiR6sY8SEOg3k/m+M/rt6vPpIvB815I29picWk4ejyL6axawDQ+U3ElT1ytSA4dHXvEEJJ3yMP9+fOFSHYcKSEFFQK8xJPUegiR3oovKqaW0YHaUkG2y0AQLIngk+Yg+4aQYnEZSg/zURwQdBWfyKkvjprVsrFVDBGt++bwKBDrt1VsAyEPxATb3nFw== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HFFQlPEUJVE+QJ1pjyS8FVwtoekBjQzXU9rNNQdxpRA=; b=BUL87YN0dFPcUH0tkQ0D4rchICZi7mn+XEm5UMixGjDNnW3Ro0tOUVqln2qHZ1NE5tG6V1DiD9zQISOcVgKszUhr6hYZZW4ooyJWF1bUP2v1zGtB+sausxiteJ6mf/zdsnzicgpIobSS+RBdcbqJKwqLf01047IpqkMQ3Q166FA= Received: from AS4P189CA0036.EURP189.PROD.OUTLOOK.COM (2603:10a6:20b:5dd::7) by AS2PR08MB10177.eurprd08.prod.outlook.com (2603:10a6:20b:64f::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:20:46 +0000 Received: from AM2PEPF0001C711.eurprd05.prod.outlook.com (2603:10a6:20b:5dd:cafe::3) by AS4P189CA0036.outlook.office365.com (2603:10a6:20b:5dd::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:20:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by AM2PEPF0001C711.mail.protection.outlook.com (10.167.16.181) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:20:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DF/hMr68u5yQ01GlMun4N7Q2gwWtiEAzrWO/NdopK5ieb78Ke8W8wmzTjcUTPEkVfG9haK2rfaUmphj21gZmGwd9KOaoyKEqdncj1z66QvRo84vJ/3Dp68uGOxmWY9jG5gpPXwmiR9CspjXeWzkTqkBJ7XUdKUbgtpwylpiwMtgFlE34+HmCzXICj/VeO9rIkJqYJQjchPH50HxhMN2JC1EsJEZcTCTSvqDn0/fedknEEYmwSAOIplEfF0GCAMqsjsSuR4upTnfwIFf71+H81OOmzGqMwBzfrV7EdYEChiVGEMuh3nwgSs5Xvbvod52LMHqctKCfyNo6dJXhVOlexg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HFFQlPEUJVE+QJ1pjyS8FVwtoekBjQzXU9rNNQdxpRA=; b=yPdAOLg3XG7a419IiJqnpFxUgTNcKlkJ+nkvFp8GnHxcavV3QTZfYnw5cThPMxBPmRnjrsBjOBRTU7Z616AcCppjop0ztbpboav+2QMBlRLPNxsEbqElMmnXyTO4VZayZUG5FE/bvWKuCYpw8MrpACgpyUSFrTCQh3m2HFlkNS9V4XyYMTDTp0QbqGRKJ6trX6oNHsTUjvP8vDxj7iq+nJlr+Bta6oWuEmiri3USaDUVJd0dq+bPAPR+xYNUVj3Dr+QLn059BAoSEzB/klMRg0jyGtgRKARoZ6PTOLDSNSXI0wu+iTxAF4zsyXq3FxLZOiZhdrtObLFZ2nUgPjeG5Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HFFQlPEUJVE+QJ1pjyS8FVwtoekBjQzXU9rNNQdxpRA=; b=BUL87YN0dFPcUH0tkQ0D4rchICZi7mn+XEm5UMixGjDNnW3Ro0tOUVqln2qHZ1NE5tG6V1DiD9zQISOcVgKszUhr6hYZZW4ooyJWF1bUP2v1zGtB+sausxiteJ6mf/zdsnzicgpIobSS+RBdcbqJKwqLf01047IpqkMQ3Q166FA= Received: from DU2PR04CA0157.eurprd04.prod.outlook.com (2603:10a6:10:2b0::12) by GV2PR08MB11277.eurprd08.prod.outlook.com (2603:10a6:150:2f2::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:19:38 +0000 Received: from DU6PEPF00009528.eurprd02.prod.outlook.com (2603:10a6:10:2b0:cafe::75) by DU2PR04CA0157.outlook.office365.com (2603:10a6:10:2b0::12) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009528.mail.protection.outlook.com (10.167.8.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:38 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:34 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:34 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 03/12] Update constant creation for BB SLP with predicated tails Date: Wed, 3 Jun 2026 15:19:15 +0000 Message-ID: <20260603151924.53706-4-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009528:EE_|GV2PR08MB11277:EE_|AM2PEPF0001C711:EE_|AS2PR08MB10177:EE_ X-MS-Office365-Filtering-Correlation-Id: 69864971-0be6-425e-f9c3-08dec183af92 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|1800799024|36860700016|82310400026|376014|6133799003|3023799007|22082099003|18002099003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info-Original: efAs63MEkESdk9v/vhCW3LjDA6wiqwf9rAI05vQyZnKmxlpkcE6kSQthlG2ac2wgdxaP7pvFk9N4Hz0XjVVRp+jmsRSUfrAgA7YnATpYCW7WofPy8TlP2hFYvYt7wZADqhgzwsjXl/2cuF7KoPwS3irNTj8I+J/i8FFvGyfcambPRzq04sJOlTyxWOYISDgWYVE07+AkGCsg1Heg42mXazhPAsIjvXK7CaowRNLAULwXAiS5lMnE9NnRb6Y4rZ1VLKlJEmwTzzu/bMWW8ohoS5lBhdd+KFA3MLxyixfOYeqBUV6EXnpWOIlao7qDcTz1n4zPYh9CXR4mnCaiYjhCmd7NEKwHWZFBPSgzPt8n5l0kS5RXVuuzL9JM1HrJXlY7+CtSaP5aZhm2E0k5msVjtg7OazE9yNAwo38WvpgObuiHe0v0w6zMUagqWqkylfDnjyhx86SuHYETTDt6P0WvfMvnx/2cxpVV+nSTNdluTQik2ZyfnqSUm3EPZufpfRdv5IRyBIGU/gCjDWleLBIM3QXufEGfPC1QrOxb0787aRMYiQtV1dgBW6vLQ9xJCvfozDitXykYhEK8z4GNS1xxhmeisROTGt5w3PTq1QgBidIlvEFodPd1+2LPhy8iufHCAiu4nf6o8J3+51xpn1bSAuX0k8nCYaX6az3K03kLrpnIZ/BJd2o9SuBuR9XdcZXc7g7Tg9u9agf4in1nn1a1fefvNvG7+hmOvjLtqAkhxFQ= X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(36860700016)(82310400026)(376014)(6133799003)(3023799007)(22082099003)(18002099003)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: UM1inwU5q6xkifqLfbH0+M98lhxES/1h8cIdhPuC9HTu1g8AW5Ens3rjo3PssdwE2yWLXB9ucpIF8R70fUptzfdfA8ONRlAcZ0VxA/P7lvEE8AhDOadbNohB1BeNV2NPWJuuuL6oNL184YNp1aFit56IlOQbm9rUMhBHXo6HOltukuhqd8EU6H8vR+IJUIiysUGtbSWVRLy73sXzzwiz/V5Q+a/Kf8mp7r75bUXfbn7VZQiY26DvkaYwj4W9G/zZw/B/ost4UnjjrpUpq6cc809AjfdAniqdXULA4DQ1sJKe6ex6uruf6Tm8095WCM0I0U0Gs116KIaceR62KyDUzA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV2PR08MB11277 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM2PEPF0001C711.eurprd05.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 350b329b-0987-4c84-fe05-08dec18386ed X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|82310400026|376014|14060799003|35042699022|36860700016|22082099003|6133799003|18002099003|3023799007|11063799006|56012099006; X-Microsoft-Antispam-Message-Info: SFYqkizpuNugI3a7voX/jmqHCkfoM4wVN0mWTrKxuoN3m0PPoK6FduQWxOl32cOI4V31KygC5HtNAWcVYnzW6Diq2j2VUaRmrp/+eadCvXflvx9t5kDGV/SuknAcTa2eAhzD8LWpiuPVEF11oG9+lKSJQS9lQpU34pvC9b6BUfx0aGWy8n/io+MtKE+1oqJGAxyAyK99H1e34YzgJJ9LOLtzkhyXL8OOJW4WeRRBXKQvMqtur+0zpHH56/XEj32RG46CHjCfhg3cFyRzjdYZ3attB+PI60/nQAiNHnv8jEZVcE2+VujPqcFKWqKz+FHeur/ftHiV/gaIUxUOEh1xjAezBCX/3M9/t8wQ72ofnABHQO3SLPoBheSylpI9zSX8HPRS02VHIRri0ep69R80zB1cCdtFfUm+LJH3bTqyqkSUGWJJOlB3Dn+tsdiYN6+8dxgTrCBiV3TjP/1jW41SQZ9AmJHQ8nS33OfTtFMCXi9IYDWVN0PxGJ5JvSVbUyG2SRmxX8ZeMV3E0c1/5POIURGg7R1PtPefjlG3mzf2PVqYiPMaNj+6ZaR8zgGMr2ZGgJOnimz3wu3iP3JNhMdBq9GLfJMk5D/WrSsao2J09ATMtIQgi8fM4kqLYg58Ax5+TtqIItADJf9Ulz2jbG12i+0SchaeRibuMomEOX5OHEU9zq1Gob5qzCTwLYyXgmiJpyCcbOHt5cmnqUKoGtLHE2HgzbpF9JR9xnLZdLXgfC4= X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(82310400026)(376014)(14060799003)(35042699022)(36860700016)(22082099003)(6133799003)(18002099003)(3023799007)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: KhJTB30Qh03NwOn3Mxt04MbJr0yjB0glI14W1P5Wr6mSFOp5+q89rA2a6BhCNHAaIquMvTtB8a84WcyXa0RbZCBd1cMP3uxCrsed09OvxNiCoE29oYNpejgrAwjtqiZYnNehx4/JH7ElzIUGPOUKcOe7afJyhWrThKDj9IwzVJMj12xpc0QCtquJZgOYmjimd9Ms5gDJx9jBufb50psMYbMZYqVWn8HpMxZP7FKnVm+L5SHtR/EdgbF7jcbtIsOKxd0Zw1IXiaTV35kUvyTtnFmy8rcB3AdqMr8VZIYRl4GrWXD4p33Silw7iJpRDnm0nTxVSzlPGmqfRrBNRtJY8HyZ8AaSSt3r4w25ZjUiOFCNXT0F/LMgJntmSdZ363DvfZXrv65GV7jXcORZpCdbkhH27GKF/XI7dfUjkXH/glm31KJI5tAhVoNM9JKOHQvZ X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:20:46.2717 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 69864971-0be6-425e-f9c3-08dec183af92 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: AM2PEPF0001C711.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR08MB10177 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, RCVD_IN_DNSWL_BLOCKED, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Created a new function, gimple_build_vector_from_elems, for use when creating vectorized definitions for basic block vectorization in vect_create_constant_vectors. The existing gimple_build_vector function cannot be used for SVE vector types because it relies on the type associated with the tree_vector_builder having a constant number of subparts. Even if that limitation were lifted, the possibility of tree_vector_builder patterns being used is inappropriate. The new function takes a vector type and vec of tree nodes giving the element values to put into the built vector, instead of an instance of tree_vector_builder. If the number of values is zero then a zero constant is built. If all values are constant then a vector constant is built. Otherwise, a new constructor node is created. gcc/ChangeLog: * gimple-fold.cc (gimple_build_vector_from_elems): Define a new function to build a vector from a list of elements that need not be complete. * gimple-fold.h (gimple_build_vector_from_elems): Declare a new function and a simpler overloaded version with fewer parameters. * tree-vect-slp.cc (vect_create_constant_vectors): Use gimple_build_vector_from_elems instead of duplicate_and_interleave to create non-uniform constant vectors for BB SLP vectorization. --- gcc/gimple-fold.cc | 55 ++++++++++++++++++++++++++++++++++++++++++++ gcc/gimple-fold.h | 14 +++++++++++ gcc/tree-vect-slp.cc | 40 +++++++++++++++++++++++++------- 3 files changed, 101 insertions(+), 8 deletions(-) diff --git a/gcc/gimple-fold.cc b/gcc/gimple-fold.cc index 1ceb5aa5fba..3462c5acb6e 100644 --- a/gcc/gimple-fold.cc +++ b/gcc/gimple-fold.cc @@ -11425,6 +11425,61 @@ gimple_build_vector (gimple_stmt_iterator *gsi, return builder->build (); } +/* Build a vector of type VECTYPE from a partial list of ELTS, handling the case + in which some elements are non-constant. The list of values may be shorter + than the minimum number of subparts implied by VECTYPE. (When the vector + type is variable-length, the actual number of subparts may not be known.) + Omitted elements are implicitly zero. + + Return a gimple value for the result, inserting any new instructions + to GSI honoring BEFORE and UPDATE. */ + +tree +gimple_build_vector_from_elems (gimple_stmt_iterator *gsi, bool before, + gsi_iterator_update update, location_t loc, + tree vectype, const vec &elts) +{ + unsigned int encoded_nelts = elts.length (); + gimple_seq seq = NULL; + gcc_assert (TREE_CODE (vectype) == VECTOR_TYPE); + unsigned int lower_bound + = constant_lower_bound (TYPE_VECTOR_SUBPARTS (vectype)); + gcc_assert (encoded_nelts <= lower_bound); + + if (encoded_nelts == 0) + return build_zero_cst (vectype); + + /* Prepare a vector of constructor elements and find out whether all + of the element values are constant. */ + vec *v; + vec_alloc (v, encoded_nelts); + bool is_constant = true; + + for (unsigned int i = 0; i < encoded_nelts; ++i) + { + if (!CONSTANT_CLASS_P (elts[i])) + is_constant = false; + + CONSTRUCTOR_APPEND_ELT (v, NULL_TREE, elts[i]); + } + + /* If all element values are constant then we can return a new VECTOR_CST + node. Any elements for which no value is supplied will be zero. */ + if (is_constant) + return build_vector_from_ctor (vectype, v); + + tree res; + if (gimple_in_ssa_p (cfun)) + res = make_ssa_name (vectype); + else + res = create_tmp_reg (vectype); + gimple *stmt = gimple_build_assign (res, build_constructor (vectype, v)); + gimple_set_location (stmt, loc); + gimple_seq_add_stmt_without_update (&seq, stmt); + gimple_build_insert_seq (gsi, before, update, seq); + return res; +} + /* Emit gimple statements into &stmts that take a value given in OLD_SIZE and generate a value guaranteed to be rounded upwards to ALIGN. diff --git a/gcc/gimple-fold.h b/gcc/gimple-fold.h index f1853560779..8b324be005a 100644 --- a/gcc/gimple-fold.h +++ b/gcc/gimple-fold.h @@ -243,6 +243,20 @@ gimple_build_vector (gimple_seq *seq, tree_vector_builder *builder) UNKNOWN_LOCATION, builder); } +extern tree gimple_build_vector_from_elems (gimple_stmt_iterator *, bool, + enum gsi_iterator_update, + location_t, tree vectype, + const vec &); + +inline tree +gimple_build_vector_from_elems (gimple_seq *seq, tree vectype, + const vec &elts) +{ + gimple_stmt_iterator gsi = gsi_last (*seq); + return gimple_build_vector_from_elems (&gsi, false, GSI_CONTINUE_LINKING, + UNKNOWN_LOCATION, vectype, elts); +} + extern tree gimple_build_round_up (gimple_stmt_iterator *, bool, enum gsi_iterator_update, location_t, tree, tree, diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index 4dd7e6e1e21..f91d3e723ec 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -10727,7 +10727,7 @@ vect_create_constant_vectors (vec_info *vinfo, slp_tree op_node) unsigned j, number_of_places_left_in_vector; tree vector_type; tree vop; - int group_size = op_node->ops.length (); + unsigned int group_size = op_node->ops.length (); unsigned int vec_num, i; unsigned number_of_copies = 1; bool constant_p; @@ -10757,10 +10757,23 @@ vect_create_constant_vectors (vec_info *vinfo, slp_tree op_node) (s1, s2, ..., s8). We will create two vectors {s1, s2, s3, s4} and {s5, s6, s7, s8}. */ - /* When using duplicate_and_interleave, we just need one element for - each scalar statement. */ - if (!TYPE_VECTOR_SUBPARTS (vector_type).is_constant (&nunits)) - nunits = group_size; + if (is_a (vinfo)) + { + /* We don't use duplicate_and_interleave for basic block vectorization. + We know that either the group size is exactly divisible by the vector + length or it fits within a single vector. */ + nunits = constant_lower_bound (TYPE_VECTOR_SUBPARTS (vector_type)); + gcc_checking_assert (multiple_p (group_size, nunits) + || known_le (group_size, nunits)); + nunits = MIN (nunits, group_size); + } + else + { + /* When using duplicate_and_interleave, we just need one element for + each scalar statement. */ + if (!TYPE_VECTOR_SUBPARTS (vector_type).is_constant (&nunits)) + nunits = group_size; + } number_of_copies = nunits * number_of_vectors / group_size; @@ -10860,6 +10873,11 @@ vect_create_constant_vectors (vec_info *vinfo, slp_tree op_node) ? multiple_p (type_nunits, nunits) : known_eq (type_nunits, nunits)) vec_cst = gimple_build_vector (&ctor_seq, &elts); + else if (is_a (vinfo)) + { + vec_cst = gimple_build_vector_from_elems (&ctor_seq, + elts.type (), elts); + } else { if (permute_results.is_empty ()) @@ -10925,9 +10943,15 @@ vect_create_constant_vectors (vec_info *vinfo, slp_tree op_node) NUMBER_OF_SCALARS/NUNITS or NUNITS/NUMBER_OF_SCALARS, and hence we have to replicate the vectors. */ while (number_of_vectors > SLP_TREE_VEC_DEFS (op_node).length ()) - for (i = 0; SLP_TREE_VEC_DEFS (op_node).iterate (i, &vop) && i < vec_num; - i++) - SLP_TREE_VEC_DEFS (op_node).quick_push (vop); + { + /* Guard against the outer loop never terminating because the + inner loop is never entered. */ + gcc_checking_assert (vec_num > 0); + + for (i = 0; SLP_TREE_VEC_DEFS (op_node).iterate (i, &vop) && i < vec_num; + i++) + SLP_TREE_VEC_DEFS (op_node).quick_push (vop); + } } /* Get the scalar definition of the Nth lane from SLP_NODE or NULL_TREE From patchwork Wed Jun 3 15:19:16 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136406 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 105EC4BA2E1B for ; Wed, 3 Jun 2026 15:23:38 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 105EC4BA2E1B Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=XrEQTWug; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=XrEQTWug X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from PA4PR04CU001.outbound.protection.outlook.com (mail-francecentralazlp170130007.outbound.protection.outlook.com [IPv6:2a01:111:f403:c20a::7]) by sourceware.org (Postfix) with ESMTPS id 264014BA2E2B for ; Wed, 3 Jun 2026 15:20:49 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 264014BA2E2B Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 264014BA2E2B Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c20a::7 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500049; cv=pass; b=pg/sHTaGgtQp3bawtV3wKYgiia0fUQw+9NVvzK3pBKgGdj4x06O+ntv01aNENBiWC/R+v4sH07sm+QdOjpXF/5h6fqLJSaWdsdhJUbNXrr8AbQggNn12WNzA6nZsboSFlMP+MzVhjhSPWKkK/s7ZUiSv2rr3zs/fEGKoca4jYvk= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500049; c=relaxed/simple; bh=gmuP0PgBSZYe8slkCGRCZDLgS01bLmQud48pFwsxqpQ=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=j3OHYjUB0BY/QyOkS+XWW1xeDYDCtOqc8fBoq1ESTZNe8ABl01JG10v7qahyIlzVlhC8NrTl+lt7yRng1ACc1D9c3ui5vxMm1IbL1ceqHXv+zlff4xwNfjBw03FtZtZxICAHYtTjJvs0YqlLDNd2eL0Q/M+4CIYjrrWMMS7TrpM= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=XrEQTWug; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=XrEQTWug DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 264014BA2E2B ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=XgvyN9pcRU+Dfs9t8js1DZIVWJkkcszRNs+waKpbqRyOje/uTWHg+juwMgxaPxtLqPAZQ5IGaBHKCzC5/OW9jH6G+BVxHzuTplQR5D3ykiTpC86kh8R7C3Oj8KfrUL9oCVpewlaeR6RHPIAqccOxRiZGl4mEKlyev9pTpTY6f8CXPS4EubwQ6zNq1Cc45hORv4Gp5Rkn2XmP23zNYfZSX3crPH+AqfqD1+NzWeszdT0Vv2H/ydqu9EbC4+4qefF/H1rltZAgBsoDkfQLReVs+dOFueDT0KMAtldCeNN02iaVVs0wL21k2ZrQh3a7ekqFg/UhGWwaEaCCn1fCB0lUbw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+WhTVbCcVBMM/0mPxDB0FazWrxqo9yjFeOIe++WfxD8=; b=gtSidJBN37srQtKHMXSBNFuq7liRgtySqpKihKtE+jejjjb/MQb/Ry4shfQ6qxcpjfYntpHJ/CmeC+stUIl0yW10xPy8zJJyHsEfxYv5YRQ2Rna4WGZ5Mro2kLAymtfnsr/eeRdrbeSEtnHw/B0CGWoYADy2eU1UNuICk6EwYVPFv26vLLaZevqbj5UXolX3avEhHhX6bCXNvudcZCT4gNyj8y44XniFhkcclArnf66GxP0AsIDORdIO2CENIOP8eybwhGGKArPlFPez63wF5538kM6Na+TheOAMbQnoPhYuCXzYiRw8BgKO45KZwBHKQEjuhS1rIhCHJ7+hd0d1CA== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+WhTVbCcVBMM/0mPxDB0FazWrxqo9yjFeOIe++WfxD8=; b=XrEQTWug2TX2lNLZuq/LwWn3DCiiX5mdAt4n/6MuXvyGgJteDxivbqY82Q9kyCrqKnZs3nllNK1KGAXhagOJqlX3nuEtkdNaSxniw/RKizX+vQOVW0xjAgcuYl+OZXkokasO4u6uuij0+XWerc3azz2wBcPKU8AMTd2lBW3SCnI= Received: from AM0P309CA0030.EURP309.PROD.OUTLOOK.COM (2603:10a6:20b:28f::32) by DBAPR08MB5670.eurprd08.prod.outlook.com (2603:10a6:10:1a6::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:20:45 +0000 Received: from AM3PEPF0000A797.eurprd04.prod.outlook.com (2603:10a6:20b:28f:cafe::4a) by AM0P309CA0030.outlook.office365.com (2603:10a6:20b:28f::32) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:20:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by AM3PEPF0000A797.mail.protection.outlook.com (10.167.16.102) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:20:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=wxrM5acp+NMTVljzr/hQgfQmczYzQz8eqnU0COZ2IBGbtsi2iGm0xKJf9ObfyVQIXqCEL/2DSdRhIFMGw4BxGvtZJnAhEDjutBMwgfd55yTiaVvJ36vcgD/dfYEUwyRDs3uWQIE9yAYYjzqKFqug5BJaK0LR8LENwPw9rXm6ydusqgcM5f1tARyUdBgGFPMw9qKcIGwlrEIqSK3gJrMAsTfpRwZiRZdZpAskbpkAvgtnbpraCGTfYt+VLb5/vUFfvlGBlDWooMPZ58PC9iAmd1dVuxEJkPwowXWPNx9ls2KiXjAZoYqQZej1HagqhTz2Q5pKjfSCo9ead7EhaZyTkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+WhTVbCcVBMM/0mPxDB0FazWrxqo9yjFeOIe++WfxD8=; b=q63wjkpdoQZdiOe2B9IGn5/xXHVuGAPcHPH9MKV/WGuzcAUk9loBcO8kCCPpjP5nF6+36iENFfJiNxBfnbMkm3476SGDJfkdJ2ja7y70pOrdiHQkGbBLUMKs9PmFwOQLvyBBAhdvAZt1oSQ9EdTzTJ9MmUQTcO7uSRJ2dRlEXb/zPiQtubilUMESsdZ0F3vKqzctR3TIQG/zERi9/s2LcTup97nMxHv5yfVIwqnOA6Yj+FsPxxjrKQ30sJqbiVm5ZnwQA3AwmKOgkCVE3VEL96nL1of+opu7d6QPMH+mX0gBtUSmfg8xK33e7kPDUXh8NH6FSfUalpziLxNIm2yBJQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+WhTVbCcVBMM/0mPxDB0FazWrxqo9yjFeOIe++WfxD8=; b=XrEQTWug2TX2lNLZuq/LwWn3DCiiX5mdAt4n/6MuXvyGgJteDxivbqY82Q9kyCrqKnZs3nllNK1KGAXhagOJqlX3nuEtkdNaSxniw/RKizX+vQOVW0xjAgcuYl+OZXkokasO4u6uuij0+XWerc3azz2wBcPKU8AMTd2lBW3SCnI= Received: from DU2PR04CA0155.eurprd04.prod.outlook.com (2603:10a6:10:2b0::10) by VE1PR08MB5791.eurprd08.prod.outlook.com (2603:10a6:800:1b1::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:19:38 +0000 Received: from DU6PEPF00009528.eurprd02.prod.outlook.com (2603:10a6:10:2b0:cafe::64) by DU2PR04CA0155.outlook.office365.com (2603:10a6:10:2b0::10) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009528.mail.protection.outlook.com (10.167.8.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:38 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:35 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:34 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 04/12] Refactor check_load_store_for_partial_vectors Date: Wed, 3 Jun 2026 15:19:16 +0000 Message-ID: <20260603151924.53706-5-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009528:EE_|VE1PR08MB5791:EE_|AM3PEPF0000A797:EE_|DBAPR08MB5670:EE_ X-MS-Office365-Filtering-Correlation-Id: 9b27de70-99ef-4c34-ab3b-08dec183aec6 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|82310400026|1800799024|376014|36860700016|22082099003|18002099003|6133799003|56012099006|11063799006; X-Microsoft-Antispam-Message-Info-Original: orcCdXfAvirWaI+NEItiOnO2sG5maCELzpQS/oAbZ9Uy0shRK//T3zKP6XvY3E5sKQNcgTlkcETElwJerVfqv1zOScqiKzsV+zC+m5KwHgiFsM/g6mMom5//Ohga8E6BTr5lmgEL45bdRAB4B+rsmVkWH72ETHu1Nd3fu7sWWuTp4XH4HnA0aKK9I5IJlA77tsOWnHUyX8FwEhbMJQIvAh6mbKgZZVuroIBeU8RrKKKGqg4REaB8mcEo8DizeaYhwqoHUuoguA/pkBEBsBTsSVBmhHs2aRfjU39409hPJZ2uofywLfGy/Mmi9vV+7DH3q580R6nzPo0WkBoz8MxRLrDaLXcvyrsZaXbYfvEzMxh/aPL1ldXdwF7RnFBeu9Vdfx9tqp8bP0+v6h71Z8jSCXz6GsiIL2ocurFnqUDBBcf74Q7COPooNdYVcNjbmYN6BmT71hwNkO41YNZt5GzB1UzTPRoOSHlSlxgjILW+sfnIvrgCK/Vk040Rrrg2pifiiOnJ7CyNWqUKA/yi6gnA2xCA0JdfEu2p6hKvZj1BxPzYjlxwlL4XjZDVCH/EUOq/7GSS4EU5GLPHtE2OyaYoX7zrAw4U/ha4bC+iQMRh3cE2sXWx00Sr5BzRj7wDStUfy+RDdc7CQf9aJUZiaTBvH4IoopXZ/kqpPkbGaeGYeRHMbXlZrk6z/QP5lcXKR4IWbwHsw9mjxQOhNcTx+mTrWPJz1d4GwytoDDh/8eiXDkM= X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(82310400026)(1800799024)(376014)(36860700016)(22082099003)(18002099003)(6133799003)(56012099006)(11063799006); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: dYM2AJ7zRMBNqjE49eCqjfConGD8XLTadksdrVjonrnEhua7G+ek5tGLUmCJgnvlNLX8R/6t3bqCLRXZ4HU5wrKxb2DF9j+rk27Sw6uOi3vwm1QME2BSii7WULf107exRigigMJxm7/4BkhITIMJg2TkWGnqXGNcgWQWb3bKERFaW/VTwgo63S1hpX8ADa08RXPCqmcL0Tg1Itiu1QkjNzoq9DGfExtzLBS/nWSMzPAT8xCiLhPTIfwOQ5fDOtrJv4cMrTMdu6P9ZzOH/CCgm0D/5EB6Rz1EhcaJmV11H48Kw6ZbtfQkQZKjSMs5dzLI2ziOfCLjw36y0ZsLuMTOyA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR08MB5791 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM3PEPF0000A797.eurprd04.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: d2030b01-319b-474a-6936-08dec1838725 X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700016|376014|35042699022|14060799003|1800799024|82310400026|56012099006|11063799006|6133799003|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: ElCx2+aBPELyYXni4PLB/69ktRPGE5gY6djvEjvybAvIHduhs6fjwOT9U4EOaDh5lTbna6EZ2RMCG7wgkcBQ2m3h/Xwy/1N8aELJtp4qV8SjgoBw/rxfs+e1lkXCUNyWMGgp8OkTZgsUjq7Sq6yStn2XLs7Pcrh+FMmnelClrCwus3sDe6IDfVjLqYtpM5DTer5Q15zcFLMcSlqQ4LgZrnrZHL1vHjNl+ZAv3+/bmpR6MYLN1R2rMLV8kEXtAKcUL2KKYawVTEALfDwIqJK9F8UOle8yacP31nYVT+q8M0C+BkDqnFgICYVQ+8BL4WWw3fTrTlMstcWKCCzImZRnMELOYZKZGlDwioiBRsIEeG63N2S9r1Rbk6iVwIOmhnrFxZ42o01MuVU7ZdB9cFxqXL/4KCmwdv7CpyGUss9gU8vcRBl1HPW44aPl9ApQ+Y4FLN+/gHe+Omq7i6cnP9XyW+BJDZgJwgz8p4/BDVdEC9lqMwFDSxYHiXNECmh5NCArjnoCWOqnZjfwtMH1osKMo4J/miy5xaLqVMxNqbmrnLWenrOgcW56xyBKvXgCQKpWHBPLKZcopwr8l4pZ71tUlenZ0kYkmRws7hnBiELukTLXvCQdARckjhg6uIBi2D8XdKbF+nwpLqfW6KYxz3Wq6WZktiVgIgtiYkvaWjebv3UjCd6WVhzM7a6S1Oe/OG5GkSAyJOJHLGc3p3TtYrBTNhHyO7DluVPF/FJjmZS2Kjw= X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(36860700016)(376014)(35042699022)(14060799003)(1800799024)(82310400026)(56012099006)(11063799006)(6133799003)(18002099003)(22082099003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /hD4iDw/SvpDnhovUqjDU862+VJLXRiU9J7ssxfmmrgFuw3/WNZY9LT9hworaRidYZdrftytMY520BK+f0fouITqRdRLBkD+XmRH6OoyTlUsgDb0JYY4SgpPjNBhM2lvmTT5bzO5W/zPjGK9prI9QS8EQia0ogmduogMOEvAHDUQMoW8tDeJYiXIQwpAc5T158JupdqcuZ7GNtMzQIUu8SR3Rl9R1dprJS8wHoKkUH+GYST8Ol6kit23ODnxwsYKbK0QWUeKQutc2YrlLq0LABqinclzUa+LYG0rGfPWfcv5Lw6xC7vnwxnZmkJSecJCCn8imeUyQEu8TLJ8SX91RTpu+MPuDhcMcyGyKF7ax4eDnNEnXrTgi6ZzRlxadz4OmgsO0qj3MKkhJpsCFs8IfJEVPHKlrDitWnRnSjbwpW74z0DjTdvLoHPZtcPS/eJR X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:20:44.9385 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9b27de70-99ef-4c34-ab3b-08dec183aec6 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: AM3PEPF0000A797.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBAPR08MB5670 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, RCVD_IN_DNSWL_BLOCKED, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Moved existing code to determine the partial vector style for a load or store into a new function that will be reused for BB SLP with predicated tails. gcc/ChangeLog: * tree-vect-stmts.cc (vect_get_partial_vector_style): Define a new function to get the partial vectors style supported by the target. (check_load_store_for_partial_vectors): Use the new function in place of existing custom code to do the same thing. --- gcc/tree-vect-stmts.cc | 71 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 58 insertions(+), 13 deletions(-) diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index ecad74e7cbf..1cfa2177add 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -1461,6 +1461,56 @@ vect_get_len (vec_info *vinfo, slp_tree slp_node, gimple_stmt_iterator *gsi, static tree permute_vec_elements (vec_info *, tree, tree, tree, stmt_vec_info, gimple_stmt_iterator *); +/* If target supports vector load/store (determined by IS_LOAD) with length for + VECTYPE, return VECT_PARTIAL_VECTORS_LEN; otherwise, if it supports vector + masked load/store for mode for VECTYPE, return + VECT_PARTIAL_VECTORS_WHILE_ULT; otherwise, return VECT_PARTIAL_VECTORS_NONE. + If load/store with length or mask is supported and ELSVALS is not null, + store the possible else values in the vector it points to. Unless FACTOR + is null, *FACTOR is set to the number of separate subelements that each + element of VECTYPE should be split into. (This is always 1 except in the + case of VECT_PARTIAL_VECTORS_LEN when the machine mode of VECTYPE can only + be supported by using a length in bytes instead of a length in lanes.) + */ +static enum vect_partial_vector_style +vect_get_partial_vector_style (tree vectype, bool is_load, + unsigned *factor = nullptr, + vec *elsvals = nullptr) +{ + machine_mode vecmode = TYPE_MODE (vectype); + + /* TYPE_MODE tries to map vector modes that are unsupported by the target to + same-sized scalar modes, so reject scalar modes here. */ + if (VECTOR_MODE_P (vecmode)) + { + machine_mode mask_mode, vmode; + if (get_len_load_store_mode (vecmode, is_load, nullptr, elsvals) + .exists (&vmode)) + { + if (factor) + *factor = (vecmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vecmode); + return vect_partial_vectors_len; + } + + if (factor) + *factor = 1; + + if (targetm.vectorize.get_mask_mode (vecmode).exists (&mask_mode) + && can_vec_mask_load_store_p (vecmode, mask_mode, is_load, NULL, + elsvals)) + { + return vect_partial_vectors_while_ult; + } + } + + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "the target doesn't have the appropriate partial" + " vectorization %s for vector data type %T.\n", + is_load ? "load" : "store", vectype); + return vect_partial_vectors_none; +} + /* Check whether a load or store statement in the loop or SLP group described by VINFO is possible using partial vectors. This is testing whether the vectorizer pass has the appropriate support, @@ -1626,28 +1676,23 @@ check_load_store_for_partial_vectors (vec_info *vinfo, tree vectype, poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); poly_uint64 vf = loop_vinfo ? LOOP_VINFO_VECT_FACTOR (loop_vinfo) : 1; - machine_mode mask_mode; - machine_mode vmode; - bool using_partial_vectors_p = false; - if (get_len_load_store_mode - (vecmode, is_load, nullptr, elsvals).exists (&vmode)) + unsigned factor; + vect_partial_vector_style partial_vector_style + = vect_get_partial_vector_style (vectype, is_load, &factor, elsvals); + + if (partial_vector_style == vect_partial_vectors_len) { nvectors = group_memory_nvectors (group_size * vf, nunits); - unsigned factor = (vecmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vecmode); vect_record_len (vinfo, slp_node, nvectors, vectype, factor); - using_partial_vectors_p = true; } - else if (targetm.vectorize.get_mask_mode (vecmode).exists (&mask_mode) - && can_vec_mask_load_store_p (vecmode, mask_mode, is_load, NULL, - elsvals)) + else if (partial_vector_style == vect_partial_vectors_while_ult) { nvectors = group_memory_nvectors (group_size * vf, nunits); vect_record_mask (vinfo, slp_node, nvectors, vectype, scalar_mask); - using_partial_vectors_p = true; } - - if (!using_partial_vectors_p) + else { + gcc_assert (partial_vector_style == vect_partial_vectors_none); if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, "can't operate on partial vectors because the" From patchwork Wed Jun 3 15:19:17 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136412 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 15F8A4BA2E20 for ; Wed, 3 Jun 2026 15:32:24 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 15F8A4BA2E20 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=KKUye4dF; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=KKUye4dF X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from AM0PR83CU005.outbound.protection.outlook.com (mail-westeuropeazlp170100001.outbound.protection.outlook.com [IPv6:2a01:111:f403:c201::1]) by sourceware.org (Postfix) with ESMTPS id A97054BA23DB for ; Wed, 3 Jun 2026 15:21:00 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A97054BA23DB Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org A97054BA23DB Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c201::1 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500060; cv=pass; b=T8X70gb8L+m7GJrsPhHbOTRVCqatXaO7QAzujhVceOg/5AImFyoYnbZqOA1WzgGx0VDqs3JNAKM4/qaZNCtrqRQORlNBqECY3ScW5I9PESMrEH5t0UDFgaW0Sn5+9o9JVhBETdC1vFJWhUJ2ezrZZfmxkmkTvzwRBC3fsZEWdl8= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500060; c=relaxed/simple; bh=mI5XoQC9Ij4q0GSt2M8YNeZByhbZGOWTADKbkhga+qA=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=SGcNDBEysZx5JLxW2ShDa9kVQ5CFWdG1V5I9ONxvURwKftfeet4M6rpHUt9EPMQBmrbUjqCQoU5h0B11EzJJTvsnHfA00FiYZsQC4O3A/YnNWhGwSoWX0Xnz73nBURFPGQF//tzHYnjubIAfNdaZU90kBUIVQqrWI2XHvaZukSQ= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=KKUye4dF; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=KKUye4dF DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A97054BA23DB ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=ggbu5nodSBfnXuCUGfWnqVm9fq++m+9uv+N1vc61tY+zimHRGXk03RJIDcb+BQL8Q0OtKqIl3cTF77XOHsFY6b6MepV2Z2j3iPbHcB4G+qOAKP0CSngMItxGFBcwm4VseEUEUkQPCMlAacJkxQC2IERIKLZtdiwqHXuvUZQfmBwlmtVsH5sHT6kedqtL8mpo1gY5WmIr9le4Y3ANtaJkUDj68flzDtrYxC4L4ouS0Jab+QOA3drrfawium4SZru6YSqYmGgmpRxkdt/JqWn0r1zCL83PS2VvaHn6xqrdS2eQf+q2/eGIFd2u1wp5lsYbAJ2RNaV2HGe0exgmRaLlnw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QQOHbu70dJoLb7VTM0kcQitv2yDJeD/6GqWuUz5jwJY=; b=mH/ZLu/ELGu8XPqBVwEJU24g5aGGZNDJPGHNPT8qUag8oT45GdL/zUnb+DtnISyyvoIkSLaOc8GEehMdxXkQlyKyvegYf2dtTGqJEQHmnDBsaknmdVVyYk7KANv0fMSzIRGQhBD06ciw1oh8YMtDXolLQavEWOWVTev5BNJbMSauYfZva6bCWYiDecdIMqDsGz1m2wzdiYT3qmfOVPNisUPshPj19zCYeCLPg3DXRPQzHkpCRVtdXPx0gtE0dhoID0ei7+922IcrrUV7St7jk25ZHDeIoDJjc6FgdIZ5FsV2FM7IPdqO9JRxLnsSuzWtDNzRe09wx92L+oPiRuKExg== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QQOHbu70dJoLb7VTM0kcQitv2yDJeD/6GqWuUz5jwJY=; b=KKUye4dFfio2n1fIhq39uNaoWYruY4EliSRSQDEBQ6kaUtOduzjIfylEeBK/rMxXGFabwXyTE5Oa8yLJvW0bGxAbxEhWl+mcEm1IFpVYA4b4wZC25nzbIdDaQUeg1a6D/egURYw5uej2Mi14d7KGMgoEEiUfyLmfMzIKkIClRV0= Received: from DB9PR06CA0012.eurprd06.prod.outlook.com (2603:10a6:10:1db::17) by VI1PR08MB10273.eurprd08.prod.outlook.com (2603:10a6:800:1be::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:20:49 +0000 Received: from DB5PEPF00014B9F.eurprd02.prod.outlook.com (2603:10a6:10:1db:cafe::24) by DB9PR06CA0012.outlook.office365.com (2603:10a6:10:1db::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:20:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by DB5PEPF00014B9F.mail.protection.outlook.com (10.167.8.169) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:20:48 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gNqjwD+er/WAMz68s4bryBnIpd3b0Y2kJDrrCqMtDg/9fgMyebwm16DphYmMvUxjI5m6weJEnFJ5D9Mx16hbk82d/7g3X7RpzHrIQlVAOKWHK6UIA7oUnuONrcGbjhmWKuB/yu3DHZEw43akjJgNBIT4PTHaxAV2M9hGJDiYpj7PZ362Kveeierr7isw00fiQazjRVvVKJHXS/ZffbotrdrVM8Fe1+LrqSBnZ7s+fEDrImJ4jm4EOtI0Vn/7c6i6i78r23Lg9sll04bawnjCC4gjpXUJR2A8H56HgrIpPbE43vNEnBONGL5IZhH+MeezcMiGSC5Wns7uaezbXIhTDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QQOHbu70dJoLb7VTM0kcQitv2yDJeD/6GqWuUz5jwJY=; b=L9eE0S6VbukSEY95MowG9VbX+Y3ZYATrKQB+h0Frj8EO2TVfNmfFaoLsjIWV/sa3pqHVypmh41KWhKLZfQ0hZyiG9Ei8aJKShKaB/OV4JVcTbC29ed710zXDUdvjUT5FEkGeueXQBwvnxmXcfXWltXCIyWzcYBB74xhmeJVVvAprnLqlYRuw+zgQ6BBp9dUn+3FPe9YNd1/6TmK/noyfiBhBcE0QwJ5qfhkpt30WR+i+qpTTJUlIwm2l+1CaDjQp/QXu2mR1lg157PZ+d90uY9LcJ+JLg1+QIPaVRJJ2PIIBlRrh64KcObOJ3F7o5SLenBeZbqJlBpRGprwg4oU8Jg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QQOHbu70dJoLb7VTM0kcQitv2yDJeD/6GqWuUz5jwJY=; b=KKUye4dFfio2n1fIhq39uNaoWYruY4EliSRSQDEBQ6kaUtOduzjIfylEeBK/rMxXGFabwXyTE5Oa8yLJvW0bGxAbxEhWl+mcEm1IFpVYA4b4wZC25nzbIdDaQUeg1a6D/egURYw5uej2Mi14d7KGMgoEEiUfyLmfMzIKkIClRV0= Received: from DB7PR05CA0008.eurprd05.prod.outlook.com (2603:10a6:10:36::21) by DB9PR08MB11404.eurprd08.prod.outlook.com (2603:10a6:10:60e::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:19:43 +0000 Received: from DU6PEPF00009527.eurprd02.prod.outlook.com (2603:10a6:10:36:cafe::6d) by DB7PR05CA0008.outlook.office365.com (2603:10a6:10:36::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009527.mail.protection.outlook.com (10.167.8.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:43 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:35 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:35 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 05/12] New parameter for vect_maybe_update_slp_op_vectype Date: Wed, 3 Jun 2026 15:19:17 +0000 Message-ID: <20260603151924.53706-6-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009527:EE_|DB9PR08MB11404:EE_|DB5PEPF00014B9F:EE_|VI1PR08MB10273:EE_ X-MS-Office365-Filtering-Correlation-Id: 4b1421b4-882f-43ea-4a14-08dec183b11d x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|1800799024|36860700016|82310400026|376014|6133799003|22082099003|18002099003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info-Original: z7QthNlKmPLxGSO/OQU/linfjBCWmuHgABstQaB+rTBxSBt2CR3zKA58Are86mQpd1O50MankNGVhHmVcCWGlM1wRxK0KLBr1P+iNp/NWmEVqn9xsFd5S8+z7n1m2KojQVe02FbE+SFuTlL9yt0Kum7ATjyuDb6PO8l7pa1SdN5k7/KnYZtTGqO3quDFoVaGZ+7Ul1+fFwyp4yw3MzxNsNiei4lvNYLRTzr5Ts4B07clLmTxEjHpOKXQtVEpc2/E01KH6RSW8ML4yG/iPhDdn/iypjmyNxCLXdKCAoI7FlH17VJGHM2Lss0MoePNxJAqQ+UWkI+liJovzjLL55KWUV6Je/PvYz+BZKQtm9cZ0D3j0MI3zyWwVvVszMMyP470JacgaJJ9gZ8waYVYN2my76wWWYKaUKCPRZaJ5OWOo5P6KqSI2mBkQByPZOfuJ82bCSjPHxKpknxsxEeR0/xY4ZpjQoTBB8Di+nasD83FV3wdGpkLe04GcYkGe2EcT9yD3tAGYZ1xFcPrPN18ynMqjLKyIBWOJNIIXBF8YLAdh2QJNaj68spuW6nGQXzsshMluuXUpYHCUKvoLQj9EyydbBTPP3csS0bwX8vp2/wYBW4CVa/835CFZxOt18NSjRIHgcvXMuNpJ1/vPrLeqZD2pacBf9f2XvE6X0sFHJJTNOcDvyNimgyzqtUcIrDG8WbbOnuk8HN3/5+EFsY3qQobNi96tuvefA8I4m0qFflkDis= X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(36860700016)(82310400026)(376014)(6133799003)(22082099003)(18002099003)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: MBb4GICcwlWsLLLPRYHFGx3BuqdkK+542PxLsj+9l32tlqynzkVMubY/M0oMzFiHra284Do0ypgpcRH1b2foCrZEJbE2RbRVQQut1+0MdFwU5Qb4C337+AXjj6Mmdp1b67R4uSb3jQANC4wzmvIuPtT7yQxtArlrQLZvdL5tB9yt1AeuD+2pGWflIWIXVLSz46ONat2q0wFEu48U5h7B0htaSXZj/lnraeMFr/Cgy/PozJ7KyCHePMPclr0LPhmE8oJfS86PYtUpkbI0BoRHDOzt4ez5oD5XDEUSsfcDVm5ImvjVirrSY5qwbOYMyRcYWPIIRWW+w2XIJVnjNyZXlg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB11404 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5PEPF00014B9F.eurprd02.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: c662dd79-75c5-4eb3-4ae2-08dec18389d0 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|35042699022|36860700016|14060799003|82310400026|376014|6133799003|11063799006|22082099003|18002099003|56012099006; X-Microsoft-Antispam-Message-Info: RC0/2Hcc7K21ftZcGrIUPhx2rEmH31Z8dvISqtg7p5ne2MR4QRJZoCiUHie4Osjtkjbj5a8gCH5u0GHI4om3Ss0WWyJmaeQvaqt2nKDpj/QsIhsrdt1B4ePuc3C9etfOcwFuxb7l4tmh0CMYofGUJw8PSrLZN1ZFrYHmTIXIrPoa/a0DSZKh29AbouK5G7ii/2oX4RY1TGACwGF8XyN4i0vAW1PonfefqfiWjJRoBISv3djV10uEtj+KARnrP74PWusgts13loVUgQ123RsrLkPJ0p+LfrbHQiUy1nk1d/LF27Z4B1tmwBD363iMjCvDe0yg36QFrOo1o7owChZB2vln9m8y/yyizblbV3vnnyNKEpwxNikjb6pdrZts6FZdT1M5FgJgq3zDOOheK0+0Qeo12TyVWRJVDGpcQeDZpH5YUH3uLifpVIiiiM0tEijvB3ZWRu37U9YvMIhCjkB6KxZPJAKppxnaFKElHe8/eoLCKt5QimHfbBxQOsqtl+JsVhrN3rnesCXKmPeolBLY8AZMjAY1Z4dHPBkZoh7IFYhldRIJ8/I5waCtjSqJha7lwZtQqJY+IDKJMJpH+I7s8PK3luNq+JNRYnW4MRx2ZuLbzc9PdS0Q9GyCLLO/CcK+b6eemQ2V/q3teT3H3XFIyYCDfe/2rkR3GxagVDOn9gX6MdzWllrNc7bwpasHykZ0mnav3IzL90MiLwJ/hJPC1qqIf/8Mhi4tCCO3TfzKKA0= X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(35042699022)(36860700016)(14060799003)(82310400026)(376014)(6133799003)(11063799006)(22082099003)(18002099003)(56012099006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 7O38U/o/MA7urtHh3wMRonD0inw46kwGQBggSq9KyHSPYHjsHsrBzAzGUK3dk9WO3XugrOvbZ3/8X2kAgO+nWxBRioX3f4eRbbT2v/dvOTSAHf6GruoR6Z2RzrwTdX30QOqZ6e4Be/syzlvNXc5wMwMXZ0z0n3gFvXSHdeHzBbqnJHRCqEJ1ChkjFCO4VOA/CkDspH1GWsohUuHMY2LOZ1UOZXrECjSX67uyPDduawhQHKH3az/1yssH5jOijYqxMG6VqiymhoW5LPlO/47qurej1VGRxAxN2Ea3pRHwVXraa4AE+vRtQQTqKw/mG5plg3pkOW3tshtTjiWLYmelmy4D56xVsS58P4z7c5DkwScwY0mHmhYJTbT2oaxEAbdMpytikJoEAapMmDghZjn17PSTVTi21OfNK2UBcLYRC7FgU0/6WYt5jOMf5hGjPvoP X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:20:48.8452 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4b1421b4-882f-43ea-4a14-08dec183b11d X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: DB5PEPF00014B9F.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB10273 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Update all callers to pass a pointer to the vectorizer state into this helper function. Its value is temporarily unused but will be required for BB SLP with predicated tails. gcc/ChangeLog: * tree-vect-loop.cc (vectorizable_lane_reducing): Pass loop_vinfo to vect_maybe_update_slp_op_vectype. (vectorizable_reduction): As above. (vectorizable_lc_phi): As above. (vectorizable_phi): Pass vinfo to vect_maybe_update_slp_op_vectype. (vectorizable_recurr): Pass loop_vinfo to vect_maybe_update_slp_op_vectype. (vectorizable_induction): As above. * tree-vect-slp.cc (vectorizable_slp_permutation_1): Pass vinfo to vect_maybe_update_slp_op_vectype. * tree-vect-stmts.cc (vectorizable_bswap): As above. (vectorizable_call): As above. (vectorizable_simd_clone_call): As above. (vectorizable_conversion): As above. (vectorizable_assignment): As above. (vectorizable_shift): As above. (vectorizable_operation): As above. (vectorizable_store): As above. (vectorizable_load): As above. (vectorizable_condition): As above. (vectorizable_comparison_1): As above. (vect_maybe_update_slp_op_vectype): Update definition to accept a parameter of type vec_info *. This is preparatory: the the argument is currently unused. * tree-vectorizer.h (vect_maybe_update_slp_op_vectype): Update declaration to accept a parameter of type vec_info *. --- gcc/tree-vect-loop.cc | 20 ++++++++------- gcc/tree-vect-slp.cc | 2 +- gcc/tree-vect-stmts.cc | 57 ++++++++++++++++++++++-------------------- gcc/tree-vectorizer.h | 2 +- 4 files changed, 43 insertions(+), 38 deletions(-) diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 51e7f05100d..6d602c67108 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -6843,7 +6843,7 @@ vectorizable_lane_reducing (loop_vec_info loop_vinfo, stmt_vec_info stmt_info, return false; } - if (!vect_maybe_update_slp_op_vectype (slp_op, vectype)) + if (!vect_maybe_update_slp_op_vectype (loop_vinfo, slp_op, vectype)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -6980,7 +6980,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo, unsigned j; slp_tree child; FOR_EACH_VEC_ELT (SLP_TREE_CHILDREN (slp_node), j, child) - if (!vect_maybe_update_slp_op_vectype (child, + if (!vect_maybe_update_slp_op_vectype (loop_vinfo, child, SLP_TREE_VECTYPE (slp_node))) { if (dump_enabled_p ()) @@ -7113,7 +7113,7 @@ vectorizable_reduction (loop_vec_info loop_vinfo, vectype_op = get_vectype_for_scalar_type (loop_vinfo, type_op); if (!vectype_op - || !vect_maybe_update_slp_op_vectype (op_node, + || !vect_maybe_update_slp_op_vectype (loop_vinfo, op_node, vectype_op)) return false; } @@ -7814,7 +7814,8 @@ vectorizable_reduction (loop_vec_info loop_vinfo, if (single_defuse_cycle || reduction_type == FOLD_LEFT_REDUCTION) for (i = 0; i < (int) op.num_ops; i++) - if (!vect_maybe_update_slp_op_vectype (slp_op[i], vectype_op[i])) + if (!vect_maybe_update_slp_op_vectype (loop_vinfo, slp_op[i], + vectype_op[i])) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -8512,7 +8513,8 @@ vectorizable_lc_phi (loop_vec_info loop_vinfo, /* Deal with copies from externs or constants that disguise as loop-closed PHI nodes (PR97886). */ - if (!vect_maybe_update_slp_op_vectype (SLP_TREE_CHILDREN (slp_node)[0], + if (!vect_maybe_update_slp_op_vectype (loop_vinfo, + SLP_TREE_CHILDREN (slp_node)[0], SLP_TREE_VECTYPE (slp_node))) { if (dump_enabled_p ()) @@ -8588,7 +8590,7 @@ vectorizable_phi (bb_vec_info vinfo, "PHI node with unvectorized backedge def\n"); return false; } - else if (!vect_maybe_update_slp_op_vectype (child, vectype)) + else if (!vect_maybe_update_slp_op_vectype (vinfo, child, vectype)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -8777,7 +8779,7 @@ vectorizable_recurr (loop_vec_info loop_vinfo, stmt_vec_info stmt_info, unsigned j; slp_tree child; FOR_EACH_VEC_ELT (SLP_TREE_CHILDREN (slp_node), j, child) - if (!vect_maybe_update_slp_op_vectype (child, vectype)) + if (!vect_maybe_update_slp_op_vectype (loop_vinfo, child, vectype)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -9660,8 +9662,8 @@ vectorizable_induction (loop_vec_info loop_vinfo, unsigned j; slp_tree child; FOR_EACH_VEC_ELT (SLP_TREE_CHILDREN (slp_node), j, child) - if (!vect_maybe_update_slp_op_vectype - (child, SLP_TREE_VECTYPE (slp_node))) + if (!vect_maybe_update_slp_op_vectype (loop_vinfo, child, + SLP_TREE_VECTYPE (slp_node))) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index f91d3e723ec..1850af4e753 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -11458,7 +11458,7 @@ vectorizable_slp_permutation_1 (vec_info *vinfo, gimple_stmt_iterator *gsi, FOR_EACH_VEC_ELT (children, i, child) { if ((SLP_TREE_DEF_TYPE (child) != vect_internal_def - && !vect_maybe_update_slp_op_vectype (child, op_vectype)) + && !vect_maybe_update_slp_op_vectype (vinfo, child, op_vectype)) || !types_compatible_p (SLP_TREE_VECTYPE (child), op_vectype) || !types_compatible_p (TREE_TYPE (vectype), TREE_TYPE (op_vectype))) { diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 1cfa2177add..cd3ba6fa1cb 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -3469,7 +3469,7 @@ vectorizable_bswap (vec_info *vinfo, if (cost_vec) { - if (!vect_maybe_update_slp_op_vectype (slp_op[0], vectype_in)) + if (!vect_maybe_update_slp_op_vectype (vinfo, slp_op[0], vectype_in)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -3831,7 +3831,7 @@ vectorizable_call (vec_info *vinfo, if (cost_vec) /* transformation not required. */ { for (i = 0; i < nargs; ++i) - if (!vect_maybe_update_slp_op_vectype (slp_op[i], + if (!vect_maybe_update_slp_op_vectype (vinfo, slp_op[i], vectypes[i] ? vectypes[i] : vectype_in)) { @@ -4550,7 +4550,8 @@ vectorizable_simd_clone_call (vec_info *vinfo, stmt_vec_info stmt_info, if (cost_vec) /* transformation not required. */ { for (unsigned i = 0; i < nargs; ++i) - if (!vect_maybe_update_slp_op_vectype (slp_op[i], arginfo[i].vectype)) + if (!vect_maybe_update_slp_op_vectype (vinfo, slp_op[i], + arginfo[i].vectype)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -5817,8 +5818,8 @@ vectorizable_conversion (vec_info *vinfo, if (cost_vec) /* transformation not required. */ { - if (!vect_maybe_update_slp_op_vectype (slp_op0, vectype_in) - || !vect_maybe_update_slp_op_vectype (slp_op1, vectype_in)) + if (!vect_maybe_update_slp_op_vectype (vinfo, slp_op0, vectype_in) + || !vect_maybe_update_slp_op_vectype (vinfo, slp_op1, vectype_in)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -6171,7 +6172,7 @@ vectorizable_assignment (vec_info *vinfo, if (cost_vec) /* transformation not required. */ { - if (!vect_maybe_update_slp_op_vectype (slp_op, vectype_in)) + if (!vect_maybe_update_slp_op_vectype (vinfo, slp_op, vectype_in)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -6500,13 +6501,13 @@ vectorizable_shift (vec_info *vinfo, if (cost_vec) /* transformation not required. */ { - if (!vect_maybe_update_slp_op_vectype (slp_op0, vectype) + if (!vect_maybe_update_slp_op_vectype (vinfo, slp_op0, vectype) || ((!scalar_shift_arg || dt[1] == vect_internal_def) - && (!incompatible_op1_vectype_p - || dt[1] == vect_constant_def) - && !vect_maybe_update_slp_op_vectype - (slp_op1, - incompatible_op1_vectype_p ? vectype : op1_vectype))) + && (!incompatible_op1_vectype_p || dt[1] == vect_constant_def) + && !vect_maybe_update_slp_op_vectype (vinfo, slp_op1, + incompatible_op1_vectype_p + ? vectype + : op1_vectype))) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -6932,9 +6933,9 @@ vectorizable_operation (vec_info *vinfo, } /* Put types on constant and invariant SLP children. */ - if (!vect_maybe_update_slp_op_vectype (slp_op0, vectype) - || !vect_maybe_update_slp_op_vectype (slp_op1, vectype) - || !vect_maybe_update_slp_op_vectype (slp_op2, vectype)) + if (!vect_maybe_update_slp_op_vectype (vinfo, slp_op0, vectype) + || !vect_maybe_update_slp_op_vectype (vinfo, slp_op1, vectype) + || !vect_maybe_update_slp_op_vectype (vinfo, slp_op2, vectype)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -8408,9 +8409,9 @@ vectorizable_store (vec_info *vinfo, vls_type, group_size, &ls, mask_node); - if (!vect_maybe_update_slp_op_vectype (op_node, vectype) + if (!vect_maybe_update_slp_op_vectype (vinfo, op_node, vectype) || (mask_node - && !vect_maybe_update_slp_op_vectype (mask_node, + && !vect_maybe_update_slp_op_vectype (vinfo, mask_node, mask_vectype))) { if (dump_enabled_p ()) @@ -8847,7 +8848,7 @@ vectorizable_store (vec_info *vinfo, above only handles the mask and the first store operand node. */ for (slp_tree child : SLP_TREE_CHILDREN (slp_node)) if (child != mask_node - && !vect_maybe_update_slp_op_vectype (child, vectype)) + && !vect_maybe_update_slp_op_vectype (vinfo, child, vectype)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -10044,7 +10045,7 @@ vectorizable_load (vec_info *vinfo, } if (mask_node - && !vect_maybe_update_slp_op_vectype (mask_node, + && !vect_maybe_update_slp_op_vectype (vinfo, mask_node, mask_vectype)) { if (dump_enabled_p ()) @@ -12392,13 +12393,14 @@ vectorizable_condition (vec_info *vinfo, || !expand_vec_cond_expr_p (vectype, vec_cmp_type)))) return false; - if (!vect_maybe_update_slp_op_vectype (SLP_TREE_CHILDREN (slp_node)[0], + if (!vect_maybe_update_slp_op_vectype (vinfo, + SLP_TREE_CHILDREN (slp_node)[0], comp_vectype) || (op_adjust == 1 - && !vect_maybe_update_slp_op_vectype - (SLP_TREE_CHILDREN (slp_node)[1], comp_vectype)) - || !vect_maybe_update_slp_op_vectype (then_slp_node, vectype) - || !vect_maybe_update_slp_op_vectype (else_slp_node, vectype)) + && !vect_maybe_update_slp_op_vectype ( + vinfo, SLP_TREE_CHILDREN (slp_node)[1], comp_vectype)) + || !vect_maybe_update_slp_op_vectype (vinfo, then_slp_node, vectype) + || !vect_maybe_update_slp_op_vectype (vinfo, else_slp_node, vectype)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -12841,8 +12843,8 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype, } /* Put types on constant and invariant SLP children. */ - if (!vect_maybe_update_slp_op_vectype (slp_rhs1, vectype) - || !vect_maybe_update_slp_op_vectype (slp_rhs2, vectype)) + if (!vect_maybe_update_slp_op_vectype (vinfo, slp_rhs1, vectype) + || !vect_maybe_update_slp_op_vectype (vinfo, slp_rhs2, vectype)) { if (dump_enabled_p ()) dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, @@ -14090,7 +14092,7 @@ vect_is_simple_use (vec_info *vinfo, slp_tree slp_node, for example when conflicting vector types are present. */ bool -vect_maybe_update_slp_op_vectype (slp_tree op, tree vectype) +vect_maybe_update_slp_op_vectype (vec_info *vinfo, slp_tree op, tree vectype) { if (!op || SLP_TREE_DEF_TYPE (op) == vect_internal_def) return true; @@ -14104,6 +14106,7 @@ vect_maybe_update_slp_op_vectype (slp_tree op, tree vectype) && SLP_TREE_DEF_TYPE (op) == vect_external_def && SLP_TREE_LANES (op) > 1) return false; + (void) vinfo; /* FORNOW */ SLP_TREE_VECTYPE (op) = vectype; return true; } diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index f79f04ff8ac..62f6ad320f0 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -2534,7 +2534,7 @@ extern bool vect_is_simple_use (vec_info *, slp_tree, unsigned, tree *, slp_tree *, enum vect_def_type *, tree *, stmt_vec_info * = NULL); -extern bool vect_maybe_update_slp_op_vectype (slp_tree, tree); +extern bool vect_maybe_update_slp_op_vectype (vec_info *, slp_tree, tree); extern tree perm_mask_for_reverse (tree); extern bool supportable_widening_operation (code_helper, tree, tree, bool, code_helper*, code_helper*, From patchwork Wed Jun 3 15:19:18 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136409 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8B1FC4BA23CB for ; Wed, 3 Jun 2026 15:26:48 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 8B1FC4BA23CB Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=eEKIcsee; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=eEKIcsee X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from AM0PR83CU005.outbound.protection.outlook.com (mail-westeuropeazon11010024.outbound.protection.outlook.com [52.101.69.24]) by sourceware.org (Postfix) with ESMTPS id 26B5B4BA2E19 for ; Wed, 3 Jun 2026 15:21:08 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 26B5B4BA2E19 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 26B5B4BA2E19 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.69.24 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500068; cv=pass; b=YWjync6VdqRbevcaefVQPHz7xruRiQiVdCwy/w3aM9b2XyWerN24vbfVUHtKU5VNlF3unH1khLQVpJpmKelA4lpZQBytC8xF+cK503Gh2OqRwaqCZvQ1J9V/Rk8YKh4jSG96xWT0xFE6ldrGGNiZs0EsMCHWzIxbjSF+Sz8yUb8= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500068; c=relaxed/simple; bh=woWes6CSAsJ7RfxcblkFywnic8zo6ZRYekIMXFZvi9k=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=tUK/sBdhWpdxcBwqAnmNyjWDDbA0i2hC+9AQavHNDHi1bU2Yt7oX9gPG+95hoPPkoxeSHWG0JE7I4MrsYiMc4iEA9lIvJ4vEXpnYmlj0gKjS1NIRxYymnQsP1+j82j1wjFclXEXMeThMogMr87YWt5kt4fBzotn3jRaOQOcswXU= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=eEKIcsee; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=eEKIcsee DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 26B5B4BA2E19 ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=Z+imvzYXUA8iu9yWM4mIlXf0MxVlrtK10SsydsyH8YdNzGft3eCUzrFM9wz3i3tDXZxeI1jdqguetrQS3o5AZ49EQsFmk1vveIoCzSYK5nFyICIn0nEqipZEUOX7L56oVOf3NTEDf9pRjrdH3YeF4MHeGSoT2VUUwtr96cUH5xa+E7Zl9B6UaF8j6/rAYu7DKe0maEQr8G2LABRKw1VkpkP8ZJGEYushoe3f94Y7RaZ8TJQU+IRwjLwnIQMB7LrMpVT+ueihgf1FRrbzTkUn4Gp4IxwOSYgLj4a+xsCzqV9kG5uw8JxTianrLeJyXP1IAzj0vQlWSQYfx7se0cv4Pg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Yu0UQx4JaeWw5bvoWXScM0dIxm9HTjX6DzcR3T3CwxY=; b=k+3TXOhuZnCFUuVakNjfKpaeLISAKfm5oTaaZ52X3D52QADN4EwUKspYIBiGzuwyWRbt2XtuI888cTfX3I9JAXy/c75fN9U7tyRzLDDmxv0JTTBWutxlZ0TGYUWjOqM5xOBwBJFu7A7maWhCDGq1VJ91mK+D5yY0rGwilVaC8O3uS/llRnOjtj1/PE06owTaCagMTk02WeYtDT7KZ2RQWurvsdBs7y7yVRpSOIVSq4Q6LJbs3ZkpfFUu+Ia3Ut+ATHz2Ni39sWOqqnNihGZ3tG7K3cXASviO4tdXBqksux6vb2WAdbwRwhOZ2he2NzhZMYQuN9f6I4hiBuG6/A8STA== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Yu0UQx4JaeWw5bvoWXScM0dIxm9HTjX6DzcR3T3CwxY=; b=eEKIcseek6I7/cFoLViFsJ1TvOIoJIk/k4lzZl+vnWQfVHVs55FtSWgTuQoURKYMmSLyQF6mXpD9J+pwU4VW1d16IlCeqkYaI8Rfe6JTVo+MrtrZ2MhTQUsf5gt9PIxVDwa+wfKChu/3YQiuAHq6BJOhH91x0XBGoJD8hBjQV1Y= Received: from DUZPR01CA0165.eurprd01.prod.exchangelabs.com (2603:10a6:10:4b3::6) by GVXPR08MB7823.eurprd08.prod.outlook.com (2603:10a6:150:2::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:20:59 +0000 Received: from DB3PEPF0000885B.eurprd02.prod.outlook.com (2603:10a6:10:4b3:cafe::87) by DUZPR01CA0165.outlook.office365.com (2603:10a6:10:4b3::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:20:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by DB3PEPF0000885B.mail.protection.outlook.com (10.167.242.6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:20:58 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ltNOT/3Fgktfovy7mvXu3h8UU3PDz4HMHMF3ggeFkDLFhcCtVY9TbrVkPJBOZYc7dKSfx1rw67cqtTVgoXr+e5f7V142eGE/2U1n2fTXoxMcqQb4HrAzQosiiP5/nMRM74aD1TJZXcLq+NcnhhCuCxlAvZCNzYzdWiKVGbaSBBHMwrAI5AL7+2nR9/iJoA+6VQeuc3Ttr/2Rxj3OmN6WwVJlKzUz/TJ3pBeYuaPiC6D9zrmkcoYP32Jl1NHpuyL0Gjp8pE1SmSLNMw6Va/fQiTCdaC0HyD+jetUpulDrcjjvZhvsezEJ/YYXPIkR+yDQn18PnMoGHMrDMTq21LjQ1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Yu0UQx4JaeWw5bvoWXScM0dIxm9HTjX6DzcR3T3CwxY=; b=hzjdviBmVO9I4t/EKkrvcl1MyQ6hPIaC3gmPQfgrHf2Sp/Bp/Q6QjzIc9goTGA3wm6N8mOr54rRPGme1xgnqTCyTdeZiWFz5rhdYy2KEyFFnWRPxySoJpFqQktfqJWYj0/DgQd3SWsiigRja+hycTS17El+IMBlPJNdM/M7NnjGqkN5ZDA6ysGlonWzP/BReNIo2R5Lzd7R3NRytCTaiAPGa6pAxYDF3v5chKDE0MU0V8TF39nmDkFeEN2j+K46bWqdpXwUBhzlquD0MguLavmC/o5U6CiQXNA1txpNXF1fYy5z8reCpFCLRHdCq/3rj0hIgcLgPxme8kwTO6tNu3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Yu0UQx4JaeWw5bvoWXScM0dIxm9HTjX6DzcR3T3CwxY=; b=eEKIcseek6I7/cFoLViFsJ1TvOIoJIk/k4lzZl+vnWQfVHVs55FtSWgTuQoURKYMmSLyQF6mXpD9J+pwU4VW1d16IlCeqkYaI8Rfe6JTVo+MrtrZ2MhTQUsf5gt9PIxVDwa+wfKChu/3YQiuAHq6BJOhH91x0XBGoJD8hBjQV1Y= Received: from DUZPR01CA0009.eurprd01.prod.exchangelabs.com (2603:10a6:10:3c3::16) by AMDPR08MB11459.eurprd08.prod.outlook.com (2603:10a6:20b:71a::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:19:55 +0000 Received: from DU6PEPF00009525.eurprd02.prod.outlook.com (2603:10a6:10:3c3:cafe::8d) by DUZPR01CA0009.outlook.office365.com (2603:10a6:10:3c3::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009525.mail.protection.outlook.com (10.167.8.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:54 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:35 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:35 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 06/12] Handle variable-length vector types in store_constructor Date: Wed, 3 Jun 2026 15:19:18 +0000 Message-ID: <20260603151924.53706-7-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009525:EE_|AMDPR08MB11459:EE_|DB3PEPF0000885B:EE_|GVXPR08MB7823:EE_ X-MS-Office365-Filtering-Correlation-Id: 69ac9cab-9773-4ca8-9041-08dec183b6cc x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|1800799024|376014|82310400026|36860700016|3023799007|11063799006|5023799004|56012099006|18002099003|22082099003|6133799003; X-Microsoft-Antispam-Message-Info-Original: jR94Q/IGWO/rqE8uzwjaSrum8wFABJG1Smw+QvsqhkC7vBPJ/KtdLkSZVFhEy8lNppGmfkXOFJXKFfJNJw4HujZUPlKyhKYMBqG0Jx2sZE6CKxd2ArNYc0MsXzdHpQiMGi6ADx2NAFAeYbZWU1YjFiSFG7oYxj6IEZjy4CtoCAdhYXZhXIPUKNVfcxvMgVxAoF6ZoKQlyMkaI0G7pcfl0kJdfPHqcqaL1D7cOH/Q3g34NhZsOuRwMILgCCPgmouKnUV2ZFfzvTN7bjoEq5wv7L6uvFahTEFPe24kAphOPn1a3brGbqQ1FBK0707B+oWGT/bpiKZe5bUlRR32A+kr79oFxcILkpPhIzqOJEzuWnpu3GkVyQDghaDmli3NyKFKMuu+3b9lufeDKXjGF9x93MEXwUmuCqe8mqUTkyLeTWEqMhxZ5XdrzxQWmuwF1dfoyXaqZIoxSyHeFiYNmD+LKTyYIKneEm0rxiX+6pp7J4z2rgdso5mBmuHfjgEnodW47md7l3amhz+hxkQdIITJMmc6sGdV9rct/F41rExcBHDrbwUWX2QrVme5sWDHX7j6iJF8lp3I33+aa+A3KYGQ3RTvNWOydAHpOwtSnypE69TpJU/LHYtK+mNpbJ6ja/frsVhTU8ypJukcgVF4eO6KYBAKLjEuq5aTtOaPb2XHaExOd46xBcu2Zm6jjk7qwMTtFTd8s5pqAELTvlwdAkB5kQfujAWiC7Km3mg7PywvGm0= X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(376014)(82310400026)(36860700016)(3023799007)(11063799006)(5023799004)(56012099006)(18002099003)(22082099003)(6133799003); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: sWdmSLFq59V5slVRj91WBf+LMPdhgGSlqP/zoRxayWHDhO1JGoKYor1GmUgrUpAYktdYJ0pCKERd61UjpO20Bd2msg1Kq+TZJhaXwEbOaFPHPScVDNzHB13YaPJqGcOaHa1FPnBbG7z9/1EwMGfxn8dw8LDVz3sajoQhfQRsrF1YhaQXA32GR0J6oAamxWuFuX3as6ZYz1F8lzVHdEDLznrQskFcD/lmQ6ExeyVYj8iw6RoNBwXazJVBZM5+KpMMHrwAbIvaWmjP3PRyh0aOOMKbhGCM2Z0/mll+ZtxMEL8FVE065plnxC9eAYlXztXNeW2w4Qdm+nBBJKjHbdaFIw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AMDPR08MB11459 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB3PEPF0000885B.eurprd02.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 41a0f9de-dc46-4f4e-ad40-08dec18390df X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|35042699022|36860700016|82310400026|14060799003|376014|6133799003|3023799007|5023799004|22082099003|18002099003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info: CMdHikHjklmFOeDZHRooOSSEhMF69DRdVfxTuxgxhzz4Je1N5M+2Vn6jzqsxRlLG8eDw4Jyy1j67tg4roGmMB5c/ff6JYuoBl1jTiE8gPKDznytgTM1EwccNhz2YMuXRkFzBHfSujkEgOyaftkQjb3zZc3m8lT8u8rq997pMrlRmb8jpIrqGNfEWCkcru4a33sSjW2zwiAM01YlXw435K+GyIs2qbBdSHx1/cTMABkStsymIA00/ZnU3eYRFHraHBEd5NBi2i2+5kRP8BNDkhZABwsGuHhJkGnO+tnYZlddG1FDxbkIqDoRFz4fEj7jYfk7H7kHfq53JTpRqdS5og5m4PAA+hdT5ix+1SWVRYJHnB62xmG5jM9IsYoa/VuLPsZhffEDwoTpazcPuoVCg5mEu8mLAE07R+qnltMxADOjF34AFMEaDU2hiaH6ZDfUDz+eyVF7Z6+aBHbOcK2LF7lOvndVLuLPTuEnm4kU2dVw1PJzSaIUxGVXWC0uShYI7kaLlZ9hRfCotkTuTScmlOPdfK8aZ14L6NgHKgt6vF9LjUD7/8gCP9aS5KlseD1RdlOFcGnd51ilc08juBRstJ+2xAhnZfXrSGLV0Mbfvu6GiNBroGsAPmOdDgSfFv3CVxkTHyyEVZdZLmtlG+MZAc/UFDOr009A+ZcxjhcSP/NipUFtgBThnN7igRuG2xVwQXQjrEbHFx6JkvzZXXbZkY1TCJPO/45cKPUsKn+I+euA= X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(35042699022)(36860700016)(82310400026)(14060799003)(376014)(6133799003)(3023799007)(5023799004)(22082099003)(18002099003)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: tKBoyuTY0xSJLnyiSW9Km2umubCdaFnHsamfQfllEos4rPWBcVEH8apfZtjoTj1G4KMFwCDvUBtsBfAWQbGcoT4X2BsNDGPDk2Y++OIWeaqzAAPQesh9llCCwNLntMMqu6BAvd+IqE9lcKRYW94+prv7NCrJW5LxT8g44yiuzA+g2V/tp/e/RHCidNo8Ag4PoeWVW+Jutl13ZbitL5HnwAq1wiIX/vYxt9rC8L46HLNE5ONpqEWdSI4krkC8TpXfO5aFg7JjKP7wOaG4pxBCQB9HGFY1uqgz6g4cj7AjY/pvlZS8V0f2ZXDrRzvd2gVUXC6MEVHjK2LN2EM5X/+piwsS+IggBGqmhWzWmggoSX/BYYTFsY20KRCYGy9Irs6gPyDr/XgAmpAK0cuERg0R0tkKqQvYFRzRCL9LtVDrWskwYGCV3acl+q1BLNwalWwR X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:20:58.3947 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 69ac9cab-9773-4ca8-9041-08dec183b6cc X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: DB3PEPF0000885B.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GVXPR08MB7823 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, RCVD_IN_DNSWL_BLOCKED, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org When given a constructor of variable-length vector type, the store_constructor function now builds a vector with the lower bound of the number of subparts in the vector type and uses it to emit the body of the kind of insn chosen by the convert_optab_handler function. Previously, this function used a fallback path of calling store_constructor_field upon discovering that the number of subparts in the vector type was not a constant multiple of the number of subparts in the element type. For example, this allows GCC to generate the following AArch64 assembly language output for the tail of a reduction in the slp_6 test: uaddv d31, p6, z31.b uaddv d27, p6, z27.b uaddv d26, p6, z26.b movi d30, #0 insr z30.b, b26 insr z30.b, b27 insr z30.b, b31 add z25.b, z25.b, z30.b instead of the following output (with predicated tails for basic block SLP vectorization but without this change): addvl x0, sp, #2 movi d0, #0 st1b z0.b, p6, [sp, #2, mul vl] uaddv d27, p6, z27.b uaddv d26, p6, z26.b uaddv d25, p6, z25.b str b27, [x0] addvl x0, sp, #1 add x0, x0, 1 ptrue p7.b, vl3 ld1b z0.b, p6/z, [sp, #2, mul vl] st1b z0.b, p6, [sp, #1, mul vl] str b26, [x0] ld1b z0.b, p6/z, [sp, #1, mul vl] st1b z0.b, p6, [sp] str b25, [sp, 2] ld1b z0.b, p6/z, [sp] add z28.b, z28.b, z0.b st1b z28.b, p7, [x1] addvl sp, sp, #3 or the original assembly language output (with neither predicated tails for basic block SLP vectorization nor this change): uaddv d31, p6, z31.b fmov x0, d31 uaddv d31, p6, z26.b add w6, w6, w0 fmov x0, d31 uaddv d31, p6, z27.b add w5, w5, w0 fmov x0, d31 add w4, w4, w0 gcc/ChangeLog: * expr.cc (store_constructor): Add an else block to handle cases of TREE_CODE (TREE_TYPE (exp)) == VECTOR_TYPE in which exact_div (n_elts, GET_MODE_NUNITS (eltmode)).is_constant (&const_n_elts) is false similar to the existing "element type is not a vector type" case except that const_n_elts is taken from the lower bound of the subparts of the vector type. --- gcc/expr.cc | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/gcc/expr.cc b/gcc/expr.cc index de73215ccc6..f163ea16fe2 100644 --- a/gcc/expr.cc +++ b/gcc/expr.cc @@ -7498,11 +7498,14 @@ fields_length (const_tree type) return count; } - /* Store the value of constructor EXP into the rtx TARGET. TARGET is either a REG or a MEM; we know it cannot conflict, since safe_from_p has been called. CLEARED is true if TARGET is known to have been zero'd. + If the constructor EXP has a vector type then elements of TARGET for which + there is no corresponding element in EXP are zero'd. For a variable-length + vector type, elements beyond the minimum number of subparts of the type are + not zero'd. SIZE is the number of bytes of TARGET we are allowed to modify: this may not be the same as the size of EXP if we are assigning to a field which has been packed to exclude padding bits. @@ -8075,7 +8078,12 @@ store_constructor (tree exp, rtx target, int cleared, poly_int64 size, similarly non-const type vectors. */ icode = convert_optab_handler (vec_init_optab, mode, eltmode); } - + else + { + /* Handle variable-length vector types. */ + icode = convert_optab_handler (vec_init_optab, mode, eltmode); + const_n_elts = constant_lower_bound (n_elts); + } if (const_n_elts && icode != CODE_FOR_nothing) { vector = rtvec_alloc (const_n_elts); From patchwork Wed Jun 3 15:19:19 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136411 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 2C0A14BA23C5 for ; Wed, 3 Jun 2026 15:29:24 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2C0A14BA23C5 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=kw0lWP2h; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=kw0lWP2h X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from AM0PR02CU008.outbound.protection.outlook.com (mail-westeuropeazlp170130006.outbound.protection.outlook.com [IPv6:2a01:111:f403:c201::6]) by sourceware.org (Postfix) with ESMTPS id 1FADA4BA23D4 for ; Wed, 3 Jun 2026 15:21:15 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1FADA4BA23D4 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1FADA4BA23D4 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c201::6 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500075; cv=pass; b=g3Z5QDz7jl6AaGqRvExcaIwpI+AZA5Krkv3PcWXHHUFe8R8QsY8RGT7AeKoclRHiudwGYGZ079qr3zADIx0Jrg/1+hfFsorcOcSGSyON1WaFgAcvJUu6U+pWfdknqehRUF5QgYBXm7azZR6NHDL4aPf9v2WFxHAR9+fEp/KaXJU= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500075; c=relaxed/simple; bh=26gL2T7qH+EJG4zO1DG11cBxkDf9cKZNNARxavKsEs0=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=IbHCmHOTb/IXvzF+oise3MO0sjOq2428EsmWFwgM4x2u/ljPtDTbQuifqc9PTxoY5J12MjO/o/Rdin3R326RAea5fI2oqWFWdOtEK77KqWcjSYkXzlhpenxf0lKTXpBgiBR526CpnlSw+nOK0BcesVs/3g8b9cs/zJUjKK89Q9c= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=kw0lWP2h; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=kw0lWP2h DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 1FADA4BA23D4 ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=lvle5HIDUNVe2/t3Re9aOYBmv+mwG9M0W8Uz/Ie5l4NZsH8BTvr2pQN4hlG0lCkRswIhVePj6iCbHCAc9vkKeL3sU6iLdZ0gtc2VEH01XP+lr11oMMl0aciJZP7ntu4Kq6Td+r+czISgDPHsd2ilAN6UzkTgut00/z5MeXGqd0gQPGyFQR6NZaYpLCgB+LINYFoXFKVwXySWm6ekmmCGaBtePf6pKbTFWWkoyegvlRwwxRumgu3eUti9sKE8yAROSI0e25RF6Q4DglbkrfCoeHJkSBBdZeQw2m56lgEVTt0Z2BSrlVq5cmTSQP48Dg/h2DVmbtuIlAdw4Sfyt/6NSA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5j2Z1CDQsqAN0nSzpIjxlnzcW5Wpb/ynR2dZw3MJqJs=; b=nB0Fv6KeJgSQcxDpoIjgAfD4+SmJJW/dQcEnHRO42699iXBEEWTFdVzB3uNVPZwUCWnrzlPKYCxvD0vN9HWWqAyDSolb2tXc2X+G71LMJcWVywc9d449VrPGfESqDUQk8SAoI/Veh/yHnSzZXBjuN0uaFJG/IdD5OlfAcqlGVZy1BNDM1L4IMM+F2GHSRE0NJJggDDOURV6Zhgl3BU/gA0UAKp7YJtGuI1Fi5iT7joS7GfrsZroDxG2OyElawJ5T6MmO5Xvem+cEKPd3FDGeLBMh9ICYj4V/qY/p+Wlx7FTYxUy5FHfrVN1JiHRssZzO1Aglr2KQDbnx+11CGTIUvw== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5j2Z1CDQsqAN0nSzpIjxlnzcW5Wpb/ynR2dZw3MJqJs=; b=kw0lWP2hSwLcON0PxfrXM35NHJQ6aMn+No6KlQpiOb5L6EVon0M1084AlHkbIvyPDOFm+xDZiqbI/ywv2TeJ+ohFJoH12z2OqBWBnT2ST9JRFPOzbPKkT65+1p2VG8/IrxnsvPgRMZJr1bJAkEGuDvRWG+n73PfqToOlwzYSa1s= Received: from CWLP265CA0372.GBRP265.PROD.OUTLOOK.COM (2603:10a6:401:5e::24) by GV2PR08MB8147.eurprd08.prod.outlook.com (2603:10a6:150:7b::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.16; Wed, 3 Jun 2026 15:21:02 +0000 Received: from AMS0EPF00000196.eurprd05.prod.outlook.com (2603:10a6:401:5e:cafe::30) by CWLP265CA0372.outlook.office365.com (2603:10a6:401:5e::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:21:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by AMS0EPF00000196.mail.protection.outlook.com (10.167.16.217) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:21:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=bYojWlD6N1UOncef4EdWXDmvAf0qxfJlIMl/o7GUHlS5lDdB/5SgkwwczbusRxSkh5H61rldS8OKL3MRZEvph/f4/kC9Xo3UV3rkKuA9XhIJwWpsmiJyoytZqz1edQixeveX50PsAuHVmJ+tMNVstuYMj/DVxydVbQvKKNT8xuvY2TK9MloTp+3OF4DNlBbox4dO0CVd+RraSJVcczO82cKhNoE8REYvEV39Vt6nA6pAvUjS+9aWO13IMLP68oKV7jr+K65KENWUyD+tqoWbrG20rHRPbsL4ozXtiQ9HHA/6f06JIUxCUo3Y2SA/t0fFQsc8iCu9qdBNfRPDu2WLHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5j2Z1CDQsqAN0nSzpIjxlnzcW5Wpb/ynR2dZw3MJqJs=; b=DKJ33hBnzSKETuYPMZ+zOeH9ntAk53d+2gIM2fhKk4bg9JV9/RjhKbko7xji0Te+3MEKuLD6fP5ScP2klRWn2DmRQNkXsnZvy3/o/bZI8PRif2LuT7yPc0kTX+1OjeJM9iuQL5r29gdXrNn+SOZFia2oPuYyn5iiUTwW7AYIJEVyTcn7jC2CyXTIRcVmo9+h0xqNqDBNdkf6OoggkiekCnMWPzwQR0n8sawOoc60FQoWy4tu+/7oJCyD1fewqGQm8XT554taXyReCbFafds9GKJxuBhlQkdAMiW70DSQ4WbvXg6j8tKaTg9ZOATdJk0qXBabK8ZdKDVqEv4Jvt/Rbw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5j2Z1CDQsqAN0nSzpIjxlnzcW5Wpb/ynR2dZw3MJqJs=; b=kw0lWP2hSwLcON0PxfrXM35NHJQ6aMn+No6KlQpiOb5L6EVon0M1084AlHkbIvyPDOFm+xDZiqbI/ywv2TeJ+ohFJoH12z2OqBWBnT2ST9JRFPOzbPKkT65+1p2VG8/IrxnsvPgRMZJr1bJAkEGuDvRWG+n73PfqToOlwzYSa1s= Received: from DU2PR04CA0174.eurprd04.prod.outlook.com (2603:10a6:10:2b0::29) by PAXPR08MB7672.eurprd08.prod.outlook.com (2603:10a6:102:246::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:19:56 +0000 Received: from DU6PEPF00009528.eurprd02.prod.outlook.com (2603:10a6:10:2b0:cafe::1c) by DU2PR04CA0174.outlook.office365.com (2603:10a6:10:2b0::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009528.mail.protection.outlook.com (10.167.8.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:55 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:35 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:35 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 07/12] AArch64/SVE: Relax the expectations of the popcnt-sve test Date: Wed, 3 Jun 2026 15:19:19 +0000 Message-ID: <20260603151924.53706-8-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009528:EE_|PAXPR08MB7672:EE_|AMS0EPF00000196:EE_|GV2PR08MB8147:EE_ X-MS-Office365-Filtering-Correlation-Id: e26ff48d-d840-4b16-778e-08dec183b8e1 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|1800799024|82310400026|376014|36860700016|13003099007|22082099003|18002099003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info-Original: CU8Dsv8xtkLt6JJ9Y5Ff5BZ+4eYIzaLrG2yBlwDtLChxICZhsXB17lWtvn9buD6zta463yFxB8sOO0ONoMICXXfMb44fQVCxS9qnsxeIlt582DtvQ6diMJIOYfKyrKwNOmw1yi6JgSEetMIY+rpxAQBM7n5IydGtBwwtJMcLr9tQo6hTNsA3RAaZJa9XMXvhGft9b8WhYJMOFM9Qepxx4+6310o4BTDGcPUbzBW1dovwSnDQnH8crUOz9TXrBqZvwnbhfFA1VMYOoS+YSyz9iJouSH+EzUdHDy/9H9sS5rb6TEyjrmDc3/57voJQ1tig9XpT+OKOmyLJEb9oJuHY+kAkGBZxhUsv+JlqQlNn3z4J3b6TtPcSc9ogzb1IEyzaYVGIs+ATXiiADibXuKa4Wp6bzV7SmY7mbJC0Yr0cI1TPrzw+r/HwxqLh3Wng8LLx75JCuLQ/Qmm6puLFyJlD+Ud+5rFYLCI6Pu5Py4M4wDR2fpMFNtsLsCdOCmZHz7tYt41RBoRU/RWSQ1Z5gUEfAXvLh3stkyrfMmisS+JldHW7EGyOR7HowyDihg4j3Assq8vSxXU18xdhGOZSSq2TqXSbqrlclN2XGu6IuGJ0bcnubhWF5Ys8rTthcUONp2lJWtdjIic/n5dnee73Zoihw2yyM50ZgUtaR/5Ceti9gyV0h6kHRiI+Lu0tHyhgSdG4 X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(82310400026)(376014)(36860700016)(13003099007)(22082099003)(18002099003)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: HZnrJ6ex0TbYQUlgEmcGwvqLU8urZdwCrdS3QznVnGtddHlv99FqY0hbqfCy7INXhGenvTRxEF73n3cPcUFbSAfikYNkB2piNc4DmR9xAiuGajp/byKw5lMARnynyXn+21+n2NscwQWCerPoAIfLkIu5YF6T/JYgtMv2sj942R2Yo72g1EY4Y2hLorlWDzZagooYXSrZGSVXy3TuzNOPd8HMjdKH4CCalxGH4vvA9hqHqDEk/mOsteXIHnmSDrl8NH98Wi3geNAuupdAPCyoFZESTeR08jD9NKzywrduroUV1RWaFHtFcZi7Rwxs4cmVzSzd4H+94LV8xz2ZKS/Udg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAXPR08MB7672 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AMS0EPF00000196.eurprd05.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 369101cd-a3ae-49d7-02c2-08dec183918d X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|82310400026|35042699022|36860700016|1800799024|14060799003|13003099007|11063799006|18002099003|56012099006|22082099003; X-Microsoft-Antispam-Message-Info: acFnr6ObqaEXGhIgMnVscPQVP59G9j+pZunNHwygCGd+z26ug5vujEV2evbpWxnISVq6zTBF7eFy/mPmUMGIil3BXIQYw9rxueF1RyFmgs4+es53Dd5RiH24uotD5wgpMTZhJHYHgumXdQ459iBXzlW4j8XxoCSJSCHfmQopPbmNeghNux9RbD0wtFHqi3WatfOeJyd0Trh6js+L9Iy5ARWq6idpIfHJ57jqY+pB2r5HEgXZqKXQNJ3HZXzlDRFL5fbQRZAR0Fh5PTntzZf0fS5phNoU1TTyQoZk+YJX3eqQqvWdiyWqMlE7dockAYD2vnhSWkwobL29CYBpwTRRt+5z3N1OMGBs3q6qGyoRbvin1LkaCtxFjaGv1f+ejjFyhyqEymFh8SxevxKMOw0b0prMtlLhfN54Ik65CXMbLZ9zaLcRwgPChI8nEGm+Ji7UpsGRBcIBSi+l/1XFRDdce2INFUh0OHnfcjZD4mpCNhmOeoDscR6P2OFWB87C21pg/ZPqZzA33BYFs6uPUKhgPC8jMOTcXGc9muJrb2cslaj4rSQzpLKuJLK7yBGorMn+NrQ3ylezlJnfTSQZfn+WfHWB8p8GJ2YLSwzRPH+e0MnaaIUsHmKK8MynC3tkdmYxmKEm7Xg0ty6YZWX8oHsnOwE15FZfV6Gn/Ml4BunU1LFEsQJ84ujgg1LFbg6RwjBT X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(376014)(82310400026)(35042699022)(36860700016)(1800799024)(14060799003)(13003099007)(11063799006)(18002099003)(56012099006)(22082099003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ZcFKPh8shzEEJtNVjm7VL5FWV/nJsKKJ7pVchBLRiLq9MJ26CYqoxFTSf2KPMmsleMP3w5LG33Y1X8dy7ICbUA98OGgwx/VkGPoUEAi4GIfP7kgiHlmAgg6qLMNFkkvQ8bAsOSmARO2y/4/kuUxLGtdtKLlgSRXLqGAAvSjLtFvyCMb4Lqx5jcDdC5IEmJWaQC+iA+enMTUuSu0867vL7+LTMDU+TvoU4XNMpfZc33WXj/XR/PWMcWzkOapyiMYD7rUy1MubAYrv81t+qhPZ01op4npohzZVlNNpaMzyb7QCfYQeKMT1ugPfJzoqaNkHHTLkaqdk8vjRUi0PZK+7tf1i7rB5ihf6ZdsF1b7TWBXjFnQw5VEyCXFXcbpX4ozy2cYtf5//nTw9zVTTzr2pCiYmTTAto1DrHDbSj1l2Aumni+uQIJ9Ho947d79dP6jY X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:21:01.8990 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e26ff48d-d840-4b16-778e-08dec183b8e1 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: AMS0EPF00000196.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV2PR08MB8147 X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_BLOCKED, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org When predicated tails are enabled for basic block SLP vectorization, the assembly language generated by GCC when compiling popcnt-sve.c will change. Relax the regular expressions used by this test in preparation. Currently, analysis of f_v8hi succeeds with vector mode V16QI and the following GIMPLE is produced: vector(8) short unsigned intD.19 vect__1.18D.4648; ... vect__1.18_69 = MEM [(short unsigned intD.19 *)vectp.17_68 clique 1 base 1]; vect_patt_60.19_70 = .POPCOUNT (vect__1.18_69); With predicated tails, analysis instead succeeds with a variable-length vector mode and the following GIMPLE is produced: vector([8,8]) short unsigned intD.19 vect__1.18D.4649; ... slp_mask_45 = .WHILE_ULT (0, 8, { 0, ... }); # VUSE <.MEM_25(D)> vect__1.18_46 = .MASK_LOAD (vectp.17_44, 16B, slp_mask_45, { 0, ... }); vect_patt_36.19_47 = .POPCOUNT (vect__1.18_46); When lowered to RTL, the WHILE_ULT is replaced by reinterpretation of a V16QI as VNx8HI: (insn 7 4 8 2 ( set (reg:V16QI 107) (mem:V16QI (reg/v/f:DI 103 [ b ]) [1 S16 A16]) ) "gcc.target/aarch64/popcnt-sve.c":33:8 discrim 1 -1 (nil)) (insn 8 7 9 2 ( set (reg:VNx8HI 106) (subreg:VNx8HI (reg:V16QI 107) 0)) "gcc.target/aarch64/popcnt-sve.c":33:8 discrim 1 -1 (nil)) A mask is still required to lower POPCOUNT, so an all-ones mask is synthesized: (insn 9 8 10 2 (set (reg:VNx16BI 108) (const_vector:VNx16BI repeat [(const_int 1 [0x1]) ])) "gcc.target/aarch64/popcnt-sve.c":69:8 discrim 1 -1 (nil)) (insn 10 9 11 2 (set (reg:VNx4SI 105) (unspec:VNx4SI [ (subreg:VNx4BI (reg:VNx16BI 108) 0) (popcount:VNx4SI (reg:VNx4SI 106)) ] UNSPEC_PRED_X)) "gcc.target/aarch64/popcnt-sve.c":69:8 discrim 1 -1 (nil)) However, this mask is not the same as the specific-width mask currently expected by the tests. gcc/testsuite/ChangeLog: * gcc.target/aarch64/popcnt-sve.c: Update test expectations to allow both current and alternative valid mask specifications. --- gcc/testsuite/gcc.target/aarch64/popcnt-sve.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/gcc/testsuite/gcc.target/aarch64/popcnt-sve.c b/gcc/testsuite/gcc.target/aarch64/popcnt-sve.c index c3b4c69b4b4..117a5ca8f1b 100644 --- a/gcc/testsuite/gcc.target/aarch64/popcnt-sve.c +++ b/gcc/testsuite/gcc.target/aarch64/popcnt-sve.c @@ -4,7 +4,7 @@ /* ** f_v4hi: -** ptrue (p[0-7]).b, vl8 +** ptrue (p[0-7]).b, (?:vl8|all) ** ldr d([0-9]+), \[x0\] ** cnt z\2.h, \1/m, z\2.h ** str d\2, \[x1\] @@ -21,7 +21,7 @@ f_v4hi (unsigned short *__restrict b, unsigned short *__restrict d) /* ** f_v8hi: -** ptrue (p[0-7]).b, vl16 +** ptrue (p[0-7]).b, (?:vl16|all) ** ldr q([0-9]+), \[x0\] ** cnt z\2.h, \1/m, z\2.h ** str q\2, \[x1\] @@ -42,7 +42,7 @@ f_v8hi (unsigned short *__restrict b, unsigned short *__restrict d) /* ** f_v2si: -** ptrue (p[0-7]).b, vl8 +** ptrue (p[0-7]).b, (?:vl8|all) ** ldr d([0-9]+), \[x0\] ** cnt z\2.s, \1/m, z\2.s ** str d\2, \[x1\] @@ -57,7 +57,7 @@ f_v2si (unsigned int *__restrict b, unsigned int *__restrict d) /* ** f_v4si: -** ptrue (p[0-7]).b, vl16 +** ptrue (p[0-7]).b, (?:vl16|all) ** ldr q([0-9]+), \[x0\] ** cnt z\2.s, \1/m, z\2.s ** str q\2, \[x1\] @@ -74,7 +74,7 @@ f_v4si (unsigned int *__restrict b, unsigned int *__restrict d) /* ** f_v2di: -** ptrue (p[0-7]).b, vl16 +** ptrue (p[0-7]).b, (?:vl16|all) ** ldr q([0-9]+), \[x0\] ** cnt z\2.d, \1/m, z\2.d ** str q\2, \[x1\] From patchwork Wed Jun 3 15:19:20 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136413 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 705B74BA2E23 for ; Wed, 3 Jun 2026 15:34:28 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 705B74BA2E23 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=DBhi8Ibj; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=DBhi8Ibj X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from DU2PR03CU002.outbound.protection.outlook.com (mail-northeuropeazon11011053.outbound.protection.outlook.com [52.101.65.53]) by sourceware.org (Postfix) with ESMTPS id 6BDC24BA2E38 for ; Wed, 3 Jun 2026 15:21:10 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6BDC24BA2E38 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 6BDC24BA2E38 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.65.53 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500070; cv=pass; b=gb0eTeTT3m7PaOmljo6z4lR4PRPnrRRrjl7ZIa+iNendCndwvAGD/9bFa9/3HuiO/YLmH8NQQoeyAQ2GiwPyvpwJCAASSC+ms4u7LyLbpLOp4BuJugKkUf6j6WKOlrbWe/+if3AA/eSLzYRWfya2nh1+3XLN1miOIzIierd/NHM= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500070; c=relaxed/simple; bh=XQCGLoi1qKntUT8OR6JY3n8WF6ZZ7iIhi8fVJ28dgMg=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=jP0of94eB9HZC4c1oKIJFoYoe6x+XcFonkWkIF+5Z1k5hS/JmeFiGtlhyVd8KRapXzmcm0/HW/74iJBh4GR9CnLvKPY4SKkx0QlH6jSJ4XZVaQ9V6KGLvUDRzzlOWry882xXk5zXs6qY0ObG1gk3b9i2a6H/1pKps+ZOEToo3RI= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=DBhi8Ibj; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=DBhi8Ibj DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 6BDC24BA2E38 ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=N+hsy53OmrV+iUl7b2MSk5q5kzJqsMifqPInTu3tgXWaq4aX25M1SmUaksuMmYix54R72extm3AEGmdwHnQFNkG8YlM17SxtAARpQPldjuB3Ng+oMBpFsmB5biMlvmkmBcyAGDhMkAvDNY/JZcoch+mEa/iNnvbjk3UUIgRZeJvuLP0XpCn+/rfzwPZ4qnhYo9nTH9MdLIRaaTkmDCB9tOMXbDWH6dlVd8P3BzxM23EGctOBHVViuV0nz6DeJG/iDyBnt1B1LKFRZWdHelMTFYRXDCwsdausu+83hJc86UHRbhaH8sho+s+KXfGRQqQNMOVHDV7fPD1+vTBxY65g7A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eu1MhsBAJeIwooclwULT1d9fe+DvUQ82K0cetPK6nrs=; b=wQDh828INh1vIWG5jEPS+nBeYGWgzD0YwuzhA1k8bKRyjbQeTLK0FWR0N2BOzu8pLblPanO+2KtmChBlC3Lo/CGW7EDmz3NgaUrA/WFIw0PBi5ENFsxN0ZR4sgQ+PPkbeKaaCxfcQq6iSzdT9emxulXesDKevWBDCk8nSdaSmYxCO8hij2jrbYzOOW+ji/jojmRECFlMT47b1sQ7krhu/Vb/amTdhrtiXpmIURKMKz2zOZpb157oP4Daar/fTjGUQYuui8hnp//BIS+mt2mxEHK8en2pyR3QX1VVVewzwO4LKd1UN1S5hWjAHCqor2APEeNVaLWjg1wB81ttR/GKrQ== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eu1MhsBAJeIwooclwULT1d9fe+DvUQ82K0cetPK6nrs=; b=DBhi8Ibj0DAG9jv+AYPCAAXD2XxkHO9QRKGMSmm3DxY6SnZRV/ZAjw5pa8SVl8G7fLaMshEJDHTsmgHIZLvu1I0EhZaMDQjVf9JF7d0M5s+LW0nbcZrUZiD6Rm29q2Aw6UWOOj2nZ5vf7KhWwF3TL9FG04TWFgdFjH8gFDql6BY= Received: from AS4P250CA0026.EURP250.PROD.OUTLOOK.COM (2603:10a6:20b:5e3::14) by GVXPR08MB11227.eurprd08.prod.outlook.com (2603:10a6:150:289::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.17; Wed, 3 Jun 2026 15:21:02 +0000 Received: from AM1PEPF000252DA.eurprd07.prod.outlook.com (2603:10a6:20b:5e3:cafe::89) by AS4P250CA0026.outlook.office365.com (2603:10a6:20b:5e3::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:21:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by AM1PEPF000252DA.mail.protection.outlook.com (10.167.16.52) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:21:02 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=JkUjwY2CJcwoTsgeXId55Z7i+xLPgyBV10PCc6/jzLdn0HSCtTxLrOKP5m9eiguxI6hwlbXWzgcUey5jgEnUDPrsPwQjpD7/MmdULn4rXHGQybHXQOiul7Qs0/uTTYda75jQVu5XIi7+0Y/gH6KZqULTADpUM1bkU8vvRLu2RgqwpacXd6xH3FP/rEqYsL3ornhqq1awfrGcxA3XnQY8n7ejd7Acyycmfj6Mpp2xNuqxJRWf2n2SSTItm7gyOuDxXLwzlIoPGckqgYVwUYewjk8mnkstqvqcpc28KzN4fXOajfSBInA2PMHXLC5AhooyxsHIItICVyzpSJvOEwQ66g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eu1MhsBAJeIwooclwULT1d9fe+DvUQ82K0cetPK6nrs=; b=CpwfaOoumUWcR1zWM5yMxzZNA45axwPOfuN6yHFKqClpHCtnlISXt91Zn/Bes8GDdi27mzL0z3wtPZclaeTmT6VwEJzp0AXGHVdmrExc0LKE4B8+mZT9pnb1bkqoGWE6YDdbBNy/lh2ekg89/8Za6yw2SDXUqOdu61xf87epLxi1m3F1UnZc7MkxjNuZvfq4Nsgy/O4MAI8l2r3TwbaSB2z2RKAHb5x6GooTZljKThtpsUuKPyo6WFOZmaPLRcQGAsUpHVlzXI1kQD/+LmjOwI0MLTwa0esnefbewILBpszJHa/hubO9tHjqP06WXLTLQ7XQqUwVpnadAs8Jlvu58g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eu1MhsBAJeIwooclwULT1d9fe+DvUQ82K0cetPK6nrs=; b=DBhi8Ibj0DAG9jv+AYPCAAXD2XxkHO9QRKGMSmm3DxY6SnZRV/ZAjw5pa8SVl8G7fLaMshEJDHTsmgHIZLvu1I0EhZaMDQjVf9JF7d0M5s+LW0nbcZrUZiD6Rm29q2Aw6UWOOj2nZ5vf7KhWwF3TL9FG04TWFgdFjH8gFDql6BY= Received: from DU2PR04CA0151.eurprd04.prod.outlook.com (2603:10a6:10:2b0::6) by PAWPR08MB10135.eurprd08.prod.outlook.com (2603:10a6:102:35a::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.71.15; Wed, 3 Jun 2026 15:19:58 +0000 Received: from DU6PEPF00009528.eurprd02.prod.outlook.com (2603:10a6:10:2b0:cafe::50) by DU2PR04CA0151.outlook.office365.com (2603:10a6:10:2b0::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009528.mail.protection.outlook.com (10.167.8.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:58 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:35 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:35 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 08/12] AArch64/SVE: Optimize vec_init for partial SVE vector modes Date: Wed, 3 Jun 2026 15:19:20 +0000 Message-ID: <20260603151924.53706-9-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009528:EE_|PAWPR08MB10135:EE_|AM1PEPF000252DA:EE_|GVXPR08MB11227:EE_ X-MS-Office365-Filtering-Correlation-Id: 67f6184e-ec34-48e1-7df0-08dec183b920 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|82310400026|1800799024|376014|36860700016|56012099006|22082099003|18002099003|6133799003|11063799006; X-Microsoft-Antispam-Message-Info-Original: gHJlGpXMbhca8nRs9ij42E6zPQxmGknOuohplE8ti72qFNlGdOfh4Ozy/jsK7ST3xebjqk4Q0Iu66EKBCotCfuTS/mclQWCgSX+Hd7KC7pt0Iwqh3cjMwe1A3OazPza2+HY3hmXpfeAzpMsTKAgYsQoceiW1pzZYn37cr8/ZfNNaxgstO+/XJ/dN9m8Ou+iVFEqn6D+0pJ/ktlDTQO9sydUB2/lqf34UvWWiJGgE3OwbnpcdRMmlRLyjK4r9p8IMbgmLLOy8kS5UG3bx8h/9lpP0mi/ZkpgefeCC3lyxoMNU5EOblasDkpAkTR/QV32EpfdPNVjokzXWvi19D60oIOkx3rXiR5tuOTGdyRBw24gCaqc9dY7Z7K78BRogykbx3WhT3OK3T4tcOgyO+LDBQxQ2h9i0XRelgnG2ktnXFcKfVlgSMaM/W46OtFhRFCj8TK4Qb+TpnZScazssfoBLCe9hqru89D9LhRmvDqvaKEgMKl4DRveJ7rQ4qVpJqxTtZ3fOD4oje8OfKLhFhS4DjloneclYy6S4UcUXFRHJFQI/Zs9xe/GM4gFdBA/6rzZUl/DSM/Ej0C89QsydIPuSOhn1VMS4Y/h+t1/NHGEuLyLRkHkZ/wZX/YyOOh/tEosChBiQoBrhSstaDIQG1upY+RgZ3/Q8LO77ufGslR+xqCl8EcyVTqlldFwRgQZLtrXJ X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(82310400026)(1800799024)(376014)(36860700016)(56012099006)(22082099003)(18002099003)(6133799003)(11063799006); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: Rm5J+kkwDOzoDVqEb4Av1520i5neKjOIMUZJUESNHGWONhkldSoRxvAo7lqn5Q2tGys55nPWi0ofnKl6qRdmDFbdTS70o0JUXUK+F1n/NKtpZwbWKZP55tvUACqKxadEgYhE/FSlUcXruVznPPFewRTYGuzuYwyjJowJ2EIoVs/suANoHRhtChiuwKLCz+Ug5lGZRU1RDoL+cBm4lzkK6Uhz6ErBCChrRuxCdaJoP2KBj2qYcE1TTBgm/3+f+cPDSkerbQNKWksSkqGgy2VESF9n72b6Id6zRUW6b61bZF6LUmQda6sb7sO4rQEL2PPq+jYoDyzxkwEoUfWXnxzvsw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAWPR08MB10135 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM1PEPF000252DA.eurprd07.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: ea277d63-387e-403f-32ee-08dec18392cc X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|36860700016|82310400026|14060799003|1800799024|35042699022|56012099006|11063799006|18002099003|6133799003|22082099003|13003099007; X-Microsoft-Antispam-Message-Info: K3tA6eQnkFBIk0b55ymcBOsuaoA80QAXchEeKWkdbOEgBQffpU+L36RgqM3764OTePPe0lKz+PZkIFSZ1CkCyO5+IR/xBTvMHUdta/xnqk6aSKgf+ygwqB5479WWOQWk+zr8uYEsgU9Gb5XDcl6kjptqNmVkRkXiBQBkLch32/jXEHIa5aw64jsSgchdNwtfi37fiLf5d6tI5eBj0YcAQUI1u6ZYcsB0kErR6CoEOvQ8xVLb+qI8MUh7ZqNa+F2cdi3BQUP6NNZUIbZHJyoqtZdDpp+JVzSwsJB6GpkxQr44EHdiCmYKd7qH0Oc5bxnLErA/Z4LVym+ND2F0LjB8zvLCE3XMvyOhsKoQo0GpzDr/HSfPuhfbalI56vHtWyL5/1TYgSelJ+Gvqg+QbyCAqn/NVKTPmIjcuIdGApd/yhd5GhLKHwp1/bElpuos+ma4uQoHUW+jcWDKab8F3KbcZVbrrGXTufT3f/V8IHe0kjtPcDmtgDQVPruUsBfF70y0+CnRhkqRTHh2d8yyiBjpJWcpx1lyE0B/XkYYmuv9xZ8ndGhLh7EJRSMCojb8fCpD9nYwwKzLEAEScinLsfa3gWCkCfosTkZdRjUizWMa13KNsRhv9SmVRtW8YrRm0x0EHUc6Ly+NJ43ZNh5O6CWO9mE1vhFdqU/NXdhct0Vi6r9I+/Na0cZdykSTWVggO8KA X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(376014)(36860700016)(82310400026)(14060799003)(1800799024)(35042699022)(56012099006)(11063799006)(18002099003)(6133799003)(22082099003)(13003099007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ac7Tjm9ZUdN1wAgaLjMmDCuuVrLdEzGzPtV3cfcBUh8bKnu9O9tW+oR7FqJRamP27rtL2xv2s9FhUF7ImCquAR9uJ/gLr5L12S8KTRdt0VGfkF/XK9Z/y7x0BZKHQwAE7mzJhx6XeRzh+ftvJdbHCt4tVGrObyWK1UwprWfYiOefmWrsCSx/cqGUNt4i7+tytq38ZtAgACLGPkweV6BqLB3iNU/gtCmzthA8ChdkioMlMI4gZ8oo2gEopKFwNyS3xis6OzizFQ/fziSnNoAkxFae53wB7wzuEQTAceA/vr8HVZKOlT/fUhcj0j3/j31SIg6fAN6OkLGYM8XxTOBblhDxr6bxJycwmH7CWyGUFcI5vutDtgIijKvRgWwPn2NkpGkrKF8Wp0goaOUrK/DQxpHnWyxLV26yZxEugTkf3BmiOmnjxARJ+24ZkYjzDjCR X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:21:02.2923 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 67f6184e-ec34-48e1-7df0-08dec183b920 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: AM1PEPF000252DA.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GVXPR08MB11227 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_BLOCKED, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org When basic block vectorization is extended to support predicated vector tails, it attempts to vectorize more stores. This is only done if the cost model deems it profitable, but the cost model assumes that vec_init is cheap; in practice, that was not always true. For example, uint8_t * vectp.2689; vector([4,4]) unsigned char _846; vector([4,4]) slp_mask_848; ... _846 = {_20, _30, _40, _50}; vectp.2689_847 = src_61(D) + 64; slp_mask_848 = .WHILE_ULT (0, 4, { 0, ... }); .MASK_STORE (vectp.2689_847, 8B, slp_mask_848, _846); was expected to have a vector cost of 4 (the same as the scalar cost) but the code actually generated for _846 = {_20, _30, _40, _50}; was a repetitive series of write-modify-read operations using four stack locations for temporary storage: (set (reg:VNx16BI Y) (const_vector:VNx16BI repeat [ (const_int 1 [0x1]) ])) (set (mem/c:VNx4QI (plus:DI (reg/f:DI 96 virtual-stack-vars) (const_poly_int:DI [-O, -O])) [0 S[O, O] A8]) (unspec:VNx4QI [ (subreg:VNx4BI (reg:VNx16BI Y) 0) (reg:VNx4QI 205 [ _845 ]) ] UNSPEC_PRED_X)) (set (reg:QI Z) (subreg:QI (reg:SI X [ _W ]) 0)) (set (mem/c:QI (plus:DI (reg/f:DI 96 virtual-stack-vars) (const_poly_int:DI [-O, -O])) [0 S1 A8]) (reg:QI Z)) (set (reg:VNx4QI 205 [ _845 ]) (unspec:VNx4QI [ (subreg:VNx4BI (reg:VNx16BI V) 0) (mem/c:VNx4QI (plus:DI (reg/f:DI 96 virtual-stack-vars) (const_poly_int:DI [-O, -O])) [0 S[O, O] A8]) ] UNSPEC_PRED_X)) (repeated four times) which compiled to something like: addpl x5, sp, #6 st1b {z27.s}, p7, sp, #3, mul vl strb w10, [x5] ld1b {z28.s}, p7/z, sp, #3, mul vl (repeated four times) With these changes, the compiled code is instead: mov z31.b, w0 insr z31.s, s28 insr z31.s, s29 insr z31.s, s30 which is not yet optimal but is a great improvement. To achieve that, "vec_init" was modified to accept all SVE vector modes, which means that the associated function aarch64_sve_expand_vector_init must now handle all partial modes (namely, VNx8QI, VNx4QI, VNx2QI, VNx4HI, VNx2HI, VNx2SI, VNx2HF, VNx4HF, VNx2SF, VNx2BF, VNx4BF). I verified that the following dependencies already handle partial vector modes: - "@aarch64_sve_" (for ZIP1) - "*vec_duplicate_reg" - maybe_code_for_aarch64_sve_rev I did not verify that emit_move_insn (which is a dependency of aarch64_sve_expand_vector_init_handle_trailing_constants) handles partial vector modes, but it seems highly likely. "vec_shl_insert_" has been modified to accept SVE_ALL instead of only SVE_FULL and operate on container instead of element types. gcc/ChangeLog: * config/aarch64/aarch64-sve.md: Update vec_init and vec_shl_insert_ to accept all SVE vector modes. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sve/slp_stack.c: New test. --- gcc/config/aarch64/aarch64-sve.md | 16 +++++------ .../gcc.target/aarch64/sve/slp_stack.c | 27 +++++++++++++++++++ 2 files changed, 35 insertions(+), 8 deletions(-) create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_stack.c diff --git a/gcc/config/aarch64/aarch64-sve.md b/gcc/config/aarch64/aarch64-sve.md index 585a587d8cf..6750428255c 100644 --- a/gcc/config/aarch64/aarch64-sve.md +++ b/gcc/config/aarch64/aarch64-sve.md @@ -2959,7 +2959,7 @@ (define_insn "@aarch64_sve_ld1ro" ;; ------------------------------------------------------------------------- (define_expand "vec_init" - [(match_operand:SVE_FULL 0 "register_operand") + [(match_operand:SVE_ALL 0 "register_operand") (match_operand 1 "")] "TARGET_SVE" { @@ -3003,17 +3003,17 @@ (define_expand "vec_initvnx16qivnx2qi" ;; Shift an SVE vector left and insert a scalar into element 0. (define_insn "vec_shl_insert_" - [(set (match_operand:SVE_FULL 0 "register_operand") - (unspec:SVE_FULL - [(match_operand:SVE_FULL 1 "register_operand") + [(set (match_operand:SVE_ALL 0 "register_operand") + (unspec:SVE_ALL + [(match_operand:SVE_ALL 1 "register_operand") (match_operand: 2 "aarch64_reg_or_zero")] UNSPEC_INSR))] "TARGET_SVE" {@ [ cons: =0 , 1 , 2 ; attrs: movprfx ] - [ ?w , 0 , rZ ; * ] insr\t%0., %2 - [ w , 0 , w ; * ] insr\t%0., %2 - [ ??&w , w , rZ ; yes ] movprfx\t%0, %1\;insr\t%0., %2 - [ ?&w , w , w ; yes ] movprfx\t%0, %1\;insr\t%0., %2 + [ ?w , 0 , rZ ; * ] insr\t%0., %2 + [ w , 0 , w ; * ] insr\t%0., %2 + [ ??&w , w , rZ ; yes ] movprfx\t%0, %1\;insr\t%0., %2 + [ ?&w , w , w ; yes ] movprfx\t%0, %1\;insr\t%0., %2 } [(set_attr "sve_type" "sve_int_general")] ) diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_stack.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_stack.c new file mode 100644 index 00000000000..76be816e0d6 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_stack.c @@ -0,0 +1,27 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize -mcpu=neoverse-n2 --param=aarch64-autovec-preference=sve-only -msve-vector-bits=scalable" } */ + +#include + +/* Without an efficient implementation of vec_init for partial SVE types, a + decision to vectorize a group in the basic block vectorizer can result in + code that repeatedly stores a whole vector on the stack, overwrites one + element, reloads the whole vector, stores it to another location, + overwrites another element, etc. This is a fairly minimal reproducer. */ +void +vec_slp_pathological_stack (uint8_t *src) +{ + int lt = src[-33]; + int l0 = src[-1]; + int l1 = src[31]; + int t0 = src[-32]; + int t1 = src[-31]; + int t2 = src[-30]; + src[64] = (l1 + (2 * l0) + lt + 2) >> 2; + src[65] = (lt + t0 + 1) >> 1; + src[66] = (t0 + t1 + 1) >> 1; + src[67] = (t1 + t2 + 1) >> 1; +} + +/* { dg-final { scan-assembler-not {sp} } } + */ From patchwork Wed Jun 3 15:19:21 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136415 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 85F904BA2E08 for ; Wed, 3 Jun 2026 15:37:45 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 85F904BA2E08 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=aW9FdkOH; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=aW9FdkOH X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from DU2PR03CU002.outbound.protection.outlook.com (mail-northeuropeazon11011035.outbound.protection.outlook.com [52.101.65.35]) by sourceware.org (Postfix) with ESMTPS id 433C04BA2E3D for ; Wed, 3 Jun 2026 15:21:14 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 433C04BA2E3D Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 433C04BA2E3D Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.65.35 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500074; cv=pass; b=iybh93cVhGUmX3odF9eUrMEFK4tYSJLL+9FV7tDSUcz6311ux91vfQlu8/BgEHvoiU0VVqYuZ23loK/mLIrAu3rFdOlF3s+1cRR0OGaMgDlNveIPr/AD6YPdKcuGRbVW2S9Q2/N4EnoF6p+hURgu8ODAaNBCFYsE9nP5pNSSKiE= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500074; c=relaxed/simple; bh=MrgcI0QtFu8oB8tZPcEumJivBd24SYpEzHsji5LOdJY=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=wgcNqWFZnw1SVd8wX4P0WuFsAa/cv3OQB3QZW28GMYokGxk1m2NPglj+6g/icBr9m2vkEvSBvghDjg4udae83B9OZugbkg+VQ08UaA0fKaxI2BbnG+HI3Rn1zes+Kvbvldp84WjvLGj/ezdVM6vgZJhGeveniXObXpdFoWEoNNw= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=aW9FdkOH; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=aW9FdkOH DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 433C04BA2E3D ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=P8llRsSWW4dDB/DWVy2t60GHJlQl+0IGuRwBFEFBzlUTYwckUgtE15Vd/HbcOXoKpLXxLSosuLri9iAHXOiCQKusTeouX1BUYO3Fs2mSsuc2YFodjvasc9EPNyjoJL56Ap+oj8bZ1iIJK9FZbQbQkqp74NRBN+GuXx3MjuErnP1aAmVxff3TAT1oAEc4i+riCIs/Xrlz022cBsutumMnPqo8hydB4NlDbQ3X5i8zjXFOganI9SIMCweRuBQQAufJd6v63mbpb7zQGSnNEs4tQBOSDHLOXzSSITScagE0vI+KnCxL4B/kZF774qjY1newPOEk4Zp5j77lS+W4JnxLIQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oGaRm3vmKuE6i5VJl5Ag5UttgYexWqWCNBJaVG4Ju3Y=; b=j5K8lTH1CmaqFI2gUFA7ERLZALsvVj7XtSF/cOel6HEU4HbjGRO10BDbNOAfVR1CZzHrQRQb0kt49mBLea6jsDh3Kd4uTHK3kAfFPZ7Of0KOLSSDIcwozEZkLYKNjlKHoHP1dST2SzHzPpDINiBLzSaNH1h0Pd6lDno/ehgILBisRDmWGhMgKxjTj6WyrdORO+aa1dEKtiOe3hfMN7EkLqMsVPItWKcraId4ac8SwD0vBxCaHoikZR3Cjw0T4CD59Z4I8opGM05G34vFf2Q/fXZxMH2nw6JdXUKbvkTr/QZvP3+/gTrq5kT6CSak2kJLTkT5R5WLAFqRBb8mFFa/hA== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oGaRm3vmKuE6i5VJl5Ag5UttgYexWqWCNBJaVG4Ju3Y=; b=aW9FdkOHxvO7AqDfY+NORitQt9q1vJNXK/Gbm21otl7uRgZhkOoaqT2ZvM6PgHNm4DPkchgjirqi0UY4JY9w6xSfNVAWmT2D6jo4IZZqETlKE/HwIebB4LNl4evMYioRlswl86JRvVW0Q8eNrj+1XwiW8x1q2Ia01V2R4EpMHlw= Received: from AS4P190CA0012.EURP190.PROD.OUTLOOK.COM (2603:10a6:20b:5de::12) by PAVPR08MB9881.eurprd08.prod.outlook.com (2603:10a6:102:32c::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:21:06 +0000 Received: from AMS0EPF00000191.eurprd05.prod.outlook.com (2603:10a6:20b:5de:cafe::50) by AS4P190CA0012.outlook.office365.com (2603:10a6:20b:5de::12) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:21:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by AMS0EPF00000191.mail.protection.outlook.com (10.167.16.216) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:21:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Xn4kyqwNtC0JxtJeSXbZdXICZr4aEMkCxSMozupAaKNqxDU5gdTy/EsHKT3x4f61CXFfGRUxD9Cr0XbcegphVolaGEWlxdVnhBpl95qnjV/ycwrTjRIunBYVnLVpLZ3wi1K91MNtM8Ad3B+X/INzPb9zNEa3FZ9fHeumctMHGkgu/OkD+iDh2oMjStK4xbOqgPwlnaMrC7+S3hcJzfgy8wYGqTGPgbA1ljlVz8jBmSDUAQpG6ajSqP2SaDBLFd4L+B8IGw7M7lHS0Axe1RghEUQWQZrG03IfKr1t8uTLNhxbuly/N69YtYyda1lUGgyrUZUGHmGcQkXXoPp4KcNHHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oGaRm3vmKuE6i5VJl5Ag5UttgYexWqWCNBJaVG4Ju3Y=; b=DBmRVDAcUTpnZ07nmrcIQs4iyyQ5OgCscSjwkBURZ1lpsm9KFI834VFEGFndbdkPuMW0QwTXj7gZMp3i7YF7KyYyCY9i+lFhahwcrZenz7btcAB2nqBh6tcWqKd4/tGwaWxb0oHprHSzT4Q6HSdAH1brW0h+hKPuoH86KGTy646CKJ+ZHg2AmYtY7DPAQ0vLQ/88iDY9niUpBjtRLsQKZbDFRgxmffCnLq1Fsgwy/31ZFm23EP2KS9dqqyE6v48v6av/fBsmxankdxWALhEs7SuPgaBecCWfAv8Jes+wq/+z6sMWlCijnj/K2o7B41sBDTJfOIKaFIRtDdOOzxvdrw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oGaRm3vmKuE6i5VJl5Ag5UttgYexWqWCNBJaVG4Ju3Y=; b=aW9FdkOHxvO7AqDfY+NORitQt9q1vJNXK/Gbm21otl7uRgZhkOoaqT2ZvM6PgHNm4DPkchgjirqi0UY4JY9w6xSfNVAWmT2D6jo4IZZqETlKE/HwIebB4LNl4evMYioRlswl86JRvVW0Q8eNrj+1XwiW8x1q2Ia01V2R4EpMHlw= Received: from DU2PR04CA0172.eurprd04.prod.outlook.com (2603:10a6:10:2b0::27) by AS8PR08MB8993.eurprd08.prod.outlook.com (2603:10a6:20b:5b4::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:19:58 +0000 Received: from DU6PEPF00009528.eurprd02.prod.outlook.com (2603:10a6:10:2b0:cafe::75) by DU2PR04CA0172.outlook.office365.com (2603:10a6:10:2b0::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009528.mail.protection.outlook.com (10.167.8.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:58 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:35 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:35 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 09/12] Extend BB SLP vectorization to use predicated tails Date: Wed, 3 Jun 2026 15:19:21 +0000 Message-ID: <20260603151924.53706-10-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009528:EE_|AS8PR08MB8993:EE_|AMS0EPF00000191:EE_|PAVPR08MB9881:EE_ X-MS-Office365-Filtering-Correlation-Id: 58089bf7-cc3d-40bf-1ee1-08dec183bb8e x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|36860700016|376014|1800799024|82310400026|56012099006|11063799006|5023799004|6133799003|18002099003|22082099003|3023799007|13003099007; X-Microsoft-Antispam-Message-Info-Original: e6KaheFk0Sr4d2bwAUO3pGojbhYTWvjyQ1rfCeF/fGY4LpNnd6V5npFuh188Y5sFHoZIOwioDJZG5oovufAyTPzlA1CRUZcHAyu28qeetGSHrqieN2jnFmBHhdaBX4xieOBMk0Ci+Xyf1HS4BUM1d2X0IJh2NYqMrbtL6C/sqIcORLHShF6jFlSka36p0UfLYV2Rnu82/u0HQuLabLZfgRfWs5ijYChInERvDA6yDRfdb35N0HG3AKSg7/PJqMSPXHexxZR8EcTElsPItsW/+QgS/6R1QtFc5fXEopKs4MDCm95iMMe1HrEps+tY2l8TXKfoEwWwJSvURPGzA2KVhQzBtSFHvF12LgRyeo5yA1S3cGzupGDNFPIn2Vua3SG2Z2J4Nty1mURVouy+KotcQWgJaR+DfFK/DfZ4t7REBN2bUbim1yNZ9B8mEAMsLBDLvjUOF1drGZJNusDv3UU7gMU0s1SD2fU0UxM/tJ09LYuG0fIZpZ8RkaG1mIcVFIaNwylLLKiUtbR0ZGb/LaO0m7HgUvMVYQdT4h49Bsy3SsX8a96knJUaAXQt+mdO9XTnu9MkHEt6eP6WChqphlz4gjjM4eMQZrGEUDc/sV1vFm4ePyP8qPz80f7PZxOnKbXUyfbSdTPMa4iU2Q9s2/2IIb+TyI8JR9qOwahYeeBpagcVEpsR5vpqXJRm2ps35OXE X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(36860700016)(376014)(1800799024)(82310400026)(56012099006)(11063799006)(5023799004)(6133799003)(18002099003)(22082099003)(3023799007)(13003099007); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: kCA42eJB25Ny20ArgxUcQ1rm7YB5Tv7ME3qqrb25Zx3JXGt3tWnZpLYQcKkNgrYhKiH16WVP5jknVcDfieKVdX8IwMc5RFlNTSXnLv7Oa/0AIsWbzoVO+LUbQrq16QIddOxf150/x3YIrCNiLuN0fYfyIiW5PtXBkEqWktkAMQ5Qn9W76C4pCkSXF8QrxK5VAufLz4tWkuU6JsiIRc/t2IaFnqxkegbKHYzhsbd8ab5WUTOEuxvXkRB23Cg1x7qJ9J66ySSkWrWon0VOV13Ac7fzySt1EeojWekLRezaqlRI9a+78G9a/cJkoXNPXpC0r8YIhs+/PfljeMqK2c+HFA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB8993 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AMS0EPF00000191.eurprd05.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: bd20a2b0-8c0a-4118-ec0c-08dec1839309 X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|1800799024|82310400026|35042699022|14060799003|36860700016|56012099006|18002099003|22082099003|3023799007|11063799006|5023799004|6133799003|13003099007; X-Microsoft-Antispam-Message-Info: Vp7Gtu+YhNMDKgdethH41w2/GcSH+jaP4EQRoJzEPVhM6Vjj8VVfvwGFP1gDVOuj4Qcg5O6LktksrfZxr3tkuuvn7SyDE/X8QXPR4V37iQ18QwadRMswcbV+h982W55QIq3MyvLhhkwNIn2gQEtFGraoN5/ojJxSt/Ror9ZigOuyG2/5NwbNJR8+VWUhbAUG99cPl4FFZKJtT7pCvpOyT9WJSQAF65POpkFu+j5WI22YdxDhxMDgNtf8KEff5fU6wbFykebFQ/FeZdnnGSg5A5ClobW4mGR0iAXPa3dNEPML1BIqzrDuOt3oMooU0uQ12AxUygbYqjU7J93nUHCB3auPMeSiXykesHgK5qEdYYexGhV3Q+n8r4TXo6IETBhBJ0EjZi0sb9N/RmZC1aEpKwwvnQhvQG6L0ApRskXYyPsrS5J7WPMQSdMd7pxPrQwtbp12HggtUyzLJq2p8ztJuFTnfwv25sGTDJ7Ft7o4+yG87JKEriW7xQqeBMU7ShgfCfra4o/v83mhKu2eEU+/tCVYKi+yOUonOwgtzlFUNjhvoUwZ6IQFx2ovKdufS1pAXdx4vD6tnYEwboGM7H8cW3PKfKABENAhoXwCS5OOU4jVh3bc9F+HB/6rpwUcBkylcUh1ku4o//fcH0hnW9l5nhadYY8X6U5EsGBK/jgSLJ9yU73UhqafHzvRx81YBkOH X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(376014)(1800799024)(82310400026)(35042699022)(14060799003)(36860700016)(56012099006)(18002099003)(22082099003)(3023799007)(11063799006)(5023799004)(6133799003)(13003099007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ngWc4SzxLAFjzBJ3/86Ang0k7H0fGCW4TSFW8tCbpMvU9jlySV0wr0wgR1eyV4DwUpnDm4we2OO5Mkq9zS00zIcV1n/AuPs1uzQ1/qGD+CX8Ki/PboYOZ7NbmNDqTZtArz4r27jWyMP0bMbQh2puwNoTao2Qb3xb4XxpbZv0PyhoPfsovkGSE9A3s32MZ3v6gVnq+QmeRWEJotwIVjkx99c7OYdNQZbx4XgJdmn/obqpXRsi2ScYWC9Wvhyo0zcGI5pYwMf6zn2LbQzWt9ArkcwCQDaoW9M7pW46yfhEO0/jThmE6wZuMyoxcqcVowndtvtAuNXUa3LmYSW8a5llrGLA+rcegKo7vFqp06n9MQfyZ/tLDaiRwJx9R+a/xpkoW6Io2pEbMWRqRrqJsF6mOpCL7xK3CuuJqXjx/C35l2ngdvarrORdavInWNCPQPTy X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:21:06.3654 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 58089bf7-cc3d-40bf-1ee1-08dec183bb8e X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: AMS0EPF00000191.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAVPR08MB9881 X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_BLOCKED, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org This enables use of a predicate mask or length limit for vectorization of basic blocks in cases where previously only the equivalent rolled (i.e. loop) form of some source code would have been vectorized. Predication is used for groups whose size is not neatly divisible into vectors of lengths that can be supported directly by the target. The initial vector mode for an SLP region is "autodetected" by calling aarch64_preferred_simd_mode, which prefers SVE modes if supported and unless configured otherwise (e.g. VNx4SI for int). If at least one profitable subgraph can be scheduled then GCC does not try to vectorise the region using any other modes, even though their estimated costs might otherwise have been lower. For example, if analysis of a 24-byte group succeeds with vector mode V16QI (using types vector(16) and vector(8) char) then the estimated cost of the vectorised code is 11+11=22. If analysis of the same group succeeds with vector mode VNx16QI (using type vector([16,16]) char for both subtrees) then the estimated cost is 15+15=30. In both cases, the estimated vectorised cost would beat the estimated scalar cost of 96+48=144, so vector([16,16]) wins because VNx16QI is tried first. This is mitigated by the fact that a sequence of GIMPLE stmts such as: vectp.14_86 = x_50(D) + 16; slp_mask_87 = .WHILE_ULT (0, 8, { 0, ... }); .MASK_STORE (vectp.14_86, 8B, slp_mask_87, vect__34.12_85); are lowered to a fixed-length vector store (e.g., str d30, [x0, 16]) if possible, instead of a more literal interpretation such as: add x0, x0, 16 ptrue p7.b, vl7 st1b z30.b, p7, [x0] The vect_record_nunits function used during building of an SLP tree is updated to prevent it returning failure for BB SLP if the group size is not an integral multiple of the number of lanes in the vector type; it now allows such cases if the vector type might be more than long enough. Instead of giving up if vect_get_vector_types_for_stmt fails for the specified group size, vect_build_slp_tree_1 now calls vect_get_vector_types_for_stmt again without a group size (which defaults to 0) as a fallback. If this succeeds then the initial failure is treated as a 'soft' failure that results in the group being split. Consequently, assertions that "For BB vectorization, we should always have a group size once we've constructed the SLP tree" were deleted in get_vectype_for_scalar_type and vect_get_vector_types_for_stmt. For BB SLP, vect_analyze_slp_instance previously gave up after building an SLP tree if it could not prove that the group size was at least the maximum lane count across all of the vector types in the SLP tree (which is unprovable for scalable vector types), or attempted to split the group if it could prove that the group size was greater than this maximum but not exactly divisible by it (which is also unprovable for scalable vector types). This function will now provisionally create a new SLP instance if the group size definitely does not exceed the minimum number of lanes, even if the group size otherwise satisfies conditions that would require a loop to be unrolled (e.g., a group of size 3 that uses a mixture of V4SI and V8HI types). If the group size lies between the minimum and maximum number of lanes then vectorization is still abandoned (e.g., a group of size 3 that uses a mixture of V2DI and V4SI types). With BB SLP, there is no need for agreement between different SLP nodes about whether to use masks or lengths to support partial vectors. Instead, that decision is made early and per individual SLP node, by vect_analyze_stmt. If a partial vector is required (i.e. if the number of subparts in the vector type may be greater than the number of active lanes for the node) then vect_analyze_stmt now requires SLP_TREE_CAN_USE_PARTIAL_VECTORS_P to be true; otherwise it clears any SLP_TREE_PARTIAL_VECTORS_STYLE that could have been set. The vect_get_num_copies function used during statement analysis is updated to return early with 1 if a vector type is long enough for the specified SLP tree node. This avoids an ICE in vect_get_num_vectors, which cannot cope with SVE vector types. When checking whether a value that is used outside the vectorized region can be supported, the vectorizable_live_operation function calculates which vector contains the result, and which lane of that vector we need. Previously, this calculation gave the wrong answer for BB SLP with a variable-length vector type (eventually generating invalid offsets such as BIT_FIELD_REF <_251, 32, POLY_INT_CST [96, 128]> to access the third element of a group using type VNx4SI) because it reused logic intended for loop vectorization, which selects the 'last' occurrence of a scalar index relative to the group size (which is a multiple of the vector length). For BB SLP with a predicate mask, only the first SLP_TREE_LANES elements are well defined. vect_create_vectorized_promotion_stmts no longer pushes more stmts than implied by vect_get_num_copies because it could previously overrun the number of slots allocated for an SLP node (based on its number of lanes and type). e.g., four defs were pushed for a promotion of V8HI to V2DI (8/2=4) even if only two lanes of the V8HI were active. Allowing it later caused ICE in vectorizable_operation for a parent node, because binary ops require both operands to be the same length. Since promotion no longer produces redundant definitions, vectorizable_conversion also had to be modified so that demotion no longer relies on an even number of defs being produced. If necessary, it now pushes a single constant zero def. The whole change is enabled by wiring the wrapper function vect_can_use_partial_vectors_p to SLP_TREE_CAN_USE_PARTIAL_VECTORS_P when invoked for BB SLP vectorization. Update test expectations for gcc.dg/vect/vect-over-widen-*.c, gcc.target/aarch64/sve/slp_6.c and gcc.target/aarch64/sve/vec_construct_*.c. The vec_construct_*.c tests previously expected their output to use Advanced SIMD instead of SVE despite their use of vector length agnostic types such as svint16_t and despite the fact that they are in the aarch64/sve directory. Since BB SLP can now vectorize these tests using VLA types such as 'vector([8,8]) char', and because (with one exception) the resultant code is deemed profitable relative to scalar code, GCC no longer considers vectorizing using non-VLA types such as 'vector(8) char' (although the estimated cost with non-VLA types might have been lower, had it been calculated). Instruction selection is not the focus of these tests therefore I updated them to expect SVE instead (e.g. st1b instead of str) and added --param=aarch64-autovec-preference=sve-only to reduce future churn. Because the cost model takes into account predicate mask generation for BB SLP with VLA types, the threshold at which vectorized code wins against scalar code is higher than before. The number of elements stored by vec_construct_3.c was increased just enough to allow for that. gcc/ChangeLog: * tree-vect-loop.cc (vectorizable_live_operation): Simplify the calculation of the index of the final result to avoid generating invalid polynomial offsets relative to the end of variable-length vector types, which is what happens if the code for loop vectorization is reused for basic block SLP. * tree-vect-slp.cc (vect_record_nunits): Allow group sizes that are indivisible by the vector length. (vect_build_slp_tree_1): In case of failure of vect_get_vector_types_for_stmt, try to get fallback vector types and continue analysis to allow splitting of groups. (vect_build_slp_tree_2): Don't call can_duplicate_and_interleave_p when doing basic block SLP vectorization. (vect_update_slp_min_nunits_for_node): New recursive function. Update min_nunits to reflect the minimum number of subparts for all of the vector types used by an SLP subgraph. (vect_slp_tree_min_nunits): New function. Initialize min_nunits then call vect_update_slp_min_nunits_for_node. (vect_analyze_slp_instance): For BB SLP vectorization, create a new SLP instance if the group size definitely does not exceed the minimum number of subparts for all of the vector types used in the SLP tree, even if the group size otherwise satisfies conditions that would require a loop to be unrolled. (vectorizable_slp_permutation_1): Instead of asserting that an SLP tree node's number of lanes is compatible with the chosen vector width, return a failure indication if incompatible. * tree-vect-stmts.cc (check_load_store_for_partial_vectors): When calculating the number of vectors, get the group size from SLP_TREE_LANES instead of a parameter (e.g., DR_GROUP_SIZE) if doing BB SLP vectorization. Don't assume it can be divided by the number of subparts in the vector type to get a compile-time constant. (vect_get_data_ptr_increment): Require a parameter of type loop_vec_info instead of vec_info *. (vect_create_vectorized_promotion_stmts): Require an SLP tree node to be passed by the caller, for use by vect_get_num_copies. Stop pushing more stmts than implied by vect_get_num_copies. (vectorizable_conversion): Pass SLP tree node to vect_create_vectorized_promotion_stmts. Demotion no longer relies on an even number of definitions being produced by promotion. If necessary, push a single constant zero definition. (vectorizable_load): Pass loop_vec_info instead of vec_info * when calling vect_get_data_ptr_increment. (vect_analyze_stmt): For BB SLP vectorization, check whether the group needs partial vectors. If it does then return a failure indication if SLP_TREE_CAN_USE_PARTIAL_VECTORS_P was cleared by a callee of this function; if it doesn't need partial vectors then clear any partial vectors style that might have been chosen by callees of this function. (get_vectype_for_scalar_type): For BB SLP vectorization, allow invocation of this function with a group size of zero even if one or more SLP instances have been created. If the number of subparts in the natural choice of vector type could be greater than the group size then pick a shorter vector type only if the target does not support partial vectors. (vect_maybe_update_slp_op_vectype): Reject external definitions that have a number of lanes not divisible by the number of subparts in a vector type naively inferred from the scalar type. (vect_get_vector_types_for_stmt): Add a new output parameter of Boolean type. Set it to true if the statement can't be vectorized because it uses a data type that the target doesn't support in vector form for a group of the given size, otherwise false. * tree-vectorizer.h (vect_get_num_copies): Return early with 1 if a vector type is long enough for the specified SLP tree node to avoid an ICE in vect_get_num_vectors. (vect_get_vector_types_for_stmt): Update function declaration. (vect_can_use_partial_vectors_p): Handle the BB SLP use-case by returning the result of SLP_TREE_CAN_USE_PARTIAL_VECTORS_P. gcc/testsuite/ChangeLog: * gcc.dg/vect/vect-over-widen-10.c: Update test expectations to avoid spurious matching of scan-tree-dump-not pattern. * gcc.dg/vect/vect-over-widen-13.c: As above. * gcc.dg/vect/vect-over-widen-14.c: As above. * gcc.dg/vect/vect-over-widen-17.c: As above. * gcc.dg/vect/vect-over-widen-18.c: As above. * gcc.dg/vect/vect-over-widen-5.c: As above. * gcc.dg/vect/vect-over-widen-6.c: As above. * gcc.dg/vect/vect-over-widen-7.c: As above. * gcc.dg/vect/vect-over-widen-8.c: As above. * gcc.dg/vect/vect-over-widen-9.c: As above. * gcc.target/aarch64/sve/vec_construct_1.c: Expect SVE instead of ASIMD instructions and add --param=aarch64-autovec-preference=sve-only to stop flip-flopping. * gcc.target/aarch64/sve/vec_construct_1.c: As above. * gcc.target/aarch64/sve/vec_construct_2.c: As above. * gcc.target/aarch64/sve/vec_construct_3.c: Expect SVE instead of ASIMD instructions and add --param=aarch64-autovec-preference=sve-only to avoid flip-flopping. Increase the number of elements stored to ensure vectorization using SVE is deemed profitable despite predicate mask costs. * gcc.target/aarch64/sve/vec_construct_4.c: Expect SVE instead of ASIMD instructions and add --param=aarch64-autovec-preference=sve-only to stop flip-flopping. * gcc.target/aarch64/sve/vec_construct_5.c: As above. --- .../gcc.dg/vect/vect-over-widen-10.c | 2 +- .../gcc.dg/vect/vect-over-widen-13.c | 2 +- .../gcc.dg/vect/vect-over-widen-14.c | 2 +- .../gcc.dg/vect/vect-over-widen-17.c | 2 +- .../gcc.dg/vect/vect-over-widen-18.c | 2 +- gcc/testsuite/gcc.dg/vect/vect-over-widen-5.c | 2 +- gcc/testsuite/gcc.dg/vect/vect-over-widen-6.c | 2 +- gcc/testsuite/gcc.dg/vect/vect-over-widen-7.c | 2 +- gcc/testsuite/gcc.dg/vect/vect-over-widen-8.c | 2 +- gcc/testsuite/gcc.dg/vect/vect-over-widen-9.c | 2 +- gcc/testsuite/gcc.target/aarch64/sve/slp_6.c | 3 - .../gcc.target/aarch64/sve/vec_construct_1.c | 6 +- .../gcc.target/aarch64/sve/vec_construct_2.c | 4 +- .../gcc.target/aarch64/sve/vec_construct_3.c | 20 +- .../gcc.target/aarch64/sve/vec_construct_4.c | 5 +- .../gcc.target/aarch64/sve/vec_construct_5.c | 6 +- gcc/tree-vect-loop.cc | 14 +- gcc/tree-vect-slp.cc | 141 ++++++++++-- gcc/tree-vect-stmts.cc | 202 +++++++++++++----- gcc/tree-vectorizer.h | 13 +- 20 files changed, 317 insertions(+), 117 deletions(-) diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-10.c b/gcc/testsuite/gcc.dg/vect/vect-over-widen-10.c index f0140e4ef6d..6efcf739db9 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-10.c +++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-10.c @@ -16,5 +16,5 @@ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* >> 1} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* >> 2} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern: detected:[^\n]* \(unsigned char\)} "vect" } } */ -/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */ +/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */ /* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-13.c b/gcc/testsuite/gcc.dg/vect/vect-over-widen-13.c index 08a65ea5518..720353716cf 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-13.c +++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-13.c @@ -48,5 +48,5 @@ main (void) /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* \+} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* / 2} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern: detected:[^\n]* = \(signed char\)} "vect" } } */ -/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */ +/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */ /* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-14.c b/gcc/testsuite/gcc.dg/vect/vect-over-widen-14.c index dfa09f5d2ca..f1d5f95c543 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-14.c +++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-14.c @@ -15,5 +15,5 @@ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* \+} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* >> 1} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern: detected:[^\n]* = \(unsigned char\)} "vect" } } */ -/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */ +/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */ /* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-17.c b/gcc/testsuite/gcc.dg/vect/vect-over-widen-17.c index 53fcfd0c06c..ac1a0f86727 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-17.c +++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-17.c @@ -46,5 +46,5 @@ main (void) adopts realign_load scheme. It requires rs6000_builtin_mask_for_load to generate mask whose return type is vector char. */ /* { dg-final { scan-tree-dump-not {vector[^\n]*char} "vect" { target vect_hw_misalign } } } */ -/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */ +/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */ /* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-18.c b/gcc/testsuite/gcc.dg/vect/vect-over-widen-18.c index aa58cd1c957..3ebfaa78270 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-18.c +++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-18.c @@ -47,5 +47,5 @@ main (void) /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* |} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* <<} "vect" } } */ /* { dg-final { scan-tree-dump {vector[^\n]*char} "vect" } } */ -/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */ +/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */ /* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-5.c b/gcc/testsuite/gcc.dg/vect/vect-over-widen-5.c index c2ab11a9d32..1d89789a86d 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-5.c +++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-5.c @@ -49,5 +49,5 @@ main (void) /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* \+ } "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* >> 1} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern: detected:[^\n]* \(signed char\)} "vect" } } */ -/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */ +/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */ /* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-6.c b/gcc/testsuite/gcc.dg/vect/vect-over-widen-6.c index bda92c965e0..62d5a52587e 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-6.c +++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-6.c @@ -13,5 +13,5 @@ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* \+ } "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* >> 1} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern: detected:[^\n]* \(unsigned char\)} "vect" } } */ -/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */ +/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */ /* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-7.c b/gcc/testsuite/gcc.dg/vect/vect-over-widen-7.c index 1d55e13fb1f..6e09631009a 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-7.c +++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-7.c @@ -51,5 +51,5 @@ main (void) /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* \+ } "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* >> 2} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern: detected:[^\n]* \(signed char\)} "vect" } } */ -/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */ +/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */ /* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-8.c b/gcc/testsuite/gcc.dg/vect/vect-over-widen-8.c index 553c0712a79..b6d650beab4 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-8.c +++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-8.c @@ -16,5 +16,5 @@ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* \+ } "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* >> 2} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern: detected:[^\n]* \(unsigned char\)} "vect" } } */ -/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */ +/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */ /* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-9.c b/gcc/testsuite/gcc.dg/vect/vect-over-widen-9.c index 36bfc68e053..e82f8a571da 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-9.c +++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-9.c @@ -56,5 +56,5 @@ main (void) /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* >> 1} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern: detected:[^\n]* >> 2} "vect" } } */ /* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern: detected:[^\n]* \(signed char\)} "vect" } } */ -/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */ +/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */ /* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_6.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_6.c index 44d128477d2..1c9ac15a699 100644 --- a/gcc/testsuite/gcc.target/aarch64/sve/slp_6.c +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_6.c @@ -37,9 +37,6 @@ vec_slp_##TYPE (TYPE *restrict a, TYPE *restrict b, int n) \ TEST_ALL (VEC_PERM) /* These loops can't use SLP. */ -/* { dg-final { scan-assembler-not {\tld1b\t} } } */ -/* { dg-final { scan-assembler-not {\tld1h\t} } } */ -/* { dg-final { scan-assembler-not {\tld1w\t} } } */ /* { dg-final { scan-assembler-not {\tld1d\t} } } */ /* { dg-final { scan-assembler {\tld3b\t} } } */ /* { dg-final { scan-assembler {\tld3h\t} } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_1.c b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_1.c index 2f8ce6808a9..eea13c28e49 100644 --- a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_1.c +++ b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_1.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -ftree-slp-vectorize" } */ +/* { dg-options "-O2 -ftree-slp-vectorize --param=aarch64-autovec-preference=sve-only" } */ /* Test that a group of stores of 8 elements derived from a horizontal reduction is vectorized by constructing a vector and storing it. @@ -30,8 +30,8 @@ foo (svint8_t src0, svint8_t src1, svint8_t src2, svint8_t src3, svint8_t src4, s.h = svaddv_s8 (all, src7); } -/* { dg-final { scan-assembler-times {\tins\tv[0-9]+\.b\[[0-9]+\], v[0-9]+\.b\[[0-9]+\]\n} 7 } } */ -/* { dg-final { scan-assembler-times {\tstr\td[0-9]+, } 1 } } */ +/* { dg-final { scan-assembler-times {\tinsr\tz[0-9]+\.h, h[0-9]+\n} 7 } } */ +/* { dg-final { scan-assembler-times {\tst1b\tz[0-9]+\.h, p[0-9]+, \[x[0-9]+\]\n} 1 } } */ /* { dg-final { scan-assembler-not {\tstr\tb[0-9]+, } } } */ /* { dg-final { scan-assembler-not {\tstrb\tw[0-9]+, } } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_2.c b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_2.c index 6715118d7b0..2bf537e13e2 100644 --- a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_2.c +++ b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_2.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -ftree-slp-vectorize" } */ +/* { dg-options "-O2 -ftree-slp-vectorize --param=aarch64-autovec-preference=sve-only" } */ /* Test that a group of stores of 8 elements derived from the results of calls to a function that has only vector parameters and returns a scalar result is @@ -40,3 +40,5 @@ foo (svint8_t src0, svint8_t src1, svint8_t src2, svint8_t src3, svint8_t src4, /* { dg-final { scan-assembler-not {\tins\tv[0-9]+\.b\[[0-9]+\], w[0-9]+\n} } } */ /* { dg-final { scan-assembler-not {\tstr\td[0-9]+, } } } */ +/* { dg-final { scan-assembler-not {\tfmov\th[0-9]+, h[0-9]+\n} } } */ +/* { dg-final { scan-assembler-not {\tinsr\tz[0-9]+\.b, w[0-9]+\n} } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_3.c b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_3.c index 8143d0050ad..ccadaccbcb4 100644 --- a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_3.c +++ b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_3.c @@ -1,7 +1,7 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -ftree-slp-vectorize" } */ +/* { dg-options "-O2 -ftree-slp-vectorize --param=aarch64-autovec-preference=sve-only" } */ -/* Test that a group of stores of 8 elements derived from a horizontal +/* Test that a group of stores of 14 elements derived from a horizontal reduction is vectorized by constructing a vector and storing it even if the results of the reductions are narrowed. Since there are no GPR-to-SIMD register transfers, there is no @@ -13,12 +13,14 @@ struct S { - char a, b, c, d, e, f, g, h; + char a, b, c, d, e, f, g, h, i, j, k, l, m, n; } s; void foo (svint16_t src0, svint32_t src1, svint16_t src2, svint32_t src3, - svint32_t src4, svint16_t src5, svint32_t src6, svint16_t src7) + svint32_t src4, svint16_t src5, svint32_t src6, svint16_t src7, + svint16_t src8, svint32_t src9, svint16_t src10, svint32_t src11, + svint32_t src12, svint16_t src13) { svbool_t all16 = svptrue_b16 (); svbool_t all32 = svptrue_b32 (); @@ -30,10 +32,16 @@ foo (svint16_t src0, svint32_t src1, svint16_t src2, svint32_t src3, s.f = svminv_s16 (all16, src5); s.g = svlastb_s32 (svptrue_pat_b32 (SV_VL1), src6); s.h = svaddv_s16 (all16, src7); + s.i = svmaxv_s16 (all16, src8); + s.j = svminv_s32 (all32, src9); + s.k = svlastb_s16 (svptrue_pat_b16 (SV_VL1), src10); + s.l = svaddv_s32 (all32, src11); + s.m = svmaxv_s32 (all32, src12); + s.n = svminv_s16 (all16, src13); } -/* { dg-final { scan-assembler-times {\tins\tv[0-9]+\.b\[[0-9]+\], v[0-9]+\.b\[[0-9]+\]\n} 7 } } */ -/* { dg-final { scan-assembler-times {\tstr\td[0-9]+, } 1 } } */ +/* { dg-final { scan-assembler-times {\tinsr\tz[0-9]+\.b, b[0-9]+\n} 13 } } */ +/* { dg-final { scan-assembler-times {\tst1b\tz[0-9]+\.b, p[0-9]+, \[x[0-9]\]\n} 1 } } */ /* { dg-final { scan-assembler-not {\tstr\tb[0-9]+, } } } */ /* { dg-final { scan-assembler-not {\tstrb\tw[0-9]+, } } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_4.c b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_4.c index 49f8114b64c..3d41af684a3 100644 --- a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_4.c +++ b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_4.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -ftree-slp-vectorize" } */ +/* { dg-options "-O2 -ftree-slp-vectorize --param=aarch64-autovec-preference=sve-only" } */ /* Test that a group of stores of 8 elements derived from a horizontal reduction is not vectorized by constructing a vector and storing it @@ -33,5 +33,6 @@ foo (svint16_t src0, svint8_t src1, svint16_t src2, svint8_t src3, /* { dg-final { scan-assembler-times {\tstp\tw[0-9]+, w[0-9]+,} 4 } } */ /* { dg-final { scan-assembler-not {\tins\tv[0-9]+\.s\[[0-9]+\], w[0-9]+\n} } } */ -/* { dg-final { scan-assembler-not {\tfmov\ts[0-9]+, w[0-9]+\n} } } +/* { dg-final { scan-assembler-not {\tfmov\ts[0-9]+, w[0-9]+\n} } } */ /* { dg-final { scan-assembler-not {\tstp\tq[0-9]+, q[0-9]+,} } } */ +/* { dg-final { scan-assembler-not {\tinsr\tz[0-9]+.s, w[0-9]+\n} } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_5.c b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_5.c index 983d6c69ebc..89e57406c0e 100644 --- a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_5.c +++ b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_5.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -ftree-slp-vectorize" } */ +/* { dg-options "-O2 -ftree-slp-vectorize --param=aarch64-autovec-preference=sve-only" } */ /* Test that a group of stores of 8 elements derived from lane extractions is vectorized by constructing a vector and storing it. Since there are no @@ -30,8 +30,8 @@ foo (svint8_t src0, svint8_t src1, svint8_t src2, svint8_t src3, svint8_t src4, s.h = svlastb_s8 (p, src7); } -/* { dg-final { scan-assembler-times {\tins\tv[0-9]+\.b\[[0-9]+\], v[0-9]+\.b\[[0-9]+\]\n} 7 } } */ -/* { dg-final { scan-assembler-times {\tstr\td[0-9]+, } 1 } } */ +/* { dg-final { scan-assembler-times {\tinsr\tz[0-9]+\.h, h[0-9]+\n} 7 } } */ +/* { dg-final { scan-assembler-times {\tst1b\tz[0-9]+\.h, p[0-9]+, \[x[0-9]+\]\n} 1 } } */ /* { dg-final { scan-assembler-not {\tstr\tb[0-9]+, } } } */ /* { dg-final { scan-assembler-not {\tstrb\tw[0-9]+, } } } */ diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 6d602c67108..7503fd084cf 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -10227,12 +10227,16 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info, gcc_assert (slp_index >= 0); - /* Get the last occurrence of the scalar index from the concatenation of - all the slp vectors. Calculate which slp vector it is and the index - within. */ - int num_scalar = SLP_TREE_LANES (slp_node); int num_vec = vect_get_num_copies (vinfo, slp_node); - poly_uint64 pos = (num_vec * nunits) - num_scalar + slp_index; + poly_uint64 pos = slp_index; + if (loop_vinfo) + { + /* Get the last occurrence of the scalar index from the concatenation of + all the slp vectors. Calculate which slp vector it is and the index + within. */ + int num_scalar = SLP_TREE_LANES (slp_node); + pos += (num_vec * nunits) - num_scalar; + } /* Calculate which vector contains the result, and which lane of that vector we need. */ diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index 1850af4e753..6af13e65e19 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -1117,8 +1117,12 @@ vect_record_max_nunits (vec_info *vinfo, stmt_vec_info stmt_info, } /* If populating the vector type requires unrolling then fail - before adjusting *max_nunits for basic-block vectorization. */ + before adjusting *max_nunits for basic-block vectorization. + Allow group sizes that are indivisible by the vector length only if they + are known not to exceed the vector length. We may be able to support such + cases by generating constant masks. */ if (is_a (vinfo) + && maybe_gt (group_size, TYPE_VECTOR_SUBPARTS (vectype)) && !multiple_p (group_size, TYPE_VECTOR_SUBPARTS (vectype))) { if (dump_enabled_p ()) @@ -1170,12 +1174,29 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, tree soft_fail_nunits_vectype = NULL_TREE; tree vectype, nunits_vectype; + bool unsupported_datatype = false; if (!vect_get_vector_types_for_stmt (vinfo, first_stmt_info, &vectype, - &nunits_vectype, group_size)) + &nunits_vectype, &unsupported_datatype, + group_size)) { - /* Fatal mismatch. */ - matches[0] = false; - return false; + /* Try to get fallback vector types and continue analysis, producing + matches[] as if vectype was not an issue. This allows splitting of + groups to happen. */ + if (unsupported_datatype + && vect_get_vector_types_for_stmt (vinfo, first_stmt_info, &vectype, + &nunits_vectype, + &unsupported_datatype)) + { + gcc_assert (is_a (vinfo)); + maybe_soft_fail = true; + soft_fail_nunits_vectype = nunits_vectype; + } + else + { + /* Fatal mismatch. */ + matches[0] = false; + return false; + } } if (is_a (vinfo) && known_le (TYPE_VECTOR_SUBPARTS (vectype), 1U)) @@ -1705,16 +1726,22 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char *swap, if (maybe_soft_fail) { - unsigned HOST_WIDE_INT const_nunits; - if (!TYPE_VECTOR_SUBPARTS - (soft_fail_nunits_vectype).is_constant (&const_nunits) - || const_nunits > group_size) + /* Use the known minimum number of subparts for VLA because we still need + to choose a splitting point although the choice is more arbitrary. */ + unsigned HOST_WIDE_INT const_nunits = constant_lower_bound ( + TYPE_VECTOR_SUBPARTS (soft_fail_nunits_vectype)); + + if (const_nunits > group_size) matches[0] = false; else { /* With constant vector elements simulate a mismatch at the point we need to split. */ + gcc_assert ((const_nunits & (const_nunits - 1)) == 0); unsigned tail = group_size & (const_nunits - 1); + if (tail == 0) + tail = const_nunits; + gcc_assert (group_size >= tail); memset (&matches[group_size - tail], 0, sizeof (bool) * tail); } return false; @@ -2446,13 +2473,21 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node, /* Check whether we can build the invariant. If we can't we never will be able to. */ tree type = TREE_TYPE (chains[0][n].op); - if (!GET_MODE_SIZE (vinfo->vector_mode).is_constant () - && (TREE_CODE (type) == BOOLEAN_TYPE - || !can_duplicate_and_interleave_p (vinfo, group_size, - type))) + if (!GET_MODE_SIZE (vinfo->vector_mode).is_constant ()) { - matches[0] = false; - goto out; + if (TREE_CODE (type) == BOOLEAN_TYPE) + { + matches[0] = false; + goto out; + } + + if (!is_a (vinfo) + && !can_duplicate_and_interleave_p (vinfo, group_size, + type)) + { + matches[0] = false; + goto out; + } } } else if (dt != vect_internal_def) @@ -2881,7 +2916,7 @@ out: uniform_val = NULL_TREE; break; } - if (!uniform_val + if (!uniform_val && !is_a (vinfo) && !can_duplicate_and_interleave_p (vinfo, oprnd_info->ops.length (), TREE_TYPE (op0))) @@ -4993,6 +5028,53 @@ vect_analyze_slp_reductions (loop_vec_info loop_vinfo, return true; } +/* Update MIN_NUNITS to reflect the minimum number of subparts for all of the + vector types used by the SLP subgraph rooted at NODE. VISITED is used to + avoid reevaluating any node in the subgraph; it thereby prevents infinite + recursion should a cycle be encountered. The value of MIN_NUNITS will only be + updated if any node in the subgraph has a vector type with a number of + subparts that is smaller than the passed-in value of MIN_NUNITS. Before + calling this function for the first time, initialize MIN_NUNITS to + UINT64_MAX. */ + +static void +vect_update_slp_min_nunits_for_node (slp_tree node, poly_uint64 &min_nunits, + hash_set &visited) +{ + if (!node || SLP_TREE_DEF_TYPE (node) != vect_internal_def) + return; + + if (visited.add (node)) + return; + + for (slp_tree child : SLP_TREE_CHILDREN (node)) + vect_update_slp_min_nunits_for_node (child, min_nunits, visited); + + tree vectype = SLP_TREE_VECTYPE (node); + if (!vectype) + return; + + /* All unit counts have the form vec_info::vector_size * X for some + rational X, therefore we know the values are ordered. */ + poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); + min_nunits = known_eq (min_nunits, UINT64_MAX) + ? nunits + : ordered_min (min_nunits, nunits); +} + +/* For NODE, return the minimum number of subparts for all of the vector + types used in the given SLP graph. */ + +static poly_uint64 +vect_slp_tree_min_nunits (slp_tree node) +{ + poly_uint64 min_nunits = UINT64_MAX; + hash_set visited; + vect_update_slp_min_nunits_for_node (node, min_nunits, visited); + gcc_checking_assert (known_ne (min_nunits, UINT64_MAX)); + return min_nunits; +} + /* Analyze an SLP instance starting from a group of grouped stores. Call vect_build_slp_tree to build a tree of packed stmts if possible. Return FALSE if it's impossible to SLP any stmt in the group. */ @@ -5062,8 +5144,8 @@ vect_analyze_slp_instance (vec_info *vinfo, poly_uint64 unrolling_factor = calculate_unrolling_factor (max_nunits, group_size); - if (maybe_ne (unrolling_factor, 1U) - && is_a (vinfo)) + if (maybe_ne (unrolling_factor, 1U) && is_a (vinfo) + && !known_ge (vect_slp_tree_min_nunits (node), group_size)) { unsigned HOST_WIDE_INT const_max_nunits; if (!max_nunits.is_constant (&const_max_nunits) @@ -5148,9 +5230,10 @@ vect_analyze_slp_instance (vec_info *vinfo, = TREE_TYPE (DR_REF (STMT_VINFO_DATA_REF (stmt_info))); tree vectype = get_vectype_for_scalar_type (vinfo, scalar_type, 1 << floor_log2 (i)); - unsigned HOST_WIDE_INT const_nunits; - if (vectype - && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&const_nunits)) + unsigned HOST_WIDE_INT const_nunits + = vectype ? constant_lower_bound (TYPE_VECTOR_SUBPARTS (vectype)) + : 0; + if (const_nunits > 1 && (i % const_nunits) == 0) { /* Split into two groups at the first vector boundary. */ gcc_assert ((const_nunits & (const_nunits - 1)) == 0); @@ -11596,7 +11679,21 @@ vectorizable_slp_permutation_1 (vec_info *vinfo, gimple_stmt_iterator *gsi, unpack_factor = 1; } unsigned olanes = unpack_factor * ncopies * SLP_TREE_LANES (node); - gcc_assert (repeating_p || multiple_p (olanes, nunits)); + + /* With fully-predicated BB-SLP, an external node's number of lanes can be + incompatible with the chosen vector width (e.g., lane packs of 3 with a + natural 2-lane vector type). */ + if (!repeating_p && !multiple_p (olanes, nunits)) + { + if (dump_p) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "unsupported permutation %p: vector type %T," + " nunits=" HOST_WIDE_INT_PRINT_UNSIGNED + " ncopies=%" PRIu64 ", lanes=%u and unpack=%u\n", + (void *) node, vectype, estimated_poly_value (nunits), + ncopies, SLP_TREE_LANES (node), unpack_factor); + return -1; + } /* Compute the { { SLP operand, vector index}, lane } permutation sequence from the { SLP operand, scalar lane } permutation as recorded in the diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index cd3ba6fa1cb..367a9c63ea4 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -1671,23 +1671,27 @@ check_load_store_for_partial_vectors (vec_info *vinfo, tree vectype, unsigned int nvectors; if (can_div_away_from_zero_p (size, nunits, &nvectors)) return nvectors; - gcc_unreachable (); + + gcc_assert (known_le (size, nunits)); + return 1u; }; poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); - poly_uint64 vf = loop_vinfo ? LOOP_VINFO_VECT_FACTOR (loop_vinfo) : 1; + poly_uint64 size = loop_vinfo + ? group_size * LOOP_VINFO_VECT_FACTOR (loop_vinfo) + : SLP_TREE_LANES (slp_node); unsigned factor; vect_partial_vector_style partial_vector_style = vect_get_partial_vector_style (vectype, is_load, &factor, elsvals); if (partial_vector_style == vect_partial_vectors_len) { - nvectors = group_memory_nvectors (group_size * vf, nunits); + nvectors = group_memory_nvectors (size, nunits); vect_record_len (vinfo, slp_node, nvectors, vectype, factor); } else if (partial_vector_style == vect_partial_vectors_while_ult) { - nvectors = group_memory_nvectors (group_size * vf, nunits); + nvectors = group_memory_nvectors (size, nunits); vect_record_mask (vinfo, slp_node, nvectors, vectype, scalar_mask); } else @@ -3362,12 +3366,11 @@ vect_get_strided_load_store_ops (stmt_vec_info stmt_info, slp_tree node, static tree vect_get_loop_variant_data_ptr_increment ( - vec_info *vinfo, tree aggr_type, gimple_stmt_iterator *gsi, + loop_vec_info loop_vinfo, tree aggr_type, gimple_stmt_iterator *gsi, vec_loop_lens *loop_lens, dr_vec_info *dr_info, vect_memory_access_type memory_access_type) { - loop_vec_info loop_vinfo = dyn_cast (vinfo); - tree step = vect_dr_behavior (vinfo, dr_info)->step; + tree step = vect_dr_behavior (loop_vinfo, dr_info)->step; /* gather/scatter never reach here. */ gcc_assert (!mat_gather_scatter_p (memory_access_type)); @@ -3411,7 +3414,7 @@ vect_get_data_ptr_increment (vec_info *vinfo, gimple_stmt_iterator *gsi, loop_vec_info loop_vinfo = dyn_cast (vinfo); if (loop_vinfo && LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)) - return vect_get_loop_variant_data_ptr_increment (vinfo, aggr_type, gsi, + return vect_get_loop_variant_data_ptr_increment (loop_vinfo, aggr_type, gsi, loop_lens, dr_info, memory_access_type); @@ -5297,7 +5300,7 @@ vect_create_vectorized_demotion_stmts (vec_info *vinfo, vec *vec_oprnds, call the function recursively. */ static void -vect_create_vectorized_promotion_stmts (vec_info *vinfo, +vect_create_vectorized_promotion_stmts (vec_info *vinfo, slp_tree slp_node, vec *vec_oprnds0, vec *vec_oprnds1, stmt_vec_info stmt_info, tree vec_dest, @@ -5310,37 +5313,39 @@ vect_create_vectorized_promotion_stmts (vec_info *vinfo, gimple *new_stmt1, *new_stmt2; vec vec_tmp = vNULL; - vec_tmp.create (vec_oprnds0->length () * 2); + const unsigned ncopies = vect_get_num_copies (vinfo, slp_node); + vec_tmp.create (ncopies); + gcc_assert (vec_oprnds0->length () <= ncopies); FOR_EACH_VEC_ELT (*vec_oprnds0, i, vop0) { + if (vec_tmp.length () >= ncopies) + break; + if (op_type == binary_op) vop1 = (*vec_oprnds1)[i]; else vop1 = NULL_TREE; /* Generate the two halves of promotion operation. */ - new_stmt1 = vect_gen_widened_results_half (vinfo, ch1, vop0, vop1, - op_type, vec_dest, gsi, - stmt_info); - new_stmt2 = vect_gen_widened_results_half (vinfo, ch2, vop0, vop1, - op_type, vec_dest, gsi, - stmt_info); - if (is_gimple_call (new_stmt1)) - { - new_tmp1 = gimple_call_lhs (new_stmt1); - new_tmp2 = gimple_call_lhs (new_stmt2); - } - else + new_stmt1 + = vect_gen_widened_results_half (vinfo, ch1, vop0, vop1, op_type, + vec_dest, gsi, stmt_info); + new_tmp1 = is_gimple_call (new_stmt1) ? gimple_call_lhs (new_stmt1) + : gimple_assign_lhs (new_stmt1); + vec_tmp.quick_push (new_tmp1); + + if (vec_tmp.length () < ncopies) { - new_tmp1 = gimple_assign_lhs (new_stmt1); - new_tmp2 = gimple_assign_lhs (new_stmt2); + new_stmt2 + = vect_gen_widened_results_half (vinfo, ch2, vop0, vop1, op_type, + vec_dest, gsi, stmt_info); + new_tmp2 = is_gimple_call (new_stmt2) ? gimple_call_lhs (new_stmt2) + : gimple_assign_lhs (new_stmt2); + vec_tmp.quick_push (new_tmp2); } - - /* Store the results for the next step. */ - vec_tmp.quick_push (new_tmp1); - vec_tmp.quick_push (new_tmp2); } + gcc_assert (vec_tmp.length () <= ncopies); vec_oprnds0->release (); *vec_oprnds0 = vec_tmp; } @@ -5553,6 +5558,7 @@ vectorizable_conversion (vec_info *vinfo, from the scalar type. */ if (!vectype_in) vectype_in = get_vectype_for_scalar_type (vinfo, rhs_type, slp_node); + if (!cost_vec) gcc_assert (vectype_in); if (!vectype_in) @@ -5961,12 +5967,15 @@ vectorizable_conversion (vec_info *vinfo, stmt_info, this_dest, gsi, c1, op_type); else - vect_create_vectorized_promotion_stmts (vinfo, &vec_oprnds0, - &vec_oprnds1, stmt_info, - this_dest, gsi, + vect_create_vectorized_promotion_stmts (vinfo, slp_node, + &vec_oprnds0, &vec_oprnds1, + stmt_info, this_dest, gsi, c1, c2, op_type); } + gcc_assert (vec_oprnds0.length () + == vect_get_num_copies (vinfo, slp_node)); + FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0) { gimple *new_stmt; @@ -5990,6 +5999,16 @@ vectorizable_conversion (vec_info *vinfo, generate more than one vector stmt - i.e - we need to "unroll" the vector stmt by a factor VF/nunits. */ vect_get_vec_defs (vinfo, slp_node, op0, &vec_oprnds0); + + /* Promotion no longer produces redundant defs (since support was + added for length/mask-predicated BB SLP of awkward-sized groups), + therefore demotion now has to handle that case too. */ + if (vec_oprnds0.length () % 2 != 0) + { + tree vectype = TREE_TYPE (vec_oprnds0[0]); + vec_oprnds0.safe_push (build_zero_cst (vectype)); + } + /* Arguments are ready. Create the new vector stmts. */ if (cvt_type && modifier == NARROW_DST) FOR_EACH_VEC_ELT (vec_oprnds0, i, vop0) @@ -10803,7 +10822,7 @@ vectorizable_load (vec_info *vinfo, aggr_type = build_array_type_nelts (elem_type, group_size * nunits); if (!costing_p) - bump = vect_get_data_ptr_increment (vinfo, gsi, dr_info, aggr_type, + bump = vect_get_data_ptr_increment (loop_vinfo, gsi, dr_info, aggr_type, memory_access_type, loop_lens); unsigned int inside_cost = 0, prologue_cost = 0; @@ -13460,6 +13479,38 @@ vect_analyze_stmt (vec_info *vinfo, " live stmt not supported: %G", stmt_info->stmt); + if (bb_vinfo) + { + unsigned int group_size = SLP_TREE_LANES (node); + tree vectype = SLP_TREE_VECTYPE (node); + poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); + bool needs_partial = maybe_lt (group_size, nunits); + if (needs_partial) + { + /* If partial vectors are required then they must be supported by the + target; however, don't assume that a partial vectors style has + been set because a mask or length may not be required for the + statement. */ + if (!SLP_TREE_CAN_USE_PARTIAL_VECTORS_P (node)) + return opt_result::failure_at (stmt_info->stmt, + "not vectorized: SLP node needs but " + "cannot use partial vectors: %G", + stmt_info->stmt); + } + else + { + /* If we don't need partial vectors then we don't care about whether + they are supported or not; however, we need to clear any partial + vectors style that might have been chosen because it will be used + to control generation of lengths or masks. */ + SLP_TREE_PARTIAL_VECTORS_STYLE (node) = vect_partial_vectors_none; + SLP_TREE_NUM_PARTIAL_VECTORS (node) = 0; + } + + if (maybe_gt (group_size, nunits)) + gcc_assert (multiple_p (group_size, nunits)); + } + return opt_result::success (); } @@ -13767,13 +13818,7 @@ tree get_vectype_for_scalar_type (vec_info *vinfo, tree scalar_type, unsigned int group_size) { - /* For BB vectorization, we should always have a group size once we've - constructed the SLP tree; the only valid uses of zero GROUP_SIZEs - are tentative requests during things like early data reference - analysis and pattern recognition. */ - if (is_a (vinfo)) - gcc_assert (vinfo->slp_instances.is_empty () || group_size != 0); - else + if (!is_a (vinfo)) group_size = 0; tree vectype = get_related_vectype_for_scalar_type (vinfo->vector_mode, @@ -13787,10 +13832,18 @@ get_vectype_for_scalar_type (vec_info *vinfo, tree scalar_type, vinfo->used_vector_modes.add (TYPE_MODE (vectype)); /* If the natural choice of vector type doesn't satisfy GROUP_SIZE, - try again with an explicit number of elements. */ - if (vectype - && group_size - && maybe_ge (TYPE_VECTOR_SUBPARTS (vectype), group_size)) + try again with an explicit number of elements. A vector type satisfies + GROUP_SIZE if it is definitely not too long to store the whole group, + or we are able to generate masks to handle the unknown number of excess + lanes that might exist. Otherwise, we must substitute a vector type that + can be used to carve up the group. + */ + if (vectype && group_size + && maybe_gt (TYPE_VECTOR_SUBPARTS (vectype), group_size) + && (vect_get_partial_vector_style (vectype, true) + == vect_partial_vectors_none + || vect_get_partial_vector_style (vectype, false) + == vect_partial_vectors_none)) { /* Start with the biggest number of units that fits within GROUP_SIZE and halve it until we find a valid vector type. @@ -14106,7 +14159,36 @@ vect_maybe_update_slp_op_vectype (vec_info *vinfo, slp_tree op, tree vectype) && SLP_TREE_DEF_TYPE (op) == vect_external_def && SLP_TREE_LANES (op) > 1) return false; - (void) vinfo; /* FORNOW */ + + /* When the vectorizer falls back to building vector operands from scalars, + it can create SLP trees with external defs that have a number of lanes not + divisible by the number of subparts in a vector type naively inferred from + the scalar type. Reject such types to avoid ICE when later computing the + prologue cost for invariant operands. */ + if (SLP_TREE_DEF_TYPE (op) == vect_external_def) + { + poly_uint64 vf = 1; + + if (loop_vec_info loop_vinfo = dyn_cast (vinfo)) + vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo); + + vf *= SLP_TREE_LANES (op); + + if (maybe_lt (TYPE_VECTOR_SUBPARTS (vectype), vf) + && !multiple_p (vf, TYPE_VECTOR_SUBPARTS (vectype))) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "lanes=" HOST_WIDE_INT_PRINT_UNSIGNED + " is not divisible by " + "subparts=" HOST_WIDE_INT_PRINT_UNSIGNED ".\n", + estimated_poly_value (vf), + estimated_poly_value ( + TYPE_VECTOR_SUBPARTS (vectype))); + return false; + } + } + SLP_TREE_VECTYPE (op) = vectype; return true; } @@ -14814,27 +14896,32 @@ vect_gen_while_not (gimple_seq *seq, tree mask_type, tree start_index, - Set *NUNITS_VECTYPE_OUT to the vector type that contains the maximum number of units needed to vectorize STMT_INFO, or NULL_TREE if the - statement does not help to determine the overall number of units. */ + statement does not help to determine the overall number of units. + + - Set *UNSUPPORTED_DATATYPE to false. + + On failure: + + - Set *UNSUPPORTED_DATATYPE to true if the statement can't be vectorized + because it uses a data type that the target doesn't support in vector form + for a group of the given GROUP_SIZE. + */ opt_result vect_get_vector_types_for_stmt (vec_info *vinfo, stmt_vec_info stmt_info, tree *stmt_vectype_out, tree *nunits_vectype_out, + bool *unsupported_datatype, unsigned int group_size) { gimple *stmt = stmt_info->stmt; - /* For BB vectorization, we should always have a group size once we've - constructed the SLP tree; the only valid uses of zero GROUP_SIZEs - are tentative requests during things like early data reference - analysis and pattern recognition. */ - if (is_a (vinfo)) - gcc_assert (vinfo->slp_instances.is_empty () || group_size != 0); - else + if (!is_a (vinfo)) group_size = 0; *stmt_vectype_out = NULL_TREE; *nunits_vectype_out = NULL_TREE; + *unsupported_datatype = false; if (gimple_get_lhs (stmt) == NULL_TREE /* Allow vector conditionals through here. */ @@ -14907,10 +14994,13 @@ vect_get_vector_types_for_stmt (vec_info *vinfo, stmt_vec_info stmt_info, } vectype = get_vectype_for_scalar_type (vinfo, scalar_type, group_size); if (!vectype) - return opt_result::failure_at (stmt, - "not vectorized:" - " unsupported data-type %T\n", - scalar_type); + { + *unsupported_datatype = true; + return opt_result::failure_at (stmt, + "not vectorized:" + " unsupported data-type %T\n", + scalar_type); + } if (dump_enabled_p ()) dump_printf_loc (MSG_NOTE, vect_location, "vectype: %T\n", vectype); diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 62f6ad320f0..1b9103b6f5f 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -2353,6 +2353,8 @@ vect_get_num_copies (vec_info *vinfo, slp_tree node) vf *= SLP_TREE_LANES (node); tree vectype = SLP_TREE_VECTYPE (node); + if (known_ge (TYPE_VECTOR_SUBPARTS (vectype), vf)) + return 1; return vect_get_num_vectors (vf, vectype); } @@ -2621,9 +2623,9 @@ extern tree vect_gen_while (gimple_seq *, tree, tree, tree, const char * = nullptr); extern void vect_gen_while_ssa_name (gimple_seq *, tree, tree, tree, tree); extern tree vect_gen_while_not (gimple_seq *, tree, tree, tree); -extern opt_result vect_get_vector_types_for_stmt (vec_info *, - stmt_vec_info, tree *, - tree *, unsigned int = 0); +extern opt_result vect_get_vector_types_for_stmt (vec_info *, stmt_vec_info, + tree *, tree *, + bool *, unsigned int = 0); extern opt_tree vect_get_mask_type_for_stmt (stmt_vec_info, unsigned int = 0); /* In tree-if-conv.cc. */ @@ -2956,9 +2958,8 @@ vect_can_use_partial_vectors_p (vec_info *vinfo, slp_tree slp_node) loop_vec_info loop_vinfo = dyn_cast (vinfo); if (loop_vinfo) return LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo); - - (void) slp_node; /* FORNOW */ - return false; + else + return SLP_TREE_CAN_USE_PARTIAL_VECTORS_P (slp_node); } /* If VINFO is vectorizer state for loop vectorization then record that we no From patchwork Wed Jun 3 15:19:22 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136405 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 6F78D4BA23CE for ; Wed, 3 Jun 2026 15:23:20 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 6F78D4BA23CE Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=ZbunRcHK; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=ZbunRcHK X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from DUZPR83CU001.outbound.protection.outlook.com (mail-northeuropeazon11012043.outbound.protection.outlook.com [52.101.66.43]) by sourceware.org (Postfix) with ESMTPS id 72DD14BA2E2E for ; Wed, 3 Jun 2026 15:21:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 72DD14BA2E2E Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 72DD14BA2E2E Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.66.43 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500065; cv=pass; b=lpcBSadYB4No1SITc/uuoZx01jbvY6xWXaEIEFB2lc6NDhDTL3ObEazTVGDcrkBAehgsSYfaDmuo2fMN1GHAOYogSioS8ivdkI72VxlYlq2H0MPycG7enEiTxxKnbQJ7mGEHvdCmQjieAnAh1B7e2NoCFMLyVA/dXqATrfoNiok= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500065; c=relaxed/simple; bh=sQEutcpT0P+swUEJ7UDbb/+3f+LAE7E2MkQmszBw/Pg=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=cipor8CzQkRzmwnJSwhvi1KbEQHfgnCggD9eHhaOr5TtrD/dynOezcM+Ce08W2N4hRyIt0K0QrvfTyt4Xx9XT94dF3lE8uhqhigm2g04/8lOPsAWD0EYdD5KA5YYkFHVyauOHzANm2AZeAlQR/g3tAV49vosXG42hCHp3Q+PQWo= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=ZbunRcHK; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=ZbunRcHK DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 72DD14BA2E2E ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=l5qTPQ5h6sWHU/AJPG7obJpJ9z24mmbdUxUnC3/xDziC8FV4KnvKXA7EGudd6ZP2z/1eVzJirI2A7xrVXdJy+En+BtsfdL4BjqxK/C7H5AIqdk/KTadf12rSyAGgTREOIEoNMIYAcqV9wgMcDUgJ5hRKoB3rriQfV/3oC25nn77MrfGqav+ANg896Y6sBCn3mNAyafquMWQivC+pzssYhuEJOw6amAUkdclWN1yRRz23bVjctW3mHjnstRcCVgFM7sys+mebcgVy5YtK7+woOr8wBscKM5Ef1piRATU9/M8/LYiL9H2bNiRaRMPhBR/rM2qycVo/2RWm8oKzfv1cBA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VlRA/78I6M9EAu93R4+qfn3Gfc3tEcCpd1ndw31ixCY=; b=eysUCP/Z9SAw1V5XGTbCZRdLQ9MracD8nZ2nHrupPlydOfPaE7fL7D4GKjh2rgUK9paKrSo/bd6gYBdxCWmRIHfGQ30YCa1AuC810ODgvLuZs4uUJDbKs8DKPPrYtivudpRT1o/uJ3RFfyjtbZxxmsVFHeWt7rTm5UjZu7YCo9OKRGu4eJqEI0/0mVS2r43ZkW3tMJCL3t9VkULrFpTUE203JQ4iDZhjhaqxWqA2bqmMzvk9mNYrc7z+IRyWxBVIO8cO+iE+RDkfrHWx9rEqrN//15mbBRRgqL0/FKJdFO1HvXdigQf1ZYAVQBWICHot9rjuOue2cC+v3QXVcQJNnA== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VlRA/78I6M9EAu93R4+qfn3Gfc3tEcCpd1ndw31ixCY=; b=ZbunRcHKV7hQhFdcdVCg6wQzYNrc2SewBJgka4CCmNNJoVyFNHt9kzQjjCdR8HHTbuFlYBg4tyycid04ROppm6ZCUEXvTiFxpbRa+wCgmpHaiUo+W6d7lF+A4NWxxK9q7Q57XMO6nbO7ENdwR2Ksiw1naiNrc/enSjJ0ZU9J258= Received: from CWLP265CA0390.GBRP265.PROD.OUTLOOK.COM (2603:10a6:400:1d6::23) by VI0PR08MB11303.eurprd08.prod.outlook.com (2603:10a6:800:2fe::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:21:02 +0000 Received: from AM2PEPF0001C710.eurprd05.prod.outlook.com (2603:10a6:400:1d6:cafe::e1) by CWLP265CA0390.outlook.office365.com (2603:10a6:400:1d6::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:21:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by AM2PEPF0001C710.mail.protection.outlook.com (10.167.16.180) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:21:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kmWu/yIg/veLv6vnJBvVKTZC42cylAwD446MdG92lVch+GnwjXmME+9WYv4wkBVdRAEemYJdSIC0X9gaQZh+dbxSdQO6830K6L82rOoJ8+/oC6TbZbsIFLALuUoVcEYZdcJNGTyKvPKHSeV70FeXrddEnAzCWJ09DZaMZA/RG3kgoQvQ6pR53ZmMorq5XU7oj2FF7wv/SsOV0KJ5eIc2jjZ6OXEdCrc9UQt7OoOGY9MZF9zJOvhgyw4VTsmJjCad2W9UTt0zGK1U9Su9ujpqgXJpNgLlf1FWpjSqTHEnQtH7es0PCHTjvU9KivU7UHFnXpgqnJ8Mpxw1h3jqYI+eCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VlRA/78I6M9EAu93R4+qfn3Gfc3tEcCpd1ndw31ixCY=; b=UMFgnfihty4wzhrF4Mw8JY1WYuudQDdT7cIiKrkx7N3LQvHWGf+k3Sgu+T+mOc2Vc0T7gAnzXtXNyNKTKvR9aJfn4oVH0ehkxvC4938+3wZoWX+9mAVv246/ps4Bn3RaFI9OGoypy/o66inSE9x2u8BIw6I2/9+Zm1aQ5XJtPRxWAoPvH2jL6YQDqZu3raKg5Y0e6JG1raH82Ee0TP820dfRnAxtE5mJjuDjsCbKxbSxjFItQGy7TSQp2JTVnGJAg4HeIS2wN9hG1bA4YvDSX8zvMSOrMxypwKKQtV2Rh6n4QQ7til2V+J8WYuTE3v6YVaMQzOiz/wrWboj3Bon+aQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VlRA/78I6M9EAu93R4+qfn3Gfc3tEcCpd1ndw31ixCY=; b=ZbunRcHKV7hQhFdcdVCg6wQzYNrc2SewBJgka4CCmNNJoVyFNHt9kzQjjCdR8HHTbuFlYBg4tyycid04ROppm6ZCUEXvTiFxpbRa+wCgmpHaiUo+W6d7lF+A4NWxxK9q7Q57XMO6nbO7ENdwR2Ksiw1naiNrc/enSjJ0ZU9J258= Received: from DU2PR04CA0156.eurprd04.prod.outlook.com (2603:10a6:10:2b0::11) by AMDPR08MB11481.eurprd08.prod.outlook.com (2603:10a6:20b:719::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:19:59 +0000 Received: from DU6PEPF00009528.eurprd02.prod.outlook.com (2603:10a6:10:2b0:cafe::30) by DU2PR04CA0156.outlook.office365.com (2603:10a6:10:2b0::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:19:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009528.mail.protection.outlook.com (10.167.8.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:19:59 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:35 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:35 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 10/12] AArch64/SVE: Tests for use of predicated vector tails for BB SLP Date: Wed, 3 Jun 2026 15:19:22 +0000 Message-ID: <20260603151924.53706-11-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009528:EE_|AMDPR08MB11481:EE_|AM2PEPF0001C710:EE_|VI0PR08MB11303:EE_ X-MS-Office365-Filtering-Correlation-Id: 8b0ce553-4674-480f-0be3-08dec183b8e8 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|1800799024|82310400026|376014|36860700016|18002099003|22082099003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info-Original: g5bxthYixiMNn6iduOAyJbvS4n0dnKLlvtOAp3J+yDHyvY8dVeTOaEeynnu+bzVXn3r3W/plMrkrpE/AOzq+uTqkRnkpV3PYFJfvll3WzKoGi4MESAM+7o87zwA7XNimZD0bNolRvpW3zRR8ROghiFCkbvehmYws787AqYEzNzXNhN/xxLUv6CKIkweJGsJ/8Q4AFtlPtDGnGYNmXVTMA8voxIjOqaO0CSHzqm3ZUJs7NMQFMeGGaDrq4LqNXwu1JjRLK/poraOjc2/VUZ5yo8+AKKdD1o/QMcF9vEHtd3wQ0EXzRzMnnchtpJd9V7KwaZ88j0FxbdzSWcHS7Za01elNowVMGg1BKSIFbFaie3XxtbuepfoiM42tViauH6hQNUHjoIdOafCX+rLUx9kPAi35V47Ytnnz2H4U8a8Lm3D/9qHo9NPY1jRhKCaAP8KCWsQnFDmMUuhf2XiRkTgaDEDrlAKojmtIlVg2YG0zY7hZZOb4xDx0ce8ABvbYczoOk1egnBaSft1YyMLpTpTc9z/r1KYLHB/cBFeHc1ztP0BXwo7Ksh7kAD2J6O3clw6LXq/x+sGtETzbjnpD6ioaqSCopNSfPLdqKvEjhbhfM/dhPIKVywaMCbUlw1uVCeAnUVVXbG8Aid8uEdOpu/C78SkGYO0/wEK2qJzoG80j6EYv1+/PvKV9VbrcqA5kGRXO X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(82310400026)(376014)(36860700016)(18002099003)(22082099003)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: WGlhmAwIsZqMYHsCPRtTq9lXpvOhfcqOEH8j6Hq9+Ub5gu3vL8yDEuI9ZJSqqKM8HGg+9+27ybrpoUtAsCO/cgQP+7VRbDCckSW1+09+6uAq6GLE/5wdOLzXI2bw3kojm238k1OuRCP6E8TX9AyJYYm1g8kfDNklqPVTcyTP0QOoYs/ZCRB5OlwqtuShmTCRMSAsWUUNHPe8yhpdvoXj1P7nPM4Y6i/ikSvDBFFHoeKUbD3Oxmg+sKuo2RO0TvOeAhVp5gE62Qr65IO6Euz/Ed953H6aZev+FOU1FXqhlhPPbUk+rbGwbeHwfBZW6SKsPTP8kRaQYi+Akjhd0ysUGg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AMDPR08MB11481 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM2PEPF0001C710.eurprd05.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 65652fcc-3c53-4e94-7976-08dec18393bb X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|35042699022|36860700016|82310400026|14060799003|376014|22082099003|18002099003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info: WNuzWic7q/joUOwvrIVotXc26PkUcE4cCkEDuLxAn8QZIDCwJZW/xYzKQJivg7nelCOrtCiZzaoY45GoqlExOY1/iRVUezEOt6CshuKL1CTLGEjhxetA3X/3KH5xXFCOB4Jk499s56g6WPPRecqaB2TsifdpAGHkhPfg4AI3mgQ5F38Y0t3LIyzudMzB7LJy2NpiLsffe/hOtsjQtUTBJeg2ru24tQ+qp7VWB4lbop3O1BYmgydqPZCA4lM2aOhSluDneW0hgurfRPJVMinv63vDYHg8ks+UyHzoLEzv/M32gC6Uvuq38fj89PTWh6kA/Fl6+Xn2/U9UAn5XSdfPY9juWJx6r7gCmzz7EMRdZdSNKZhFED4ZX0esxWKjQh3TyjO3Nw98BStElN12AH842s+htnC75ZcHbjRbjkuQwMUzahgHVhKae4ugLZaxq8XfCw3EVhbs+H9SZst2h2LPNT4nELnuebjshMFTpVb7F6o6DI+iY5IYQ1o0OkCyk3uxD8B/tuGmpBuJsMDHDgiDimIFYSQlmFk7KIJZCgsIB6Pk7e7jq/txYqITmy6yjwO50TQonTBSZWFRzDLr3iai5SNALHz3cDkexZkjPbNFpxM/bvLc9NcmQeDhPAeNspFG4CfPiN+fACH9EBAeGHx6XCzTUk6tfrurLA04qMATE3s7O19CMnM8XBcmEQ5IUzUU X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(35042699022)(36860700016)(82310400026)(14060799003)(376014)(22082099003)(18002099003)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: OAZKHfkyw0S7ELfzgvlGqW6hJ3jFCDigBVLqxZ9jyM3nMaBD0tfU4sxfjFtydm1aMVZ5nxhvkG6VvgGYq1qWz0zmVTchwhY5yP1uKoDDV0MU3v5Mk31rtSzKicDMCX4+rZi25rD+6UVYN9pygBsbpn7qt2y1JadgU+LYQkoyVpk9Gu3kfFrodfs2QjrvQmUoo8e1pcvSWDphmLz5dSBAl9gRlvwk1ACaIIfCp16jNj4lNyl4fkeo75uCnO5Nbsa9c+f2kLX6sg9jv7wSW2Vuu/IJaJ1rsmmsusCsVMPO5YExgndtfjyxeRPQMkmfR5Un5xLlgIgx2EAwWu5cUBTlEaMLQa50+ytjb7frJbc9hhBUByqfebrb3ha2DNUlYYQqfN/HkJqLHcuhfqOq+gRy9Nkisxlt+JFAFCMfXjXqUKg805EQKTHcbgYKn6QOLYmk X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:21:01.9288 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8b0ce553-4674-480f-0be3-08dec183b8e8 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: AM2PEPF0001C710.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI0PR08MB11303 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_BLOCKED, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org New tests verify that GCC can generate predicated vector-length specific code for AArch64 if the specified vector length is shorter than, equal to, or longer than the number of elements to be processed (including if the specified length is sufficient but the minimum length would not be); other tests verify that GCC can generate predicated vector-length agnostic code for AArch64 if the minimum length (of 16 bytes) is shorter than, equal to, or longer than the number of elements to be processed. The eighth new test verifies a surprising invariant: According to the default tuning model, the cost of a copy of two contiguous bytes is the same whether the operation is vectorized or not, therefore it is vectorised. That decision is reached despite the fact that the model pessimistically allows for predicate masks that are not ultimately produced as part of the vectorized code. (2x scalar_store + 2x scalar_load = cost 4 vs. 1x vector_load + 1x vector_store + 2x vector_stmt = cost 4.) gcc/testsuite/ChangeLog: * gcc.target/aarch64/sve/slp_pred_1.c: New test. * gcc.target/aarch64/sve/slp_pred_1_run.c: New test. * gcc.target/aarch64/sve/slp_pred_2.c: New test. * gcc.target/aarch64/sve/slp_pred_3.c: New test. * gcc.target/aarch64/sve/slp_pred_3_run.c: New test. * gcc.target/aarch64/sve/slp_pred_4.c: New test. * gcc.target/aarch64/sve/slp_pred_5.c: New test. * gcc.target/aarch64/sve/slp_pred_6.c: New test. * gcc.target/aarch64/sve/slp_pred_6_run.c: New test. * gcc.target/aarch64/sve/slp_pred_7.c: New test. * gcc.target/aarch64/sve/slp_pred_8.c: New test. * gcc.target/aarch64/sve/slp_pred_harness.h: Test harness shared between tests for vectorization with SVE predication. --- .../gcc.target/aarch64/sve/slp_pred_1.c | 33 ++++++++++++++++ .../gcc.target/aarch64/sve/slp_pred_1_run.c | 6 +++ .../gcc.target/aarch64/sve/slp_pred_2.c | 33 ++++++++++++++++ .../gcc.target/aarch64/sve/slp_pred_3.c | 33 ++++++++++++++++ .../gcc.target/aarch64/sve/slp_pred_3_run.c | 6 +++ .../gcc.target/aarch64/sve/slp_pred_4.c | 33 ++++++++++++++++ .../gcc.target/aarch64/sve/slp_pred_5.c | 36 +++++++++++++++++ .../gcc.target/aarch64/sve/slp_pred_6.c | 39 +++++++++++++++++++ .../gcc.target/aarch64/sve/slp_pred_6_run.c | 6 +++ .../gcc.target/aarch64/sve/slp_pred_7.c | 38 ++++++++++++++++++ .../gcc.target/aarch64/sve/slp_pred_8.c | 23 +++++++++++ .../gcc.target/aarch64/sve/slp_pred_harness.h | 28 +++++++++++++ 12 files changed, 314 insertions(+) create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_1_run.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_2.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_3.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_3_run.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_4.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_5.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_6.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_6_run.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_7.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_8.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/slp_pred_harness.h diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_1.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_1.c new file mode 100644 index 00000000000..cfb732abd55 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_1.c @@ -0,0 +1,33 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only -msve-vector-bits=scalable" } */ + +#include + +/* Test that we can vectorize with SVE predication when generating vector-length + agnostic code if the minimum possible vector length (of 16 bytes) is larger + than the number of elements to be processed. */ + +void +f (uint8_t *x) +{ + x[0] += 1; + x[1] += 2; + x[2] += 1; + x[3] += 2; + x[4] += 1; + x[5] += 2; + x[6] += 1; + x[7] += 2; + x[8] += 1; + x[9] += 2; + x[10] += 1; + x[11] += 2; + x[12] += 1; + x[13] += 2; + x[14] += 1; // one less than the minimum vector length +} + +/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7].b, xzr, x[0-9]\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tld1b\tz[0-9]+\.b, p[0-7]/z, \[x[0-9]\]\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tadd\tz[0-9]+\.b, z[0-9]+\.b, z[0-9]+\.b\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tst1b\tz[0-9]+\.b, p[0-7], \[x[0-9]\]\n} 1 } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_1_run.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_1_run.c new file mode 100644 index 00000000000..b76f8d52a1f --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_1_run.c @@ -0,0 +1,6 @@ +/* { dg-do run { target aarch64_sve_hw } } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only -msve-vector-bits=scalable" } */ +#include "slp_pred_harness.h" +#include "slp_pred_1.c" + +HARNESS (15) diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_2.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_2.c new file mode 100644 index 00000000000..464f251f955 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_2.c @@ -0,0 +1,33 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only -msve-vector-bits=128" } */ + +#include + +/* Test that we can vectorize with SVE predication when generating vector-length + specific code if the configured vector length is larger than the number of + elements to be processed. */ + +void +f (uint8_t *x) +{ + x[0] += 1; + x[1] += 2; + x[2] += 1; + x[3] += 2; + x[4] += 1; + x[5] += 2; + x[6] += 1; + x[7] += 2; + x[8] += 1; + x[9] += 2; + x[10] += 1; + x[11] += 2; + x[12] += 1; + x[13] += 2; + x[14] += 1; // one less than the configured vector length +} + +/* { dg-final { scan-assembler-times {\tptrue\tp[0-7].b, mul3\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tld1b\tz[0-9]+\.b, p[0-7]/z, \[x[0-9]\]\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tadd\tz[0-9]+\.b, z[0-9]+\.b, z[0-9]+\.b\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tst1b\tz[0-9]+\.b, p[0-7], \[x[0-9]\]\n} 1 } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_3.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_3.c new file mode 100644 index 00000000000..7d16fdcad3e --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_3.c @@ -0,0 +1,33 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only -msve-vector-bits=scalable" } */ + +#include + +/* Test that we can vectorize with SVE predication when generating vector-length + agnostic code if the minimum possible vector length (of 16 bytes) is equal to + the number of elements to be processed. */ + +void +f (uint8_t *x) +{ + x[0] += 1; + x[1] += 2; + x[2] += 1; + x[3] += 2; + x[4] += 1; + x[5] += 2; + x[6] += 1; + x[7] += 2; + x[8] += 1; + x[9] += 2; + x[10] += 1; + x[11] += 2; + x[12] += 1; + x[13] += 2; + x[14] += 1; + x[15] += 2; // exactly fits the minimum vector length +} + +/* { dg-final { scan-assembler-times {\tldr\tq[0-9]+, \[x[0-9]\]\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tadd\tz[0-9]+\.b, z[0-9]+\.b, z[0-9]+\.b\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tstr\tq[0-9]+, \[x[0-9]\]\n} 1 } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_3_run.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_3_run.c new file mode 100644 index 00000000000..eb3093059d1 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_3_run.c @@ -0,0 +1,6 @@ +/* { dg-do run { target aarch64_sve_hw } } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only -msve-vector-bits=scalable" } */ +#include "slp_pred_harness.h" +#include "slp_pred_3.c" + +HARNESS (16) diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_4.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_4.c new file mode 100644 index 00000000000..915ddb74fd1 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_4.c @@ -0,0 +1,33 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only -msve-vector-bits=128" } */ + +#include + +/* Test that we can vectorize with SVE predication when generating vector-length + specific code if the configured vector length is equal to the number of + elements to be processed. */ + +void +f (uint8_t *x) +{ + x[0] += 1; + x[1] += 2; + x[2] += 1; + x[3] += 2; + x[4] += 1; + x[5] += 2; + x[6] += 1; + x[7] += 2; + x[8] += 1; + x[9] += 2; + x[10] += 1; + x[11] += 2; + x[12] += 1; + x[13] += 2; + x[14] += 1; + x[15] += 2; // exactly fits the configured vector length +} + +/* { dg-final { scan-assembler-times {\tldr\tq[0-9]+, \[x[0-9]\]\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tadd\tz[0-9]+\.b, z[0-9]+\.b, z[0-9]+\.b\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tstr\tq[0-9]+, \[x[0-9]\]\n} 1 } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_5.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_5.c new file mode 100644 index 00000000000..e7dd8a832c5 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_5.c @@ -0,0 +1,36 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only -msve-vector-bits=256" } */ + +#include + +/* Test that we can vectorize with SVE predication when generating + vector-length specific code if the number of elements to be + processed is greater than the minimum possible vector length + (of 16 bytes) but less the configured vector length. */ + +void +f (uint8_t *x) +{ + x[0] += 1; + x[1] += 2; + x[2] += 1; + x[3] += 2; + x[4] += 1; + x[5] += 2; + x[6] += 1; + x[7] += 2; + x[8] += 1; + x[9] += 2; + x[10] += 1; + x[11] += 2; + x[12] += 1; + x[13] += 2; + x[14] += 1; + x[15] += 2; + x[16] += 1; // one more than the minimum vector length +} + +/* { dg-final { scan-assembler-times {\twhilelo\tp[0-7].b, xzr, x[0-9]+\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tld1b\tz[0-9]+\.b, p[0-7]/z, \[x[0-9]+\]\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tadd\tz[0-9]+\.b, z[0-9]+\.b, z[0-9]+\.b\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tst1b\tz[0-9]+\.b, p[0-7], \[x[0-9]+\]\n} 1 } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_6.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_6.c new file mode 100644 index 00000000000..98e91b1af9a --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_6.c @@ -0,0 +1,39 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only -msve-vector-bits=scalable" } */ + +#include + +/* Test that we do not attempt to vectorize with SVE predication when + generating vector-length agnostic code if the minimum possible + vector length (of 16 bytes) is smaller than the number of elements + to be processed. */ + +void +f (uint8_t *x) +{ + x[0] += 1; + x[1] += 2; + x[2] += 1; + x[3] += 2; + x[4] += 1; + x[5] += 2; + x[6] += 1; + x[7] += 2; + x[8] += 1; + x[9] += 2; + x[10] += 1; + x[11] += 2; + x[12] += 1; + x[13] += 2; + x[14] += 1; + x[15] += 2; + x[16] += 1; // one more than the minimum vector length +} + +/* { dg-final { scan-assembler-times {\tldr\tq[0-9]+, \[x[0-9]\]\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tadd\tz[0-9]+\.b, z[0-9]+\.b, z[0-9]+\.b\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tstr\tq[0-9]+, \[x[0-9]\]\n} 1 } } */ + +/* { dg-final { scan-assembler-times {\tldrb\tw[0-9]+, \[x[0-9], 16\]\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tadd\tw[0-9]+, w[0-9]+, 1\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tstrb\tw[0-9]+, \[x[0-9], 16\]\n} 1 } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_6_run.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_6_run.c new file mode 100644 index 00000000000..d6b6537ac23 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_6_run.c @@ -0,0 +1,6 @@ +/* { dg-do run { target aarch64_sve_hw } } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only -msve-vector-bits=scalable" } */ +#include "slp_pred_harness.h" +#include "slp_pred_6.c" + +HARNESS (17) diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_7.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_7.c new file mode 100644 index 00000000000..ac9508301a3 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_7.c @@ -0,0 +1,38 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only -msve-vector-bits=128" } */ + +#include + +/* Test that we do not attempt to vectorize with SVE predication when + generating vector-length specific code if the configured vector + length is smaller than the number of elements to be processed. */ + +void +f (uint8_t *x) +{ + x[0] += 1; + x[1] += 2; + x[2] += 1; + x[3] += 2; + x[4] += 1; + x[5] += 2; + x[6] += 1; + x[7] += 2; + x[8] += 1; + x[9] += 2; + x[10] += 1; + x[11] += 2; + x[12] += 1; + x[13] += 2; + x[14] += 1; + x[15] += 2; + x[16] += 1; // one more than the configured vector length +} + +/* { dg-final { scan-assembler-times {\tldr\tq[0-9]+, \[x[0-9]\]\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tadd\tz[0-9]+\.b, z[0-9]+\.b, z[0-9]+\.b\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tstr\tq[0-9]+, \[x[0-9]\]\n} 1 } } */ + +/* { dg-final { scan-assembler-times {\tldrb\tw[0-9]+, \[x[0-9], 16\]\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tadd\tw[0-9]+, w[0-9]+, 1\n} 1 } } */ +/* { dg-final { scan-assembler-times {\tstrb\tw[0-9]+, \[x[0-9], 16\]\n} 1 } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_8.c b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_8.c new file mode 100644 index 00000000000..c21e1acdc81 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_8.c @@ -0,0 +1,23 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize --param=aarch64-autovec-preference=sve-only" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include + +/* According to the default tuning model, the cost of this function is the same + whether it is vectorized or not, therefore it is vectorised. That decision + is reached despite pessimistically allowing for predicate masks that are not + ultimately produced as part of the vectorized code. */ + +/* +** f: +** ldr (h[0-9]+), \[x1\] +** str \1, \[x0\] +** ret +*/ +void +f (uint8_t (*dst)[2], uint8_t (*__restrict src)[2]) +{ + (*dst)[0] = (*src)[0]; + (*dst)[1] = (*src)[1]; +} diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_harness.h b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_harness.h new file mode 100644 index 00000000000..ac569fc670c --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_pred_harness.h @@ -0,0 +1,28 @@ +/* Test harness shared between tests for vectorization with SVE predication. */ + +#define HARNESS(N) \ + int __attribute__ ((optimize (1))) main (void) \ + { \ + uint8_t a[N], b[N]; \ + for (unsigned int i = 0; i < N; ++i) \ + { \ + a[i] = i * 2 + i % 5; \ + b[i] = a[i]; \ + asm volatile ("" ::: "memory"); \ + } \ + f (a); \ + for (unsigned int i = 0; i < N; i += 2) \ + { \ + b[i]++; \ + if (a[i] != b[i]) \ + __builtin_abort (); \ + if (i + 1 < N) \ + { \ + b[i + 1] += 2; \ + if (a[i + 1] != b[i + 1]) \ + __builtin_abort (); \ + } \ + asm volatile ("" ::: "memory"); \ + } \ + return 0; \ + } From patchwork Wed Jun 3 15:19:23 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136414 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 02F314BA2E33 for ; Wed, 3 Jun 2026 15:36:04 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 02F314BA2E33 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=TEr6+JIc; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=TEr6+JIc X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from MRWPR03CU001.outbound.protection.outlook.com (mail-francesouthazon11011064.outbound.protection.outlook.com [40.107.130.64]) by sourceware.org (Postfix) with ESMTPS id 04C804BA2E2F for ; Wed, 3 Jun 2026 15:21:14 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 04C804BA2E2F Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 04C804BA2E2F Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=40.107.130.64 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500074; cv=pass; b=vn9qzxI8FRcGUuWZs17PhbsYiYeY86Z/hzrQ2zU92nVXBmqLFC1x7Mci/KHookLAYNU2AIPULMJuyk/IBBZJWoXHdty+MhSTGS5siQVL4Gu2krT9L9nNYyg+UFP2AXpCCT2/+zcGl2Yft8qao3Vmq2uEewheYyLcTy71cwfoXkQ= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500074; c=relaxed/simple; bh=HSJo5mBZP/jgMWhrOjbzOLExfcFZuQWNR25bTCVNJoo=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=SGrtruyNqLPqGj6UT6YBxoJeDDjhwNSdXgW24vjyBqyrx2BOowxZop0IzODGLvkuEMo3CvN710R4WFxg9CK+RJdldo7JsC8lBW3BzljC4/izfZOfJ1Si3YHGS5qEJaPzlrYUWQf4MLAfmzGgpZ7JLKJUr9RVZ/ZZ8WmG9Jd2BfI= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=TEr6+JIc; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=TEr6+JIc DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 04C804BA2E2F ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=wGlqdhmalWVHXojbyDz+dq+jXQr4thXBDyChDuc5m/yg0qwy1SnyoDa4ua85BsuL9nMqdfCN6s5XI0HwLm8vC7NDNcGt7B8zvewADfB+QwAd4w0UQg6XcTMahjUm0YZ+fQ6iHawVDunrIZeWgxqShewHN0WWBofd6bBCt4kuBfX8mkbz+I6yR+q/vHNwa0LqCVy9+VPtmrZf9gQb7GbJFNzPbglt8qzmvkG8q26LVD7NGCiW0p3pp77FpldyVKMjfz19FzQXEWxpwLwoh9TaWIWq1CL4lTSiODeXNUISyaruXjhqG8VYUPdLm8jYHGvrQcbI162F96DJW4WDzhfIbw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ng0+8jYQr12P6u0FYwf6zEGLXTJ29/2YdULAbM0ychM=; b=yr+TIt3/f5K7Pwb3UmwEGlHgdG88DXWO3Vklk53UiSKNs2DpU7xH2r/WZWxTttJ4IKO1xnkCygIyYijPVgilHiQ4F33mHc9Uu1PaXKxE9ajbcmQguZIC7d0trhAuA+8bQwknFcFj+0cvzMuuwqqfnjVkG3/Rel7U81HXR9LsEoUwsFOLrmYuwQv+6n5wSy558DSmDOK8XEFZ7FMvfEhr+4jMzkdAzjiG2ZH6MmHIY1F4tE1H6MZ4pcEUhU7p3s69Mu+1khYapN0kLAtGKSRQeGMbrzZUlIBniTU1HsvP/whkuD6yKYHV88Mnbk8oahqR6yLHbqDV54Y9mVAAwihstw== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ng0+8jYQr12P6u0FYwf6zEGLXTJ29/2YdULAbM0ychM=; b=TEr6+JIcUN42ftDu+EWwKp35b9LbWnzJWrWp/paDTJOdwxfp+U8pn5UavuZPihoq0z7z2xrzgbcOeNTeuvMA9UvmuWGB4S5etQWDeIZNKItbrN6FA0pZ9ThuHeoMLQ3janpKnBj2RNNpsuqVPsDE3O140lm0GnPmoOcWDqZ2fU4= Received: from AM0P309CA0018.EURP309.PROD.OUTLOOK.COM (2603:10a6:20b:28f::11) by GV1PR08MB8665.eurprd08.prod.outlook.com (2603:10a6:150:82::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:21:09 +0000 Received: from AM3PEPF0000A797.eurprd04.prod.outlook.com (2603:10a6:20b:28f:cafe::2) by AM0P309CA0018.outlook.office365.com (2603:10a6:20b:28f::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:21:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by AM3PEPF0000A797.mail.protection.outlook.com (10.167.16.102) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:21:09 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CYi3AN1CrVnNB6orkU7B7LfOWFKRVJ01tpGbY41sFVTR3p5VE7eyLdxK8Mk6Rt6/ncjbyrPAfbGQNL7K/kd0V/K+1FXnIvkV2HuQOXdJSi0n+QZC85OfUy/Gvhm9hVu2n7jrr44eN2gHLhsT6qieyPHnCT5baqwVGcpuYXR+YVhs37nV+y36q/LeDmvPdLDjo/BZj68AfHBCcYNzLchqAewjgk/p0LrMQxL2BmD4XynYZKLW9/0QNvUgbdu16Z3e6PPUjgi1LgJpjH1OW4GHxhzdqMAN0fAjvSjxzWVdgylfOudZNpTKFhWyXyLI2nR51gWBXE5oA14Mym7geF7sXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ng0+8jYQr12P6u0FYwf6zEGLXTJ29/2YdULAbM0ychM=; b=C3kcnBUgzD9iyIE/2bxGUPpFSdTuYRDzZ1veVR/Acfli0WWT9Y9Pq4ydsg47gStx+FIsV6wIxe/AQ1hU2POMkvTP36hv2BPrvE34y9oQg73SF2xdKLEIvEYER09PKTtHXSquYyr3Pxf8C5cpbVbHCsvdvy7L5TrKoGAVpTNj18V4NInlzZ5ujmS07DshihcgyNejGsmdiX5b+Hj8xQb/I9KpcOqFhDyQG0RirJGoPnYhPgaYivxAKsqNNTj9ZD/0epJbgist6lkemmM2Ks27Vmzqrye59XIa8Qw9dvIOminYXkS+shog2ffIYPQhWcDq0etXLpblM/ZRiCuOuRGE0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ng0+8jYQr12P6u0FYwf6zEGLXTJ29/2YdULAbM0ychM=; b=TEr6+JIcUN42ftDu+EWwKp35b9LbWnzJWrWp/paDTJOdwxfp+U8pn5UavuZPihoq0z7z2xrzgbcOeNTeuvMA9UvmuWGB4S5etQWDeIZNKItbrN6FA0pZ9ThuHeoMLQ3janpKnBj2RNNpsuqVPsDE3O140lm0GnPmoOcWDqZ2fU4= Received: from DUZPR01CA0184.eurprd01.prod.exchangelabs.com (2603:10a6:10:4b3::7) by FRZPR08MB10902.eurprd08.prod.outlook.com (2603:10a6:d10:13b::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:20:00 +0000 Received: from DU6PEPF00009526.eurprd02.prod.outlook.com (2603:10a6:10:4b3:cafe::85) by DUZPR01CA0184.outlook.office365.com (2603:10a6:10:4b3::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:20:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009526.mail.protection.outlook.com (10.167.8.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:20:00 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:35 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:35 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 11/12] Add extra conditional dump output to the vectorizer Date: Wed, 3 Jun 2026 15:19:23 +0000 Message-ID: <20260603151924.53706-12-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009526:EE_|FRZPR08MB10902:EE_|AM3PEPF0000A797:EE_|GV1PR08MB8665:EE_ X-MS-Office365-Filtering-Correlation-Id: 0c69d1a1-f17b-414b-b1c2-08dec183bd22 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|1800799024|36860700016|82310400026|376014|6133799003|3023799007|22082099003|18002099003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info-Original: Sz89V1YzcmhrgFRaUFmpr8tP6lSxHJqLb9P21pn+9qjKkRkWc+ykETqnt8dQIsqw3xiwBNCBt535gm+K7CA3v6l4wZVmLfZhEK2PRFDYKnNPLmWu2jHnbSmCVXtaSPfVRX1kCXHfjvNA/6hKBPL5sH2YXEPgvHz1fMI2xWdxRs2D4n98uGA1ODoqDhaCM0YRik9rWPy9rkvP0VSC9vlBtxFRPXqIpgEVFJxUHFmFxfPIDe9KTpdP3lH6eYCg0y0gxVCRUdnGfDkMPM/baD5Qq5OhFsA0tSPK+rOok5sCBGY0wg1Lc5WNd15BlnNSPmUpPH0RKmbhWZdernj4rjcA3zV9VtL6HtMHaO9EhnDJ6SynThIDgPIAHpGiZQrenjpxdZ6EsCP5z63mtz0pT6ZtpwUMUuLeVCII8XeQhofMdDr246CxidKwJR2gi2I+sLTGvQIB7qq7sHqKkVNpYkAhDRQdKqvSDhVLl+JXzhrBFeghJznAe68AOc1t0tEb82H8H78zREFFLFgVetOy2PziwpgQYHvfw398a7YnkJSnr0fA2LTj+X22F/8ZTeNntRth8PUQTgTnt+2FSfygQt24yXz1R+g/mNcfCIQJyasumd/ndJRui6zrQiD1lc8lw9kUhM0Jt4PWP+Vz+gNxLwWxklDv+gNRQUCXPICMiAE+erBkK0fndsVthwzzy0T7CFm/TdyrKEC3gILoznoWMnuDBQHHdjRZI36hxCjCNO9Crcc= X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(36860700016)(82310400026)(376014)(6133799003)(3023799007)(22082099003)(18002099003)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: D+wGBGQz8v5CRkgdk0vO9aWVoyZIaYCySv9jNuS5N5BMcLcvqoSTUH3eviQQu3ULX9iy8aOW7Ix61VKdumQUnLmrlQhH4KhVfeEcllWzogjdDAOkASjmh0RRlkS50yPhg8DfYYlQ9KnWdZ3MY30DX7vba9OGgJPvf568hev55F3tU7udX5qbJyDKEKVPGpTvixT3c3WUPc3Pxx9pDyWg3qbgBu9BXhCaXjvSo2fnFCiWSC6qfbLhSssyCM2e1ercBJvxCFRnLJPNbQBW7b6XQRYjLCiFVvoBpWaa6rnYIWrcuDY713wIUVWwhki/PBONQOhhhWgtTxBrCXQO0FZF/w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: FRZPR08MB10902 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM3PEPF0000A797.eurprd04.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 2f8f0d17-5f81-43bd-0fc8-08dec1839401 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|1800799024|376014|35042699022|14060799003|36860700016|3023799007|22082099003|18002099003|6133799003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info: cVpj0tifcV/xcwFtndlMqr5eK3dLpdo1mWBDrj/rYNMBLohxuJuirYsl6R/q6vDbGyFpaeRCpwTksUoibnxxn4nd7h8zZtqaBX17QEK81455NvQ/QuUZUiiFWMIIFh+ErkQxMzA6r01PESadsoU0aU2rAxHnn0zkLKzhM1UquwJtn6Ji5LB9/Ygo3hn/4C2vUkLLYeQ78KSqOiAqQRQY9TINJe9cf34zgvYAqPkmSxSaHG/eEADSMEKOEXx733FBCMMlcwPtiTp18bjQUo0cbGFg0DF1RmjNdLWWGQOqDS5jfWBw0Wz/AF23dr60yrIS24FHGsiU9yAKuYV7Il6Eq2VbFpcjAbfRR8VsqQV7vdz+YJH6D1FD74ai82uiJI96qtp7DN7Uh6xiC5ju2NlBMVlPNcbSSt5Fpqks/hznSuAygUUiJS53L8TGXNyU80QJHLf9frnAvos3J7iwrYmrFMwWFTR+Zmz/SZ3lOzjSOC7x2gHxfszGF/npFWNn6CnTA9StbNlHjcKl6AXgoCiDoe558HEoMsZ8rjWRCZZ8Z46T4LRJzO9MdfFs4jIQGJNqIxHH/Dq7dNiJvUd/r9M0zIFYwK+ULp8g5if4tF+wVuRXY4D3HeANvbpLrOo3dSqgiHWU/YIAvHdSYP1KNK9FAb49zCpKlFAdBTkm02qPl0xxgev3s+WRmh8bz7Hwp+XJw5HXGTMrKzy4dlFF7aJ/qwIjJquRIGbqiUhIu7QzsFo= X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(82310400026)(1800799024)(376014)(35042699022)(14060799003)(36860700016)(3023799007)(22082099003)(18002099003)(6133799003)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: VDeeBUDgNDLH/T9o+nQ2SKd7IdP9wGe6zYhlRnA0N97sj+V0uwrAyHs7NRsvzoblEaI5fajE3jP3ZhDPJ33mUc8bCHip7ZlRtCE6X1pBzhKho3grraF/EKHAxbZiNRzeNu37Wn5ThTH+KSbizSWun9sBcT/YnTIWMiMGhCGMPI1DJj2l7ZNLTGUrTyDm9onbNQh1rJs3zXrYGiWIuvbvzR9nqBdvbx8k+pyS6KPe0M1MtsvlFHaMc7MYmKRy2GMXQ22ATQ79BG3e93p1JCG5EqeK5KuidHTfN47LOkIydRDPN04AzJPfjfzPWpy5k8fjH+OY9Sy36owQfwODBBQmijX/exe7zIFDFB7DVRy61xnY7iHU/svNdZa0YU6kCDVuNDGyX7sxookvodq9MBvf6UZIDplsEz9JvQD4KaeXDrqGdVenhxC6UNk8gs9HVrwY X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:21:09.0332 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0c69d1a1-f17b-414b-b1c2-08dec183bd22 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: AM3PEPF0000A797.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR08MB8665 X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, RCVD_IN_DNSWL_BLOCKED, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Instruments those things that needed to be instrumented in order to develop predicated tails for basic block SLP (superword-level parallelism). gcc/ChangeLog: * tree-vect-loop.cc (vect_get_max_nscalars_per_iter): Dump the result. (vect_verify_full_masking): Dump the number of vectors and number of scalars per iteration. * tree-vect-slp.cc (vect_prologue_cost_for_slp): Dump the SLP tree node's address, vector type and group size. * tree-vect-stmts.cc (vectorizable_conversion): Dump the vector type inferred from the scalar type. (vectorizable_comparison_1): Dump the vector type guessed for an invariant comparison. (get_vectype_for_scalar_type): Dump the natural choice of vector type and instrument the fallback loop which searches for a narrower type. (vect_maybe_update_slp_op_vectype): Dump the new vector type and number of lanes of an operand when its vector type is updated. --- gcc/tree-vect-loop.cc | 10 ++++++++++ gcc/tree-vect-slp.cc | 6 ++++++ gcc/tree-vect-stmts.cc | 38 +++++++++++++++++++++++++++++++++++++- 3 files changed, 53 insertions(+), 1 deletion(-) diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 7503fd084cf..f8e33a1d846 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -922,6 +922,11 @@ vect_get_max_nscalars_per_iter (loop_vec_info loop_vinfo) rgroup_controls *rgm; FOR_EACH_VEC_ELT (LOOP_VINFO_MASKS (loop_vinfo).rgc_vec, i, rgm) res = MAX (res, rgm->max_nscalars_per_iter); + + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "max_nscalars_per_iter=%u\n", + res); + return res; } @@ -1023,6 +1028,11 @@ vect_verify_full_masking (loop_vec_info loop_vinfo) unsigned int nscalars_per_iter = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype), LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant (); + if (dump_enabled_p ()) + dump_printf_loc ( + MSG_NOTE, vect_location, + "verify_full_masking: nvectors=%u, nscalars_per_iter=%u\n", nvectors, + nscalars_per_iter); if (rgm->max_nscalars_per_iter < nscalars_per_iter) { diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index 6af13e65e19..03435dc653d 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -8876,6 +8876,12 @@ vect_prologue_cost_for_slp (vec_info *vinfo, slp_tree node, When all elements are the same we can use a splat. */ tree vectype = SLP_TREE_VECTYPE (node); unsigned group_size = SLP_TREE_SCALAR_OPS (node).length (); + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "vect_prologue_cost_for_slp: node %p, vector type %T, " + "group_size %u\n", + (void *) node, vectype, group_size); + unsigned HOST_WIDE_INT const_nunits; unsigned nelt_limit; unsigned nvectors = vect_get_num_copies (vinfo, node); diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 367a9c63ea4..773cc9ed37f 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -5557,7 +5557,12 @@ vectorizable_conversion (vec_info *vinfo, /* If op0 is an external or constant def, infer the vector type from the scalar type. */ if (!vectype_in) - vectype_in = get_vectype_for_scalar_type (vinfo, rhs_type, slp_node); + { + vectype_in = get_vectype_for_scalar_type (vinfo, rhs_type, slp_node); + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "inferred vector type %T\n", + vectype_in); + } if (!cost_vec) gcc_assert (vectype_in); @@ -12786,6 +12791,16 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype, if (!vectype) { vectype = get_vectype_for_scalar_type (vinfo, TREE_TYPE (rhs1), slp_node); + if (dump_enabled_p ()) + { + if (vectype) + dump_printf_loc (MSG_NOTE, vect_location, + "invariant comparison, guessed type %T\n", + vectype); + else + dump_printf_loc (MSG_NOTE, vect_location, + "invariant comparison, no usable vector type\n"); + } if (!vectype || maybe_ne (TYPE_VECTOR_SUBPARTS (vectype), nunits)) return false; } @@ -13829,7 +13844,19 @@ get_vectype_for_scalar_type (vec_info *vinfo, tree scalar_type, /* Register the natural choice of vector type, before the group size has been applied. */ if (vectype) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "get_vectype_for_scalar_type: natural type for %T " + "(ignoring group size %u): %T\n", + scalar_type, group_size, vectype); vinfo->used_vector_modes.add (TYPE_MODE (vectype)); + } + else if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "get_vectype_for_scalar_type: no natural type for %T " + "(ignoring group size %u)\n", + scalar_type, group_size); /* If the natural choice of vector type doesn't satisfy GROUP_SIZE, try again with an explicit number of elements. A vector type satisfies @@ -13862,6 +13889,11 @@ get_vectype_for_scalar_type (vec_info *vinfo, tree scalar_type, { vectype = get_related_vectype_for_scalar_type (vinfo->vector_mode, scalar_type, nunits); + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "get_vectype_for_scalar_type: trying %u elements " + "of type %T: %T\n", + nunits, scalar_type, vectype); nunits /= 2; } while (nunits > 1 && !vectype); @@ -14190,6 +14222,10 @@ vect_maybe_update_slp_op_vectype (vec_info *vinfo, slp_tree op, tree vectype) } SLP_TREE_VECTYPE (op) = vectype; + if (dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, + "updated vectype of operand %p with %u lanes to %T\n", + (void *) op, SLP_TREE_LANES (op), vectype); return true; } From patchwork Wed Jun 3 15:19:24 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Bazley X-Patchwork-Id: 136407 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4C3CF4BA2E3D for ; Wed, 3 Jun 2026 15:23:46 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 4C3CF4BA2E3D Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=ex9gtXmo; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=ex9gtXmo X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from AM0PR83CU005.outbound.protection.outlook.com (mail-westeuropeazlp170100001.outbound.protection.outlook.com [IPv6:2a01:111:f403:c201::1]) by sourceware.org (Postfix) with ESMTPS id 5C5D54BA23C6 for ; Wed, 3 Jun 2026 15:21:12 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 5C5D54BA23C6 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 5C5D54BA23C6 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c201::1 ARC-Seal: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500072; cv=pass; b=YXDwC6VFjMJlkUASYa5C9jhZbATBo2S0uEvZLpT7sKyHI9LScnMnstFN6uKKraRB1kllLaj1+5JDo0IZc+Qte7TpPwVDGXz+jnpglY5w+KXhoWjlt5GwSt9SR+OdDjcBXtPQBocoirX9u1FQP20ItRSyj/yMsDRR2acGF96tkjQ= ARC-Message-Signature: i=3; a=rsa-sha256; d=sourceware.org; s=key; t=1780500072; c=relaxed/simple; bh=qNWIidd+VgXFe5aJlatJ0F2wijjfv143J61nGe30wQY=; h=DKIM-Signature:DKIM-Signature:From:To:Subject:Date:Message-ID: MIME-Version; b=ASYHcEuG2eEXauqt+yMiQbJmJ1O7AizV3mYhi7726eNuUfDQPv8Y6D8OKnwV0SzGftN6ui/dm6xn3XI2bsk3LX5+IXRnvLcV8jxQTp0kqqy9qd8Wbqfnu92Yd0ZdjL5jkcwKl6lIYorngAYwxkr8pAeLBnWvZOO63cb1avF7pLE= ARC-Authentication-Results: i=3; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=ex9gtXmo; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.a=rsa-sha256 header.s=selector1 header.b=ex9gtXmo DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 5C5D54BA23C6 ARC-Seal: i=2; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=pass; b=mBAyIp8bYf2WdQbaUT0F7KU+VWoHYhokTqG7hV40E3BR1rzPcus311fEITVDg4JanUQ6WTtnG4P9+jdYlX/heEgnPGYON4cnpY+C++KZO50q6TTIsHZmXVZMtieUQThCsFguOtiKaQESrZshvdz+tAbRdyD85bZ6SIavxfSAf4zMNGDPuI5wwp/oAmFtB3aeO5vWspB36qXy2x5LxOAfiOT0UZW/ANNvXpKb6lzK7UongBIeygpF6E4/CFuAE/0CiNvtMJ0didbr4NB2Y82mVKzqmKj36UH2M8i452dC2ArmFT1h3WPNLTWttIHILVVPxA/KdZeB7T2AWbtA+ZWC7A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XBlMatWjrlrNh1Uzx8z1cHdJsyJwjc9Dj0HnS5cOdl0=; b=rdyWGZb7MCuyWl08gJd2FwFaGN3RDbKBazseTqEu04BqF3VCw7lLefJk1tf48OZ+lAlq797B+bwdQg5Kbm5WPQ9NE4bHneSHWK+cn/bJrm5VOCRzS5MUXvHedFBmqeHCzKGPjpDdXEn8g1DNjEq/J7st/Pp8CzzviPCZKi68IxS0yqSJbgBzQoifphvNOCPBYoWnk1WjMetvQUwP46feWXBCnzGOxTJ+9eakNlnTUFOHam+hOZGkM8My7+CXc5HRe5FIuIuFs5QBtrQja7qxpVmDCIBybYygb2JMBzL90zhVsEIu+pwaABL5C9ag8C+OFb2K0N0rvdsMEJ/zWn2F9A== ARC-Authentication-Results: i=2; mx.microsoft.com 1; spf=pass (sender ip is 4.158.2.129) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=pass (signature was verified) header.d=arm.com; arc=pass (0 oda=1 ltdi=1 spf=[1,1,smtp.mailfrom=arm.com] dmarc=[1,1,header.from=arm.com]) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XBlMatWjrlrNh1Uzx8z1cHdJsyJwjc9Dj0HnS5cOdl0=; b=ex9gtXmowR31NfdTvCNIJf9/MV4ylJ39owXtGY7JP0rZLtphJNPbVp4lhwPJ/stp4P9tywpFuzWkL6te5ymt2aonUQzXGF568X0mbDiyvisTwXwX26zEJhYAT9gTZ0EG1sXyo8GqBeMEokXy8iSQ1hnjKtOoyiqS+KPZENa+pcw= Received: from CWLP265CA0495.GBRP265.PROD.OUTLOOK.COM (2603:10a6:400:18b::7) by AS2PR08MB8669.eurprd08.prod.outlook.com (2603:10a6:20b:55c::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:21:08 +0000 Received: from AMS0EPF000001B7.eurprd05.prod.outlook.com (2603:10a6:400:18b:cafe::17) by CWLP265CA0495.outlook.office365.com (2603:10a6:400:18b::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:21:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 4.158.2.129) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=arm.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 4.158.2.129 as permitted sender) receiver=protection.outlook.com; client-ip=4.158.2.129; helo=outbound-uk1.az.dlp.m.darktrace.com; pr=C Received: from outbound-uk1.az.dlp.m.darktrace.com (4.158.2.129) by AMS0EPF000001B7.mail.protection.outlook.com (10.167.16.171) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:21:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NpUQi3euzdRlPtSoxI/X+nMeSgZtyJ9xzIfJM+Jq+JCVld1o45DDPbgXDnXkbbpPhYu9fPxLxFL4Pg7zZ7bds6iL1ofaz0Y+aBh45Ve6uG0EidVeF3xM1/Um8PbGnWb9auDf9F+vTqZUxsfPPxc7dOX/op0rios8mqwZLsOZPKs7eITWppbMSJ85Qc2vI9mg0NEV8ZWL/ZQC7m5rlsHDjvNyhvA2Zugg7NqvI2mLbr34vzC/vVXSMFBIAx3PBwVSRbyHiPAZKU92LC1nxRT6oIl3elN62fx+zBEQ1JZLKqIHkIG3jyIXlJaUIs6thba+0E3bLfLsZdvaMmgLZVJofA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XBlMatWjrlrNh1Uzx8z1cHdJsyJwjc9Dj0HnS5cOdl0=; b=QO/rgGmstRswrR2/6JP1v80/v2/jyeJcp4WpZcvqoaMp3c0WJphTmjNfdlTbE2naatODmgNH6Jfy0vp4EmE5FUZkKhrWzSjA1zVIJWF9tsyLRP/Bx+BOjpXgbqXEETxbDudQFwMw+ITe2V0U9qQ4+O2zkqLjCSlpyV6UH38lVwkZTTy9NN6XGBo7o3DkQAbqOe/Gv+ZT1d+L7RA9b5G+ylDkH16Hygorx1pjkNsejxIqMmNdVjIPtaqafWqdliFNL3bNhNDfLLfq1giLXRJ5LLKL6HAKwUzmDqwAN2F4sNKKkF5XSPrJjc8Kpb4ZhV04+ddv3RLHc2pTlQJdjygILA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 172.205.89.229) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arm.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XBlMatWjrlrNh1Uzx8z1cHdJsyJwjc9Dj0HnS5cOdl0=; b=ex9gtXmowR31NfdTvCNIJf9/MV4ylJ39owXtGY7JP0rZLtphJNPbVp4lhwPJ/stp4P9tywpFuzWkL6te5ymt2aonUQzXGF568X0mbDiyvisTwXwX26zEJhYAT9gTZ0EG1sXyo8GqBeMEokXy8iSQ1hnjKtOoyiqS+KPZENa+pcw= Received: from DUZPR01CA0178.eurprd01.prod.exchangelabs.com (2603:10a6:10:4b3::20) by PAVPR08MB9356.eurprd08.prod.outlook.com (2603:10a6:102:30f::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.7; Wed, 3 Jun 2026 15:20:03 +0000 Received: from DU6PEPF00009526.eurprd02.prod.outlook.com (2603:10a6:10:4b3:cafe::66) by DUZPR01CA0178.outlook.office365.com (2603:10a6:10:4b3::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.92.7 via Frontend Transport; Wed, 3 Jun 2026 15:20:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 172.205.89.229) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 172.205.89.229 as permitted sender) receiver=protection.outlook.com; client-ip=172.205.89.229; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (172.205.89.229) by DU6PEPF00009526.mail.protection.outlook.com (10.167.8.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.92.5 via Frontend Transport; Wed, 3 Jun 2026 15:20:03 +0000 Received: from AZ-NEU-EX04.Arm.com (10.240.25.138) by AZ-NEU-EX03.Arm.com (10.240.25.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.29; Wed, 3 Jun 2026 15:19:35 +0000 Received: from ip-10-248-139-165.eu-west-1.compute.internal (10.248.139.165) by mail.arm.com (10.240.25.138) with Microsoft SMTP Server id 15.2.2562.29 via Frontend Transport; Wed, 3 Jun 2026 15:19:35 +0000 From: Christopher Bazley To: CC: , , , Christopher Bazley Subject: [PATCH v11 12/12] Specify behavior of vec_init for scalable vector modes Date: Wed, 3 Jun 2026 15:19:24 +0000 Message-ID: <20260603151924.53706-13-chris.bazley@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260603151924.53706-1-chris.bazley@arm.com> References: <20260603151924.53706-1-chris.bazley@arm.com> MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DU6PEPF00009526:EE_|PAVPR08MB9356:EE_|AMS0EPF000001B7:EE_|AS2PR08MB8669:EE_ X-MS-Office365-Filtering-Correlation-Id: 587b1c2d-2161-4fa6-cb02-08dec183bc8c x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; ARA:13230040|1800799024|376014|36860700016|82310400026|11063799006|56012099006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info-Original: 3J5/OZR9dmZg1ERxkIViToiRYVMNAIQdk27wgBZZDrwfySSZD57DKhD779/dS6IBPD0nInGGHn8psGhv8n+isGBdgjeV2u8o85pl8i6RsW86DLvQDzKZuGC084oyU5nzn60+pKwqPKwGKxlaoIfg15je4GDJzeC4C5vy3PMydtmD2T935aSf/hqNwMC1ZPoZL3EIn0idfkfbLIPJZ0sc9/KFFRPO3CSKs3pFs0Z6nwt2dHJuNgZgZE3uJ8M+24gIP+M1tbxbQqzrBVOte08yAid5dSc7P5ywyhhw8T01+tzC7dNIeBzyW6IBuR21Y2aMDnsVrA8Rla05MWVNPzYDHRsvtGLQUPWmI5iimAwJnmDStHrN2GiC4aOTiETxgsT5oRyd45VSH8TCbypo3/7NFz/widS3+BMsJSPwUWy1IWlihvU593By0PuEeIrTiFjs0FnlUKChalGCdDL7wM8AVQTcI73ILoh8nnOFX2k4bObmo79059ce22wWGJma6lREDIYkJjSzWBdcfpvXuBmeSg76dNySXg3hXQuC7+CKTcAv2WAENlEFZHWzHivz0O5e/L6wJONEJXq9i1r/GyEk4FfKpbZSXZpWyLM49L7TCz3FQcOvFyHrBU7YZG+J+XFkGffnKuO96/M8QPno4F/QK8kOpwOOtVNjjWWKfhjbSHQWmXaurGiSvPVouzWlWBUduuEYsIFP595JnZ1s8h0fMHUjW52tD5jxF3O0/jh/Wxw= X-Forefront-Antispam-Report-Untrusted: CIP:172.205.89.229; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(376014)(36860700016)(82310400026)(11063799006)(56012099006)(22082099003)(18002099003); DIR:OUT; SFP:1101; X-Exchange-RoutingPolicyChecked: PUT1HqHZiRm3HO5OAb8eqmhwq0si/6W+RABCMvbjmA5meDaw8ou2D3I2Rr3aDORfx0fknKgG7JB/ZRzGGom0iZT4SajjtDqrFkwKB+dSmAbAe6j4Ji++70D8TZ3Hp6y16hKKzrLxN0ugHSPQzPuOFHJeUf/Bj4udYDh++9xo2THP6GGJii66uWVv8SLxy31yu3iT62uywzGlbCItBpNg2nxtp8NCaLI807VGPLiZJXNW9CWHD5Q76WM8DvkIMw8pQ8OB53MzdeCLex9ibn7znyJu2Cfk9k/WmxkqVWv+DT3kSSAfPPf9x7iEYlo1hyFa4ljAQcQl8CrqUWVj86KrDg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAVPR08MB9356 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AMS0EPF000001B7.eurprd05.prod.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 6db0dba7-29d3-41cb-8031-08dec1839632 X-Microsoft-Antispam: BCL:0; ARA:13230040|1800799024|82310400026|14060799003|376014|35042699022|36860700016|18002099003|22082099003|11063799006|56012099006; X-Microsoft-Antispam-Message-Info: IFV+jY8WBwr6Lp0Jhle8HAL90owA9tuTgPCLoyjj50w7czrCyf1aJm4JbA5f340KicDvV0FRwPedqOHik5Hrl7id5a0GXD3ZquxBUY+l0slGybBK6mi0Zo/bEGvSEVEfqSq1K3ZpPtTyyuDXUiOMdXeFKb/RFNpDktjUJmbYFHM0IXFjy2Ja/blfXivmOEH98B6M8gJv0co2Jk+rVJgCRcfwrDxl6q89hWtTY3DnQCpucFEvm/zJbvMeC7L96e50Ff3ckl8MB3lkHtz0XzG5LcTzR4Kfhgzgqit6wzExYmxJZ6xdSuCTZ7BoYe4fxy99Dz8N8ilYVWOokDrthfAGvkEg139ewglR6AaKIeImdKUaFZCRbwSZQhlqisTgyG4EzjbzoBxg/7B/VkEj/bJ53NOYfHSmoVLINTW5Rve9y4b+zokX6Ms/uAT2MaxHlGw39PevmLUy1W8qclFOGnFTAKJuUOQVdkFMZ8IqWjz2mgZgZNdAa/BHJMtrWBwa5uKRSbIGPZgEhAzE1ck02QfeJywyg042lubYLgzayMOt8nASZ0ughlV1MdPX//buO8cs7BwkGqWFAQiIoxvO8yGkne07Gr7NK6f0rgLSHjqxV9XcAAV0Z91DT3Sf0JtLksP4H1AAuzbc2nj4scokHNF/BzTzylHYtk29BK8h0H6oZKSrHFDRd+8Moqayu+ydSKfHXUS24Id5yQRpn7VZeAfFFcGaouXg8IT0CNsezCmAGgE= X-Forefront-Antispam-Report: CIP:4.158.2.129; CTRY:GB; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:outbound-uk1.az.dlp.m.darktrace.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230040)(1800799024)(82310400026)(14060799003)(376014)(35042699022)(36860700016)(18002099003)(22082099003)(11063799006)(56012099006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: kicroVo62kLgejydTJSy8LfWri4B0wrPSrVap4ZFV/8yFZFc5WzRKVn792sMPMfGzv3HIdKkRDNA9xcf4sq/HEJuY2aiY6FGRPnMiWidGDqOzS0Dsy3LwIFN2ouEPyo2lzmO2PpsaqB6k73dveJzUbpkBi6v4momJ1CmCDnlnfHYEZvH3zpG9mTIqzGTO3SjaglX0ZQrvc6bzuKXXW+WXhrIyNv3PRwiShHNPoJOCwKJhZ99SgN8GXseMD+g0a6Ynv9ERqCylI6dO8rjacYwrSP5iPgAl/6pvGy9HuLf64LxWx2uCA2+mEZMKfjmo2kQoAVPrVLoZX7iZNszNmoHLP/P4ugAUlnQc+9rHQ8zE4Wh7//eJVnHOh0eFr6dyLhD2pI83fVPJBEEoNumxcYUc61ETNGywEZz1zEtW/EWHclzvrTPGohNtknhG1hCx5Oq X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jun 2026 15:21:08.0465 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 587b1c2d-2161-4fa6-cb02-08dec183bc8c X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[4.158.2.129]; Helo=[outbound-uk1.az.dlp.m.darktrace.com] X-MS-Exchange-CrossTenant-AuthSource: AMS0EPF000001B7.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR08MB8669 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS, SPF_NONE, TXREP, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org gcc/ChangeLog: * doc/md.texi: Update description of vec_init. --- gcc/doc/md.texi | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi index 5a2625ce986..dd11c5adaf5 100644 --- a/gcc/doc/md.texi +++ b/gcc/doc/md.texi @@ -7501,7 +7501,9 @@ Initialize the vector to given values. Operand 0 is the vector to initialize and operand 1 is parallel containing values for individual fields. The @var{n} mode is the mode of the elements, should be either element mode of the vector mode @var{m}, or a vector mode with the same element mode and -smaller number of elements. +smaller number of elements. If @var{m} specifies a scalable vector mode, +then elements beyond the minimum number of elements implied by @var{m} are +initialized to zero instead of using values from operand 1. @mdindex vec_duplicate@var{m} @item @samp{vec_duplicate@var{m}}