From patchwork Wed Sep 2 14:52:42 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142725 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 6FE0B4BAE7D2 for ; Wed, 2 Sep 2026 14:57:03 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 6FE0B4BAE7D2 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=YRBHLBLL X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from CO1PR03CU002.outbound.protection.outlook.com (mail-westus2azon11010020.outbound.protection.outlook.com [52.101.46.20]) by sourceware.org (Postfix) with ESMTPS id 2101F4BA9036 for ; Wed, 2 Sep 2026 14:53:43 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 2101F4BA9036 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 2101F4BA9036 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.46.20 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360823; cv=pass; b=QpyXSIOUm3TtJH2DAejUkPcsqvN0vwpLmKzGeZnXv9kMrsOKjLk/xnQCCa+UiURB7B/uKLEfcQ0fzUbdLhjW1Rm1jeUh5Pjwqbp6+H8BFWeAmTl77bKDp4caJsuc+yJwCrq9k/sousZJS2HM/G2zMMpy+OELf0iJNZupZUvDoIw= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360823; c=relaxed/simple; bh=1dVgl3ui8UouvKrQNH+wDaDk1ZcLCU7/ATZF7XBCLA0=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=dEEp4NmlZwGbZEF+NTYQawwPS5Gt/XHzE65YMClX81kmoljcP7mEX4xPor+Xm4tHEVi/rxGpI7nguCUxNBrVQ0Culypx1A6Sa7JQviNiN2ENwsgecjB60/QiJPl879TqwL8Ei3Le9Hlvt+KqvhZcaoh0aeiYVd+M15HgR4LvJFw= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=YRBHLBLL DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2101F4BA9036 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fY4GbhOmCwfv3tbGRW+7gNPimB6tbfvmjZCSFn6LhwWMLs1etmA8k44S5yLNrbFiKT0S45mhbiJoMiZQU49IneIGV9DgY2OecVetDZlxj9kgL4PGYagLyH8zOyOrAUll1jWpLrOkjcawylrH2IHMi8YOw52XSDeJfV2714YfhanpPI9igMDZFGQJm0c617yI/1xwZYbuitvw1/vFBGr5zz+moQOOEv67gnSSk12s+crySFda10xS+MlI38GsPki9dyHkR+fWQNQNn5ISk7xkScIEtE0mEMzvwn9ZYpe2Hh1CEeau16Yos+/fXneWrPbOObLfeQ1s6aFK4K5SLxZAPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=frAbTHIShWGqcDrlgeafNn5UVnQ5zV7sYD5A/2C1fE8=; b=QdfdTvbnPMG7Kkheb/zcLjcBjIUwmrg7HthEqIyLR1HIxzs8TC/Fv2sksE4V7juQ9uzGYCsVDZoiU1XTSAbKA48kQ8b2AC6FzWi4vLIqrd7Iw8pzJQPgrQCQltcyCufAnIUfKmEd21+3iUwGg/Dvo/ySQANn+dEB2d0phFdMZ3pdvOIBoPMwJBfpIS7gqM9iafKaX48LBfFRJZDQ2HsCWZu4BpdxT6sAVsp+6k4EB8X6cbz4HrjzCjUMIaVkAhDp4y/ukTCVbCl6o/2+c9NegY93ERPiDH4f22DMo9e2kUXbDQl8pnlgRP7iB8OSLjdI+C7TpD/HPeHxbUke4TbbRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=frAbTHIShWGqcDrlgeafNn5UVnQ5zV7sYD5A/2C1fE8=; b=YRBHLBLLtN4Ki5mKnthJB0zBCV6fpkKfrv+DDH6rGPN/taMKnCcTdGLOi0AtryrjxIopyyGiE2/hkrWuhH3gUdR5vzfzd/SL+3wLDbwnFhkC6SurHSaQD55tZYmKKuQ6y1hNGZwnKmA3j6Uo5/9OF7SUBsp/WvOLfAOUOxPOTzvXuXLBf+Xz2HmX3ZA6Mitkr586Tjaxq5hhEYzGK9gGd1JJ8kRDbtCVqen1Clq+K0CF1/3EeLGPAYEsHPLT8mh7NAe9YVl2hl3pA9euyb6cKHr8OQuqm94kc6DazKIJyufmAaa9LeZkal7nzQ+//1ylbWNWmUtzBRxl79SefwG/FQ== Received: from BN9PR03CA0422.namprd03.prod.outlook.com (2603:10b6:408:113::7) by SN7PR12MB7021.namprd12.prod.outlook.com (2603:10b6:806:262::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.12; Wed, 2 Sep 2026 14:53:36 +0000 Received: from BN3PEPF0000B06E.namprd21.prod.outlook.com (2603:10b6:408:113:cafe::7) by BN9PR03CA0422.outlook.office365.com (2603:10b6:408:113::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.11 via Frontend Transport; Wed, 2 Sep 2026 14:53:36 +0000 X-MS-Exchange-Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN3PEPF0000B06E.mail.protection.outlook.com (10.167.243.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.1 via Frontend Transport; Wed, 2 Sep 2026 14:53:36 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:08 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:07 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 01/13] match-sat-alu.pd: Allow wider ranges for signed saturating truncation Date: Wed, 2 Sep 2026 16:52:42 +0200 Message-ID: <20260902145254.77832-3-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B06E:EE_|SN7PR12MB7021:EE_ X-MS-Office365-Filtering-Correlation-Id: 292b79b9-302c-4feb-6a81-08df0901f784 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|23010399003|376014|1800799024|36860700016|82310400026|13003099007|10067099003|6133799003|56012099006|11063799006|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: R4NvteFRaYTej3Wt4QOw6iV1czN476cSB4rDOZ9GG4Jw6gnds6A236mnSI+FOpuuaS1/s2wx69VLvYRADH18Ja93jhRr6sB/oD5DGbw8dkQHgFDy+hkEl67fLTpE96CZVZAwIoE7fz0CUEE33zKyT7h7YZoOCw42VeaFQIoxB64VEj0LayotdVDaNcIqvbzd88d9E+8PHzODFXdbcmNYqrEqD0o+zQeFRLiHV22cSKnlj4EOqhjK1b209naEVBziXoEKuyV0DM1Voz0wMTsE4DC0JALFMmHLzBLOyV75Hfn8j9Nb8lHOYI8FoA6I4dWf1o0pBjimjFy+UmDRpkEp+qSqT1B54lAG1PpPEXjYu7wi6mDI7795Hv+ZOSGG/Qvmaew4leWrXfH7Bvi5Q6nKwHy5LNr0p4CJCm99dN+6BQBq009IFGnUKnhgEiNYJYjEDPR0uYTjLZWuYHD0phVb6wS8yTewESrlNWUc98lmRxglshYwcYCeqK0PxIJXjfBM0lucje2hUpwc/bW5O3jHNaRevUP1NDe5SWq3n4bMrBGMtguFA5ox4/fcxC+cpN82xRQjnPRC+jicfymDwZWCOdAEav4btnVXCgRBP3uCzT6xEH89O1L33tUKrmcBR61v8nTB0cm9F9RxeRn0yGbUxC7ZIsebNP0uUEyUJVTLGxbTgVR6rma+mIlVB8f3z7ecD2QsO7dpd+vyAHLj/diIrg== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(23010399003)(376014)(1800799024)(36860700016)(82310400026)(13003099007)(10067099003)(6133799003)(56012099006)(11063799006)(22082099003)(18002099003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: g1Tm0j+MZYkWUE9iv5D+OudQcn1jM8cs8Qf+izRC9ZAS7ygroMUbpasqEYuPlKGjVK543inAeXHZ102F3h2HhLvm4wuUkG1/X8KWeJKjJmWjW7ntwVNbArwctIxNlejm9dlOhIb0ztB0Wnr3ok3hYIk60+sh0FtF5alRPPjF2cfWi3PDu/VmfsZbxA0jXqnSV90yl6HG69Mj14Jsux/CITWMEqkXyi5aZ3Dy/4Gv8bjzzSVoBkhcrI/hXmZiW1BULsi2GZsVG716zpohBxttwnAJtAT45nUf3HNsLLTWzp7tVBFcJ4lk0N/QLbaLg4aG4iB3L4iDiHumNENvgEbBmkERLfUDXDy84j8vbBGDlZ41n7DQqXdZUPMKWt18kSFeM2Fb5Hc3+VNWQfaSYoISOkWs/ykgdOroEhVNrRe1DWloHzG3Kfy2h1Awyl7XvBbw X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:36.0184 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 292b79b9-302c-4feb-6a81-08df0901f784 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B06E.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7021 X-Spam-Status: No, score=-8.1 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov Signed saturating truncation currently requires the unsigned range type to have the same precision as the signed input. This rejects a wider range type even though it preserves every input value. Allow the range type to be at least as wide as the input. Continue to require a result narrower than the input. Construct the offset and limit constants at the range precision. Use wide-int operations that do not depend on host-width shifts. Tests cover a valid 16-bit input with a 32-bit range and an 8-bit result. They also verify that a 17-bit range for a 32-bit input remains rejected because it can discard input bits. With AArch64 -O3 -march=armv8-a -mmax-vectorization and --param=vect-epilogues-nomask=0: before, 16 elements per iteration: ldp q28, q27, [x3], 32 uzp1 v26.16b, v28.16b, v27.16b saddl v0.4s, v28.4h, v31.4h saddl2 v24.4s, v28.8h, v31.8h saddl v23.4s, v27.4h, v31.4h saddl2 v25.4s, v27.8h, v31.8h ushr v28.8h, v28.8h, 15 ushr v27.8h, v27.8h, 15 cmhi v0.4s, v0.4s, v30.4s cmhi v24.4s, v24.4s, v30.4s cmhi v23.4s, v23.4s, v30.4s cmhi v25.4s, v25.4s, v30.4s uzp1 v27.16b, v28.16b, v27.16b uzp1 v24.8h, v0.8h, v24.8h uzp1 v25.8h, v23.8h, v25.8h neg v27.16b, v27.16b uzp1 v25.16b, v24.16b, v25.16b eor v27.16b, v27.16b, v29.16b bif v27.16b, v26.16b, v25.16b str q27, [x4], 16 after, 8 elements per iteration: ldr q31, [x1, x4] sqxtn v31.8b, v31.8h str d31, [x0, x3] Bootstrapped and tested on aarch64-none-linux-gnu. Tested on x86_64-pc-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (signed_integer_sat_trunc): Allow a wider lossless unsigned range type. Continue to require a narrowing result. Construct the expected constants at the range precision. gcc/testsuite/ChangeLog: * gcc.dg/vect/vect-sat-trunc-range-precision-1.c: New test. * gcc.target/aarch64/vect-sat-trunc-range-precision-1.c: Likewise. * gcc.target/aarch64/vect-sat-trunc-range-precision-negative-1.c: Likewise. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 29 ++--- .../vect/vect-sat-trunc-range-precision-1.c | 122 ++++++++++++++++++ .../vect-sat-trunc-range-precision-1.c | 7 + ...ect-sat-trunc-range-precision-negative-1.c | 24 ++++ 4 files changed, 167 insertions(+), 15 deletions(-) create mode 100644 gcc/testsuite/gcc.dg/vect/vect-sat-trunc-range-precision-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-range-precision-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-range-precision-negative-1.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index 22855726d90..7156529f589 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -437,25 +437,24 @@ along with GCC; see the file COPYING3. If not see (negate (nop_convert? (convert (lt @0 integer_zerop))))) INTEGER_CST@3) (convert @0)) - /* The comparison has to be the unsigned reinterpretation of X, and the - conversion has to narrow, otherwise the constants below do not have the - precision the comparison is carried out at. */ - (if (!TYPE_UNSIGNED (TREE_TYPE (@0)) && TYPE_UNSIGNED (TREE_TYPE (@4)) - && TYPE_PRECISION (TREE_TYPE (@4)) == TYPE_PRECISION (TREE_TYPE (@0)) - && TYPE_PRECISION (type) < TYPE_PRECISION (TREE_TYPE (@0))) + /* The comparison type has to be unsigned and at least as wide as X. A + narrower type would examine only the low bits of X. The result + conversion has to narrow X. */ + (if (!TYPE_UNSIGNED (TREE_TYPE (@0)) + && TYPE_UNSIGNED (TREE_TYPE (@4)) + && (TYPE_PRECISION (TREE_TYPE (@4)) + >= TYPE_PRECISION (TREE_TYPE (@0))) + && TYPE_PRECISION (type) < TYPE_PRECISION (TREE_TYPE (@0))) (with { - unsigned itype_prec = TYPE_PRECISION (TREE_TYPE (@0)); unsigned otype_prec = TYPE_PRECISION (type); - wide_int offset = wi::uhwi (HOST_WIDE_INT_1U << (otype_prec - 1), - itype_prec); // Aka 128 for int8_t - wide_int limit_0 = wi::mask (otype_prec, false, itype_prec); // Aka 255 - wide_int limit_1 = wi::uhwi ((HOST_WIDE_INT_1U << otype_prec) - 3, - itype_prec); // Aka 253 - wide_int limit_2 = wi::uhwi ((HOST_WIDE_INT_1U << otype_prec) - 2, - itype_prec); // Aka 254 + unsigned range_prec = TYPE_PRECISION (TREE_TYPE (@4)); + wide_int offset = wi::set_bit_in_zero (otype_prec - 1, range_prec); + wide_int limit_0 = wi::mask (otype_prec, false, range_prec); // Aka 255 + wide_int limit_1 = limit_0 - 2; // Aka 253 + wide_int limit_2 = limit_0 - 1; // Aka 254 wide_int otype_max = wi::mask (otype_prec - 1, false, otype_prec); - wide_int itype_max = wi::mask (otype_prec - 1, false, itype_prec); + wide_int itype_max = wi::mask (otype_prec - 1, false, range_prec); wide_int int_cst_1 = wi::to_wide (@1); wide_int int_cst_2 = wi::to_wide (@2); wide_int int_cst_3 = wi::to_wide (@3); diff --git a/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-range-precision-1.c b/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-range-precision-1.c new file mode 100644 index 00000000000..b4e11f3a973 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-range-precision-1.c @@ -0,0 +1,122 @@ +/* { dg-do run { target bitint } } */ +/* { dg-additional-options "-O3 -fvect-cost-model=unlimited" } */ + +typedef __INT8_TYPE__ int8_t; +typedef __INT16_TYPE__ int16_t; +typedef __INT32_TYPE__ int32_t; +typedef __UINT16_TYPE__ uint16_t; +typedef __UINT32_TYPE__ uint32_t; +typedef signed _BitInt(16) int16b_t; +typedef unsigned _BitInt(17) uint17_t; + +#define N 259 +#define SAT_VALUE(OUT, X, MAX) ((OUT) (-((OUT) ((X) < 0)) ^ (OUT) MAX)) + +__attribute__((noipa)) +static void +wide_range (int8_t *__restrict out, const int16b_t *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + int16b_t source = in[i]; + uint32_t range = (uint32_t) source + 128U; + out[i] = (range > 255U + ? SAT_VALUE (int8_t, source, 127) : (int8_t) source); + } +} + +__attribute__((noipa)) +static void +narrow_range (int16_t *__restrict out, const int32_t *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + uint17_t range = (uint17_t) source + (uint17_t) 32768; + out[i] = (range > (uint17_t) 65535 + ? SAT_VALUE (int16_t, source, 32767) : (int16_t) source); + } +} + +__attribute__((noipa, optimize ("O0"))) +static void +wide_range_ref (int8_t *out, const int16b_t *in, int n) +{ + for (int i = 0; i < n; ++i) + { + int16b_t source = in[i]; + if (source < -128) + out[i] = -128; + else if (source > 127) + out[i] = 127; + else + out[i] = (int8_t) source; + } +} + +__attribute__((noipa, optimize ("O0"))) +static void +narrow_range_ref (int16_t *out, const int32_t *in, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + uint17_t range = (uint17_t) source + (uint17_t) 32768; + out[i] = (range > (uint17_t) 65535 + ? SAT_VALUE (int16_t, source, 32767) : (int16_t) source); + } +} + +static void +check_wide_range (void) +{ + int8_t out[N]; + int8_t ref[N]; + int16b_t in[N]; + + for (int i = 0; i < N; ++i) + in[i] = (uint16_t) i * 40503U + 97U; + + for (int n = 0; n <= N; ++n) + { + for (int i = 0; i < N; ++i) + out[i] = ref[i] = 23; + wide_range (out, in, n); + wide_range_ref (ref, in, n); + for (int i = 0; i < N; ++i) + if (out[i] != ref[i]) + __builtin_abort (); + } +} + +static void +check_narrow_range (void) +{ + int16_t out[N]; + int16_t ref[N]; + int32_t in[N]; + + for (int i = 0; i < N; ++i) + in[i] = (int32_t) ((uint32_t) i * 2654435761U + 1013904223U); + in[0] = 131072; + + for (int n = 0; n <= N; ++n) + { + for (int i = 0; i < N; ++i) + out[i] = ref[i] = 23; + narrow_range (out, in, n); + narrow_range_ref (ref, in, n); + for (int i = 0; i < N; ++i) + if (out[i] != ref[i]) + __builtin_abort (); + } +} + +int +main (void) +{ + check_wide_range (); + check_narrow_range (); + return 0; +} + diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-range-precision-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-range-precision-1.c new file mode 100644 index 00000000000..c52b309cbe1 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-range-precision-1.c @@ -0,0 +1,7 @@ +/* { dg-do compile { target bitint } } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +#include "../../gcc.dg/vect/vect-sat-trunc-range-precision-1.c" + +/* { dg-final { scan-tree-dump-times "sat_trunc pattern recognized" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-range-precision-negative-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-range-precision-negative-1.c new file mode 100644 index 00000000000..8c30f8d7632 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-range-precision-negative-1.c @@ -0,0 +1,24 @@ +/* { dg-do compile { target bitint } } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +typedef __INT16_TYPE__ int16_t; +typedef __INT32_TYPE__ int32_t; +typedef unsigned _BitInt(17) uint17_t; + +#define SAT_VALUE(X) ((int16_t) (-((int16_t) ((X) < 0)) ^ 32767)) + +__attribute__((noipa)) +void +narrow_range (int16_t *__restrict out, const int32_t *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + uint17_t range = (uint17_t) source + (uint17_t) 32768; + out[i] = (range > (uint17_t) 65535 + ? SAT_VALUE (source) : (int16_t) source); + } +} + +/* { dg-final { scan-tree-dump-not "sat_trunc pattern recognized" "vect" } } */ From patchwork Wed Sep 2 14:52:43 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142719 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 71A204B9DB4B for ; Wed, 2 Sep 2026 14:54:24 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 71A204B9DB4B Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=Ib597/Rv X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazlp170120005.outbound.protection.outlook.com [IPv6:2a01:111:f403:c105::5]) by sourceware.org (Postfix) with ESMTPS id C7D954BA9020 for ; Wed, 2 Sep 2026 14:53:42 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org C7D954BA9020 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org C7D954BA9020 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c105::5 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360822; cv=pass; b=WcROGBJGSUcK2DmUmhly2QiZ50vO9xt+98aJPE6mCJO3L2dwzLmweL/AZi4IDLARJKhdhddGE5ncdSSODy8hcZzfKEOXRTIxiDA9NUSQHBBEEHtBT0uhF5Qm3kfbrjkLpT8FAXVqd9q0LDgQVxwpgtVn7H7SeiUDfuGoKDKK7yg= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360822; c=relaxed/simple; bh=+e5m6poy2f4jemhqXGFLbSaUAK2+QtrCtTtEqcqBgGk=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=q/1s82ewaT3k605qgAJr5kpg3I+zvSDYGk0wPRg9Gw3W0plnAXa1RUrbu1VHEWaBZP5pU0LYdmBj1F3r6aE3BEHGWDVEfBbeNAfNqb62xhnG7NOYD1krKRjlF0Uy14fa77vTkpmPOoj7YOaReJ9sICyVkm8Zn78Lfr4eRt8M8jI= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=Ib597/Rv DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org C7D954BA9020 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lfA/VjGBxn4sOY0s7z7kJtV/KNCjtKzFml5hb+nxEzPp84feruot9q/btDmlXrJowwr4E+bz1e5j3uPy8RkE4wwxJmDymoL6Vt6LxHxXqbxl67dvLFl3G+W6avL3Tnoz0XS+ajWPZvNM5uRfn+Eb4oG3kFhOM+Ighc3PjHdSBnGuua+DTBV/hmA+VQlOzjxllc3pOcEdRiG1Grjbm52GrK3LsTOmDmy9hufl0i+gJel4WlizOKz29vk5iLhZ89uqy5WZUUx18Kl7TKUrdwMAR+Rid7cj8ztgCWD9KEHH0tIIS3ELnBjLyOS8f39SbPc76XiSIWXKttbLYExpFLXGvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Brq6hIhwNbic0BPfyWkTL2o/YBoWUVmySl/sw6E6QGU=; b=m9W/frzhgCVN+wA6k4w6Mtt3WcoCzACWOBva8w7A7l+PtJkzulZLTqIqTQL4HCccciYKQa8NTaqm2y1MH7hS+l52th7jlTmYKlYYanFOgJFwKrwDWYmXJxrUIgz7F2vr0vTrF8kejeowKrsr4fkO0Saw4xiYILJzIlZuHO7ydxINpOHCGb8WSne1HkcffKvJlFu5XW76QQX0vxVAt7Z9/LUkhSfZoPytgFrExrcuuTV4DEDpdZP5b1u8fgCOBZn3KOmWLgvc9wftVwVPZ9bbxX8pkbDYqORqGaswebXjvvwUwBXlhv+CZ90AkVUmfFl4RGiOogmTwJ3ojyJ7emVerQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Brq6hIhwNbic0BPfyWkTL2o/YBoWUVmySl/sw6E6QGU=; b=Ib597/RvAzJ7RgukNPOQsJ218CBO3rSuRlSFVl9GW0eceZnUSp2pTe7jAEFB6745VwcUlt3vlVUSRi51enMcvPyawt6h3B+GJ54yLJTwzVtEzXOQeIKG01XbPa5kjjLVhTozBTQLRO2N6JBxGzFHLIBKYKumopke4LA6kbJAFamIeCRHnRCrJnv+qwS5/KMpntexiUr8vGecGZoWGR50PJozQ0TAE8pda1XcAY6jTkNtRGlVgLZ8h8ow/4BifulKhrFe8EZv1sygxLPJDxoPmB9IqArO3G+Zh04rB6gIyx2E8zi6CXWSS7DxzEslVPQto85t0UhwTvpDWQt8MjNU4g== Received: from BN9PR03CA0450.namprd03.prod.outlook.com (2603:10b6:408:113::35) by CH2PR12MB4261.namprd12.prod.outlook.com (2603:10b6:610:a9::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Wed, 2 Sep 2026 14:53:37 +0000 Received: from BN3PEPF0000B06E.namprd21.prod.outlook.com (2603:10b6:408:113:cafe::5) by BN9PR03CA0450.outlook.office365.com (2603:10b6:408:113::35) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.10 via Frontend Transport; Wed, 2 Sep 2026 14:53:36 +0000 X-MS-Exchange-Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN3PEPF0000B06E.mail.protection.outlook.com (10.167.243.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.1 via Frontend Transport; Wed, 2 Sep 2026 14:53:36 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:09 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:08 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 02/13] match-sat-alu.pd: Recognize X - MIN (X, Y) as saturating subtraction Date: Wed, 2 Sep 2026 16:52:43 +0200 Message-ID: <20260902145254.77832-4-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B06E:EE_|CH2PR12MB4261:EE_ X-MS-Office365-Filtering-Correlation-Id: ddb205a2-296b-4b82-e37f-08df0901f7f0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|23010399003|36860700016|82310400026|1800799024|376014|13003099007|6133799003|10067099003|56012099006|11063799006|3023799007|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: E5aEqAmYV+ncjHyqHc1DBISP+IsyTahb4oHZmNzSfAjOre4I5nKcQRJD1ZKPCvQeTde7uxwre/WTf0qZzOH+/Cpq41cqU4ld2wjOdkX5x6Y4EdFzGXKQbco4CrnxzOj09wYt9TMNATGyHAcMj6B4ktyGKm0HNKznXmYUBxS0WhbPo70hwzHPCtTJ7BzQulXEXWsHujTPN9xSsAyHHrmNKabfUDaTwzUn53WAMcZtKdp5RnMMKZLtjRR0JtgHWpe5dvUS9/Yt5hD8+FfTPaswm5pm5NBVpg4+e/D+h+S6HBPX0FfcAIsQ//xkOUIXl2Zhm6ORSB+f3izf/p45mFwLTtpAFUODerGG4I4LakZ2uBDFgvvwBtP/a3Ad7CNM7xjlBMRwETALKBiZgCAn1y3ROcEUpnb0gK4cl0T55pWInoPxEB58tCzpHruSWP36Coft2jE8JP/NDPYz1MgPFvxlTfBAkwFiAHUbgpJNdHhhcTOJnUR4/9vbH81vGkn1PYjrmgyTTiSzaVDNDwkhbH+cbghl+VA8N9EylWhJroAoMERczjMygnUndiogsxL9lHAsvi6Agmyr5IGXh6/2YPSI6wmx8KsJnpMPo4c4nqXDzfUkKlDQ4X89WwHarNd3G+uECZAbupNqJ4r9dRnao7j72fA9P3q7OzhI940Wuft8iyAWF7mHWqnvO1MqnczZS7DFof2gOLYrzpldtigH8Gw+yA== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(23010399003)(36860700016)(82310400026)(1800799024)(376014)(13003099007)(6133799003)(10067099003)(56012099006)(11063799006)(3023799007)(22082099003)(18002099003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: q3cAapguC1an48nAy80nfJTjr2lNfaADhLLxn9SBItEOq63WZOxW6JWy596ZsBE04K+FxNjmbDbbk+7sJPWl6DajgjLawQGrrfpEKTSyefT/Z+9pvSHYPU7mDFVhJc2x0uEXj+92bxJ+MHQs41vFD5Nknj2xjUuTYJcmGBQWCRV2m9jKLKtiNGcoQoc9siVTpuBg7X2QTVNHDGL7Tey4n4T7g6WyI88ydyk8FzwMsg26tp9W7AMer1W8QJDYcrQfMwC/EQyI+8b3Z10dsVSXq23Ej14hPdV81Hrr0WJAd/YmFWSY20ilfwME866Bp1LHdM31YKL3xcC7cxNHZfU0aVOHs0riirnZI6TjBBVat8trR2kIO/tTv9dG5WmSbrbKUPql4isqU1MkczjEsgqB1elnMcyudaNKNYF3Z1rcwHgNNwWtX6B+tnI6EkLC+1Ep X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:36.7447 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ddb205a2-296b-4b82-e37f-08df0901f7f0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B06E.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4261 X-Spam-Status: No, score=-8.1 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov X - MIN (X, Y) is unsigned saturating subtraction, but subtraction statements were not checked for this form. Recognize the form and check MINUS_EXPR statements for saturation. Require the MIN to have one use so that the replacement removes it. If a replacement changes the current statement, do not try to widen that statement. AArch64 -O2: before: cmp w1, w0 csel w1, w1, w0, ls sub w0, w0, w1 after: subs w0, w0, w1 csel w0, w0, wzr, cs The vector form becomes one UQSUB instead of a comparison, a select, and a subtraction. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (unsigned_integer_sat_sub): Add the X - MIN (X, Y) form. * tree-ssa-math-opts.cc (math_opts_dom_walker::before_dom_children): Call match_unsigned_saturation_sub for MINUS_EXPR as well, and only widen while the statement is unchanged. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sat_u_sub_minmax-1.c: New test. * gcc.target/aarch64/sat_u_sub_minmax-2.c: New test. * gcc.target/aarch64/sat_u_sub_minmax-3.c: New test. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 6 ++++ .../gcc.target/aarch64/sat_u_sub_minmax-1.c | 18 +++++++++++ .../gcc.target/aarch64/sat_u_sub_minmax-2.c | 32 +++++++++++++++++++ .../gcc.target/aarch64/sat_u_sub_minmax-3.c | 27 ++++++++++++++++ gcc/tree-ssa-math-opts.cc | 5 +-- 5 files changed, 86 insertions(+), 2 deletions(-) create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-2.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-3.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index 7156529f589..1b35b35cff8 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -131,6 +131,12 @@ along with GCC; see the file COPYING3. If not see /* SAT_U_SUB = (X - Y) * (X >= Y) */ (mult:c (minus @0 @1) (convert (ge @0 @1))) (if (types_match (type, @0, @1)))) + (match (unsigned_integer_sat_sub @0 @1) + /* SAT_U_SUB = X - MIN (X, Y). The MIN has to be single use: while it + stays live the saturating subtract is computed beside it instead of + replacing it, which costs an instruction. */ + (minus @0 (min:c@2 @0 @1)) + (if (single_use (@2)))) (match (unsigned_integer_sat_sub @0 @1) /* DIFF = SUB_OVERFLOW (X, Y) SAT_U_SUB = REALPART (DIFF) | (IMAGPART (DIFF) + (-1)) */ diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-1.c b/gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-1.c new file mode 100644 index 00000000000..aa1d7b6f031 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-1.c @@ -0,0 +1,18 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-optimized" } */ + +#define DEF_MIN(T) \ + T min_##T (T a, T b) { return a - (a < b ? a : b); } \ + T nim_##T (T a, T b) { return a - (b < a ? b : a); } + +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned int u32; +typedef unsigned long long u64; + +DEF_MIN (u8) +DEF_MIN (u16) +DEF_MIN (u32) +DEF_MIN (u64) + +/* { dg-final { scan-tree-dump-times "\\.SAT_SUB " 8 "optimized" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-2.c b/gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-2.c new file mode 100644 index 00000000000..dcf8ab46309 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-2.c @@ -0,0 +1,32 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -fdump-tree-optimized" } */ + +typedef unsigned char u8; +typedef unsigned int u32; + +/* The recognition must reach the vectoriser, so that the loop becomes a + single uqsub rather than a compare, a select and a subtraction. */ + +void +min_loop (u8 *__restrict d, u8 *__restrict a, u8 *__restrict b, int n) +{ + for (int i = 0; i < n; i++) + { + u8 x = a[i], y = b[i]; + d[i] = x - (x < y ? x : y); + } +} + +void +min_loop32 (u32 *__restrict d, u32 *__restrict a, u32 *__restrict b, int n) +{ + for (int i = 0; i < n; i++) + { + u32 x = a[i], y = b[i]; + d[i] = x - (y < x ? y : x); + } +} + +/* { dg-final { scan-tree-dump "\\.SAT_SUB " "optimized" } } */ +/* { dg-final { scan-assembler "uqsub\tv\[0-9\]+\\.16b" } } */ +/* { dg-final { scan-assembler "uqsub\tv\[0-9\]+\\.4s" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-3.c b/gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-3.c new file mode 100644 index 00000000000..029c136b7d0 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_sub_minmax-3.c @@ -0,0 +1,27 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-optimized" } */ + +typedef unsigned int u32; + +/* While the MIN stays live the saturating subtract would be computed beside + it rather than instead of it, so the rules do not fire. */ + +u32 +min_live (u32 a, u32 b, u32 *o) +{ + u32 m = a < b ? a : b; + *o = m; + return a - m; +} + +u32 +add_min_live (u32 a, u32 b, u32 *o) +{ + u32 t = ~a; + u32 m = b < t ? b : t; + *o = m; + return a + m; +} + +/* { dg-final { scan-tree-dump-not "\\.SAT_SUB " "optimized" } } */ +/* { dg-final { scan-tree-dump-not "\\.SAT_ADD " "optimized" } } */ diff --git a/gcc/tree-ssa-math-opts.cc b/gcc/tree-ssa-math-opts.cc index b371b5b7cff..ed3abb7d5c4 100644 --- a/gcc/tree-ssa-math-opts.cc +++ b/gcc/tree-ssa-math-opts.cc @@ -7332,10 +7332,11 @@ math_opts_dom_walker::after_dom_children (basic_block bb) case PLUS_EXPR: match_saturation_add_with_assign (&gsi, as_a (stmt)); - match_unsigned_saturation_sub (&gsi, as_a (stmt)); /* fall-through */ case MINUS_EXPR: - if (!convert_plusminus_to_widen (&gsi, stmt, code)) + match_unsigned_saturation_sub (&gsi, as_a (stmt)); + if (gsi_stmt (gsi) == stmt + && !convert_plusminus_to_widen (&gsi, stmt, code)) { match_arith_overflow (&gsi, stmt, code, m_cfg_changed_p); if (gsi_stmt (gsi) == stmt) From patchwork Wed Sep 2 14:52:44 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142727 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 25FFE4BA2E14 for ; Wed, 2 Sep 2026 14:58:38 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 25FFE4BA2E14 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=OhJ8knPg X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from BYAPR05CU005.outbound.protection.outlook.com (mail-westusazon11010056.outbound.protection.outlook.com [52.101.85.56]) by sourceware.org (Postfix) with ESMTPS id B9E964BA9019 for ; Wed, 2 Sep 2026 14:53:49 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B9E964BA9019 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org B9E964BA9019 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.85.56 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360830; cv=pass; b=VjvkpHmGKYbZ+SyyrD3FM1Co3fVaMBsttn0kExp6nGRu3TQo+ki2lYo+VxuiJAeuqoVcDyRmlUqICrSgKvMNMt+QqW1kXCyVOhv+5UQGax015CNwYo3BEqheVO3swEv7sjoSOZJ/iVY+gxmXYLH2c3sm1mIWcMVp1ZZaJZYFEfs= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360830; c=relaxed/simple; bh=juwLeTRZ7u+jQcdKZ83ydXF1e6IifuTSDoKw0F0L7eA=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=wNdFMKRpwmbfu9CLYB6Zb0LHfQiLlYsYCtkjuiqGjZ9L8Z3/o/bY9v6CfBZ/YXfi2qQB98ju19yVJrLqtrlAsc1RBNkOtx5L/5tjBDWMzy40L4n24jN8vvCylpwbGqZwzWGoYdytHFXM7Du0NEGAyzPsQdOYY0kVXzUPl/qSL4w= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=OhJ8knPg DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B9E964BA9019 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MOrL+uoK00r9/Bv83SiXboOkm2li7AC8Pg9fP1RaUNeH9SCTmR1Ot5KxfVKooY+8QMFRi8Su9Ayzu9TNluVvSPOQjvZiUHVbGsLhiG3xgpgBkLktr2wfpaqMfo2A8I1lEBU9NbIgkoQ05AanUWbCEeufR15lRI8p0dWfGBoJFomuGVQ/9HRgHNXzIn5494IdBUI+Ph3nviR5mRaejTjWIwNBIOMVfnlVAj4TQq0uRkX2dYlOVKFCxfAwpTKo7KUbg83M01CpGpodfIpfRGxOfh3UhbpsAbd5MF5wkHezgDm+9virDP/XjS1/Y2328ZU5lYsqjX4WIxwAYiRdJNqgkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JW6TCWVrkoU2syRDgOyABK0w2f98tdvtzxfvdSplpEI=; b=C1Dat0dYQs829cDpif7t5vXnUuviMnvp5pwdJApnxE53/OgDXmr1P8rHK4NAvYxWUWXSho71GaQOsDryzL/Njcb+5zIPWZEJRNzody5KaP7tjUL4+cyffsp5GpxOoSbn7OdMlQh1y2rOb0AFaMpk+je49IegcO8FjeNAQGKFpdO3GA9drfmvpDD6IctAmr94qOqsFHE/M7SH+JJuT1CBlFye6WfynDIIbfEmtDwk69wvI4YxYz9rEJh3Me7TTYAbJIygO4onQZPk6ZXywcIY+WM9fsHUKxEDeVoSn9PWPcaJTmz6shbvuv8qZlZza5+HSkv8mSZ6i3Jlbc7olDglrw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JW6TCWVrkoU2syRDgOyABK0w2f98tdvtzxfvdSplpEI=; b=OhJ8knPgR7EUesFzORqLRcwsP2lllADOUgxxPb9cxzaLG6fXQn/rhXtWz5lQwZ9/FqMFeA2eIqNh2CC2o/YQonIvberDrx6pWyspdJ3iJ7ke8PG+9agqls4TnbmUb52Q6orh9mWKydo/BaSxmVPgSVMtJCL51f6pvZCuLMaRmuqj6UHj/GxanhCDFPg4t3ogqmF177PNBcc/No5tT4dyADqVKS57vjQ1YY+w/ngHn8ToU8ueDF2zDP0LubiLUK9b7ALMISdMgCQWS/psOSFe5XsptZpwFFfTsyjaUUEMhRWFMg73i87VATqbrfl/snkLnB2XQU+HT7VoenRoJaXS6g== Received: from SA9PR13CA0030.namprd13.prod.outlook.com (2603:10b6:806:21::35) by DS0PR12MB7828.namprd12.prod.outlook.com (2603:10b6:8:14b::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.382.10; Wed, 2 Sep 2026 14:53:38 +0000 Received: from SA2PEPF00001508.namprd04.prod.outlook.com (2603:10b6:806:21:cafe::5c) by SA9PR13CA0030.outlook.office365.com (2603:10b6:806:21::35) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.7 via Frontend Transport; Wed, 2 Sep 2026 14:53:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SA2PEPF00001508.mail.protection.outlook.com (10.167.242.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.382.8 via Frontend Transport; Wed, 2 Sep 2026 14:53:37 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:10 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:09 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 03/13] match-sat-alu.pd: Recognize signed saturation clamps Date: Wed, 2 Sep 2026 16:52:44 +0200 Message-ID: <20260902145254.77832-5-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001508:EE_|DS0PR12MB7828:EE_ X-MS-Office365-Filtering-Correlation-Id: 3f42816d-72a1-48b9-08b4-08df0901f89b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|1800799024|82310400026|23010399003|36860700016|10067099003|6133799003|18002099003|22082099003|56012099006|11063799006|13003099007; X-Microsoft-Antispam-Message-Info: m+sbIohMdFvbbOFYYKOolDVPzlFH52EsI2WOzb5lthAPM6W4wJvz1XhMj4tZycKqB40JOPpYLMjrrM+P4x6tsLcWA6uYJDRIulSlqBiyVluWbe/qY4AcGk8D68PD+Ua2DJUVMKOxSva06LWBE2a5UMYZFBSsCWBR5GiB2qhvfIqgtM4c51G0+w3czsa5EMKMi2F/4jQeyFU2d7sZ251qg4X0CPqaKM7+78VrhgkwzNBlqnTuG2zLqP1jxT+2dxEo7ydTsRX0xKlMxoT9j4Xh+EJKngEU2aQz9Qs+TlCGAAiHoZodbjWtWrg6FSmgbNqDC2UFjBIrEyR3KB2UF5VVzTNlFDnj+x4sRGZE6wwTkws261yvoWHj0YdctCgGZo22nPi54HojAc6ZEZtDoeqCyjNZHs0iuiFuXHPa15fmT4lYTaOkdDnjVFWjAq6PjEZdIiGffRJ2WUU1/RN/7i4GpwyrFOSCPVNT+BaKQCaLdkG+zuj5uFPi6x91J92oI96lJ0yI2rnQMJkvLyHPZxPn7zr+DnGRX1IV6OIQUVM6/UBMC6UpFa6VbQoKRl469zZdj19d+Mp8vhKVulZJxtfHbPfMUXPRLsVKmBOWSVhg9HGKuVpoHhrnOCcWKc0JuEaeMv1nco1xvLVSCLv0kQ3M2SOaGQ3S0Bbz6dPzkroRcywnY1iA/HxB6l1kYAjr49dueN8L3utLI/aAeE50hq9anw== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(376014)(1800799024)(82310400026)(23010399003)(36860700016)(10067099003)(6133799003)(18002099003)(22082099003)(56012099006)(11063799006)(13003099007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: lsmap9+ja8afCqozP5ONwsEuZTfiSfzhEL7bSY2EXLumfI9wDGgMju1NyU0Hz3hsHFVndGf4yS9E90l6u/eEGSJue6mXpdIDwGwkkutUxXT3YIgZh4XmuOTSwJAkjKRPuHYbz/NyqiVQ7Oo02OkaLE1jIEJMVUN2XD9YBiA3jCJawLIgRvEgIxHGwLyWvDlVV7lsGWS7er+D0sMIZMf/tlL85m+BCCjZ/6sX4xTZQmrOXtWPQ2D2Qn0T5D8a7NAy35ta/w3TujpVpuPg+vNqVutQD8VmkTQ5SbSWrmt79ZQ+v5jTqzLPqjyRnWijbXTlBMVjjw8MLp1pbAKlfGVUKOAnw2uxyZ8Tzc76peiGrLUUbv0Y5o8+HhKOQmjXFkFBK8a/2aK4IG3aPkrP917zlJIJHK+ddRkapb7qgLerfGF5A7uZAcfaszsXQQlKhXLn X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:37.9019 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3f42816d-72a1-48b9-08b4-08df0901f89b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001508.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7828 X-Spam-Status: No, score=-8.1 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov Signed saturating addition and subtraction can use a wider intermediate that is clamped to the result type bounds: (T) MIN (MAX ((WT) X + (WT) Y, T_MIN), T_MAX) -> .SAT_ADD (X, Y) Recognize addition and subtraction in both clamp nesting orders. Require a signed intermediate that is strictly wider than the result type. This keeps the arithmetic exact before the clamp. Require the outer clamp to have one use so that the replacement removes it. Check signed saturation subtraction when the clamp is rooted at a conversion. AArch64 -O2: before: sxtw x1, w1 add x0, x1, w0, sxtw mov x1, -2147483648 cmp x0, x1 csel x0, x0, x1, ge mov w1, 2147483647 cmp x0, x1 csel x0, x0, x1, le after: fmov s31, w0 fmov s30, w1 sqadd s31, s31, s30 fmov w0, s31 The other clamp forms have the same reduction. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (signed_integer_sat_add): Add the two sided clamp on a wider intermediate, in both nesting orders. (signed_integer_sat_sub): Likewise. * tree-ssa-math-opts.cc (match_unsigned_saturation_sub): Rename to ... (match_saturation_sub_with_assign): ... this, and ask the signed predicate as well. (math_opts_dom_walker::before_dom_children): Update the callers, and call it for NOP_EXPR. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sat_s_clamp-1.c: New test. * gcc.target/aarch64/sat_s_clamp-2.c: New test. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 87 ++++++++++++++++++- .../gcc.target/aarch64/sat_s_clamp-1.c | 78 +++++++++++++++++ .../gcc.target/aarch64/sat_s_clamp-2.c | 43 +++++++++ gcc/tree-ssa-math-opts.cc | 14 +-- 4 files changed, 214 insertions(+), 8 deletions(-) create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_s_clamp-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_s_clamp-2.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index 1b35b35cff8..8c7bc0782a0 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -395,7 +395,52 @@ along with GCC; see the file COPYING3. If not see integer_zerop) (signed_integer_sat_val @0) @2) - (if (wi::eq_p (wi::to_wide (@1), wi::to_wide (@3)))))) + (if (wi::eq_p (wi::to_wide (@1), wi::to_wide (@3))))) + + /* A signed saturating add written as a two sided clamp on a wider + intermediate, which is a common source form and what the integer + promotions leave for a narrow type. Both nesting orders occur: + an if followed by an if or an else if gives MAX then MIN, while + max (min (t, HI), LO) gives MIN then MAX. */ + (match (signed_integer_sat_add @0 @1) + /* WT SUM = (WT)X + (WT)Y + SAT_S_ADD = (T)MIN (MAX (SUM, T_MIN), T_MAX) */ + (convert (min@4 (max (plus (convert @0) (convert @1)) INTEGER_CST@2) + INTEGER_CST@3)) + (if (types_match (type, @0, @1) && single_use (@4)) + (with + { + unsigned precision = TYPE_PRECISION (type); + tree wide_type = TREE_TYPE (@2); + unsigned wide_precision = TYPE_PRECISION (wide_type); + wide_int lo = wide_int::from (wi::min_value (precision, SIGNED), + wide_precision, SIGNED); + wide_int hi = wide_int::from (wi::max_value (precision, SIGNED), + wide_precision, SIGNED); + } + /* One extra bit is enough for a sum of two values of the narrow type, + and at equal precision the clamp is not a saturating add at all. */ + (if (wide_precision > precision && !TYPE_UNSIGNED (wide_type) + && wi::eq_p (wi::to_wide (@2), lo) + && wi::eq_p (wi::to_wide (@3), hi)))))) + (match (signed_integer_sat_add @0 @1) + /* SAT_S_ADD = (T)MAX (MIN (SUM, T_MAX), T_MIN) */ + (convert (max@4 (min (plus (convert @0) (convert @1)) INTEGER_CST@3) + INTEGER_CST@2)) + (if (types_match (type, @0, @1) && single_use (@4)) + (with + { + unsigned precision = TYPE_PRECISION (type); + tree wide_type = TREE_TYPE (@2); + unsigned wide_precision = TYPE_PRECISION (wide_type); + wide_int lo = wide_int::from (wi::min_value (precision, SIGNED), + wide_precision, SIGNED); + wide_int hi = wide_int::from (wi::max_value (precision, SIGNED), + wide_precision, SIGNED); + } + (if (wide_precision > precision && !TYPE_UNSIGNED (wide_type) + && wi::eq_p (wi::to_wide (@2), lo) + && wi::eq_p (wi::to_wide (@3), hi))))))) /* Saturation sub for signed integer. */ (if (INTEGRAL_TYPE_P (type) && !TYPE_UNSIGNED (type)) @@ -432,7 +477,45 @@ along with GCC; see the file COPYING3. If not see (cond^ (ne (imagpart (IFN_SUB_OVERFLOW@2 @0 @1)) integer_zerop) (signed_integer_sat_val @0) (realpart @2)) - (if (types_match (type, @0, @1))))) + (if (types_match (type, @0, @1)))) + /* The same two sided clamp on a wider intermediate, for a subtraction. */ + (match (signed_integer_sat_sub @0 @1) + /* WT DIFF = (WT)X - (WT)Y + SAT_S_SUB = (T)MIN (MAX (DIFF, T_MIN), T_MAX) */ + (convert (min@4 (max (minus (convert @0) (convert @1)) INTEGER_CST@2) + INTEGER_CST@3)) + (if (types_match (type, @0, @1) && single_use (@4)) + (with + { + unsigned precision = TYPE_PRECISION (type); + tree wide_type = TREE_TYPE (@2); + unsigned wide_precision = TYPE_PRECISION (wide_type); + wide_int lo = wide_int::from (wi::min_value (precision, SIGNED), + wide_precision, SIGNED); + wide_int hi = wide_int::from (wi::max_value (precision, SIGNED), + wide_precision, SIGNED); + } + (if (wide_precision > precision && !TYPE_UNSIGNED (wide_type) + && wi::eq_p (wi::to_wide (@2), lo) + && wi::eq_p (wi::to_wide (@3), hi)))))) + (match (signed_integer_sat_sub @0 @1) + /* SAT_S_SUB = (T)MAX (MIN (DIFF, T_MAX), T_MIN) */ + (convert (max@4 (min (minus (convert @0) (convert @1)) INTEGER_CST@3) + INTEGER_CST@2)) + (if (types_match (type, @0, @1) && single_use (@4)) + (with + { + unsigned precision = TYPE_PRECISION (type); + tree wide_type = TREE_TYPE (@2); + unsigned wide_precision = TYPE_PRECISION (wide_type); + wide_int lo = wide_int::from (wi::min_value (precision, SIGNED), + wide_precision, SIGNED); + wide_int hi = wide_int::from (wi::max_value (precision, SIGNED), + wide_precision, SIGNED); + } + (if (wide_precision > precision && !TYPE_UNSIGNED (wide_type) + && wi::eq_p (wi::to_wide (@2), lo) + && wi::eq_p (wi::to_wide (@3), hi))))))) /* Saturation truncate for signed integer. */ (if (INTEGRAL_TYPE_P (type) && !TYPE_UNSIGNED (type)) diff --git a/gcc/testsuite/gcc.target/aarch64/sat_s_clamp-1.c b/gcc/testsuite/gcc.target/aarch64/sat_s_clamp-1.c new file mode 100644 index 00000000000..a8790699ac8 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_s_clamp-1.c @@ -0,0 +1,78 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-optimized" } */ + +#include + +static inline int imin (int a, int b) { return a < b ? a : b; } +static inline int imax (int a, int b) { return a > b ? a : b; } + +/* A signed saturating add or subtract written as a two sided clamp on a + wider intermediate. Both nesting orders appear, and the narrow types + reach this shape through the integer promotions. */ + +int +add_ll (int x, int y) +{ + long long t = (long long) x + y; + if (t > INT_MAX) t = INT_MAX; + if (t < INT_MIN) t = INT_MIN; + return (int) t; +} + +int +add_ll_else (int x, int y) +{ + long long t = (long long) x + y; + if (t > INT_MAX) t = INT_MAX; + else if (t < INT_MIN) t = INT_MIN; + return (int) t; +} + +int +sub_ll (int x, int y) +{ + long long t = (long long) x - y; + if (t > INT_MAX) t = INT_MAX; + if (t < INT_MIN) t = INT_MIN; + return (int) t; +} + +short +add_hi (short x, short y) +{ + int t = x + y; + return (short) imax (imin (t, 32767), -32768); +} + +short +sub_hi (short x, short y) +{ + int t = x - y; + return (short) imin (imax (t, -32768), 32767); +} + +signed char +add_qi (signed char x, signed char y) +{ + int t = x + y; + return (signed char) imax (imin (t, 127), -128); +} + +signed char +sub_qi (signed char x, signed char y) +{ + int t = x - y; + return (signed char) imin (imax (t, -128), 127); +} + +signed char +add_qi_shared (signed char x, signed char y, int *p) +{ + int t = x + y; + int lo = imax (t, -128); + *p = lo; + return (signed char) imin (lo, 127); +} + +/* { dg-final { scan-tree-dump-times "\\.SAT_ADD " 5 "optimized" } } */ +/* { dg-final { scan-tree-dump-times "\\.SAT_SUB " 3 "optimized" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sat_s_clamp-2.c b/gcc/testsuite/gcc.target/aarch64/sat_s_clamp-2.c new file mode 100644 index 00000000000..5bdde16fb25 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_s_clamp-2.c @@ -0,0 +1,43 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -fdump-tree-optimized" } */ + +#include + +static inline int imin (int a, int b) { return a < b ? a : b; } +static inline int imax (int a, int b) { return a > b ? a : b; } + +void +add_loop (short *__restrict d, short *__restrict a, short *__restrict b, int n) +{ + for (int i = 0; i < n; i++) + { + int t = a[i] + b[i]; + d[i] = (short) imax (imin (t, 32767), -32768); + } +} + +void +sat_loop (int *__restrict d, int *__restrict a, int *__restrict b, int n) +{ + for (int i = 0; i < n; i++) + { + long long t = (long long) a[i] + b[i]; + if (t > INT_MAX) t = INT_MAX; + if (t < INT_MIN) t = INT_MIN; + d[i] = (int) t; + } +} + +/* The clamp of an equal precision sum is not a saturating add: the + comparisons are dead and the whole thing is a plain addition. */ + +int +not_saturating (int x, int y) +{ + int t = x + y; + return t > INT_MAX ? INT_MAX : t < INT_MIN ? INT_MIN : t; +} + +/* { dg-final { scan-tree-dump "\\.SAT_ADD " "optimized" } } */ +/* { dg-final { scan-assembler "sqadd\tv\[0-9\]+\.8h" } } */ +/* { dg-final { scan-assembler "sqadd\tv\[0-9\]+\.4s" } } */ diff --git a/gcc/tree-ssa-math-opts.cc b/gcc/tree-ssa-math-opts.cc index ed3abb7d5c4..95f6e53f157 100644 --- a/gcc/tree-ssa-math-opts.cc +++ b/gcc/tree-ssa-math-opts.cc @@ -4252,7 +4252,7 @@ match_saturation_add (gimple_stmt_iterator *gsi, gphi *phi) } /* - * Try to match saturation unsigned sub. + * Try to match saturation sub with assign. * _1 = _4 >= _5; * _3 = _4 - _5; * _6 = _1 ? _3 : 0; @@ -4260,12 +4260,13 @@ match_saturation_add (gimple_stmt_iterator *gsi, gphi *phi) * _6 = .SAT_SUB (_4, _5); */ static void -match_unsigned_saturation_sub (gimple_stmt_iterator *gsi, gassign *stmt) +match_saturation_sub_with_assign (gimple_stmt_iterator *gsi, gassign *stmt) { tree ops[2]; tree lhs = gimple_assign_lhs (stmt); - if (gimple_unsigned_integer_sat_sub (lhs, ops, NULL)) + if (gimple_unsigned_integer_sat_sub (lhs, ops, NULL) + || gimple_signed_integer_sat_sub (lhs, ops, NULL)) build_saturation_binary_arith_call_and_replace (gsi, IFN_SAT_SUB, lhs, ops[0], ops[1]); } @@ -7327,14 +7328,14 @@ math_opts_dom_walker::after_dom_children (basic_block bb) continue; } match_arith_overflow (&gsi, stmt, code, m_cfg_changed_p); - match_unsigned_saturation_sub (&gsi, as_a (stmt)); + match_saturation_sub_with_assign (&gsi, as_a (stmt)); break; case PLUS_EXPR: match_saturation_add_with_assign (&gsi, as_a (stmt)); /* fall-through */ case MINUS_EXPR: - match_unsigned_saturation_sub (&gsi, as_a (stmt)); + match_saturation_sub_with_assign (&gsi, as_a (stmt)); if (gsi_stmt (gsi) == stmt && !convert_plusminus_to_widen (&gsi, stmt, code)) { @@ -7375,13 +7376,14 @@ math_opts_dom_walker::after_dom_children (basic_block bb) case COND_EXPR: case BIT_AND_EXPR: - match_unsigned_saturation_sub (&gsi, as_a (stmt)); + match_saturation_sub_with_assign (&gsi, as_a (stmt)); break; case NOP_EXPR: match_unsigned_saturation_mul (&gsi, as_a (stmt)); match_unsigned_saturation_trunc (&gsi, as_a (stmt)); match_saturation_add_with_assign (&gsi, as_a (stmt)); + match_saturation_sub_with_assign (&gsi, as_a (stmt)); /* fall-through */ case CONVERT_EXPR: /* The long-multiply recognizer's high-part emit ends in an From patchwork Wed Sep 2 14:52:45 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142720 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 166A84BA79BA for ; Wed, 2 Sep 2026 14:54:42 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 166A84BA79BA Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=RvmdZ3o+ X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from CO1PR03CU002.outbound.protection.outlook.com (mail-westus2azlp170100005.outbound.protection.outlook.com [IPv6:2a01:111:f403:c005::5]) by sourceware.org (Postfix) with ESMTPS id 8C6634B9DB66 for ; Wed, 2 Sep 2026 14:53:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 8C6634B9DB66 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 8C6634B9DB66 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c005::5 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360828; cv=pass; b=K1eFSlpkT9pDCCclEo+iaZxpnc4fV7evSRIgYfzC/9Sbe5/5LDNhMha7YNa07fnOuAKoOrcsHm/4BbsFq60i7Ws5bf3wHPB/KQMv96tYrotN9b3KD8jvw6kqsQBjh7R1Ib8ctVXZcpFanyhEGPuGIuCJ5502rKkDs6/8iPgcOu8= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360828; c=relaxed/simple; bh=NnAQhXLD61PW7vwlh2HV6PYxSZEph/cDGTaGuIx2/dg=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=ifIQPUDsecUH22EJjXLKKQHjy0P8q2YX8SK+OJhrMlwUrWpODiaSAuso1opOK8SYD6RJ9xWJWaf127baAhqL83JXgys0e3Y56ODrm03aOsaW0tFnIBjUJ/el/Rt7PJLW3ec2Bh+FOMJa6OINB/gC3FYBPs+X42zUfTbf/fVYMsQ= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=RvmdZ3o+ DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 8C6634B9DB66 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=GA3h+pz5z96y7AP7x1pchiMIpqv9ylmqgEVyrxR+aoNrFas3UCYNVDfsmtG5QJ4OqcWAJ2ecK+mWTvQ4wHKKu8Ibq1LLp6JsDPi835JtjApAUt5U1bEF2sKFht/aI2n0RM7mW92qN2UYU1PKCZb3ZujonZymokP1e+GAyc45H1z85qkht7XIr/tqhG6CqqLDcCw8lFITPSzZ75wM4WF9mL92285Zk09jL5g/qlZsZW02ZgbGmwT2wExAmA1Zmlx3b0JKtPbleMs8vaqH/DoVEiE0XWxL3RK7pjVixWfJxSdPbtAP+Sn+enFv66IlfvGzraQFTUngzX+kcK0yvu0I8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=olVPoo8AikYy4Q17j67lvkXpHMRRH3k3RbAUlX5D2JM=; b=tHgWYpMTQYB0RcrPhzeAbnneH3PNXaOhgpCPj4x5J96U0TXfMv/GZ4a5aWSvVwxvZVA3hxp22SavRvO0hlPcpzuPDtAxesekpgPOdU1VuaFWdXQ/owFJJmvyJ95sOXoryIj2jut31FFHs/GcGkUPU9KTRjRglpAYQ0qWA2TmfXGB9bVDDIi7l4zBichJkKjRTf/6PAQeIPZu/4ZAxw4zbHFVFSVO0udGKcTaeyrJ54MToERliuhmTBbVVm0hIgr49E9FgccLPltidmcE/k+6WAmzq/1MT0KfdkBxHQoP5aiBklsgvPv+LEDwWw2LR3jG3OfzfbBYj2Lj9JqtEA4F8A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=olVPoo8AikYy4Q17j67lvkXpHMRRH3k3RbAUlX5D2JM=; b=RvmdZ3o+R1Yohr6IJK4UQBgUyhjrlPxI2Vpy0Q2k+E/7kssgdR32QDVIBfmnD6Kxg67+C00q17VQx/JM14sqRuj4EVB2hugEPq+S3L/rpYSLRa5m4Wx8+M3ZcymwdaTYYRPw4ZA061dzMbeuW92crJSf4T1PCEwYhFm2riCj59tGm4qNCmSF4+Tyc5voAIDcJp6Me/Ew5QE0POZzT43NDMhop2GTIbcAi5zCdr+gzbWS0tVG8YLAezhDQ3+KV7hOs933C6CmypM5Kgh0aWnDFClRTR70mT4/Uv/j8awRPTkH0sWu8R75AxYrtU9wEZ846cDGTLkvZykMhVi9H1mJ3A== Received: from SA9PR03CA0016.namprd03.prod.outlook.com (2603:10b6:806:20::21) by DM6PR12MB4172.namprd12.prod.outlook.com (2603:10b6:5:212::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Wed, 2 Sep 2026 14:53:40 +0000 Received: from SA2PEPF00001509.namprd04.prod.outlook.com (2603:10b6:806:20:cafe::12) by SA9PR03CA0016.outlook.office365.com (2603:10b6:806:20::21) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.11 via Frontend Transport; Wed, 2 Sep 2026 14:53:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SA2PEPF00001509.mail.protection.outlook.com (10.167.242.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.382.8 via Frontend Transport; Wed, 2 Sep 2026 14:53:39 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:11 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:10 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 04/13] match-sat-alu.pd: Recognize borrow-test saturating subtraction Date: Wed, 2 Sep 2026 16:52:45 +0200 Message-ID: <20260902145254.77832-6-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001509:EE_|DM6PR12MB4172:EE_ X-MS-Office365-Filtering-Correlation-Id: 0647bbbf-9a37-47b4-b359-08df0901f994 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|1800799024|376014|23010399003|36860700016|13003099007|6133799003|10067099003|56012099006|11063799006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: LKwYSKvvz9JIWNtroy+e12qGHtbC19mSLDesUvLHI1pzsI7qyHmk732Lei8dW4FOQsY6EN3gSfjctP4unOn6rdWuDZEmOqgrcJ+dvb5qMO7wzZYRacQRUtp+ubJyK6ryOz67PWDgbiEdrnBI8+siEv0R9kvefHtweaSrtYyxv+x7iDhXG1oi+ent4+6ozIymo/FPBIcwAYkY1+R0epi9IDTM4OL6TorqxkjaZQV6Ti3IjFLFKBInRjcvZW8ECu3ReWcmYQ4CCoM3LYrYxaV48zbJ94dDQYS0+HT11g79wZMgvKezoQ5OQk3N0LK5oJsWQMYzJiJOOZQin0O+lYuoKP8TXP+0Hkp2VbVgpbBv3B/l5cex9FUk99Xu328/G0dMFJeSJAz5ItMrPL3aM5TPTprX+/rYswoysF0AMBHyehpS5gK/quccvRFFCsmpIFeMH8o9tLkr/Q8eTfS1obONaQtApbfxFCnoIliFRU9DhoyfvMqx9WZRvwLi+WYSuREhuNV1BTIF2rIVhrzdxjJa5a+GXfe7vnF2Hip4wq83YKUfV4zfodk1yf6RH5Bv9m3R2TQdehuOBaG/zaUnZeQNRHPMQc0bo0I+/TchsUwhsRWm2I7dglZlbt5ofGYVCFI/JnkWzOORq1TogZpFiwYFcCytY7b7C2jf/XhjXMFfDeOmsZvBBber/zklGp+GhbpSpJOooF4zXDv3OFXEpx9A4w== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(82310400026)(1800799024)(376014)(23010399003)(36860700016)(13003099007)(6133799003)(10067099003)(56012099006)(11063799006)(18002099003)(22082099003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9H5kmH8NFB4w+GRGMMsNKAThLGA3ulALjqkUigrWCaSEmPv9RFVHgMx4yZoiJqy9cwdNIMmiByYpkrfwFT31obzwdNna/WqjvDFQlFZ8EKssQ0tP4j69V/ONPO9qliaT1PS1WfniDproZtIoABRBl99oaBmu9uAXdU5M5jvlVQLSyFosMPfqAEg8sYpGcBrmkedcrJGGwZ/x2JTp8FCUl5bh73jHDyED9X7CKitI5/olMMfUdvHXEgPWaZOobL+AIw1xeVrZL5jLYS9EI8Ez/UjGjB7q6cpJOkubkJvAo5+ciknmAAmz4uE/zJoIwf5JmZCILW3ytiv6FzvGk6wQYoVuT23RQj6snk01GWFSJtApOZ68oadyDea0uf4SV5hL/m0rF6JdpHEnW5tQJwMcK0rolshw9p2qhu8m1ive4/CB3cVdGxqDy1RdKjoXuLdw X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:39.5437 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0647bbbf-9a37-47b4-b359-08df0901f994 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001509.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4172 X-Spam-Status: No, score=-8.1 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov For unsigned subtraction, the wrapped difference is greater than the minuend exactly when the subtraction borrows. These forms are therefore saturating subtraction: R = X - Y R > X ? 0 : R R <= X ? R : 0 Recognize both comparison operand orders and both result arm orders. AArch64 -O3, vector body: before: sub v30.4s, v31.4s, v30.4s cmhs v31.4s, v31.4s, v30.4s and v31.16b, v31.16b, v30.16b after: uqsub v30.4s, v31.4s, v30.4s Three vector operations become one. Remove four existing expected-failure markers for the supported forms. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (unsigned_integer_sat_sub): Add the four forms that compare the difference with the minuend. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sat_u_sub_borrow-1.c: New test. * gcc.target/aarch64/saturating_arithmetic_1.c: Remove the xfail on usub. * gcc.target/aarch64/sve/saturating_arithmetic_1.c: Remove the xfail on usubq. * gcc.target/aarch64/sve/saturating_arithmetic_3.c: Likewise. * gcc.target/aarch64/sve/saturating_arithmetic_4.c: Likewise. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 9 ++++++++ .../gcc.target/aarch64/sat_u_sub_borrow-1.c | 22 +++++++++++++++++++ .../aarch64/saturating_arithmetic_1.c | 2 +- .../aarch64/sve/saturating_arithmetic_1.c | 2 +- .../aarch64/sve/saturating_arithmetic_3.c | 2 +- .../aarch64/sve/saturating_arithmetic_4.c | 2 +- 6 files changed, 35 insertions(+), 4 deletions(-) create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_sub_borrow-1.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index 8c7bc0782a0..dfb2eb9800f 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -131,6 +131,15 @@ along with GCC; see the file COPYING3. If not see /* SAT_U_SUB = (X - Y) * (X >= Y) */ (mult:c (minus @0 @1) (convert (ge @0 @1))) (if (types_match (type, @0, @1)))) + /* The wrapped difference exceeds the minuend exactly when the subtraction + borrows, so a source that names the difference and then tests it against + the minuend is a saturating subtract. */ + (match (unsigned_integer_sat_sub @0 @1) + /* SAT_U_SUB = (X - Y) > X ? 0 : X - Y, and the commuted comparison. */ + (cond^ (gt:c (minus@2 @0 @1) @0) integer_zerop @2)) + (match (unsigned_integer_sat_sub @0 @1) + /* SAT_U_SUB = (X - Y) <= X ? X - Y : 0, and the commuted comparison. */ + (cond^ (le:c (minus@2 @0 @1) @0) @2 integer_zerop)) (match (unsigned_integer_sat_sub @0 @1) /* SAT_U_SUB = X - MIN (X, Y). The MIN has to be single use: while it stays live the saturating subtract is computed beside it instead of diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_sub_borrow-1.c b/gcc/testsuite/gcc.target/aarch64/sat_u_sub_borrow-1.c new file mode 100644 index 00000000000..8b86cb92922 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_sub_borrow-1.c @@ -0,0 +1,22 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-optimized" } */ + +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned int u32; +typedef unsigned long long u64; + +/* The wrapped difference is larger than the minuend exactly when the + subtraction borrowed, so each of these is a saturating subtract. */ + +#define DEF(N, T) \ + T f1_##N (T a, T b) { T r = a - b; return r > a ? 0 : r; } \ + T f2_##N (T a, T b) { T r = a - b; if (r > a) r = 0; return r; } \ + T f3_##N (T a, T b) { T r = a - b; return r <= a ? r : 0; } + +DEF (8, u8) +DEF (16, u16) +DEF (32, u32) +DEF (64, u64) + +/* { dg-final { scan-tree-dump-times "\\.SAT_SUB " 12 "optimized" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/saturating_arithmetic_1.c b/gcc/testsuite/gcc.target/aarch64/saturating_arithmetic_1.c index 8fc1569845b..c73a024edec 100644 --- a/gcc/testsuite/gcc.target/aarch64/saturating_arithmetic_1.c +++ b/gcc/testsuite/gcc.target/aarch64/saturating_arithmetic_1.c @@ -19,7 +19,7 @@ ** ret */ /* -** usub: { xfail *-*-* } +** usub: ** dup v([0-9]+).8b, w[01] ** dup v([0-9]+).8b, w[01] ** uqsub b([0-9]+), b\1, b\2 diff --git a/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_1.c b/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_1.c index 6936e9a2704..4cfab740c40 100644 --- a/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_1.c +++ b/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_1.c @@ -39,7 +39,7 @@ ** ... */ /* -** usubq: { xfail *-*-* } +** usubq: ** ... ** ld1b\tz([0-9]+)\.b, .* ** ld1b\tz([0-9]+)\.b, .* diff --git a/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_3.c b/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_3.c index 14e2de59b1e..def7c82e256 100644 --- a/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_3.c +++ b/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_3.c @@ -40,7 +40,7 @@ ** ... */ /* -** usubq: { xfail *-*-* } +** usubq: ** ... ** ld1w\tz([0-9]+)\.s, .* ** ld1w\tz([0-9]+)\.s, .* diff --git a/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_4.c b/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_4.c index 05a5786b4ab..5b9f8740ed4 100644 --- a/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_4.c +++ b/gcc/testsuite/gcc.target/aarch64/sve/saturating_arithmetic_4.c @@ -40,7 +40,7 @@ ** ... */ /* -** usubq: { xfail *-*-* } +** usubq: ** ... ** ld1d\tz([0-9]+)\.d, .* ** ld1d\tz([0-9]+)\.d, .* From patchwork Wed Sep 2 14:52:46 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142723 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E77954BAE7CE for ; Wed, 2 Sep 2026 14:56:40 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E77954BAE7CE Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=eg5qOvDj X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazlp170100005.outbound.protection.outlook.com [IPv6:2a01:111:f403:c111::5]) by sourceware.org (Postfix) with ESMTPS id EDD984BA79BD for ; Wed, 2 Sep 2026 14:53:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EDD984BA79BD Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org EDD984BA79BD Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c111::5 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360826; cv=pass; b=sR6m4WqrGYvPZU3cRiPehawGZlgG36Vmvu8iAlYKzloZGTRu7vCOrwjnK1LVqTWim54JqU7c85LPW9rjbJXdnby97DaKxDjYRd6JUMkDtCb7djnCGYarkcy9x5FczBV/ZZimhaQuhQd4+5nRdafWmeiK4La6aBbZECaK3ABWa6w= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360826; c=relaxed/simple; bh=nKzw1JUMKa/GtfsoiRu31IBQNOAGupzL3hkFj8tEajY=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=WMFboKYnbeUsL2Rbcj4Qjw8mMxVQCWp1HbMxK/ihUBeaMUTcbQgAAuabr3JFTl7w1vdimjPSVDJihgn2N7xnPcnIeA7ejEB4nvRcx/8+MwQb1gcqhJeacnIwvbhmqAc1C8vDJTb5hI2Lkis7wbUEni8CP5pYGLBP9cuasGk1MXU= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=eg5qOvDj DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org EDD984BA79BD ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BGLxYGkJ8JlbzOIh2eTOG4LNvEQWbIQtnrKbUM/UpUEi5m5c8VPfdFwjbMMBALRfsRej3qtZARPdnL8R/O6ApMvLBHAYj2lJiKSliz7KFToAX7N1c2I/7idKTjlC53i+YKn09DWYxCABq4VPJdkwvjF7+COReJNt23wC5AGy3ifsyR2Vjkg1r9nEUuEUTaFj6YGT7J3sMu3IF7UzCCj847DdlX0ivO0364iOxZmcLiaE4jFZ11gZcssLqRyXNQwzy9P1RYRR02aknCaRRyiRU1QtMXHrzucjz1gLjJVQBIR4t4gYSa+tUXl04JKPF/9JdcX7ySFdubVQaUMGIJwuYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=a6tYjBMFfjMc9PuXn+kKAf4iUuIen4ASz0ZmKXXkXxY=; b=eM1fwuhROeYrBZwL6mtTF9TdXJc0OCUoX0qH81rRcSZH3oFfjqcyhIk6ds1GFW+6H/HI840yjjsDq+A7Y5y9YeblEMQCF+l+uBwL5lwvDKovyYMgiK4aSWhylSddmbSH2hbux7aBDe6yZxrLJI7UXaVl8cEwB3boekRJWlkcMtVwQcwkT57kfLDZWwKcV1+KLKOPL2yy/GLbxKK422nxpnnNVuYZHHerjEcz7hrCb/ygoF+YAvzzywEMEXIQ+WraAcbTpR5a6jcohvCof2LnBZvFsSkBenEQa/dBbeBWLUzGNF6/8Pdx2g+13xgZJBcvEq4Q5tE8xODWtRIr7V4pmg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a6tYjBMFfjMc9PuXn+kKAf4iUuIen4ASz0ZmKXXkXxY=; b=eg5qOvDjiHixQlsQj2xun/BVCJatUJLVZVDWvstJ8GqLeNKl+p1OXA3ZncQ7RCh4rO7If4SALRG5Y2KTfdwMvxE3/6l2Ypr/Y9nMAz9HkR+VAwoT5toEa4mD9BGh3w65lPB3fikPOZxRhrfpCFLE3elz0GWV5RQeQkr6juyb7CDNUMxDwtZyG9BHgr91xoccM7aVr+i85WpimzicFX4ypIuld+kpCmhpdGR18/8eCXkkyoLHhislqMMzpVxjqO8TCehCZTC+PNQGLuCKyO89rJMBu81O9mqnAixas4mqyraH+IFajDxzi6n2g4QhsIEbXE+tJCu9L8scXFtTl/UMlA== Received: from MN2PR07CA0028.namprd07.prod.outlook.com (2603:10b6:208:1a0::38) by BY5PR12MB4097.namprd12.prod.outlook.com (2603:10b6:a03:213::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Wed, 2 Sep 2026 14:53:40 +0000 Received: from BN3PEPF0000B070.namprd21.prod.outlook.com (2603:10b6:208:1a0:cafe::4a) by MN2PR07CA0028.outlook.office365.com (2603:10b6:208:1a0::38) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.12 via Frontend Transport; Wed, 2 Sep 2026 14:53:40 +0000 X-MS-Exchange-Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN3PEPF0000B070.mail.protection.outlook.com (10.167.243.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.1 via Frontend Transport; Wed, 2 Sep 2026 14:53:40 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:13 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:11 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 05/13] match-sat-alu.pd: Recognize more unsigned saturating subtraction forms Date: Wed, 2 Sep 2026 16:52:46 +0200 Message-ID: <20260902145254.77832-7-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B070:EE_|BY5PR12MB4097:EE_ X-MS-Office365-Filtering-Correlation-Id: 1089288c-ee67-47c5-4b69-08df0901fa29 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|23010399003|82310400026|36860700016|1800799024|10067099003|6133799003|11063799006|18002099003|22082099003|56012099006|13003099007; X-Microsoft-Antispam-Message-Info: +wo9iCaOJ+6WV58Bc1Vhds5CXwuM2cwTm0QPll7+OGPzcyM8LrbOB6Fd5ncspic0nCKFx4ld5+GUNAvlVKBlF9npUyjVV803TGgRB0SF8aM836ju+rStozA+xcLf4V0XaXQBnvsWlqaCGETFERKpLw5Nu5NQDopl2nW89xUHu550rOTG8+75kRvPP95eLr9++wqm84zeRhHmI0RYxeqsLIzXFWTZ1mzrocJ5zGciCj+yf+O8hriUPcGAlvkGfDdBa6QTjjH+CxjLxdigYZtH7+k0HAidljg6wdRdq7zVCYtedhigx/kswz4FM1vYth70WcxZwiao1PjgyoHpy8vCqb1WKgWlPs5cifgdRd/z1TBFyzHnh3SaueZhLdeSJJ63xSOOgTwMOTGC4MnmJLLJogZiImzVEutnj6r7ebSaoLKO/Wp4QP8DDpZ58tgw78rsInnULNbEVJKgNKL7GD23006tGtsfzpDJlxYxTOl261gfWR1Af415+vgvIIbVx3OrRkLmQ68nj1XpMhRIaEFLvLldlFdLnRNf7FjFJIRHwFTZ//rz9cUmKNe3d8hwv0TI1Seg9/uP1Ie4iqExCxEshAsuuZIsNsuGfyrdGJiJZCLpW6OT2cDMlapmv++P8Ep2MPhh9yw1mvxurDTlCYfvLDmvekHv1efnm5YV1oguiFQ7hhhufrFabvQxjOwxnh1OJYP2bLrvRWR7b6kRzDbYZA== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(376014)(23010399003)(82310400026)(36860700016)(1800799024)(10067099003)(6133799003)(11063799006)(18002099003)(22082099003)(56012099006)(13003099007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: zdkVDol9A04Qkz1EzlwmCpSf/GDa9Uqb4dZ8wHguZ1lktNjgtS/F6jBC2LYIrAiCOpjNiGe9gdUT5YmoUwUWOKX8fL/qN1jQ6skMHPVzQXcP3jpBOQEzjNzvfzweVGBCmuVgAN1u2nRImqnPevC0XgSregFMiNxPWbqzrw/uHomOJ2wEp6bocsRT+1/lLXWiLj5BKiVvj6ztJdBEjOUsoyP3v2d+jks0YuipJdgGUcF2AYhXrK7l3u1nYz2Eoji/CgCbvZKrTBQSEDEI2w52bfMssWW5Vxxe9H/NybTrJnCtzgNX5atibIX8ADWgmOHrkAPSBQkzlcSuPM511O/7Up03HOmzcCqHAaLiUK8bZqOw7WsmUDlWJwEzHhdAdV3NLcXVGLiOKqfkFuRyHxN+iE+FMfkAYdOW4fgjXlKuK+uiLnlST9AiagwOH4OpIUoF X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:40.4576 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1089288c-ee67-47c5-4b69-08df0901fa29 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B070.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4097 X-Spam-Status: No, score=-8.1 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov Equivalent unsigned saturating subtraction forms can reverse the comparison operands or the conditional arms: B < A ? A - B : 0 A < B ? 0 : A - B The middle end can keep either comparison operand order. Control flow with two returns can also reverse the conditional arms. Recognize both forms and reversed comparisons in the multiply-by-predicate form. AArch64 -O2: before: sub w2, w0, w1 cmp w1, w0 csel w0, w2, wzr, cc after: subs w0, w0, w1 csel w0, w0, wzr, cs The vector form replaces a comparison, a subtraction, and an AND with one UQSUB. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (unsigned_integer_sat_sub): Accept commuted comparison operand orders and the arm order from a two-return source. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sat_u_sub_swapped-1.c: New test. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 29 +++++++++++++++++++ .../gcc.target/aarch64/sat_u_sub_swapped-1.c | 27 +++++++++++++++++ 2 files changed, 56 insertions(+) create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_sub_swapped-1.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index dfb2eb9800f..b3dd8db0d27 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -131,6 +131,35 @@ along with GCC; see the file COPYING3. If not see /* SAT_U_SUB = (X - Y) * (X >= Y) */ (mult:c (minus @0 @1) (convert (ge @0 @1))) (if (types_match (type, @0, @1)))) + /* Accept the comparison with its operands in the opposite order. */ + (match (unsigned_integer_sat_sub @0 @1) + (cond^ (lt @1 @0) (minus @0 @1) integer_zerop) + (if (types_match (type, @0, @1)))) + (match (unsigned_integer_sat_sub @0 @1) + (cond^ (le @1 @0) (convert? (minus (convert1? @0) (convert1? @1))) + integer_zerop) + (if (TYPE_UNSIGNED (TREE_TYPE (@0)) && types_match (@0, @1)))) + (match (unsigned_integer_sat_sub @0 @1) + (mult:c (minus @0 @1) (convert (lt @1 @0))) + (if (types_match (type, @0, @1)))) + (match (unsigned_integer_sat_sub @0 @1) + (mult:c (minus @0 @1) (convert (le @1 @0))) + (if (types_match (type, @0, @1)))) + /* A source with two returns can keep zero on the taken arm. */ + (match (unsigned_integer_sat_sub @0 @1) + (cond^ (lt @0 @1) integer_zerop (minus @0 @1)) + (if (types_match (type, @0, @1)))) + (match (unsigned_integer_sat_sub @0 @1) + (cond^ (le @0 @1) integer_zerop + (convert? (minus (convert1? @0) (convert1? @1)))) + (if (TYPE_UNSIGNED (TREE_TYPE (@0)) && types_match (@0, @1)))) + (match (unsigned_integer_sat_sub @0 @1) + (cond^ (gt @1 @0) integer_zerop (minus @0 @1)) + (if (types_match (type, @0, @1)))) + (match (unsigned_integer_sat_sub @0 @1) + (cond^ (ge @1 @0) integer_zerop + (convert? (minus (convert1? @0) (convert1? @1)))) + (if (TYPE_UNSIGNED (TREE_TYPE (@0)) && types_match (@0, @1)))) /* The wrapped difference exceeds the minuend exactly when the subtraction borrows, so a source that names the difference and then tests it against the minuend is a saturating subtract. */ diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_sub_swapped-1.c b/gcc/testsuite/gcc.target/aarch64/sat_u_sub_swapped-1.c new file mode 100644 index 00000000000..fb42b6b07e3 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_sub_swapped-1.c @@ -0,0 +1,27 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-optimized" } */ + +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned int u32; +typedef unsigned long long u64; + +/* The same saturating subtract with the subtrahend written on the left of + the comparison. GIMPLE orders comparison operands by SSA version, so + which spelling survives does not follow the source. */ + +#define DEF(N, T) \ + T f1_##N (T a, T b) { return b < a ? a - b : 0; } \ + T f2_##N (T a, T b) { return b <= a ? a - b : 0; } \ + T f3_##N (T a, T b) { return (T) ((a - b) * (T) (b < a)); } \ + T f4_##N (T a, T b) { return (T) ((a - b) * (T) (b <= a)); } \ + T f5_##N (T a, T b) { if (b >= a) return 0; return a - b; } \ + T f6_##N (T a, T b) { return b >= a ? 0 : a - b; } \ + T f7_##N (T a, T b) { if (a < b) return 0; return a - b; } + +DEF (8, u8) +DEF (16, u16) +DEF (32, u32) +DEF (64, u64) + +/* { dg-final { scan-tree-dump-times "\\.SAT_SUB " 28 "optimized" } } */ From patchwork Wed Sep 2 14:52:47 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142728 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 9B1BD4B99F5B for ; Wed, 2 Sep 2026 14:59:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9B1BD4B99F5B Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=p0wP+Vsq X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012000.outbound.protection.outlook.com [52.101.48.0]) by sourceware.org (Postfix) with ESMTPS id BB5AB4B9DB52 for ; Wed, 2 Sep 2026 14:53:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org BB5AB4B9DB52 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org BB5AB4B9DB52 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.48.0 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360829; cv=pass; b=GyEUWbnLKXWdbO5KmIyZ1nC3lZDyhtfWO0j4/8xvHnvzggJ7hb49w8TGYOMvhvVL2f5p/xINvigGmlRCl3LDnrAj9T7VrtLqsNEy6qMpuFOR1iI5rXtVepmNBsYI2J0QHoF1hrsqqyATmS/q7Ej7TtAx0OZZgWbPKh5OwAqmGek= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360829; c=relaxed/simple; bh=CtWHw2RnBu8TuCvhRLmrRywtscxHzXuQcjtJK/zm7AQ=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=RCsZ98aU/8LVqZYkn2kzC8Odi1pQbFJ0TlF6MpMPV4rl0frKiDQQHfmnNEwikrmOG9Oj/Bo8uYI0xyqIIjgACfjXcOIPj8l/EYuom6UHu5Rum6RhzYjyuSJLdLm4arYjxtVNVP/XOQpuzh8RhlydsRDyZ54H0CIpR+5ehhwXkcg= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=p0wP+Vsq DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BB5AB4B9DB52 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Nh2NfFtq0aSOlo8PngK4/L+v34l7OZo4SZgexJrS5Qs7Q+eO0uxf8jMHyafC8PJMGZYftJNtykJJ+ZFaZrktNDNlTSrNXdIDVpVztOAUhh2fK95aLWeUc1AumzammbUgsBXJ4YqSusxpwVKaKv99RGBS5FbO/enwTD/WQ7Ntumd1oLkVCxaIQUO8ezd8FEkU/unFS0F7Af0Ywi0fE+RWEPAaCkAZdw4Lh+17kk4OC3xkAccCQGnSc9CEkRGkqPS+NA4q7YGEc6WSMtxbT/XUW+zh0JWU7sDM97ex0fOfFRhu8vjqVYEFB3rXqMfjJWPmqnhYk2nVuCnG7iozxQoYtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AyGrBeKywUFa6WLOQvZkZp2BKMOfSjukT/JpyWDMJK8=; b=Hk8c0cGlVLHKnqwcYue12iMbZNFqIDn+D9JgMtwdwFBHOqUwn08FQ1OQ7Km6z/+7ON/qsr4Hv+L+sOgI9xXLcE7WWebDAu93IwT8zXQLzFFUdGvuMur9xG+VSBPhvGVz1Km2tQcXxoh40YLrTHGELuV5s3RWFKuKoy/tBceM8uLRsiyV5y4mtgGy2XNVFL+gcIiXht9y7isiu8l8SXdP8MmJSjbPyDk7PkDCA9/wzv5yAVXMgRpeFe5+SwwlI1zvuR3F3O3LmMO/mvS3SjdnyVISq7/6GeVpbOUtYAnjfxSyrBRc8ZldvFo5Yh50j4S1WdCfkHMZjWN4J+wdM136wA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AyGrBeKywUFa6WLOQvZkZp2BKMOfSjukT/JpyWDMJK8=; b=p0wP+VsqndAcv8CSk/Fa3/u1mvLtXujNMZ77P4rbdNM8bQ2bsQNwnKw+hQ5b/z46+0X2eEFmWYY9W9XUvm28DvRr45BaG5dKFrd1mzaRhnuVw7sfY4D3javsJuXP17bxGzz9uETxM3a3sbaYEivaPYpwLqLUwPIJj5Wi1DqbD3uTDZVXLVWFR/8WwnW/IHjE4cn1hZFJqKwhHXHWoHXw8uPlRzD2D62/IQ1oh2XbDcFxTVoqvRNiaFXiV1A+f5IZeXztrZAJ1ntJVs/4NGSO47HEaOqbfJC6KBQ93pbsXpkPnfR/AqhJbyplNqvnbUKCk673QVKnfGbkvIorm9Du5A== Received: from SA9PR13CA0023.namprd13.prod.outlook.com (2603:10b6:806:21::28) by PH0PR12MB8029.namprd12.prod.outlook.com (2603:10b6:510:26c::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Wed, 2 Sep 2026 14:53:41 +0000 Received: from SA2PEPF00001508.namprd04.prod.outlook.com (2603:10b6:806:21:cafe::3b) by SA9PR13CA0023.outlook.office365.com (2603:10b6:806:21::28) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.8 via Frontend Transport; Wed, 2 Sep 2026 14:53:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SA2PEPF00001508.mail.protection.outlook.com (10.167.242.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.382.8 via Frontend Transport; Wed, 2 Sep 2026 14:53:41 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:14 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:13 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 06/13] match-sat-alu.pd: Recognize complement tests for unsigned saturating addition Date: Wed, 2 Sep 2026 16:52:47 +0200 Message-ID: <20260902145254.77832-8-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001508:EE_|PH0PR12MB8029:EE_ X-MS-Office365-Filtering-Correlation-Id: 80cab2e0-20c0-4c50-e789-08df0901fae9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700016|1800799024|376014|23010399003|82310400026|11063799006|10067099003|6133799003|22082099003|18002099003|56012099006|13003099007; X-Microsoft-Antispam-Message-Info: Y0k6sa+JYBXvo3iX8ZfAhAHNHlR6tZ1QEJfVXAmvdl8VDWM4bQkd2X26dkXuv4Y5TYZQTjbKtzCAcGKsZBF0LrHANQvWkihHp3LBKqDfIJUOSG2jQ65k8nd52EsCU8HhDgg0fgQzx7eMkP3zelDn+1hYcrIEo4zVLeZwGBAJOe1+ua1dQOdXm5g5s1HDmV3oYE3jy+1fvToMwwzi2jXApi11wzRsPSbrEG7cE0TIk0KEqjTRvln+C36lYLMZJGBYvxY7shTzT/6NF4xhyJxvdQPXj+6RKmeczBOhoWUbvFVEvI1CDeWzCAXV47zhvwjg+CvXCJsBXIbs0iT+qI1mm8YtWIor6C7i12gFSnML+VT1gDkD1YNE61u+LhT5ZGn8f5IHRszmMiYZLHjibBoMSANCSx14dn0xmEn004Rx77sb2Efjr945M9+7nU6EeSpjGHsqRaexCcsd4YaUql6zkGWhnj5eF60jTunrKPrrFY8or1+dVWwXCb8G9x0a0in2iLIztSxoPaoIhQ62p4ggz/qBGkw35SbySFyrNbGZfViiOnR/B62IR+0g+ZcAaYTUiZBsUGGAHF0tlOihvrJ35XxAsGEXD27Qv0rwKYmDRZ81KY5BenGtn96dZcSwqhzEU/dPYy++Rb2bZNABdk2pbVzQWSdUqWA90HDrh3AUHcQmmhBDIHqBnZYCGEifo+Qrp+xoUuVYsrXXIK3Rqt7vIA== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(36860700016)(1800799024)(376014)(23010399003)(82310400026)(11063799006)(10067099003)(6133799003)(22082099003)(18002099003)(56012099006)(13003099007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: wSTFk3ZLSwggiMUxWHX0uvx8uQ1YB/EMh/pw2+uq/44pNqxA1aEHYImsjrcWMG1I6M1OaHJKuZnbMA63nFMd7kEpTVCUQ5TnoOztIFwDCln6kJWVqIlpqaugbUysfGvgFOJuA5rbFhGJ37nb9kcRILKUjKrotnFMMI/Z++6clg3Y8/iZL+wnLFlVLjeDUsAl/G4y07XVgOacxg3rhuRGdfRKHURX53qO9Dg53iDuz9A0sqz3J9eqgqLkHToT0IxpO16Lm4VcOxuGg+nlJdCPCxszERXP4Bn5JokVERnAtKpVVF/zMX9AArTkI1wvtvQXmrfXt0ApQjKTDgFqorEstpzQZuj6FzpGDtG7hq7QI7KeFoobwIWtf8a0c+L46otyCgUNgeL8KNefjzSIFWQ1/iEYGj6HD0l7bIL4yKCxZ3nHT6EjSvqKAGHtWlyujn7+ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:41.7818 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 80cab2e0-20c0-4c50-e789-08df0901fae9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001508.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB8029 X-Spam-Status: No, score=-8.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov For an unsigned type, MAX - Y is ~Y. An overflow test can therefore use X > ~Y ? MAX : X + Y Recognize strict and non-strict comparisons with either operand or result arm order. Equality is safe because both arms produce MAX. Also recognize ADD_OVERFLOW forms that select a separate X + Y expression. AArch64 -O3, vector body: before: add v29.4s, v31.4s, v30.4s not v30.16b, v30.16b cmhs v30.4s, v30.4s, v31.4s orn v30.16b, v29.16b, v30.16b after: uqadd v30.4s, v31.4s, v30.4s Four vector operations become one. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (unsigned_integer_sat_add): Recognise comparisons against the complement of an operand. Also recognise ADD_OVERFLOW with a separate addition. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sat_u_add_not_cmp-1.c: New test. * gcc.target/aarch64/sat_u_add_not_cmp-2.c: New test. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 32 +++++++++++++++++++ .../gcc.target/aarch64/sat_u_add_not_cmp-1.c | 30 +++++++++++++++++ .../gcc.target/aarch64/sat_u_add_not_cmp-2.c | 17 ++++++++++ 3 files changed, 79 insertions(+) create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_add_not_cmp-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_add_not_cmp-2.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index b3dd8db0d27..9564366d32f 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -97,6 +97,38 @@ along with GCC; see the file COPYING3. If not see (cond^ (ne (imagpart (IFN_ADD_OVERFLOW@2 @0 INTEGER_CST@1)) integer_zerop) integer_minus_onep (realpart @2)) (if (types_match (type, @0) && int_fits_type_p (@1, type)))) + (match (unsigned_integer_sat_add @0 @1) + /* SUM = ADD_OVERFLOW (X, Y) + SAT_U_ADD = IMAGPART (SUM) == 0 ? X + Y : -1. */ + (cond^ (eq (imagpart (IFN_ADD_OVERFLOW @0 @1)) integer_zerop) + (plus:c @0 @1) integer_minus_onep) + (if (types_match (type, @0, @1)))) + (match (unsigned_integer_sat_add @0 @1) + /* SUM = ADD_OVERFLOW (X, Y) + SAT_U_ADD = IMAGPART (SUM) != 0 ? -1 : X + Y. */ + (cond^ (ne (imagpart (IFN_ADD_OVERFLOW @0 @1)) integer_zerop) + integer_minus_onep (plus:c @0 @1)) + (if (types_match (type, @0, @1)))) + /* A source that tests for overflow against the complement of an operand, + such as a > MAX - b ? MAX : a + b, reaches the recognisers as a compare + with ~Y, since ~Y is MAX - Y. The strict and non-strict forms are both + exact because at X == ~Y the sum is MAX and the two arms agree. */ + (match (unsigned_integer_sat_add @0 @1) + (cond^ (ge (bit_not @1) @0) (plus:c @0 @1) integer_minus_onep)) + (match (unsigned_integer_sat_add @0 @1) + (cond^ (gt (bit_not @1) @0) (plus:c @0 @1) integer_minus_onep)) + (match (unsigned_integer_sat_add @0 @1) + (cond^ (le @0 (bit_not @1)) (plus:c @0 @1) integer_minus_onep)) + (match (unsigned_integer_sat_add @0 @1) + (cond^ (lt @0 (bit_not @1)) (plus:c @0 @1) integer_minus_onep)) + (match (unsigned_integer_sat_add @0 @1) + (cond^ (lt (bit_not @1) @0) integer_minus_onep (plus:c @0 @1))) + (match (unsigned_integer_sat_add @0 @1) + (cond^ (le (bit_not @1) @0) integer_minus_onep (plus:c @0 @1))) + (match (unsigned_integer_sat_add @0 @1) + (cond^ (gt @0 (bit_not @1)) integer_minus_onep (plus:c @0 @1))) + (match (unsigned_integer_sat_add @0 @1) + (cond^ (ge @0 (bit_not @1)) integer_minus_onep (plus:c @0 @1))) (match (unsigned_integer_sat_add @0 @1) /* WIDEN_SUM = (WT)X + (WT)Y SAT_U_ADD = WIDEN_SUM > MAX ? MAX : (NT)WIDEN_SUM */ diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_add_not_cmp-1.c b/gcc/testsuite/gcc.target/aarch64/sat_u_add_not_cmp-1.c new file mode 100644 index 00000000000..88041f53f61 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_add_not_cmp-1.c @@ -0,0 +1,30 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-optimized" } */ + +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned int u32; +typedef unsigned long long u64; + +/* Every one of these tests for overflow against MAX - b, which reaches the + middle end as a comparison with ~b. The strict and the non-strict form + agree, because at a == MAX - b the sum is MAX and so is the other arm. */ + +#define DEF(N, T, MX) \ + T f1_##N (T a, T b) { return a > (T) (MX - b) ? MX : a + b; } \ + T f2_##N (T a, T b) { return a >= (T) (MX - b) ? MX : a + b; } \ + T f3_##N (T a, T b) { return b > (T) ~a ? MX : a + b; } \ + T f4_##N (T a, T b) { return b >= (T) ~a ? MX : a + b; } \ + T f5_##N (T a, T b) { return a <= (T) (MX - b) ? a + b : MX; } \ + T f6_##N (T a, T b) { return a < (T) (MX - b) ? a + b : MX; } \ + T f7_##N (T a, T b) { return (T) (MX - a) < b ? MX : a + b; } \ + T f8_##N (T a, T b) { return (T) (MX - a) <= b ? MX : a + b; } \ + T f9_##N (T a, T b) { return (T) ~b >= a ? a + b : MX; } \ + T f10_##N (T a, T b) { return (T) ~b > a ? a + b : MX; } + +DEF (8, u8, 255) +DEF (16, u16, 65535) +DEF (32, u32, 0xffffffffu) +DEF (64, u64, ~0ull) + +/* { dg-final { scan-tree-dump-times "\\.SAT_ADD " 40 "optimized" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_add_not_cmp-2.c b/gcc/testsuite/gcc.target/aarch64/sat_u_add_not_cmp-2.c new file mode 100644 index 00000000000..bf93656885f --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_add_not_cmp-2.c @@ -0,0 +1,17 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -fdump-tree-optimized" } */ + +typedef unsigned int u32; + +/* Unrecognised, this vectorises as an add, a complement, a compare and a + select instead of one uqadd. */ + +void +f (u32 *__restrict d, u32 *__restrict a, u32 *__restrict b, int n) +{ + for (int i = 0; i < n; i++) + d[i] = a[i] > 0xffffffffu - b[i] ? 0xffffffffu : a[i] + b[i]; +} + +/* { dg-final { scan-tree-dump "\\.SAT_ADD " "optimized" } } */ +/* { dg-final { scan-assembler "uqadd\tv\[0-9\]+\.4s" } } */ From patchwork Wed Sep 2 14:52:48 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142726 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C489E4BAE7D4 for ; Wed, 2 Sep 2026 14:58:27 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org C489E4BAE7D4 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=QaadAsES X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11010051.outbound.protection.outlook.com [52.101.61.51]) by sourceware.org (Postfix) with ESMTPS id 326DC4B9DB5B for ; Wed, 2 Sep 2026 14:53:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 326DC4B9DB5B Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 326DC4B9DB5B Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.61.51 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360828; cv=pass; b=lHyPbCHQLw/owp+Z/ecijM1dxhvk3/0oCExCx6dBaspq595xFCYOgaV29ph3arNzwc38sl2biazO0z9Bjw2oT0P7+4o5oLm2PwYG2Hjs+FDjAF8xrufg8W5TJFPtB2/N4HJWDmtGh8x/yjwa4qemNm9MytUvSgs9wa38ke8IjuA= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360828; c=relaxed/simple; bh=GB2OG777EgOtX6zmatDc4GnhE23pu6aCnvw1YzufEgg=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=SyFSL8CnMKX73Tx56+iEagQbsCUlyFaabyUe1F0Hybhl2hlOiRV//lUITRyIc/H0efM2k2BWwhsk1KF78K27CKXNH5xkQdVv4k6oEXRnHAZDzy6XkmNo/93B0/F61PEi/YZwDecspPe1dxf8EeU0g2VdktKLBgw+QWrDj/V/Tc8= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=QaadAsES DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 326DC4B9DB5B ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NF1eqEoMl7vvayFk9PE3768S6hI8pyZh9QhEQw7RI5yh+omFw9LnitZFld50Fot7bQjgZwa/9PbMcfdIB4SPascJirc7qWBDKY0/OmPU6fcQYryTAq+8ZJq3HUgeaCoFr82JU8mRe9v6b77gAwWwgJO1E1PZuI9kf87IpcVo5A8/2ngWgKvKlxLAYkwOjTdY+Dt62M6CWSNubKvjc/bFkbxDJtlnVrT8aYyuPJ/e59h0jR6p7COBnYA9d/pXUsSmQ+TkMNCZ28N48imt3HGAqfzjQMd6aO21uuJTe9FYh4BFxX9SMyGThyFAZaA9sRhPgB/WQQT1CUWjzBxt6VGRDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Hj3Y6mkDQ4jLOpHPG+SxObnDJ4zS8a8nNrX5dR0peeo=; b=O9ZZb5hP+uNWANLU8Qjl4bKhZxDxuOtdrPP45QtJZn7t30xior3Ccukm8Nvep+WP5kcNBKFlywiWRnSkS6S36tRcYeptvfhmfuRVOwT04xrhZjbtT/IYCHEa14qMba6QNymGCwGCYel/y87fj0zv6DQR3NPdzCrc5WeH2KFMUOAPHGlJclcA6XoVHIMyEnVktP5+0F+sQL0D7rRxjexqyplL0GM4hR7wkgdJsIpmX9d2zzc5TlxbFE4HRCvE6OqumJA1gXes74uA586EtemN1G/TNNh8BotapaiGn11zjSOtsqnr0ktkD0qy8E/fgnhT2wuPkMnGd8NZaq2k0G1NDQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Hj3Y6mkDQ4jLOpHPG+SxObnDJ4zS8a8nNrX5dR0peeo=; b=QaadAsESnlx1yb19aSfyb25qsfwcVYwJSXebs9yM8oJ30S2qqAoYdINcytoUlm4OR34C3A8bVzylvW7ei0Qh00snnPYs3iy2tZGECnNjNg5Nja84QoeuQFU0EH3bzz0k+ngz0EbpkNPfJzt92TNQTwgE95/PqkBebNmYOLyEzcMCcwL8JXIQegqNp0KGGwCKvjYvdlC22jLfwyBP1VOtYeDC5fILWVT2ELQbeZYV6g39KDlUm5LmTwCNHcusWj3pMrXlW7VOXr+vnuPmIY1unxTNEIVo5K62DsTWgdFrBIi6RnVUoyvo8/c6Y1DoLOq63MszBalqQTu2pFH/0D85TA== Received: from SA9PR03CA0029.namprd03.prod.outlook.com (2603:10b6:806:20::34) by SN7PR12MB7978.namprd12.prod.outlook.com (2603:10b6:806:34b::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Wed, 2 Sep 2026 14:53:44 +0000 Received: from SA2PEPF00001509.namprd04.prod.outlook.com (2603:10b6:806:20:cafe::96) by SA9PR03CA0029.outlook.office365.com (2603:10b6:806:20::34) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.11 via Frontend Transport; Wed, 2 Sep 2026 14:53:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SA2PEPF00001509.mail.protection.outlook.com (10.167.242.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.382.8 via Frontend Transport; Wed, 2 Sep 2026 14:53:43 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:15 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:14 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 07/13] match-sat-alu.pd: Recognize X + MIN (~X, Y) as unsigned saturating addition Date: Wed, 2 Sep 2026 16:52:48 +0200 Message-ID: <20260902145254.77832-9-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001509:EE_|SN7PR12MB7978:EE_ X-MS-Office365-Filtering-Correlation-Id: a5cbe1b4-5315-426b-aafd-08df0901fc21 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|36860700016|1800799024|23010399003|376014|18002099003|22082099003|56012099006|3023799007|10067099003|6133799003|11063799006; X-Microsoft-Antispam-Message-Info: iGJMdpq7zf0Fu0UrWvVLzbhvEweAqA98ATuyerkBfiDrkFvZQWwSn2dqG/rVxheKrJMeakqW7X0dAZy1Ukj9Gbn5vJwkbe0u5haQDcewmkBQIZgRrWCCL7LdulLMHyN3iXf6xtvVahJugx8FiLfVL97Jz71yunmBB3A3RVIdJBmcouefDXVTytmqHkjCyQUncDSFrHjeVBNDnzqGqZOes+zfd5nWCR1L2XrXuPpPuH/SjNexFvvG+t0HDrT00seKwC1fFJ5aPAMpqJNiOTqBdAxUFd8z7DhYI4DfQqNEOwyWS51b/qwkpVMvE7/RB+CPaYohUtTzNtyWzZAvcwmFE1w3na+JK8cxBeF0oVYEyXPXceE4scT9kL2KN1C2r3baN8JqC3hPJqL483fYmQn9z79MAr1M9Lo+ZJ5Z9MrGmWBHrh0g+TQBjtDKpGb7Jg1pcBhUb2JxY8cpzG2DILsBecPVoM+so8NT4iFl6s11a6V+mDROVukbW2p+l+pr4h/k4iqslRks/uSOY+OTq/UJFCLqLgSKYX2Ot29Z/ar8M5g1MP1a8HO0On7XlTTNEcEOdBIFW0E+YZAFPulc9WTmzDaZ+lLgoxjOKQBuCjkCu0efN/xaiE6dkkCQKRtkvG8h2lPUJ7d3lXtJHTsFxAa6ZWPiUz9NRSMjYegUZcLc9ONvA/3M+ac0QfVrIkcwNpSQaRHi6Ef+Nhj6TWhAZYno9g== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(82310400026)(36860700016)(1800799024)(23010399003)(376014)(18002099003)(22082099003)(56012099006)(3023799007)(10067099003)(6133799003)(11063799006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Jnyhybk1FK0wFbJnXK/5x9qx5jDkIF2RHCo5eQDxhyqA1nr6meuz9iMtY6ZxGH0+/pPVvJfPp+FMU3KSUW9w0h2eOtGRbcqNrztl/485S6kn59QNf8fIx4caPF3tyM8O0JDXObTdY/UIQd8DLBWd182b7DUNtVilZtzT+Pta6erRDiesV0jp54zw36J7eVikZS2CSkvm+GoUPdvzpMdUUefKTsKyow9rffPCNcuGWO80B5alQWn8hjrgdvYzR9NFfoCIUvtA0AzN23mlj42Ee/wNTWM4hFeRrTuW1XAyyAYGUAocAKbMqyelQ/wAQsvIZSap408R7kTleD97SfWkAbXisz6Q9BaNEhUT8kEVsIEJPtBh4BTzm2qB7hSz4iZEUq57Ti79m8/0J0R91BUAromBHfKE8zFDH4cI6WCjPgJowjnqm49BqqKxyYtYqQdv X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:43.8107 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a5cbe1b4-5315-426b-aafd-08df0901fc21 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001509.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7978 X-Spam-Status: No, score=-8.1 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov For unsigned X, ~X is MAX - X. MIN (~X, Y) limits the addend to the largest value that cannot overflow. X + MIN (~X, Y) is therefore saturating addition. Recognize this form. Require the MIN to have one use so that the replacement removes it. AArch64 -O2: before: mvn w2, w0 cmp w2, w1 csel w2, w2, w1, ls add w0, w2, w0 after: adds w0, w0, w1 csinv w0, w0, wzr, cc Four instructions become two. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (unsigned_integer_sat_add): Add the X + MIN (~X, Y) form. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sat_u_add_min_not-1.c: New test. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 7 +++++++ .../gcc.target/aarch64/sat_u_add_min_not-1.c | 18 ++++++++++++++++++ 2 files changed, 25 insertions(+) create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_add_min_not-1.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index 9564366d32f..b01989c3a1c 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -68,6 +68,13 @@ along with GCC; see the file COPYING3. If not see wide_int sum = wi::add (cst_1, cst_2); } (if (wi::eq_p (max, sum)))))) + (match (unsigned_integer_sat_add @0 @1) + /* SAT_U_ADD = X + MIN (~X, Y). ~X is MAX - X, so the MIN caps Y at the + largest addend that does not overflow. The MIN has to be single use: + while it stays live the saturating add is computed beside it instead of + replacing it, which costs an instruction. */ + (plus:c (min:c@2 (bit_not @0) @1) @0) + (if (single_use (@2)))) (match (unsigned_integer_sat_add @0 @1) /* SUM = ADD_OVERFLOW (X, Y) SAT_U_ADD = REALPART (SUM) | -IMAGPART (SUM) */ diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_add_min_not-1.c b/gcc/testsuite/gcc.target/aarch64/sat_u_add_min_not-1.c new file mode 100644 index 00000000000..58ed36897ab --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_add_min_not-1.c @@ -0,0 +1,18 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-optimized" } */ + +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned int u32; +typedef unsigned long long u64; + +#define DEF(T) \ + T f_##T (T a, T b) { T t = ~a; return a + (b < t ? b : t); } \ + T g_##T (T a, T b) { T t = ~a; return (t < b ? t : b) + a; } + +DEF (u8) +DEF (u16) +DEF (u32) +DEF (u64) + +/* { dg-final { scan-tree-dump-times "\\.SAT_ADD " 8 "optimized" } } */ From patchwork Wed Sep 2 14:52:49 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142724 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D6C454B9DB72 for ; Wed, 2 Sep 2026 14:56:41 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D6C454B9DB72 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=DeyvBk2d X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azlp170120001.outbound.protection.outlook.com [IPv6:2a01:111:f403:c107::1]) by sourceware.org (Postfix) with ESMTPS id 71A044B9DB4D for ; Wed, 2 Sep 2026 14:53:50 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 71A044B9DB4D Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 71A044B9DB4D Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c107::1 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360830; cv=pass; b=l2LI+Tww598Fykq4ejfXllXItsp7mdYN86zSJWRUc8O/ejCxUqngT18Cy10RthbZW0747xyBURLEnLSfBIj1D9gd1wVlBiacYfIIryEY6BL1PjY54zQlHwrN3lOW6mU2Xuqc/Kis5dp2l7Y1yPHHn/D5BqeNN/Ou8yj7oq3Qits= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360830; c=relaxed/simple; bh=5IcEfDRaLmDWXRZyvPZum369ppinSxQ+AWlFgZ6M+FM=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=TdUz25LRtVq6VZ59hmwJ5OIOZzTM6xN7Ws4D2P+V85BRIbtuEdiCsOj5K0inLTSkaBc08JDfzu+60YTXP1CcPnCRNZyntMESP7FUUeV1fWsGrCpGfl5bYUw1QZx5+jM2kJ98rZPPzMdsYNoJiisOEGMD2pnVwBxnStJUIDSoP+Y= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=DeyvBk2d DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 71A044B9DB4D ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Dj6Tw0BS9+Id8jtdYZk4XV2hRseEpgzMH+j4zWDC3csyXuoWAI36UM2jlpnPzgTW5bLqz45zjc/V3iEPBAeKmRn1F7bdX6HXhMUXqCLoO3lEQqmRPTZwE2BHxTKSn+jvvdNDENQdeVtgqTsl8IguEs8mvFXON7B17LJ8Lj705DUJZYlXQMP5MsDdkK20ooD6s+jTrid3Q2ItDYne8i/PAYC2y0TUXL0TgPtlgnd7kPN9DF024PXxAdlw/0tjH54BGTF3K1Y1pyzi/di4vWOe1gj5QFJ1+KhDTrsAdpr4DuCGjgAc5gjLkm8Oam4XM0HjyVJ0hhSw6VasudwWSLakdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SCkuTttEC0H51Qo1stzIV3HOzfMCueEt6MEEEB+4eW0=; b=SyAHHRUZ/cwmfd1cBtkkZ6zUmm7pdsc9SfKCiEIlKNjwj7Aq74A+6LosjOU+4gE/1pflWj1WETsO0yecuKmNX022sY1xk+sBc7m2eXvrzju3DYlgmEGI5BHTOBjSRDZsIH3Kc05Khvd9uMex69kIZ0uRUGs+lDBaB5Vfr0mNW1gCTlLe1HYW+XpHIrnl9UYkSRlLt/NbE5339HVpCzVTiI1DQAIH50LD0ebw3HobcEDn54+kTYDbhgJHrwQS9799IO7qCCqyiQmt5T9RFLnhZn3MQFTZ8Nv602FbDy6FAh5EiGjIv7bN1KRr5T9GPZH2fSSW8wgQp/WLzfD6p6QHRA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SCkuTttEC0H51Qo1stzIV3HOzfMCueEt6MEEEB+4eW0=; b=DeyvBk2d6RE8vGuho/tQlNqIvAZzLpxBHHaRa8dq9fN1ahjp2czj4B28hV9AZ6Q0vMMNoPE5ENaw/5sFvu1vXz3v3Si6YCc3sPjkZtNGteXtcDEGm8XhwnL2TneYB9yST+4FPKhQBqihfiILWGKhag/2ghntxPeMbYvdyoujsNBLhsCDrjSd4oy7gEuMv1fC+Dd0MARJs0jO3Pgr/buxuEonbRy8tbh4WN38Mm9lwxUFBfjtBh2SG25OgNHDju1k2RQDz9948MrmiOCWHIHoe8U6BU+lNbd3/Y6j5HarWmeheAoBvZDaVNk5DX8nwbMBAIybWVr0QfWA7T4fvWYTaw== Received: from MN2PR07CA0004.namprd07.prod.outlook.com (2603:10b6:208:1a0::14) by SA1PR12MB5638.namprd12.prod.outlook.com (2603:10b6:806:229::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Wed, 2 Sep 2026 14:53:44 +0000 Received: from BN3PEPF0000B070.namprd21.prod.outlook.com (2603:10b6:208:1a0:cafe::28) by MN2PR07CA0004.outlook.office365.com (2603:10b6:208:1a0::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.12 via Frontend Transport; Wed, 2 Sep 2026 14:53:44 +0000 X-MS-Exchange-Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN3PEPF0000B070.mail.protection.outlook.com (10.167.243.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.1 via Frontend Transport; Wed, 2 Sep 2026 14:53:44 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:16 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:15 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 08/13] match-sat-alu.pd: Recognize strict widened unsigned saturation clamps Date: Wed, 2 Sep 2026 16:52:49 +0200 Message-ID: <20260902145254.77832-10-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B070:EE_|SA1PR12MB5638:EE_ X-MS-Office365-Filtering-Correlation-Id: 75062a39-0496-4b5e-e17e-08df0901fc4d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|36860700016|1800799024|23010399003|376014|18002099003|22082099003|56012099006|10067099003|6133799003|11063799006; X-Microsoft-Antispam-Message-Info: 6hjdHb3+JxQHGPtRHsF80NE5Qg4iZt1Jocu3evFBjJW6c/lAoHI9qDJouS3u0JDqMx7pdVoEVW1ScqMO8yV8SBi2RotTPyeIT3mmn+m8uCUQIRw5JDDKETVa/Ys0UlfxIqcdk9hS6tl5JZe537cOHJKUQl6RQDCHD7glTwxpK642nOfH79YQ8sroe8V8ZU4OFZlLXq5LYwlNQxei4PIv8pDkQQA6kbKmhfl4Q1DbM0WSFwTuMbr3K9NYRFIXqHB88B76+6SgvhLojsckuD3imYhB/IPS1ztRkEy5B33sE7bhu1K2B7VFcSBgqUOw14XK3Mxl3uGrkbtlm02mJOIUf4S5Ga6/8hQqDKfoMbNR9bkfYjxWD2OSijRqNPJGXyk+nw3WbOunkKODqfjSUGsqwohTb046iu/hF9CEErpe1ZYXd3D+CxDzoUUEgNi21wlmfNOXvJoB9JZngaDTYxTqYm/95e18c96+XAxAA9f1HlSHKFy0UUquwW7f9jVUG8thWuYSitRh8KZiFKoCYVgqH/m8neubwvbx4oZtLVfKBkMOJCFEwiDfRihPBOOtPnNLKiNS0/hBzTxSlJ9RqM6MtN5MFSJm/pfQ6ZxkkCC4SD+g4O13gVLHec9DPbbC2L4n/NmRvoXAb/RHkIxtA1YS7FGMmO+apzTCYQdU0Fw1iiHh0xT6cjTCF+R+xvn404ZVfnQiqtoltOjMU3/TxrDEvw== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(82310400026)(36860700016)(1800799024)(23010399003)(376014)(18002099003)(22082099003)(56012099006)(10067099003)(6133799003)(11063799006); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9l3K47n9cTpoAw8LadYlzY9C5Po0wzAYkWOd4vM7Fb1Ukqy+JN4O8M3o6Q++OO2WOhOERFFaV6U+HVam+1XwUkbRkBQSg0rZmyIlVZ6bkWgUPdGIC8VlHbFGGS921JcaS9z0r6UfvXBkzKdnq7Sq/wbLf2JixIkht54owM/g9w9w5SEpZxftWi6bLDD11pzGCqbbAxubu1ZGQifMcadAvH88mEbCyH9/k945O7vHD6C1YtcjCVlId6RXvupG+er2oDxUecXSPJAuC4NcVVdkc/ekuYldjT7wncRGA9eUmq22OvgggYy98XFjb7+rI0CPEHTmUysstwQLJbvkqohlAF8BNr0Cei9f2PjNOdn3BNXXb3cT2Xtj7ejK5NXyPVGfgVLmM0hB1lv7opK3AN9T1l9h1uon7nYI391PySTAsfHIS1MlrO3VI7yb3FdUKGpm X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:44.0563 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 75062a39-0496-4b5e-e17e-08df0901fc4d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B070.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB5638 X-Spam-Status: No, score=-8.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, SPF_HELO_NONE, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov A widened saturating addition can clamp the sum with either of these forms: SUM <= MAX ? SUM : MAX SUM < MAX ? SUM : MAX The strict comparison is canonicalized to SUM <= MAX - 1. It is equivalent because equality makes both arms produce MAX. Accept MAX and MAX - 1 as the comparison limit. AArch64 -O2: before: uxtw x2, w1 add x2, x2, w0, uxtw add w0, w0, w1 mov w1, 4294967294 cmp x2, x1 csinv w0, w0, wzr, ls after: adds w0, w0, w1 csinv w0, w0, wzr, cc Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (unsigned_integer_sat_add): Accept a compare against MAX - 1 in the widened form. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sat_u_add_widen_lt-1.c: New test. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 6 +++- .../gcc.target/aarch64/sat_u_add_widen_lt-1.c | 32 +++++++++++++++++++ 2 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_add_widen_lt-1.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index b01989c3a1c..477ae366661 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -148,8 +148,12 @@ along with GCC; see the file COPYING3. If not see unsigned widen_precision = TYPE_PRECISION (TREE_TYPE (@2)); wide_int max = wi::mask (precision, false, widen_precision); wide_int c4 = wi::to_wide (@4); + /* A source written with < rather than <= puts MAX - 1 in the compare. + The sum equal to MAX then takes the other arm, which is also MAX. */ + wide_int c4_add_1 = wi::add (c4, wi::uhwi (1, widen_precision)); } - (if (wi::eq_p (c4, max) && widen_precision > precision)))))) + (if ((wi::eq_p (c4, max) || wi::eq_p (c4_add_1, max)) + && widen_precision > precision)))))) /* Saturation sub for unsigned integer. */ (if (INTEGRAL_TYPE_P (type) && TYPE_UNSIGNED (type)) diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_add_widen_lt-1.c b/gcc/testsuite/gcc.target/aarch64/sat_u_add_widen_lt-1.c new file mode 100644 index 00000000000..114602e2f8f --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_add_widen_lt-1.c @@ -0,0 +1,32 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-optimized" } */ + +typedef unsigned int u32; +typedef unsigned short u16; +typedef unsigned long long u64; + +/* Written with < rather than <=, so the compare holds MAX - 1. The sum + equal to MAX takes the other arm, which is MAX as well. */ + +u32 +lt32 (u32 a, u32 b) +{ + u64 t = (u64) a + b; + return (u32) (t < 0xffffffffull ? t : 0xffffffffull); +} + +u32 +le32 (u32 a, u32 b) +{ + u64 t = (u64) a + b; + return (u32) (t <= 0xffffffffull ? t : 0xffffffffull); +} + +u16 +lt16 (u16 a, u16 b) +{ + u32 t = (u32) a + b; + return (u16) (t < 0xffffu ? t : 0xffffu); +} + +/* { dg-final { scan-tree-dump-times "\\.SAT_ADD " 3 "optimized" } } */ From patchwork Wed Sep 2 14:52:50 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142730 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A14B34BB3BB2 for ; Wed, 2 Sep 2026 15:01:28 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A14B34BB3BB2 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=dNyvhfUy X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from PH8PR06CU001.outbound.protection.outlook.com (mail-westus3azon11012023.outbound.protection.outlook.com [40.107.209.23]) by sourceware.org (Postfix) with ESMTPS id A1C794B9DB5B for ; Wed, 2 Sep 2026 14:54:00 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A1C794B9DB5B Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org A1C794B9DB5B Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=40.107.209.23 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360840; cv=pass; b=nDQn45Ee7vYfT0Gs2c99SfCmdqET5+yB/0dPcrP56hSSMwwGzaq86yDywE76rIVmr9xhMWIN/3OQA1LeVz5tTM9ht1PUJjzjm8sHAnC3oovOIUwBGCjvoObdvn/1apmnBYvZ6D4zzYcO1Nc4R2EUqQAKZPUugP/p5+uy5vtpQEA= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360840; c=relaxed/simple; bh=sKBTxiUf5Pa2u5DRfP1US+HpxrHlgP8/+++1T2OWj0E=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=MqdKc6x228bD7bo6PA0Gfco7OKHIuDYDmfQZ0rsQlytUrVBWEGsYgaGBktpfPqGFPiOUi/U0JfHcI64V7z+lIdAARydvtVbq521PGODnBAyukkV87Y0iLV1GBPexOiouSwC73B89jibI2Sg/CUH7P4AJR8DoczMxNKyYGpCPOPQ= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=dNyvhfUy DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A1C794B9DB5B ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Uusq8+/zKANbp8sSNd2TAgUOvR61l+xkOlKCax4vhboaEh5V7i3JTU3FgYl8wJyIlr50SH63f5+QnZl2cOMkIJNadJ2F+of7985ztpmCgmh3YkB0TGMU2y+Vnq3eQwbMedxm9rr7KGg5ftuHAqK2bJhaltd9BtFRrTyLNq8+btzLOEfPkYK8YCFtvY13v4D+keA4Klwm5ioVtwUpudBsq/A50Q4nZSfSF5Ypj+5+d+1omxboi4hJjNJoncNFJXqgJemL9bDFc5ku9K0tU6ZFtfmiA4iUALOCTApmiCrI4W7s6n8iu79m+ikz4RgHBeSGfj8CkWjSIBMmL8xBxlFz8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ba6lmilxGmLV3rhPEEwHwXDE2EJI7mBgMfFmO+0gEf8=; b=AduFiH4ZjxWOjoDmJAW9fN+yTD2tcV3z5+e2mv1Jd+XjBEda+Etd8RcekNrXyx0f17jrr8gnZ7QDk95mzjjz5DgHgPqZsrCDcptzaV4diEN/PJPGGZwonhPelwCHIKJn6mJovehGGwOIS6qPGllVnWc4uUoyjPZekfycxZVaQPQ9FdgGnFNRGSfXMEo+q4MzcC+987+Cj0whDOUf8dVB3zsVjKTsZrGZgh4La8x1Z1d4RW6pLyK83QJLFZsIQ/0WuAS9YeiILKmLbgHrPlnRmoBvzz3iPaPx87YgzsaLXYXkrcUBGGdEiy6ZkI+DGBTCZEbvFc/xOS2kihxmHg4uOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ba6lmilxGmLV3rhPEEwHwXDE2EJI7mBgMfFmO+0gEf8=; b=dNyvhfUyM4XvSkhhY6IisJuQLaLMZoPbvM0Xvkg6GiMUsQiAY9FZhYzBC8zvdUr1c7FSm3YK6dBrbQCLHW6+gxWitA0fBhnETXn4nHefK0xcUvJ5U3ujvFtvgypwJyktj1JnaTczH9Bj5vVXL14hR4wsT7/TM9JYaAE1j0/fJD5oZL5V7Rz0kDjKZgB8yQHPDUWc5Aen0vBbBfdRsvZ9aFGOExYx6Tu9Hf/9yDB4dKaafq5e5UfmGOaadJgywCL4p9yhTeG96jpmRHDKQzjGsznboCdKEhRnh2+lY+BO15cD4y4IAEd1woENVcEsczmUDaQIFbhoEiUedJjiIgmd+A== Received: from SA9PR03CA0027.namprd03.prod.outlook.com (2603:10b6:806:20::32) by CH3PR12MB8459.namprd12.prod.outlook.com (2603:10b6:610:139::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Wed, 2 Sep 2026 14:53:46 +0000 Received: from SA2PEPF00001509.namprd04.prod.outlook.com (2603:10b6:806:20:cafe::4f) by SA9PR03CA0027.outlook.office365.com (2603:10b6:806:20::32) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.11 via Frontend Transport; Wed, 2 Sep 2026 14:53:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SA2PEPF00001509.mail.protection.outlook.com (10.167.242.41) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.382.8 via Frontend Transport; Wed, 2 Sep 2026 14:53:46 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:17 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:16 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 09/13] match-sat-alu.pd: Recognize signed saturating truncation after a shift Date: Wed, 2 Sep 2026 16:52:50 +0200 Message-ID: <20260902145254.77832-11-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001509:EE_|CH3PR12MB8459:EE_ X-MS-Office365-Filtering-Correlation-Id: e7295be1-9c66-4908-ba81-08df0901fdaa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|23010399003|1800799024|36860700016|376014|11063799006|10067099003|6133799003|3023799007|56012099006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: 1Nt1eWlj10TiS5tnpEfJ5E2q6igSmw6riExULhPu7z0SMuvDdesz1+isggqyYXhdFsHFjhH9KvxEeIo8Y7R3NYaHFJ1M0YQPFydlSL/8wa4G8rF0VsuNujUyqbcIGvuT6seODDpq/+kHvdVGiftXLvMBF/DffLaM7XliQpLwYYCkEYBzpGKKhO5KX0hzlEMrmWPb2Y5679Iu9YELkZY/2C+WJHwv9T4Ai/ueHgM0iimCO0RJgBeOsYKy6xDAhcssZMXHorHvTOvMxmSkULkblietGz31beK7iLPDoZLvIJxI22FrF6E8D4s7c+Qnl/AUWyS2IwwBk9SD3B0aUgFaRumdkB35BRzncNlQ9/xn8Bwuv88zjXAQZSvTJwnxxKGh5nqtO4Z+rL8c+7v87JL0Mh6jiEPYnqa5Avsrpw7OYx8I63Tc/AcpzTPSj+LehCWGuOpnFqaCm9Hzr+Hi/H9JZXzchAIkj5YVRvBRQ7ouTV6V3vcPJ2YewEJx3sRvZjudcoCcpv4DU/fXXEtTiLSQOIQiLtm3Y+DzxE4pooB7RKnMUIIudOjErObsyC/ePT311jrR11ZTdUTflV/hp2HFtylx1nwswySQ+qr9eKpxYKPBy+xljnA+aK8SratD5L7OoDqXuuh3NK4cIIDQXNbI0Glsi/60LyN+py2mHuGn/Xq7dL+183Q3n9ECh0c22nEaKxXRtyOqcNB+lmvCKolwQA== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(82310400026)(23010399003)(1800799024)(36860700016)(376014)(11063799006)(10067099003)(6133799003)(3023799007)(56012099006)(18002099003)(22082099003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: MLPjW7YQCpDn4mqnrDtwDbq8OkLt4Oi0yAJib4i4NIBVfLt7zV16sE1VRqaxS66zihTkaxsCHehZagTJ5ki2lrW+PCZ5YrP/BoSr3Fx3By3ZakqxqGQO0JbCGbGM+rYVqPQeywK39z8JPGhU95EoeiSgb4OJtRPM9+ZDR3pse4OP76ud9p7uHFlWIYwsMoDkCk2vCSmuF3dclARebamnFq85XBXq/g3lZHmnbvyfFhhApOjvC8xHnSj0VTi9tc+GV8JqqWaio1rtSwoGO9HNWaZZRlj0KjuSrfsEr3ZongTq3ILZJrBtljmCZAtYEIiEEoeaciTCucvVit3WNRpn9GSEgjpRU5Grf3ABFmW33K/0sHTYNlLHgHIy232/8cgmYLKKtOtXV8oHr3Q/+WY3eCGMGwdXB2DaKH7cMhEgxWqZtSxzejFEJOglPu8zeXJo X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:46.3983 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e7295be1-9c66-4908-ba81-08df0901fdaa X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001509.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8459 X-Spam-Status: No, score=-8.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov An arithmetic right shift preserves its input sign. The range test and conversion can therefore use the shifted value while the saturation value uses the original input: shifted = x >> 8; range = (unsigned int) shifted + 32768; out = range > 65535 ? (short int) (x >> 31) ^ 32767 : (short int) shifted; This is equivalent to .SAT_TRUNC (shifted). Recognize a precision-minus-one sign shift and allow the saturation value to use the original input sign. Require the data shift to be a constant smaller than the input precision. Require the range type to be at least as wide as the shifted value and the output type, so the conversion and clamp constants do not lose bits. With AArch64 -O2 -march=armv8-a -fopenmp-simd: before: shifted_clip4: ldp d0, d30, [x1] movi v27.2s, 0x80, lsl 8 movi v25.2s, 0xff, msl 8 mvni v31.4h, 0x80, lsl 8 sshr v29.2s, v0.2s, 8 sshr v28.2s, v30.2s, 8 cmlt v0.2s, v0.2s, #0 cmlt v30.2s, v30.2s, #0 add v26.2s, v29.2s, v27.2s add v27.2s, v28.2s, v27.2s uzp1 v30.4h, v0.4h, v30.4h uzp1 v28.4h, v29.4h, v28.4h cmhi v26.2s, v26.2s, v25.2s cmhi v27.2s, v27.2s, v25.2s eor v31.8b, v30.8b, v31.8b uzp1 v27.4h, v26.4h, v27.4h bit v28.8b, v31.8b, v27.8b str d28, [x0] ret after: shifted_clip4: ldr q31, [x1] sqshrn v31.4h, v31.4s, 8 str d31, [x0] ret The vector sequence becomes one SQSHRN. Runtime tests cover valid and wrapping forms. Target tests cover valid forms, rejected forms, and the related signed saturating addition and subtraction forms. Bootstrapped and tested on aarch64-none-linux-gnu. Tested on x86_64-pc-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (signed_integer_sat_val): Recognize an arithmetic sign shift. (signed_integer_sat_trunc): Use signed_integer_sat_val in the existing form. Recognize a shifted value with an equivalent pre-shift sign source. gcc/testsuite/ChangeLog: * gcc.dg/vect/vect-sat-trunc-shift-sign-source-1.c: New test. * gcc.target/aarch64/sat_s_val_ashr-1.c: Likewise. * gcc.target/aarch64/vect-sat-trunc-shift-sign-source-1.c: Likewise. * gcc.target/aarch64/vect-sat-trunc-shift-sign-source-negative-1.c: Likewise. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 69 +++++- .../vect/vect-sat-trunc-shift-sign-source-1.c | 221 ++++++++++++++++++ .../gcc.target/aarch64/sat_s_val_ashr-1.c | 30 +++ .../vect-sat-trunc-shift-sign-source-1.c | 7 + ...t-sat-trunc-shift-sign-source-negative-1.c | 122 ++++++++++ 5 files changed, 438 insertions(+), 11 deletions(-) create mode 100644 gcc/testsuite/gcc.dg/vect/vect-sat-trunc-shift-sign-source-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_s_val_ashr-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-shift-sign-source-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-shift-sign-source-negative-1.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index 477ae366661..b141c87be03 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -383,7 +383,15 @@ along with GCC; see the file COPYING3. If not see (match (signed_integer_sat_val @0) (bit_xor:c (nop_convert? (negate (nop_convert? (convert (lt @0 integer_zerop))))) - max_value))) + max_value)) + /* SAT_VAL = ((T)(X >> (PRECISION (X) - 1)) ^ MAX) */ + (match (signed_integer_sat_val @0) + (bit_xor:c (convert? (rshift @0 INTEGER_CST@1)) max_value) + (if (!TYPE_UNSIGNED (TREE_TYPE (@0)) + && tree_fits_uhwi_p (@1) + && (tree_to_uhwi (@1) + == (unsigned HOST_WIDE_INT) + TYPE_PRECISION (TREE_TYPE (@0)) - 1))))) /* Saturation add for signed integer. */ (if (INTEGRAL_TYPE_P (type) && !TYPE_UNSIGNED (type)) @@ -603,9 +611,7 @@ along with GCC; see the file COPYING3. If not see (match (signed_integer_sat_trunc @0) /* SAT_S_TRUNC(X) = (unsigned)X + NT_MAX + 1 > Unsigned_MAX ? (NT)X */ (cond^ (gt (plus:c (convert@4 @0) INTEGER_CST@1) INTEGER_CST@2) - (bit_xor:c (nop_convert? - (negate (nop_convert? (convert (lt @0 integer_zerop))))) - INTEGER_CST@3) + (signed_integer_sat_val @0) (convert @0)) /* The comparison type has to be unsigned and at least as wide as X. A narrower type would examine only the low bits of X. The result @@ -623,17 +629,58 @@ along with GCC; see the file COPYING3. If not see wide_int limit_0 = wi::mask (otype_prec, false, range_prec); // Aka 255 wide_int limit_1 = limit_0 - 2; // Aka 253 wide_int limit_2 = limit_0 - 1; // Aka 254 - wide_int otype_max = wi::mask (otype_prec - 1, false, otype_prec); - wide_int itype_max = wi::mask (otype_prec - 1, false, range_prec); + wide_int otype_max_in_range + = wi::mask (otype_prec - 1, false, range_prec); wide_int int_cst_1 = wi::to_wide (@1); wide_int int_cst_2 = wi::to_wide (@2); - wide_int int_cst_3 = wi::to_wide (@3); } - (if (((wi::eq_p (int_cst_1, offset) && wi::eq_p (int_cst_2, limit_0)) - || (wi::eq_p (int_cst_1, itype_max) && wi::eq_p (int_cst_2, limit_2)) + (if ((wi::eq_p (int_cst_1, offset) + && wi::eq_p (int_cst_2, limit_0)) + || (wi::eq_p (int_cst_1, otype_max_in_range) + && wi::eq_p (int_cst_2, limit_2)) || (wi::eq_p (int_cst_1, offset) && wi::eq_p (int_cst_2, limit_2)) - || (wi::eq_p (int_cst_1, itype_max) && wi::eq_p (int_cst_2, limit_1))) - && wi::eq_p (int_cst_3, otype_max))))))) + || (wi::eq_p (int_cst_1, otype_max_in_range) + && wi::eq_p (int_cst_2, limit_1)))))))) + +/* An arithmetic right shift preserves the sign of its signed input. The + saturation value can therefore use the input sign while the range check + and truncated result use the shifted value. */ +(if (INTEGRAL_TYPE_P (type) && !TYPE_UNSIGNED (type)) + (match (signed_integer_sat_trunc @0) + (cond^ (gt (plus:c (convert@4 + (rshift@0 @5 INTEGER_CST@6)) INTEGER_CST@1) + INTEGER_CST@2) + (signed_integer_sat_val @5) + (convert @0)) + (if (!TYPE_UNSIGNED (TREE_TYPE (@0)) + && TYPE_UNSIGNED (TREE_TYPE (@4)) + && tree_fits_uhwi_p (@6) + && TYPE_PRECISION (type) < TYPE_PRECISION (TREE_TYPE (@0)) + && tree_to_uhwi (@6) < TYPE_PRECISION (TREE_TYPE (@0)) + && TYPE_PRECISION (TREE_TYPE (@4)) >= TYPE_PRECISION (type) + && (TYPE_PRECISION (TREE_TYPE (@4)) + >= TYPE_PRECISION (TREE_TYPE (@0)) - tree_to_uhwi (@6))) + (with + { + unsigned otype_prec = TYPE_PRECISION (type); + unsigned range_prec = TYPE_PRECISION (TREE_TYPE (@4)); + wide_int offset = wi::set_bit_in_zero (otype_prec - 1, range_prec); + wide_int limit_0 = wi::mask (otype_prec, false, range_prec); // Aka 255 + wide_int limit_1 = limit_0 - 2; // Aka 253 + wide_int limit_2 = limit_0 - 1; // Aka 254 + wide_int otype_max_in_range + = wi::mask (otype_prec - 1, false, range_prec); + wide_int int_cst_1 = wi::to_wide (@1); + wide_int int_cst_2 = wi::to_wide (@2); + } + (if ((wi::eq_p (int_cst_1, offset) + && wi::eq_p (int_cst_2, limit_0)) + || (wi::eq_p (int_cst_1, otype_max_in_range) + && wi::eq_p (int_cst_2, limit_2)) + || (wi::eq_p (int_cst_1, offset) + && wi::eq_p (int_cst_2, limit_2)) + || (wi::eq_p (int_cst_1, otype_max_in_range) + && wi::eq_p (int_cst_2, limit_1)))))))) /* Saturation mult for unsigned integer. */ (if (INTEGRAL_TYPE_P (type) && TYPE_UNSIGNED (type)) diff --git a/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-shift-sign-source-1.c b/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-shift-sign-source-1.c new file mode 100644 index 00000000000..f68e503b6c0 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-shift-sign-source-1.c @@ -0,0 +1,221 @@ +/* { dg-do run { target bitint } } */ +/* { dg-additional-options "-O3 -fvect-cost-model=unlimited" } */ + +typedef __INT8_TYPE__ int8_t; +typedef __INT16_TYPE__ int16_t; +typedef __INT32_TYPE__ int32_t; +typedef __INT64_TYPE__ int64_t; +typedef __UINT32_TYPE__ uint32_t; +typedef __UINT64_TYPE__ uint64_t; +typedef signed _BitInt(16) int16b_t; +typedef unsigned _BitInt(12) uint12b_t; +typedef unsigned _BitInt(16) uint16b_t; +typedef unsigned _BitInt(17) uint17b_t; +typedef unsigned _BitInt(24) uint24b_t; + +#define N 259 + +#define SHIFT_SAT_VALUE(OUT, X, SHIFT, MAX) \ + ((OUT) ((OUT) ((X) >> (SHIFT)) ^ (OUT) (MAX))) +#define COMPARE_SAT_VALUE(OUT, X, SHIFT, MAX) \ + ((OUT) (-((OUT) ((X) < 0)) ^ (OUT) (MAX))) + +#define DEF_CLIP(NAME, SAT, OUT, IN, UIN, SHIFT, SIGN_SHIFT, OFFSET, LIMIT, \ + MAX) \ + __attribute__((noipa)) \ + static void \ + NAME (OUT *__restrict out, const IN *__restrict in, int n) \ + { \ + for (int i = 0; i < n; ++i) \ + { \ + IN source = in[i]; \ + IN shifted = source >> SHIFT; \ + UIN range = (UIN) shifted + (UIN) OFFSET; \ + out[i] = (range > (UIN) LIMIT \ + ? SAT (OUT, source, SIGN_SHIFT, MAX) : (OUT) shifted); \ + } \ + } + +#define DEF_REF(NAME, OUT, IN, SHIFT, MIN, MAX) \ + __attribute__((noipa, optimize ("O0"))) \ + static void \ + NAME (OUT *out, const IN *in, int n) \ + { \ + for (int i = 0; i < n; ++i) \ + { \ + IN shifted = in[i] >> SHIFT; \ + if (shifted < (IN) MIN) \ + out[i] = (OUT) MIN; \ + else if (shifted > (IN) MAX) \ + out[i] = (OUT) MAX; \ + else \ + out[i] = (OUT) shifted; \ + } \ + } + +DEF_CLIP (clip_s16_s8, SHIFT_SAT_VALUE, int8_t, int16b_t, uint16b_t, + 3, 15, 128, 255, 127) +DEF_CLIP (clip_s32_s16, COMPARE_SAT_VALUE, int16_t, int32_t, uint32_t, + 8, 31, 32768U, 65535U, 32767) +DEF_CLIP (clip_s64_s32, SHIFT_SAT_VALUE, int32_t, int64_t, uint64_t, + 13, 63, 2147483648ULL, 4294967295ULL, 2147483647) +DEF_CLIP (clip_s16_s8_wide_unshifted, SHIFT_SAT_VALUE, int8_t, int16b_t, + uint32_t, 0, 15, 128, 255, 127) +DEF_CLIP (clip_s16_s8_wide_shifted, SHIFT_SAT_VALUE, int8_t, int16b_t, + uint32_t, 3, 15, 128, 255, 127) +DEF_CLIP (clip_s32_s16_reduced_range, SHIFT_SAT_VALUE, int16_t, int32_t, + uint24b_t, 8, 31, 32768, 65535, 32767) + +DEF_REF (ref_s16_s8, int8_t, int16b_t, 3, -128, 127) +DEF_REF (ref_s32_s16, int16_t, int32_t, 8, -32768, 32767) +DEF_REF (ref_s64_s32, int32_t, int64_t, 13, + -2147483647 - 1, 2147483647) +DEF_REF (ref_s16_s8_wide_unshifted, int8_t, int16b_t, 0, -128, 127) +DEF_REF (ref_s16_s8_wide_shifted, int8_t, int16b_t, 3, -128, 127) +DEF_REF (ref_s32_s16_reduced_range, int16_t, int32_t, 8, -32768, 32767) + +/* This range calculation can wrap a shifted value that still fits in the + shifted input type. */ + +__attribute__((noipa)) +static void +narrow_range (int16_t *__restrict out, const int32_t *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + int32_t shifted = source >> 1; + uint17b_t range = (uint17b_t) shifted + (uint17b_t) 32768; + out[i] = (range > (uint17b_t) 65535 + ? (int16_t) (source >> 31) ^ 32767 : (int16_t) shifted); + } +} + +__attribute__((noipa, optimize ("O0"))) +static void +ref_narrow_range (int16_t *out, const int32_t *in, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + int32_t shifted = source >> 1; + uint17b_t range = (uint17b_t) shifted + (uint17b_t) 32768; + out[i] = (range > (uint17b_t) 65535 + ? (int16_t) (source >> 31) ^ 32767 : (int16_t) shifted); + } +} + +/* These wrapped constants do not describe a saturating truncation. */ + +__attribute__((noipa)) +static void +wrapped_output_range (int16_t *__restrict out, const int32_t *__restrict in, + int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + int32_t shifted = source >> 20; + uint12b_t range = (uint12b_t) shifted + (uint12b_t) 32767; + out[i] = (range > (uint12b_t) 65534 + ? (int16_t) (source >> 31) ^ 32767 : (int16_t) shifted); + } +} + +__attribute__((noipa, optimize ("O0"))) +static void +ref_wrapped_output_range (int16_t *out, const int32_t *in, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + int32_t shifted = source >> 20; + uint12b_t range = (uint12b_t) shifted + (uint12b_t) 32767; + out[i] = (range > (uint12b_t) 65534 + ? (int16_t) (source >> 31) ^ 32767 : (int16_t) shifted); + } +} + +typedef void (*clip_fn) (int16_t *, const int32_t *, int); + +static void +check_invalid (clip_fn fn, clip_fn ref_fn, int32_t special) +{ + int16_t out[N]; + int16_t ref[N]; + int32_t in[N]; + + for (int i = 0; i < N; ++i) + in[i] = (int32_t) ((uint32_t) i * 2654435761U + 1013904223U); + in[0] = special; + + for (int n = 0; n <= N; ++n) + { + for (int i = 0; i < N; ++i) + out[i] = ref[i] = 23; + fn (out, in, n); + ref_fn (ref, in, n); + for (int i = 0; i < N; ++i) + if (out[i] != ref[i]) + __builtin_abort (); + } +} + +#define CHECK(NAME, REF, OUT, IN, INIT, SHIFT, MIN, MAX) \ + do \ + { \ + OUT out[N]; \ + OUT ref[N]; \ + IN in[N]; \ + IN scale = (IN) 1 << (SHIFT); \ + for (int i = 0; i < N; ++i) \ + in[i] = (IN) (INIT); \ + in[0] = ((IN) (MIN) - 1) * scale; \ + in[1] = (IN) (MIN) * scale; \ + in[2] = (IN) -1 * scale; \ + in[3] = 0; \ + in[4] = (IN) (MAX) * scale; \ + in[5] = ((IN) (MAX) + 1) * scale; \ + for (int n = 0; n <= N; ++n) \ + { \ + for (int i = 0; i < N; ++i) \ + out[i] = ref[i] = (OUT) 23; \ + NAME (out, in, n); \ + REF (ref, in, n); \ + for (int i = 0; i < N; ++i) \ + if (out[i] != ref[i]) \ + __builtin_abort (); \ + } \ + } \ + while (0) + +int +main (void) +{ + CHECK (clip_s16_s8, ref_s16_s8, int8_t, int16b_t, + (uint16b_t) i * (uint16b_t) 40503 + (uint16b_t) 97, + 3, -128, 127); + CHECK (clip_s32_s16, ref_s32_s16, int16_t, int32_t, + (uint32_t) i * 2654435761U + 1013904223U, + 8, -32768, 32767); + CHECK (clip_s64_s32, ref_s64_s32, int32_t, int64_t, + (uint64_t) i * 11400714819323198485ULL + + 13787848793156543929ULL, + 13, -2147483647 - 1, 2147483647); + CHECK (clip_s16_s8_wide_unshifted, ref_s16_s8_wide_unshifted, + int8_t, int16b_t, + (uint16b_t) i * (uint16b_t) 40503 + (uint16b_t) 97, + 0, -128, 127); + CHECK (clip_s16_s8_wide_shifted, ref_s16_s8_wide_shifted, + int8_t, int16b_t, + (uint16b_t) i * (uint16b_t) 40503 + (uint16b_t) 97, + 3, -128, 127); + CHECK (clip_s32_s16_reduced_range, ref_s32_s16_reduced_range, + int16_t, int32_t, + (uint32_t) i * 2654435761U + 1013904223U, + 8, -32768, 32767); + check_invalid (narrow_range, ref_narrow_range, 196608); + check_invalid (wrapped_output_range, ref_wrapped_output_range, 0); + return 0; +} + diff --git a/gcc/testsuite/gcc.target/aarch64/sat_s_val_ashr-1.c b/gcc/testsuite/gcc.target/aarch64/sat_s_val_ashr-1.c new file mode 100644 index 00000000000..62ce0378d0d --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_s_val_ashr-1.c @@ -0,0 +1,30 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-optimized" } */ + +#include + +/* Write the saturation value of a signed overflow as + (X >> (PREC - 1)) ^ MAX. */ + +#define DEF(N, T, MX, SH) \ + T f1_##N (T x, T y) \ + { \ + T s = (T) ((uint64_t) x + (uint64_t) y); \ + return ((x ^ s) & ~(x ^ y)) < 0 ? ((x >> SH) ^ MX) : s; \ + } \ + T f2_##N (T x, T y) \ + { \ + T r; \ + return __builtin_add_overflow (x, y, &r) ? ((x >> SH) ^ MX) : r; \ + } \ + T f3_##N (T x, T y) \ + { \ + T r; \ + return __builtin_sub_overflow (x, y, &r) ? ((x >> SH) ^ MX) : r; \ + } + +DEF (32, int32_t, INT32_MAX, 31) +DEF (64, int64_t, INT64_MAX, 63) + +/* { dg-final { scan-tree-dump-times "\\.SAT_ADD " 4 "optimized" } } */ +/* { dg-final { scan-tree-dump-times "\\.SAT_SUB " 2 "optimized" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-shift-sign-source-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-shift-sign-source-1.c new file mode 100644 index 00000000000..b3256db7b6e --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-shift-sign-source-1.c @@ -0,0 +1,7 @@ +/* { dg-do compile { target bitint } } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +#include "../../gcc.dg/vect/vect-sat-trunc-shift-sign-source-1.c" + +/* { dg-final { scan-tree-dump-times "sat_trunc pattern recognized" 6 "vect" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-shift-sign-source-negative-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-shift-sign-source-negative-1.c new file mode 100644 index 00000000000..25925c96759 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-shift-sign-source-negative-1.c @@ -0,0 +1,122 @@ +/* { dg-do compile { target bitint } } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +typedef __INT16_TYPE__ int16_t; +typedef __INT32_TYPE__ int32_t; +typedef __UINT16_TYPE__ uint16_t; +typedef __UINT32_TYPE__ uint32_t; +typedef unsigned _BitInt(12) uint12b_t; +typedef unsigned _BitInt(17) uint17b_t; + +__attribute__((noipa)) +void +different_source (int16_t *__restrict out, const int32_t *__restrict in, + const int32_t *__restrict signs, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + int32_t shifted = source >> 8; + uint32_t range = (uint32_t) shifted + 32768U; + int32_t sign_source = signs[i]; + out[i] = (range > 65535U + ? (int16_t) (sign_source >> 31) ^ 32767 + : (int16_t) shifted); + } +} + +__attribute__((noipa)) +void +logical_shift (int16_t *__restrict out, const uint32_t *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + uint32_t source = in[i]; + uint32_t shifted = source >> 8; + uint32_t range = shifted + 32768U; + out[i] = (range > 65535U + ? (int16_t) ((int32_t) source >> 31) ^ 32767 + : (int16_t) shifted); + } +} + +__attribute__((noipa)) +void +variable_count (int16_t *__restrict out, const int32_t *__restrict in, + const uint16_t *__restrict counts, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + int32_t shifted = source >> (counts[i] & 31); + uint32_t range = (uint32_t) shifted + 32768U; + out[i] = (range > 65535U + ? (int16_t) (source >> 31) ^ 32767 + : (int16_t) shifted); + } +} + +__attribute__((noipa)) +void +wrong_sign_count (int16_t *__restrict out, const int32_t *__restrict in, + int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + int32_t shifted = source >> 8; + uint32_t range = (uint32_t) shifted + 32768U; + out[i] = (range > 65535U + ? (int16_t) (source >> 30) ^ 32767 + : (int16_t) shifted); + } +} + +__attribute__((noipa)) +void +wrong_range (int16_t *__restrict out, const int32_t *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + int32_t shifted = source >> 8; + uint32_t range = (uint32_t) shifted + 32767U; + out[i] = (range > 65535U + ? (int16_t) (source >> 31) ^ 32767 + : (int16_t) shifted); + } +} + +__attribute__((noipa)) +void +narrow_range (int16_t *__restrict out, const int32_t *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + int32_t shifted = source >> 1; + uint17b_t range = (uint17b_t) shifted + (uint17b_t) 32768; + out[i] = (range > (uint17b_t) 65535 + ? (int16_t) (source >> 31) ^ 32767 + : (int16_t) shifted); + } +} + +__attribute__((noipa)) +void +wrapped_output_range (int16_t *__restrict out, + const int32_t *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + int32_t source = in[i]; + int32_t shifted = source >> 20; + uint12b_t range = (uint12b_t) shifted + (uint12b_t) 32767; + out[i] = (range > (uint12b_t) 65534 + ? (int16_t) (source >> 31) ^ 32767 + : (int16_t) shifted); + } +} + +/* { dg-final { scan-tree-dump-not "sat_trunc pattern recognized" "vect" } } */ From patchwork Wed Sep 2 14:52:51 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142722 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id DC56A4B9DB74 for ; Wed, 2 Sep 2026 14:56:38 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org DC56A4B9DB74 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=UBEtGePQ X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from BYAPR05CU005.outbound.protection.outlook.com (mail-westusazlp170100001.outbound.protection.outlook.com [IPv6:2a01:111:f403:c000::1]) by sourceware.org (Postfix) with ESMTPS id 66FE54B9DB61 for ; Wed, 2 Sep 2026 14:54:04 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 66FE54B9DB61 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 66FE54B9DB61 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c000::1 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360844; cv=pass; b=cam/pna3dQ/VsZemhK5oiKMxRSwA/mLviePTAnXMyof+vQyOUyLfLMk/dkhxMYo6jg/zxb+/HAc1A0w5ku0za8LXQeMV6QSpcxWUYd7K52yhvf2NqC9LnGvnPCZsbUXlTUUJdq6cmREDTekLX+9FmotIZ6k6OsZRstg20SNksFo= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360844; c=relaxed/simple; bh=s5HkNDw93saxsZisEEddeTaNVkd1Kal3ObtMcZSlppM=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=r/HPSJKpSHhieiUQF6hsLWDVXgmlCbITGit4Y23DDHkfwVXd4L4LnJiywmnpKWJYRC07qdUTd8dFEXw/xSCxpHqLBgP2WYw7Ys9g2dVBMKgpYEG44ubmzAcXhyiinZlfGbGPZNlXWKHHBfTYFqevoJPK2HcVzeglsp8zmNDjIRc= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=UBEtGePQ DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 66FE54B9DB61 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NhAFXhOglKgGp2wKD30Qg3gAMreun2TZoaBlBvSVnb31ItmKu1Yo61O3hnVKKb4nCHezpwEFYhDl8DVm0+KyBHHoeGhueiGkDfx7VQsr2nJ0Tidh6GxFncnbi5pszn0uLQ8KSJb+PoxVq2Ymvk0a2VHE9xoTFmI2bup04AT9Kt+O1OHMfTIb/FgEPUM2nqxJwqyNoJuvSwL31Buf/31W42d0yZFSy82x+dyDqAHiFnlGGMc9UBWsorBEMxGyHEfVhx9G6DhpyehbiXRSQRN1J/R/gG3GQXM2It1lK05AMBNuzvEffPcZ7ymHCUiv75sDHpfkYHd18GCfXMshC34B/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fY/Q+Tv0+tZsjXmNO1W1Dyxjda2ogIrXvmRtiRGmQoE=; b=qDx9r1sdQGgIPF9GC4/5UT6+nkYh0xbi/P/ZaUofmBZmIBbBsx3e97rPDkByR9obESCSSu5WbYm8r4AP1ycrpiEbOdBkeTcmMkWXj7JbvEaA673eWXARKj9jS/rMnLCT9P8dv3mQCS4v6hKZqAw2410Ixf5OGtl5AREUdQUvpt1+HDbnU3kE+FZOUZFPlp8mTzeTrLgPm0wXz8oxJ9JRp4ATNs2iU/yp8arwTrA1c/sTcSGZVTzXSb5g/5+536LYm937lZ66zjLeHqlhgsy+H3RFqzI/i2Xl93FpvSsIJjwGbUScyu029+/AekoPqvWDrQfZ0EXdM6+yZt1eDUEMpQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fY/Q+Tv0+tZsjXmNO1W1Dyxjda2ogIrXvmRtiRGmQoE=; b=UBEtGePQvR+XU+lJssOPtZ6zF8Y/6RIB/6uP0SqXByn1npqkENet80ZSSbDB/SCP/OkXuDk2tlrR2SxHoESG8QzSLDVZlbLxeONeD+SXnDyRe0rfw35lPjyNabbsxGsqXZWJAq7Wi6DWeeoTpJFCcXIIu4aSk3MYEj5JhR9PAuQwq0E8YTwxTWwe3bpsnZ5tIlxFNjNr8lXG73ijdpWFOFiWSEtG3zUGVghJlhjenrdFXLQuF5XXvHdFXA6issd5tAm7HFYtnQiLMGOziEwSnbN9k1YprzmDzv0D4AOzTWGxkssxaTHUPlMWonsZrUnpo8y7sU0nHHfxnaeV0AhQWw== Received: from BN9PR03CA0430.namprd03.prod.outlook.com (2603:10b6:408:113::15) by DS7PR12MB8420.namprd12.prod.outlook.com (2603:10b6:8:e9::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Wed, 2 Sep 2026 14:53:47 +0000 Received: from BN3PEPF0000B06E.namprd21.prod.outlook.com (2603:10b6:408:113:cafe::10) by BN9PR03CA0430.outlook.office365.com (2603:10b6:408:113::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.10 via Frontend Transport; Wed, 2 Sep 2026 14:53:47 +0000 X-MS-Exchange-Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN3PEPF0000B06E.mail.protection.outlook.com (10.167.243.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.1 via Frontend Transport; Wed, 2 Sep 2026 14:53:47 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:19 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:17 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 10/13] match-sat-alu.pd: Recognize unsigned constant-clamp truncation Date: Wed, 2 Sep 2026 16:52:51 +0200 Message-ID: <20260902145254.77832-12-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B06E:EE_|DS7PR12MB8420:EE_ X-MS-Office365-Filtering-Correlation-Id: 67f547d5-e766-419d-77ad-08df0901fe41 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700016|1800799024|82310400026|376014|23010399003|10067099003|5023799004|11063799006|56012099006|22082099003|6133799003|18002099003|13003099007; X-Microsoft-Antispam-Message-Info: 9eswgxsqFF0un2Hj+r98mVZFGevObfhEIVtVb9NtWv0fPPdve0RxZJXEgons9IuGnaeol8mrIWyh2uxVgMEmMFG6zdaewxycF8ZbKnSNmOqImDshTG50JrPIbdwztQK4yoap78c+1L6SPbdXRWH7pgbvhil3S0312JMcfySYHgUDrb+pIsL1denzxLsjQ7AUDof9LzF6FXgLKbznYESaIINh5mly/06pwIV1mcCXRqdXso1M6G7QxzFZJgOCow9iBQRTrQEii4ErjdVtKxxi+IR92GCGisvVZK0w5R5aRHHHTf3HvFwuAn8IjmwGlyBld8ReDK9MijC3S0JPabjiSUjljm7IveimGGOyXs0tx8ObNbPC4z5eEfl6QMXlK9mdWi9nGmF9wG692ZuwWJTh9XUFVNp7CLwsyK5KbBtsojGdZpaKNylfSh6imbmQNl6DWf7gDIOfP+SIkUARCao9Svg/opeBwTv+s97VApmDJYNyFo/MEANJMAPvXyxDC8zejLd5c70FjrrChUBSmPqGhBxQpLgumhRn7VUnwgl9+4Jc9ezwD7FEf059UIGfefcrLcr/7PXTFkhzGmBU2wuKqGvUhz2z4hp9UXzq2Ycv2sDCXxsJ5mF4G1h425hl7Qmw+mp6gW9c1SLepGSEpDyxe0Q029zu7gl5HTJQYOieLzT2dhEgPGXeDSww3cKtQdSQUH+ow+kpgqIe775UMoBZSg== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(36860700016)(1800799024)(82310400026)(376014)(23010399003)(10067099003)(5023799004)(11063799006)(56012099006)(22082099003)(6133799003)(18002099003)(13003099007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2YwJTejS9VS6/FqQESARxKbigAu8LOR2wGSNEteKSiJzSTGdmXBsP0P69K+jE06fu4JvbpkVrZiM/3xbSIaP5RFhrMhN/897hbL3FqwX5HNIqBS6T1veG0lxxsdVPjPPz9eIV9O/OfNLjQelh06sPSi6Aym/GWJ1q2gHA1bYm5H09yP9CaOvKKsWSswL3tj/Vrhu7VXODabymJqUzG8tsb2v3d1q0lxRF9ucEaJOhL3yhY3/TWo0+42/uSvezS/MxEIgX0AOBRJ3GWU/Zih6TYxfvWi39pX8Pv4mf5jr1SpAWxn1hCec4KkvXuCWTrmOj784IkWKJ6ENVxRo8HKbmfAv1MeRoxuhCYZc4Sn0e6jXm6vLs5vFvNfQzcVhTw3/PhE9gAtWlkpzgKPm9ScQBuQT05Ygg2Pg15foe9ed2/DrcmeQbe3ClsDGw65taWcQ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:47.3433 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 67f547d5-e766-419d-77ad-08df0901fe41 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B06E.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB8420 X-Spam-Status: No, score=-8.1 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, PROLO_LEO1, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov A signed value clamped to the range of a narrower unsigned type is unsigned saturating truncation. Recognize nested MIN and MAX operations in either order. Require exact zero and maximum bounds. Require the outer clamp to have one use so that the replacement removes it. The inner clamp can have other uses. With AArch64 -O2 -march=armv8-a -fopenmp-simd: before: clamp_u16_4: ldr q0, [x1] movi v30.4s, 0xff, msl 8 movi v31.4s, 0 smin v30.4s, v0.4s, v30.4s smax v30.4s, v30.4s, v31.4s xtn v30.4h, v30.4s str d30, [x0] ret after: clamp_u16_4: ldr q31, [x1] movi v30.4s, 0 smax v30.4s, v31.4s, v30.4s uqxtn v30.4h, v30.4s str d30, [x0] ret Runtime tests cover both clamp orders, two widths, and LTO. Target tests cover valid bounds, invalid bounds, and extra uses of each clamp. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (integer_constant_clamp): New match. (unsigned_integer_narrow_clip): Add the constant-clamp form. gcc/testsuite/ChangeLog: * gcc.dg/vect/vect-sat-trunc-clamp-1.c: New test. * gcc.target/aarch64/vect-sat-trunc-clamp-inner-use-1.c: Likewise. * gcc.target/aarch64/vect-sat-trunc-clamp-multi-use-1.c: Likewise. * gcc.target/aarch64/vect-sat-trunc-clamp-u-1.c: Likewise. * gcc.target/aarch64/vect-sat-trunc-clamp-u-negative-1.c: Likewise. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 27 +++- .../gcc.dg/vect/vect-sat-trunc-clamp-1.c | 121 ++++++++++++++++++ .../vect-sat-trunc-clamp-inner-use-1.c | 22 ++++ .../vect-sat-trunc-clamp-multi-use-1.c | 21 +++ .../aarch64/vect-sat-trunc-clamp-u-1.c | 7 + .../vect-sat-trunc-clamp-u-negative-1.c | 30 +++++ 6 files changed, 227 insertions(+), 1 deletion(-) create mode 100644 gcc/testsuite/gcc.dg/vect/vect-sat-trunc-clamp-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-inner-use-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-multi-use-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-u-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-u-negative-1.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index b141c87be03..ede949a34fe 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -23,6 +23,16 @@ along with GCC; see the file COPYING3. If not see . */ +/* Clamp @0 between the constant bounds @1 and @2. The source can put the + MIN or the MAX first. The outer operation must be single use so that + replacing its conversion also removes the complete clamp. */ +(match (integer_constant_clamp @0 @1 @2) + (max (min @0 INTEGER_CST@2) INTEGER_CST@1) + (if (single_use (t)))) +(match (integer_constant_clamp @0 @1 @2) + (min (max @0 INTEGER_CST@1) INTEGER_CST@2) + (if (single_use (t)))) + /* Saturation add for unsigned integer. */ (if (INTEGRAL_TYPE_P (type) && TYPE_UNSIGNED (type)) (match (usadd_overflow_mask @0 @1) @@ -323,7 +333,22 @@ along with GCC; see the file COPYING3. If not see && wi::eq_p (trunc_max, int_cst_1) && cmp >= 0 && expr_not_equal_to (@0, wi::min_value (itype_precision, - SIGNED)))))))) + SIGNED))))))) + (match (unsigned_integer_narrow_clip @0) + /* SAT_US_TRUNC = (NT)MAX (MIN (X, NT_MAX), 0). */ + (convert (integer_constant_clamp @0 @1 @2)) + (if (!TYPE_UNSIGNED (TREE_TYPE (@0))) + (with + { + unsigned itype_precision = TYPE_PRECISION (TREE_TYPE (@0)); + unsigned otype_precision = TYPE_PRECISION (type); + wide_int trunc_max = wi::mask (otype_precision, false, itype_precision); + wide_int lo_cst = wi::to_wide (@1, itype_precision); + wide_int hi_cst = wi::to_wide (@2, itype_precision); + } + (if (otype_precision < itype_precision + && wi::eq_p (lo_cst, 0) + && wi::eq_p (hi_cst, trunc_max))))))) /* Saturation truncate for unsigned integer. */ (if (INTEGRAL_TYPE_P (type) && TYPE_UNSIGNED (type)) diff --git a/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-clamp-1.c b/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-clamp-1.c new file mode 100644 index 00000000000..06a4428d0d2 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-clamp-1.c @@ -0,0 +1,121 @@ +/* { dg-do run } */ +/* { dg-require-effective-target vect_int } */ +/* { dg-additional-options "-O3 -fvect-cost-model=unlimited" } */ + +#include "tree-vect.h" + +typedef __UINT16_TYPE__ u16; +typedef __UINT32_TYPE__ u32; +typedef __UINT64_TYPE__ u64; +typedef __INT32_TYPE__ i32; +typedef __INT64_TYPE__ i64; + +#define N 259 + +static i32 in32[N]; +static i64 in64[N]; +static u16 out16[N]; +static u32 out32[N]; + +__attribute__((noipa)) static void +clamp_u16_lo (u16 *__restrict out, const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + out[i] = x < 0 ? 0 : (x > 65535 ? 65535 : x); + } +} + +__attribute__((noipa)) static void +clamp_u16_hi (u16 *__restrict out, const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + out[i] = x > 65535 ? 65535 : (x < 0 ? 0 : x); + } +} + +__attribute__((noipa)) static void +clamp_u32_lo (u32 *__restrict out, const i64 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i64 x = in[i]; + out[i] = x < 0 ? 0 : (x > 4294967295LL ? 4294967295LL : x); + } +} + +__attribute__((noipa)) static void +clamp_u32_hi (u32 *__restrict out, const i64 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i64 x = in[i]; + out[i] = x > 4294967295LL ? 4294967295LL : (x < 0 ? 0 : x); + } +} + +static u16 +ref_u16 (i32 x) +{ + return x < 0 ? 0 : (x > 65535 ? 65535 : x); +} + +static u32 +ref_u32 (i64 x) +{ + return x < 0 ? 0 : (x > 4294967295LL ? 4294967295LL : x); +} + +static void +check_u16 (void) +{ +#pragma GCC novector + for (int i = 0; i < N; ++i) + if (out16[i] != ref_u16 (in32[i])) + __builtin_abort (); +} + +static void +check_u32 (void) +{ +#pragma GCC novector + for (int i = 0; i < N; ++i) + if (out32[i] != ref_u32 (in64[i])) + __builtin_abort (); +} + +int +main (void) +{ + check_vect (); + + for (int i = 0; i < N; ++i) + { + in32[i] = (i32) ((u32) i * 2654435761U + 1013904223U); + in64[i] = (i64) ((u64) i * 11400714819323198485ULL + + 13787848793156543929ULL); + } + + in32[0] = -1; + in32[1] = 0; + in32[2] = 65535; + in32[3] = 65536; + in64[0] = -1; + in64[1] = 0; + in64[2] = 4294967295LL; + in64[3] = 4294967296LL; + + clamp_u16_lo (out16, in32, N); + check_u16 (); + clamp_u16_hi (out16, in32, N); + check_u16 (); + clamp_u32_lo (out32, in64, N); + check_u32 (); + clamp_u32_hi (out32, in64, N); + check_u32 (); + + return 0; +} diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-inner-use-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-inner-use-1.c new file mode 100644 index 00000000000..b93bc56188e --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-inner-use-1.c @@ -0,0 +1,22 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +typedef __UINT16_TYPE__ u16; +typedef __INT32_TYPE__ i32; + +void +clip (u16 *__restrict out, i32 *__restrict copy, + const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + i32 inner = x > 65535 ? 65535 : x; + i32 outer = inner < 0 ? 0 : inner; + out[i] = outer; + copy[i] = inner; + } +} + +/* { dg-final { scan-tree-dump-times "sat_trunc pattern recognized" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-multi-use-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-multi-use-1.c new file mode 100644 index 00000000000..0b82d40ff2f --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-multi-use-1.c @@ -0,0 +1,21 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +typedef __UINT16_TYPE__ u16; +typedef __INT32_TYPE__ i32; + +void +clip (u16 *__restrict out, i32 *__restrict copy, + const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + i32 clamped = x < 0 ? 0 : (x > 65535 ? 65535 : x); + out[i] = clamped; + copy[i] = clamped; + } +} + +/* { dg-final { scan-tree-dump-not "sat_trunc pattern recognized" "vect" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-u-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-u-1.c new file mode 100644 index 00000000000..f159a8f4dac --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-u-1.c @@ -0,0 +1,7 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +#include "../../gcc.dg/vect/vect-sat-trunc-clamp-1.c" + +/* { dg-final { scan-tree-dump-times "sat_trunc pattern recognized" 4 "vect" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-u-negative-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-u-negative-1.c new file mode 100644 index 00000000000..aa283f176ad --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-u-negative-1.c @@ -0,0 +1,30 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +typedef __UINT16_TYPE__ u16; +typedef __INT32_TYPE__ i32; + +__attribute__((noipa)) +void +bad_low (u16 *__restrict out, const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + out[i] = x < 1 ? 1 : (x > 65535 ? 65535 : x); + } +} + +__attribute__((noipa)) +void +bad_high (u16 *__restrict out, const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + out[i] = x > 65534 ? 65534 : (x < 0 ? 0 : x); + } +} + +/* { dg-final { scan-tree-dump-not "sat_trunc pattern recognized" "vect" } } */ From patchwork Wed Sep 2 14:52:52 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142729 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 179BA4BB1C0B for ; Wed, 2 Sep 2026 15:00:37 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 179BA4BB1C0B Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=I2ct8S3g X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from PH0PR06CU001.outbound.protection.outlook.com (mail-westus3azlp170110003.outbound.protection.outlook.com [IPv6:2a01:111:f403:c107::3]) by sourceware.org (Postfix) with ESMTPS id 88D6A4BA9019 for ; Wed, 2 Sep 2026 14:53:59 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 88D6A4BA9019 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 88D6A4BA9019 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c107::3 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360839; cv=pass; b=xwQen1hO7MRM2CYCrVYKgWsIzYCz6gMpY6fs53tPGoqRA5O+RTkTadV2f8JQrwJBXkm3Kvr9GoXCugiKDO15cr3hYhfY+O4/tyzvxpuH+1ZfyVdbmZDkqJJqX3o9YPePQWPOPNb/I69qL788MTaJEL4raSkAriNoB+o6k7m7CMo= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360839; c=relaxed/simple; bh=alyg69QldoHGRUDaBpS9cw6D83kvf5kQzA9R6PrhocE=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=o9avv55AZ9WEYPl0UzNu+cWZNR5sb5VgOdT4ilb3uEHQYkYRCidN6lWD847F2V4rJ9kfYgaGJsE/QpdRJ0mNSAF9UYk423HXdYyjb1tkjfZHlZjiwF6KG0sNA7Tk9qaGdXH8Fecj91nbpC6tL+19GCH+bo7zL0fJT3zXEUCAhO4= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=I2ct8S3g DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 88D6A4BA9019 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=m7683Gj5GNiKNet7MyJbPQLhbmEY2IqDCWEmWtiUGlBt+ktPEIFR679MAF8poI++41MLwy3RPcJBskXH9QK4NPbNTvpt7/uPwNMFcubE9ebiqJGcxyMvJp0mbGGTffF3Q2LUPuYhHYNW8W57zPt+aKpdrMzUYyVFVGnrrxhmlIbsIFrcs9oM50a+oGr971SRZ+Nrjpp/u0pgVMhf4stCp81SSvOgCbu75Uy0NmbabrhYdNVkAYcJxHJtP3NWTkoTqheauQCzVEHQxSjPy5VGH2b6x5U35zWlr7mfDUbRgJYbM7q7IVlrN0epFcMIVCCV+f+mqRiz61PElYY+xpZFNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tuy4i4YvBVdeJ3VC+SMZPLnZdTqfNxA5QR83MIKmxYw=; b=phdUN6QTOGbkrMnBnUXpCZ63A3Fy/yNRq01rzuG+yKpWLiajMThlaIRQWY1A4t9w74JEn0ce3kHHsmajUYkgE0f6EpSoH9n2+hVw4YRkTxp4dcKztjICk0axuI37mcLwIx2vI3ZLxBkCGa+UfKUtx/tMtf7lMIhTIeTLwIdADcAfOES1Kv45nv91LE8tB9O+QySAjyjYVeaB2V0lMjYCSWqXLTJP/H8jBr++PpQjmNpjNZ6eM0poViyC3PvZLmp38pXmM4y2UIfzlLaH9r0odLBw0t7JLgYbmjeoxU8pCyHLvlDqgyY+kKvBgpnZUuC2GyghsEwrXLVN9ivpYHglag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tuy4i4YvBVdeJ3VC+SMZPLnZdTqfNxA5QR83MIKmxYw=; b=I2ct8S3gQ/rBJ/1jFCFc4qaoKtS1J881TeJPpPu/v7WpFBrJSnNOisWh7CLw7MdB7ZYQ6IiY5M8YzsPMflw+x6rYO/D+FqGKXaxJKJ+RNhhjPouZoTYsKjZOouGay/cWLENfT5vH4RRwf0nox6mlTxUtOPxCb/nftwNvdsKNYuutM2HDH+5YtoyWZ61JHqsBxj/NYWgRehGloNKNU2xga+kDE0FSaAXBUpMbSLNMve3V7FN66xTK5kblGLrgbkDbt6I4Q06h46msV1tbX2uEqqwmc5XEin4qRrSzwilxz3NLoDUKQbfxoWwCoNOe1WxjdSKBslZvcp9kPvKyAKRI9Q== Received: from SA9PR13CA0014.namprd13.prod.outlook.com (2603:10b6:806:21::19) by PH0PR12MB5629.namprd12.prod.outlook.com (2603:10b6:510:141::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Wed, 2 Sep 2026 14:53:50 +0000 Received: from SA2PEPF00001508.namprd04.prod.outlook.com (2603:10b6:806:21:cafe::1d) by SA9PR13CA0014.outlook.office365.com (2603:10b6:806:21::19) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.9 via Frontend Transport; Wed, 2 Sep 2026 14:53:49 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SA2PEPF00001508.mail.protection.outlook.com (10.167.242.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.382.8 via Frontend Transport; Wed, 2 Sep 2026 14:53:49 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:20 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:18 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 11/13] match-sat-alu.pd: Recognize signed constant-clamp truncation Date: Wed, 2 Sep 2026 16:52:52 +0200 Message-ID: <20260902145254.77832-13-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001508:EE_|PH0PR12MB5629:EE_ X-MS-Office365-Filtering-Correlation-Id: 5c91695b-9658-4bfd-13cc-08df0901ff62 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|82310400026|1800799024|376014|23010399003|36860700016|18002099003|22082099003|3023799007|56012099006|6133799003|11063799006|10067099003; X-Microsoft-Antispam-Message-Info: TFOheMnD2plQIBaOYHtYolMLDH2dSk6QGTkUs8K40wdOBkKD2c7ucwuBYE6LOZzpw77i9JYBoJsZXuG/dDgam1ftfbkx2NBgWZB4RHJdffwLWzlhSrG6arXiNbohJNKSJEHt72okNwI+UV/QC/2jbImwIOOJe1hBXcQfs3z5VGummjjSa0Ertdo+P760I6c69Q4faAr1beG4JVg8sy7+7l8ZCnSsHZSX/ljOHziEfw9hgrIzE0ZNMfTwvzsz5AMVcnWgj1izjaSZ1E2D+lWjFZNpXnUediRQpZIJxX+GLPcMNEddO51weBvqb3LkvsevzzqtsDLxyOHhhfQ8B1OjpfsVKrclKNb7GJXsmxR3Whb6lkAMBpsGxTQe42t4jGNXcTG0H6q3j0QOFrmtKR4DZQmXALUMe9eQiXE/M5z1cHE4MxNE6HCQVCWjdgmdXNdL+Cxa32DH+0wYLTd/VDAEjvO1ihty35sR+P6ZZ9yD8Vc/wfGSXBp9GWFFxHTxjfb9RW00Pr0nxaE47TMqtewr4WLNuw5aghCYIBif89WrfabkubTGnlpYqHvvpnepnV/6kpXwe8KQ/h3mrNyYngsHzSBqoGpf1YZy6Pl0Wi3V7tkXomqXGkGHZSXBgFWI15/lbIQjz6mkkY91XdF0G62aVVUxyYgVCEYtNVUo4mAqVm0MhjoiuYvhX1kUyrmeBQoWDyXAQGssINbM6KGWfQyOdA== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(82310400026)(1800799024)(376014)(23010399003)(36860700016)(18002099003)(22082099003)(3023799007)(56012099006)(6133799003)(11063799006)(10067099003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: DNFKfFZU4mo1rkSVkgGzFWt9tIvG2sQE6POMyOYQe23H3clDMjezalTsNyesjCS/9hYv/TPbSto9KR7/4X8GZf2Zt8Hz5DBekcSYiBzmUTbXgccB8g2HCWLf427en3d5ZwJsbxVrLTI99kzDByI+3zkPJao1oGrz03yKU2KTBKtIFs4XXQbmpvfO1Cf8Vh8CtIbvvyqIWeJ/JG3ZNnj0lHzPfgscBzxkMRYg+mcPhj3st7OIjVnJMhI8YrSY0qzVrH1IN9haZjoGMwhtzjPqCNOVZKGflqWcysYzpkIEeWH3zLRv7q/D0PKMqsZFrDZ1BdTRDi2rLH7FZoMWYEnKJZvvGuTLvxQVvVtVbHkayoXi/wknD0ojoTmKimrh+iDmeKnXDtz/HmXu0ekUQhOziuw5lnRteZiXDVAAq/x84Fbu9GSfYfeDle0gLmJw31DZ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:49.2773 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5c91695b-9658-4bfd-13cc-08df0901ff62 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001508.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB5629 X-Spam-Status: No, score=-8.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, SPF_HELO_NONE, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov A signed value clamped to the bounds of a narrower signed type is signed saturating truncation. Recognize both nesting orders of the MIN and MAX operations. Require exact bounds at the input precision, signed input and output types, and a narrowing conversion. With AArch64 -O2 -march=armv8-a -fopenmp-simd: before: clamp_i16_4: ldr q0, [x1] movi v30.4s, 0x7f, msl 8 mvni v31.4s, 0x7f, msl 8 smin v30.4s, v0.4s, v30.4s smax v30.4s, v30.4s, v31.4s xtn v30.4h, v30.4s str d30, [x0] ret after: clamp_i16_4: ldr q31, [x1] sqxtn v31.4h, v31.4s str d31, [x0] ret Runtime tests cover both clamp orders, two widths, and LTO. Target tests cover valid forms and inexact bounds. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (signed_integer_sat_trunc): Add the constant-clamp form. gcc/testsuite/ChangeLog: * gcc.dg/vect/vect-sat-trunc-clamp-2.c: New test. * gcc.target/aarch64/vect-sat-trunc-clamp-s-1.c: Likewise. * gcc.target/aarch64/vect-sat-trunc-clamp-s-negative-1.c: Likewise. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 20 ++- .../gcc.dg/vect/vect-sat-trunc-clamp-2.c | 126 ++++++++++++++++++ .../aarch64/vect-sat-trunc-clamp-s-1.c | 7 + .../vect-sat-trunc-clamp-s-negative-1.c | 30 +++++ 4 files changed, 182 insertions(+), 1 deletion(-) create mode 100644 gcc/testsuite/gcc.dg/vect/vect-sat-trunc-clamp-2.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-s-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-s-negative-1.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index ede949a34fe..ec58d86486d 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -665,7 +665,25 @@ along with GCC; see the file COPYING3. If not see && wi::eq_p (int_cst_2, limit_2)) || (wi::eq_p (int_cst_1, offset) && wi::eq_p (int_cst_2, limit_2)) || (wi::eq_p (int_cst_1, otype_max_in_range) - && wi::eq_p (int_cst_2, limit_1)))))))) + && wi::eq_p (int_cst_2, limit_1))))))) + (match (signed_integer_sat_trunc @0) + /* SAT_S_TRUNC = (NT)MAX (MIN (X, NT_MAX), NT_MIN). */ + (convert (integer_constant_clamp @0 @1 @2)) + (if (!TYPE_UNSIGNED (TREE_TYPE (@0))) + (with + { + unsigned itype_precision = TYPE_PRECISION (TREE_TYPE (@0)); + unsigned otype_precision = TYPE_PRECISION (type); + wide_int otype_min = wi::mask (otype_precision - 1, true, + itype_precision); + wide_int otype_max = wi::mask (otype_precision - 1, false, + itype_precision); + wide_int lo_cst = wi::to_wide (@1, itype_precision); + wide_int hi_cst = wi::to_wide (@2, itype_precision); + } + (if (otype_precision < itype_precision + && wi::eq_p (lo_cst, otype_min) + && wi::eq_p (hi_cst, otype_max))))))) /* An arithmetic right shift preserves the sign of its signed input. The saturation value can therefore use the input sign while the range check diff --git a/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-clamp-2.c b/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-clamp-2.c new file mode 100644 index 00000000000..193979587b8 --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-clamp-2.c @@ -0,0 +1,126 @@ +/* { dg-do run } */ +/* { dg-require-effective-target vect_int } */ +/* { dg-additional-options "-O3 -fvect-cost-model=unlimited" } */ + +#include "tree-vect.h" + +typedef __UINT32_TYPE__ u32; +typedef __UINT64_TYPE__ u64; +typedef __INT16_TYPE__ i16; +typedef __INT32_TYPE__ i32; +typedef __INT64_TYPE__ i64; + +#define N 259 + +static i32 in32[N]; +static i64 in64[N]; +static i16 out16[N]; +static i32 out32[N]; + +__attribute__((noipa)) static void +clamp_s16_lo (i16 *__restrict out, const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + out[i] = x < -32768 ? -32768 : (x > 32767 ? 32767 : x); + } +} + +__attribute__((noipa)) static void +clamp_s16_hi (i16 *__restrict out, const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + out[i] = x > 32767 ? 32767 : (x < -32768 ? -32768 : x); + } +} + +__attribute__((noipa)) static void +clamp_s32_lo (i32 *__restrict out, const i64 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i64 x = in[i]; + out[i] = (x < -2147483647LL - 1 + ? -2147483647LL - 1 : (x > 2147483647LL + ? 2147483647LL : x)); + } +} + +__attribute__((noipa)) static void +clamp_s32_hi (i32 *__restrict out, const i64 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i64 x = in[i]; + out[i] = (x > 2147483647LL + ? 2147483647LL : (x < -2147483647LL - 1 + ? -2147483647LL - 1 : x)); + } +} + +static i16 +ref_s16 (i32 x) +{ + return x < -32768 ? -32768 : (x > 32767 ? 32767 : x); +} + +static i32 +ref_s32 (i64 x) +{ + return (x < -2147483647LL - 1 + ? -2147483647LL - 1 : (x > 2147483647LL ? 2147483647LL : x)); +} + +static void +check_s16 (void) +{ +#pragma GCC novector + for (int i = 0; i < N; ++i) + if (out16[i] != ref_s16 (in32[i])) + __builtin_abort (); +} + +static void +check_s32 (void) +{ +#pragma GCC novector + for (int i = 0; i < N; ++i) + if (out32[i] != ref_s32 (in64[i])) + __builtin_abort (); +} + +int +main (void) +{ + check_vect (); + + for (int i = 0; i < N; ++i) + { + in32[i] = (i32) ((u32) i * 2654435761U + 1013904223U); + in64[i] = (i64) ((u64) i * 11400714819323198485ULL + + 13787848793156543929ULL); + } + + in32[0] = -32769; + in32[1] = -32768; + in32[2] = 32767; + in32[3] = 32768; + in64[0] = -2147483647LL - 2; + in64[1] = -2147483647LL - 1; + in64[2] = 2147483647LL; + in64[3] = 2147483648LL; + + clamp_s16_lo (out16, in32, N); + check_s16 (); + clamp_s16_hi (out16, in32, N); + check_s16 (); + clamp_s32_lo (out32, in64, N); + check_s32 (); + clamp_s32_hi (out32, in64, N); + check_s32 (); + + return 0; +} diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-s-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-s-1.c new file mode 100644 index 00000000000..3a603ca18ef --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-s-1.c @@ -0,0 +1,7 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +#include "../../gcc.dg/vect/vect-sat-trunc-clamp-2.c" + +/* { dg-final { scan-tree-dump-times "sat_trunc pattern recognized" 4 "vect" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-s-negative-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-s-negative-1.c new file mode 100644 index 00000000000..b0f86653554 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-clamp-s-negative-1.c @@ -0,0 +1,30 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +typedef __INT16_TYPE__ i16; +typedef __INT32_TYPE__ i32; + +__attribute__((noipa)) +void +bad_low (i16 *__restrict out, const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + out[i] = x < -32767 ? -32767 : (x > 32767 ? 32767 : x); + } +} + +__attribute__((noipa)) +void +bad_high (i16 *__restrict out, const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + out[i] = x > 32766 ? 32766 : (x < -32768 ? -32768 : x); + } +} + +/* { dg-final { scan-tree-dump-not "sat_trunc pattern recognized" "vect" } } */ From patchwork Wed Sep 2 14:52:53 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142732 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 86BDA4BB3BF4 for ; Wed, 2 Sep 2026 15:03:19 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 86BDA4BB3BF4 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=oEIUxF64 X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from SJ2PR03CU001.outbound.protection.outlook.com (mail-westusazlp170120002.outbound.protection.outlook.com [IPv6:2a01:111:f403:c001::2]) by sourceware.org (Postfix) with ESMTPS id 407934B9DB56 for ; Wed, 2 Sep 2026 14:54:02 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 407934B9DB56 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 407934B9DB56 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=2a01:111:f403:c001::2 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360842; cv=pass; b=GDUuuHyEVnpWJqwiYBTEGOGyxLm0Rqh8c3IRwDQa87hE14rebxS/Mtht4Cq/TL2vwSm6X0zm9A3VknxGi9A3yLip4u0Jc82JM8oXQc1aaYVjoq5ac/BA5LMhXrFiXbjQQQusgac3EgfLMPce07ydhPAV/a+6sbFUVGGmRT2tsw0= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360842; c=relaxed/simple; bh=MtCweUS7rxdK+D6A+OvjZzeiFHhJ6O5RqVNomMGeeB8=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=rFrq7U2qP+DBoRTxz6rsV4CKodra06gDGjIfxIE/XvhWM0G/vMrtgizK0mdfdq62RPNyHEUcxyTkei22FTeApY528xo1VGs98+GBJjFNLNfYKw221aLZrNK7R1mQ3el6a5Yg0p49oDhvz+Dgv45Q/qqIx7sQFUcgSpS1CfhG7eE= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=oEIUxF64 DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 407934B9DB56 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=S64WP0TgsikcFTHRyyAguwIE2IKkYaX2Uc5hiej10Oop6hv1xSknYSo3bnWbfY/9tNcK1eWCdSkANZHz3aqgV+1jbCtXMsQ7kQRwnOTZz5AxJZqIR5KqiBEqEbSh/WXMphD1QP9T6PX4H09X/vSUhPZLF5wBPMPuaG4E4lAtmCoe2u/dWbvezjoP7+Ku8lu5yU0R1k5WOt8vbVnrHZViN+FUGt22zeHfA0hhlodZRTacWNwS+Y4bUP8FypqZ9HqQjVlzSaHw3cxWB8zKe6OAPvUDPwi98GEYd0qnoigV7YGIB9lOOBuwjsEj2IFW7ErgJoU2c4A7pwrOIMuU5NFDBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oLZkgbIfJ05XiWaRyOE9ndW7WhWfShxJ7I33GeXtnbM=; b=T7iOgLYlFXNp5APocgbDR+SXPpaNlLXF1A+cvhwXiNOYtYE6fVVAgI5wX9xwkXSI+5q3f6UcTo3AC5buDIeEVfPNjkOQCL5v8ZtoqBHuqBQojCKpUmU6LnNCDe1ZPOYmf1VAn1xGSmxUAPwsufQ6GKjTqaISeRVik1qrrvFjSQ2dbGemoq0rUos23gyliIEYdnozavX6+b6s1BORfLnPBBfOQunVcLysJkPeFCFnWCVUjpV6bVcyphmb1Q+xJzjJjnzwsVY0QdAzxb7/gzgKMOrrMuRj4pFDHbnento70DnRMQjJD6gcfGTmcEBV67q9hJ89zmckKtEx2G/3B8o+2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oLZkgbIfJ05XiWaRyOE9ndW7WhWfShxJ7I33GeXtnbM=; b=oEIUxF64PjEbpdhwlpbAoAMLKMfawFpJbYNUZ06ZX3Ise5KV87CsCKAUg18mesRn2OxJ61YNZGGrptYhmbDKKGqqd3mL6b8Rx01kNz5VXKkE4Rg/53ohZ3dwqG27DwdwhE747c7y0XuzkGiNULudZg83x45hzv/XFiApQFBfrIveDbwHbk6CakUqYqaDoJfcjUbYTyuoeUzL+mISQUVxDc31Yca4VUnITijngIwHhrbM3K5haNs+lN1RUEVJmpO6eY8paeYf1sn1D8EY2HA4aJ1JMx0BrJWna1iC4P6PMxK3XNd8mi3rx2N0UATTFw64fWN09S/duxurLEzMm3XeFw== Received: from SA9PR13CA0028.namprd13.prod.outlook.com (2603:10b6:806:21::33) by SA1PR12MB8120.namprd12.prod.outlook.com (2603:10b6:806:331::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.12; Wed, 2 Sep 2026 14:53:51 +0000 Received: from SA2PEPF00001508.namprd04.prod.outlook.com (2603:10b6:806:21:cafe::22) by SA9PR13CA0028.outlook.office365.com (2603:10b6:806:21::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.382.8 via Frontend Transport; Wed, 2 Sep 2026 14:53:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SA2PEPF00001508.mail.protection.outlook.com (10.167.242.40) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.382.8 via Frontend Transport; Wed, 2 Sep 2026 14:53:51 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:21 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:20 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 12/13] match-sat-alu.pd: Recognize complement masks for unsigned saturating truncation Date: Wed, 2 Sep 2026 16:52:53 +0200 Message-ID: <20260902145254.77832-14-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF00001508:EE_|SA1PR12MB8120:EE_ X-MS-Office365-Filtering-Correlation-Id: 7e8db422-45c2-4e95-f31f-08df090200ba X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700016|1800799024|82310400026|23010399003|376014|6133799003|10067099003|56012099006|11063799006|13003099007|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: 9Pddu2e3B9V2MjTiDeh1DAeg3OrSQDbp1rNLFlGOZ7/EYHVZ8JR/a8GmYudDMMZsgFaz5Mi++3hPLt9zw5sMMDjU53CIrc4VwWIV3g24KRN1uvA7So9XfAoPtGW7FGxTNhWFUZ3NxPuf+TcUQ5TJzENPq0T0EhFl3AA8UnsMm49me/YUveeXYCAftDJ87fn8JUHsvCVxQa9xabmZHvZlBA7cEpn/vctX6TCTSeDIRp+syUdvKCR0yoHQwQuACZ4iPcwKdps/0BrMNitPjiKQhvzfXkA8yO0WhbxzdyW+Rny1WgJRBouy96Y6iXEb15aH7+lX2vpjhfiqthPcHpuF2mM09FJC3jkuO37TuBMXbj0HfQW8uT+bw5EXWkMe+EFZotHwQdY5ommYNHjQucsL4UZbtNEpThsN4er+XcCjr4bweg0VIELvDl4ReTt7KVUFXPyMLmcWsUt+2jIwahg7+xRFly3of5/ZaCjzJ3CWNwvhbDZsWpVEv7EkxCCcjTZ2qAOW/RUqS6eLW+lPY6gBXIl7SPAOhD1QoHOsSqNGrXgYjaS5iMamlo8ysrFYk9Lm3x1foyqPHCapeRDdiJTLTv5MNw3I351mqlFq33oC/tO3qjlmw9eBsG+AtF7rqEVMXjFf3e3dnuT5qhLDOqCIdDtSSwplbjpYREwiSOjGPg8rKJQAizYuD7T4z96rse/RgdGWoqA+dHsvZAH5fHgd+A== X-Forefront-Antispam-Report: CIP:216.228.117.161; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge2.nvidia.com; CAT:NONE; SFS:(13230040)(36860700016)(1800799024)(82310400026)(23010399003)(376014)(6133799003)(10067099003)(56012099006)(11063799006)(13003099007)(22082099003)(18002099003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: oiNX5w6EMx3gWs2zfUZGOW8VJb8s42WHLavcx9KnFvegD5W4qsbrVSxIlq1qhjpnnFXNYN8K23dYiQS6E/gUBBoL9FQJ+bq/sUafpC0vqoam9V5E8+iofXsPHsIbnKiG25wfnx11dxUKdAe1aNzF47DhDoaixYb+BgKE7bE0PACDj3m2J1LxrEgELDqsVpyp0WYj7tbds4gC43o8Vkio6DgnMIMR1+v536STahdUqZtpM5lZxFCZyB4FcD9Czsm6aHwQf7nl0Lby1TSV8jSw22THMH9glATcESi0f+RufAfTfrMbAOFxq+wzBf0ol4zjILLiAQJNqdRSM4XFIcebvJcUGlRMMBSewiYF3As8z4PfnOKDUljtJ5WCMeF2rUCzsNRzK3HIM1NJ1wepaku6pUDj3vhDQIlJ/1WOmNQJyGl1UYylRRVkaf9hURWq4qnN X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:51.5417 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7e8db422-45c2-4e95-f31f-08df090200ba X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.161]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF00001508.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB8120 X-Spam-Status: No, score=-8.1 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FORGED_SPF_HELO, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, RCVD_IN_DNSWL_NONE, SPF_HELO_PASS, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov Unsigned saturating truncation can form its sign mask from ~X instead of -X. This avoids negating the minimum signed value. For negative X, ~X is nonnegative and its top-bit shift is zero. For positive X above the result range, ~X is negative and its top-bit shift is all ones. Recognize this form. Keep the unsigned range comparison because a signed comparison does not clamp negative inputs. With AArch64 -O2 -march=armv8-a -fopenmp-simd: before: complement_clip4: ldr q31, [x1] movi v30.4s, 0xff, msl 8 not v29.16b, v31.16b cmhi v30.4s, v31.4s, v30.4s cmlt v29.4s, v29.4s, #0 bif v29.16b, v31.16b, v30.16b xtn v29.4h, v29.4s str d29, [x0] ret after: complement_clip4: ldr q31, [x1] movi v30.4s, 0 smax v30.4s, v31.4s, v30.4s uqxtn v30.4h, v30.4s str d30, [x0] ret Tests cover both widths, extrema, bounds, LTO, and a signed-comparison counterexample. The AArch64 test covers both valid forms. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match-sat-alu.pd (unsigned_integer_narrow_clip): Add the complement form. gcc/testsuite/ChangeLog: * gcc.dg/sat-trunc-signed-cmp-2.c: New test. * gcc.dg/vect/vect-sat-trunc-complement-1.c: Likewise. * gcc.target/aarch64/vect-sat-trunc-complement-1.c: Likewise. Signed-off-by: Kyrylo Tkachov --- gcc/match-sat-alu.pd | 24 +++++++ gcc/testsuite/gcc.dg/sat-trunc-signed-cmp-2.c | 42 +++++++++++ .../gcc.dg/vect/vect-sat-trunc-complement-1.c | 70 +++++++++++++++++++ .../aarch64/vect-sat-trunc-complement-1.c | 30 ++++++++ 4 files changed, 166 insertions(+) create mode 100644 gcc/testsuite/gcc.dg/sat-trunc-signed-cmp-2.c create mode 100644 gcc/testsuite/gcc.dg/vect/vect-sat-trunc-complement-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-complement-1.c diff --git a/gcc/match-sat-alu.pd b/gcc/match-sat-alu.pd index ec58d86486d..839f2ea52e8 100644 --- a/gcc/match-sat-alu.pd +++ b/gcc/match-sat-alu.pd @@ -334,6 +334,30 @@ along with GCC; see the file COPYING3. If not see && cmp >= 0 && expr_not_equal_to (@0, wi::min_value (itype_precision, SIGNED))))))) + (match (unsigned_integer_narrow_clip @0) + /* Use the sign of ~X for a source that avoids negating the minimum signed + value. For a negative X, ~X is nonnegative and gives zero. For a + positive X above the result range, ~X is negative and gives all ones. */ + (convert (cond^ (gt (nop_convert@3 @0) INTEGER_CST@1) + (rshift:s (bit_not @0) INTEGER_CST@2) + @0)) + (if (!TYPE_UNSIGNED (TREE_TYPE (@0)) + && TYPE_UNSIGNED (TREE_TYPE (@3))) + (with + { + unsigned itype_precision = TYPE_PRECISION (TREE_TYPE (@0)); + unsigned otype_precision = TYPE_PRECISION (type); + wide_int trunc_max = wi::mask (otype_precision, false, itype_precision); + wide_int int_cst_1 = wi::to_wide (@1, itype_precision); + wide_int int_cst_2 = wi::to_wide (@2, itype_precision); + wide_int shift_amount = wi::uhwi (itype_precision - 1, + itype_precision); + int cmp = wi::cmp (int_cst_2, shift_amount, + TYPE_SIGN (TREE_TYPE (@0))); + } + (if (otype_precision < itype_precision + && wi::eq_p (trunc_max, int_cst_1) + && cmp >= 0))))) (match (unsigned_integer_narrow_clip @0) /* SAT_US_TRUNC = (NT)MAX (MIN (X, NT_MAX), 0). */ (convert (integer_constant_clamp @0 @1 @2)) diff --git a/gcc/testsuite/gcc.dg/sat-trunc-signed-cmp-2.c b/gcc/testsuite/gcc.dg/sat-trunc-signed-cmp-2.c new file mode 100644 index 00000000000..a06588ccedc --- /dev/null +++ b/gcc/testsuite/gcc.dg/sat-trunc-signed-cmp-2.c @@ -0,0 +1,42 @@ +/* The complement spelling of the narrow-clip idiom is only a clamp when the + comparison that selects the saturated arm is unsigned. With a signed + comparison a negative value takes the fall-through arm and keeps its low + bits, so it must not be turned into a saturating truncation. */ + +/* { dg-do run } */ +/* { dg-options "-O1 -ftree-vectorize -fdump-tree-vect-details" } */ + +typedef __UINT16_TYPE__ u16; +typedef __INT32_TYPE__ i32; + +#define N 32 + +static i32 a[N]; +static u16 r[N]; + +__attribute__((noipa)) static void +clip (u16 *__restrict d, const i32 *__restrict s, int n) +{ + for (int i = 0; i < n; i++) + { + i32 v = s[i]; + d[i] = v > 65535 ? (~v) >> 31 : v; + } +} + +int +main (void) +{ + for (int i = 0; i < N; i++) + a[i] = -1; + + clip (r, a, N); + + for (int i = 0; i < N; i++) + if (r[i] != (u16) -1) + __builtin_abort (); + + return 0; +} + +/* { dg-final { scan-tree-dump-not "sat_trunc pattern recognized" "vect" } } */ diff --git a/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-complement-1.c b/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-complement-1.c new file mode 100644 index 00000000000..f7bb1656a5d --- /dev/null +++ b/gcc/testsuite/gcc.dg/vect/vect-sat-trunc-complement-1.c @@ -0,0 +1,70 @@ +/* { dg-do run } */ +/* { dg-require-effective-target vect_int } */ +/* { dg-additional-options "-O3 -fvect-cost-model=unlimited" } */ + +#include "tree-vect.h" + +typedef __UINT16_TYPE__ u16; +typedef __UINT32_TYPE__ u32; +typedef __INT32_TYPE__ i32; +typedef __INT64_TYPE__ i64; + +#define N 96 + +static i32 in32[N]; +static i64 in64[N]; +static u16 out16[N]; +static u32 out32[N]; + +__attribute__((noipa)) static void +clip_u16 (u16 *__restrict out, const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + out[i] = (x & ~65535) ? (~x) >> 31 : x; + } +} + +__attribute__((noipa)) static void +clip_u32 (u32 *__restrict out, const i64 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i64 x = in[i]; + out[i] = (x & ~(i64) 0xffffffff) ? (~x) >> 63 : x; + } +} + +int +main (void) +{ + static const i32 values32[6] = + { (-2147483647 - 1), -1, 0, 65535, 65536, 2147483647 }; + static const u16 expected16[6] = + { 0, 0, 0, 65535, 65535, 65535 }; + static const i64 values64[6] = + { (-9223372036854775807LL - 1), -1, 0, 4294967295LL, + 4294967296LL, 9223372036854775807LL }; + static const u32 expected32[6] = + { 0, 0, 0, (u32) -1, (u32) -1, (u32) -1 }; + + check_vect (); + + for (int i = 0; i < N; ++i) + { + in32[i] = values32[i % 6]; + in64[i] = values64[i % 6]; + } + + clip_u16 (out16, in32, N); + clip_u32 (out32, in64, N); + +#pragma GCC novector + for (int i = 0; i < N; ++i) + if (out16[i] != expected16[i % 6] + || out32[i] != expected32[i % 6]) + __builtin_abort (); + + return 0; +} diff --git a/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-complement-1.c b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-complement-1.c new file mode 100644 index 00000000000..9c18ee063e0 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/vect-sat-trunc-complement-1.c @@ -0,0 +1,30 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -march=armv8-a" } */ +/* { dg-additional-options "-mmax-vectorization --param=vect-epilogues-nomask=0 -fdump-tree-vect-details" } */ + +typedef __UINT16_TYPE__ u16; +typedef __UINT32_TYPE__ u32; +typedef __INT32_TYPE__ i32; +typedef __INT64_TYPE__ i64; + +void +clip_u16 (u16 *__restrict out, const i32 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i32 x = in[i]; + out[i] = (x & ~65535) ? (~x) >> 31 : x; + } +} + +void +clip_u32 (u32 *__restrict out, const i64 *__restrict in, int n) +{ + for (int i = 0; i < n; ++i) + { + i64 x = in[i]; + out[i] = (x & ~(i64) 4294967295LL) ? (~x) >> 63 : x; + } +} + +/* { dg-final { scan-tree-dump-times "sat_trunc pattern recognized" 2 "vect" } } */ From patchwork Wed Sep 2 14:52:54 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 142721 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 5E5B14BAE7D0 for ; Wed, 2 Sep 2026 14:56:15 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 5E5B14BAE7D0 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=VJKCdC/q X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012049.outbound.protection.outlook.com [52.101.48.49]) by sourceware.org (Postfix) with ESMTPS id 7C3124BA79BA for ; Wed, 2 Sep 2026 14:53:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 7C3124BA79BA Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 7C3124BA79BA Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.48.49 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360835; cv=pass; b=ONzMYNOp7oKRltB+zA5pczc2w6xWGIuELxUYx39vcFGGlIsJQLN28r0i2HzjMVKJupdk9U33YMcOzhgd8nk9xZWRehvraJ7KfnpmnKmwE8+OhI4hkhQE+YHPx9+tb3P/9rPM57Ls75ZTm5kSoGEOgwkjzTMv5jDRaDNJZxhvNYo= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1788360835; c=relaxed/simple; bh=wxqwJC7u3LsDXzjcjTgabu4aRdXq7qoFcQQ2AIqvUls=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=BwxKLzUHGfXOxNCO8XBJER9iqhBQIyiCsygNWuJlsUSyQ3QLUfpDdfv5EL/xmEonujroBFHR44BTS9EZGSh731zefZUI7fu3xNJH6VdTllJPNEBir3BsNrHOCw2fTPe5b8Uh3LFrw70LBAt44Ia2A4uRqCxVW9QMbDKOrDjYpIA= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=VJKCdC/q DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7C3124BA79BA ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BHQ9bB6lFgbypRVGBsU61HWYQJ5aMuvvIJCOcEmcmZYMj3Voe+koaWeWZboXBEnj7wBqzOGlNSS7gOVGC7Y38lyxck4rDsbmjtx3l1hYbKF5PR5IYA3PBzJF+pxXiz09KWMFeV1uolSDUdaxVoXqwyK1wjKUuEFOh2a6sAc7lHk9S+1cRQXNNg66DNtmpGijVQYdfS2o4v1RlVrBnI0xou/IHseKHaE+hSbgcuVZe4VpDD9b2TcqSMbvFIiHFAI7uQH4pP6xp364VFVa98OB6djefdWK/+dXHjmdTKgzM6kCthHSk78nSrq3AK7CfP37T6hrtfRQqESk3yyS+K+Prg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=NDxVPDouu92ThNGbWm9Fn+FJ0zk13mYhwGx32FfsTOQ=; b=WfBmrz5J/eGLIm1R84QAj6P2QjpNGNEe0PpCDCQw8VCTgwDvSkK62QVYoy+DV9vuWGD4uyZsiRKCtirfEHgieEjBxnkdyfjoR0PSvhchXbL4gDuInZ/YKlQvUzibLOnPEQD21CQopX5kpaPbP7h/DMm5lAjEA0bnJRpChXUfY8YaGfAT0kQixS/TSAi0NlcaulMzgW8daTG9WeA26W0PdBROtyQfnR/TrjX6u9NFGa/JpBc2o70aiUr6Z+nLhcCx8Bok9rdMHC+NUZfKw4nPgkQg8sq635X6JotNdre992vcKXFHjPbiwWgz0wrzncg7YTCyPab0tHVcFgqtzHvsdQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NDxVPDouu92ThNGbWm9Fn+FJ0zk13mYhwGx32FfsTOQ=; b=VJKCdC/qPgnTpIISZ9u/5tEXOuhNHFpF/SGc+hR2NmpcerO06U4vP/Za0oF9A4fW0QeHjW2ezHzG+Tc3od4eV0LR6RN2b8rfOnsfZncJGKCx8PpUIcTofTN9vrSxvO1Z9+4hd3ixKQ5TfmC4w7vV3roL6u6VmodTLIr78f+bE4MH/Curzf+KkBqpWtk9F4mGjYKO8HZPGi753WUr1IL8fVwtf/2/XF4mO6M2MLYun2AqP1Gtrj7Bgbj3QTCmkpcU4YStvRSTRdRNHtbHS7nB9ucqm5bBTIyuWNo1K1pM1M8wnufdVTfon9eulr0beoqCpvas4PmAOJ41BgrLxBfaEg== Received: from MN2PR07CA0029.namprd07.prod.outlook.com (2603:10b6:208:1a0::39) by CY1PR12MB9559.namprd12.prod.outlook.com (2603:10b6:930:fd::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.360.13; Wed, 2 Sep 2026 14:53:51 +0000 Received: from BN3PEPF0000B070.namprd21.prod.outlook.com (2603:10b6:208:1a0:cafe::20) by MN2PR07CA0029.outlook.office365.com (2603:10b6:208:1a0::39) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.360.12 via Frontend Transport; Wed, 2 Sep 2026 14:53:51 +0000 X-MS-Exchange-Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN3PEPF0000B070.mail.protection.outlook.com (10.167.243.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.1 via Frontend Transport; Wed, 2 Sep 2026 14:53:51 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:22 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.46; Wed, 2 Sep 2026 07:53:21 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH 13/13] match.pd: Narrow promoted saturating addition and subtraction Date: Wed, 2 Sep 2026 16:52:54 +0200 Message-ID: <20260902145254.77832-15-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260902145254.77832-1-ktkachov@nvidia.com> References: <20260902145254.77832-1-ktkachov@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail202.nvidia.com (10.129.68.7) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B070:EE_|CY1PR12MB9559:EE_ X-MS-Office365-Filtering-Correlation-Id: db039a80-23b5-4aa1-9abc-08df09020069 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|376014|1800799024|23010399003|82310400026|36860700016|56012099006|10067099003|11063799006|6133799003|3023799007|18002099003|22082099003|13003099007; X-Microsoft-Antispam-Message-Info: /xuZZGAjkTcl74JCCxnHjhvREMpUZgQCDPN3sN8tczWLjmoe8eihK/Oawk7+cJoYSey/FDNdYDXNjlsETCZZbUMRx6yGqara/W+1Z3BTzXbV9NwJYl3xLh6OKmWvXGvO5Fh/+7gsNVFoTs5VzRXv5+iqQj9ttK8lpn3guCR1lWVLxAnB0DrdF4Pm92Ck0Yk4GpPrf8Cv0uAjBjz8/6+bFACh7bhePOiPWBAMg+heNndHqQHzlfAH528jeNZlZSt5/zLjX/CP5n9lSHa0z2M04TplqmypHckBd6bfc7mXzzpEuJmZgHuL6g4nGW4Xj8Azll95824vufdxHpYjEioNivpZ5Pp5xm3TtpGdOM7y8mDWx3A/oH8aRkBqdek/dALWNlUZWO5y8Xto71SQPGJhtFjdGD+868V551DL0McSyU8EBGVmiIo46Y/NWfKxLONlS9CWvx4Z16aJy+7PUOS0v7B1rWt6PAYjkssrZZEDi07hIIbzAdV212IvKZSjOzCPX20gJDZnATtoAoSH6HI5opwEoBWYxDTgXFQvAUvFTTZ851BH2j+hIcs/XhODUZWemfq4h406iIwwtyNkoqpuv7rYeE2sTTu+VL280eUTVZmMKOd4DuW8MNfrhbXc8LbP596OC/1uZtZgP8GftjWAl/+mwIERLGGf6v+MFW+miGbboYJDvW8iwRZBwuo/MX39WASTxJvGEGPvbIftq2EALg== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(376014)(1800799024)(23010399003)(82310400026)(36860700016)(56012099006)(10067099003)(11063799006)(6133799003)(3023799007)(18002099003)(22082099003)(13003099007); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: UhxVDOwzbT6NJlD1xgxDruQoQwiow79Ziqz6vVX0ZyesFiI18VBgihtflfYGDLf+wXrclMmjzFtOB0oMSOjH1lOpSt938BGWBrUI0YcroNR5RPWAhR1n+yo5Vi024sZlIVZC7SKRju73tpDf9pGTHMl90fO6EH5Pvn/kqW+moMUza4y8BZB5AyzLWwMr9OyYplZv+d9kfI7JvT8cQcMu1vmDxI0qYP5vGsn3DM6W72VfmGj6yHfspEQyjmPf2A7tl/dq0hNxw0KOgjM1gTraiFRSXYlpy0HM/2MNN5Gxadj1pjr5U+AidPd/bqlwK1MzPIuoYyQdWV2Hki0YhjA1lwAd0Gc8X9Msn7VWF1fiNQlWaKEbyfTcSgJvs6gTr+hmhDGZjifk6HTnkST8oyM14lp7ZvP21k5HQKJSIWBQQm5c3nLZDjdVi/4LLfgCMu/N X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2026 14:53:51.0156 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: db039a80-23b5-4aa1-9abc-08df09020069 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B070.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR12MB9559 X-Spam-Status: No, score=-8.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, LOCAL_AUTHENTICATION_FAIL_SPF, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov Integer promotions can leave a narrow saturating operation as extensions, a wide operation, a clamp, and a truncation. Simplify these forms: (type) MAX ((wide_type) A - (wide_type) B, 0) -> A - MIN (A, B) (type) MIN ((wide_type) A + (wide_type) B, MAX) -> A + MIN (~A, B) Require an intermediate type in which the operation cannot overflow. Subtraction needs a wider signed type. Addition needs one extra value bit and a further sign bit for a signed intermediate. Also require the exact maximum of the narrow type. The result exposes narrow saturating forms. AArch64 -O3, vector body: before: ldr q30, [x1, x4] ldr q29, [x2, x4] usubl v28.8h, v30.8b, v29.8b usubl2 v29.8h, v30.16b, v29.16b smax v28.8h, v28.8h, v31.8h smax v29.8h, v29.8h, v31.8h uzp1 v29.16b, v28.16b, v29.16b str q29, [x0, x4] after: ldr q31, [x1, x4] ldr q30, [x2, x4] uqsub v30.16b, v31.16b, v30.16b str q30, [x0, x4] The five computing instructions become one. Targets without a saturating instruction still use fewer operations. Scalar byte and halfword forms on AArch64 can cost one extra instruction because the backend uses SIMD registers. The direct narrow form has the same cost. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? gcc/ChangeLog: * match.pd ((type) MAX ((wide_type) a - (wide_type) b, 0)): New simplification. ((type) MIN ((wide_type) a + (wide_type) b, MAX)): Likewise. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sat_u_promoted-1.c: New test. * gcc.target/aarch64/sat_u_promoted-2.c: New test. Signed-off-by: Kyrylo Tkachov --- gcc/match.pd | 37 +++++++++++++++++++ .../gcc.target/aarch64/sat_u_promoted-1.c | 29 +++++++++++++++ .../gcc.target/aarch64/sat_u_promoted-2.c | 32 ++++++++++++++++ 3 files changed, 98 insertions(+) create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_promoted-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sat_u_promoted-2.c diff --git a/gcc/match.pd b/gcc/match.pd index 54cad1cd2bc..8afdb4105f6 100644 --- a/gcc/match.pd +++ b/gcc/match.pd @@ -4839,6 +4839,43 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT) @0 @2))) +/* Narrow a saturating add or subtract that the integer promotions widened. + For an unsigned type, + (type) MAX ((wide_type) a - (wide_type) b, 0) -> a - MIN (a, b) + (type) MIN ((wide_type) a + (wide_type) b, MAX) -> a + MIN (~a, b) + Both results are exact in the narrow type: the difference is clamped at + zero and the sum at MAX, so the truncation never loses a bit. The widening + sequence of five operations collapses to two or three narrow ones. */ +(simplify + (convert (max@4 (minus@2 (convert @0) (convert @1)) integer_zerop)) + (if (INTEGRAL_TYPE_P (type) + && TYPE_UNSIGNED (type) + && types_match (type, @0, @1) + && single_use (@4) + && !TYPE_UNSIGNED (TREE_TYPE (@2)) + && TYPE_PRECISION (TREE_TYPE (@2)) > TYPE_PRECISION (type)) + (minus @0 (min @0 @1)))) + +(simplify + (convert (min@4 (plus@2 (convert @0) (convert @1)) INTEGER_CST@3)) + (if (INTEGRAL_TYPE_P (type) + && TYPE_UNSIGNED (type) + && types_match (type, @0, @1) + && single_use (@4)) + (with + { + unsigned precision = TYPE_PRECISION (type); + tree wide_type = TREE_TYPE (@2); + unsigned wide_precision = TYPE_PRECISION (wide_type); + /* The widened sum reaches 2 * (2^precision - 1), which needs one bit + more than the narrow type, and one further bit for a sign. */ + unsigned needed = precision + (TYPE_UNSIGNED (wide_type) ? 1 : 2); + } + (if (wide_precision >= needed + && wi::eq_p (wi::to_wide (@3), + wi::mask (precision, false, wide_precision))) + (plus @0 (min @1 (bit_not @0))))))) + /* MIN (MAX (X, Y + CST), Y) -> Y for a positive CST, and the dual MAX (MIN (X, Y + CST), Y) -> Y for a negative one. The inner select is at least Y + CST, which the absence of wrapping puts strictly beyond Y, diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_promoted-1.c b/gcc/testsuite/gcc.target/aarch64/sat_u_promoted-1.c new file mode 100644 index 00000000000..424d2df3fba --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_promoted-1.c @@ -0,0 +1,29 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -fdump-tree-optimized" } */ + +typedef unsigned char u8; +typedef unsigned short u16; +typedef unsigned int u32; +typedef unsigned long long u64; + +/* The integer promotions turn a narrow saturating add or subtract written + as a clamp into a widened operation feeding a MIN or a MAX. */ + +u8 sub8 (u8 a, u8 b) { int t = a - b; return t < 0 ? 0 : t; } +u8 sub8_swapped (u8 a, u8 b) { int t = a - b; return t > 0 ? t : 0; } +u16 sub16 (u16 a, u16 b) { int t = a - b; return t < 0 ? 0 : t; } +u32 sub32 (u32 a, u32 b) { long long t = (long long) a - b; return t < 0 ? 0 : t; } + +u8 add8 (u8 a, u8 b) { int t = a + b; return t > 255 ? 255 : t; } +u16 add16 (u16 a, u16 b) { int t = a + b; return t > 65535 ? 65535 : t; } +u32 add32 (u32 a, u32 b) { u64 t = (u64) a + b; return t > 0xffffffffu ? 0xffffffffu : t; } + +u8 add8_shared (u8 a, u8 b, int *p) +{ + int t = a + b; + *p = t; + return t > 255 ? 255 : t; +} + +/* { dg-final { scan-tree-dump-times "\\.SAT_SUB " 4 "optimized" } } */ +/* { dg-final { scan-tree-dump-times "\\.SAT_ADD " 4 "optimized" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/sat_u_promoted-2.c b/gcc/testsuite/gcc.target/aarch64/sat_u_promoted-2.c new file mode 100644 index 00000000000..3370ab0706a --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sat_u_promoted-2.c @@ -0,0 +1,32 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -fdump-tree-optimized" } */ + +typedef unsigned char u8; + +/* Without the recognition these loops widen, clamp and pack again, five + vector operations where one saturating operation does the job. */ + +void +sub_loop (u8 *__restrict d, u8 *__restrict a, u8 *__restrict b, int n) +{ + for (int i = 0; i < n; i++) + { + int t = a[i] - b[i]; + d[i] = t < 0 ? 0 : t; + } +} + +void +add_loop (u8 *__restrict d, u8 *__restrict a, u8 *__restrict b, int n) +{ + for (int i = 0; i < n; i++) + { + int t = a[i] + b[i]; + d[i] = t > 255 ? 255 : t; + } +} + +/* { dg-final { scan-tree-dump "\\.SAT_SUB " "optimized" } } */ +/* { dg-final { scan-tree-dump "\\.SAT_ADD " "optimized" } } */ +/* { dg-final { scan-assembler "uqsub\tv\[0-9\]+\.16b" } } */ +/* { dg-final { scan-assembler "uqadd\tv\[0-9\]+\.16b" } } */