From patchwork Tue Aug 4 09:40:28 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kyrylo Tkachov X-Patchwork-Id: 140570 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 99D314BB1C39 for ; Tue, 4 Aug 2026 09:41:44 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 99D314BB1C39 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=NYTHws3C X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from PH7PR06CU001.outbound.protection.outlook.com (mail-westus3azon11010000.outbound.protection.outlook.com [52.101.201.0]) by sourceware.org (Postfix) with ESMTPS id E97E84BB1C29 for ; Tue, 4 Aug 2026 09:41:00 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org E97E84BB1C29 Authentication-Results: sourceware.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: sourceware.org; spf=fail smtp.mailfrom=nvidia.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org E97E84BB1C29 Authentication-Results: sourceware.org; arc=pass smtp.remote-ip=52.101.201.0 ARC-Seal: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1785836461; cv=pass; b=gdjJnhR269ZQ1x6nYviZrspEHOyNUt7FTp3N6+XneW1ERc2xiwWeCbXWMASauCOoTOBUxpGOk1fpg9TyZSqbEkIWejdbL8OnhTiNNUnCCFj5X4F4S6AAH4cmLQbO0OjHvIpyjiviGqdTZevDxhHhPetkCNww4H9xyd4v+AtHc7I= ARC-Message-Signature: i=2; a=rsa-sha256; d=sourceware.org; s=key; t=1785836461; c=relaxed/simple; bh=VG8M6O9SNwKwQart4H4Uj3HLf/9f/rM2rwqSVAX/m1w=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=b4IqiTG0MD8HJsZgDw8XEXgVRSylyJFTaJVuQcvvaiV6+ja8Yp2s94+3WKCquNrnzDC65aRkt9Cmh7OI9E6I/xoPudFGm+1TngVtFYXD3NkXnrtN4KhBqGyjCCqOHMIW/FjYzD4gPSjjX+8awYlF489zEI7ezw5ddzxTz3qCISE= ARC-Authentication-Results: i=2; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=NYTHws3C DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E97E84BB1C29 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=i8snGzFDTdJH91MyYCV/6fYrZSvFyeAXeiOcwWP+67AerxtJcdKaWOR4bBz4WE7i1ShH9YPaTF4sKYGCLPVgriLLSRTF/ljQVCE3bROIzypnu/goKbuhu8BEXR2CAxkAGlhCwbpKcjrbfs/+du0SZ0G7Zxi5Zu6EW80VGW9f0CDi5H1cfeYqQ5BS4MnnNYJ2K0l/WybNA6zNJaII7r1VhyLn/ktGNwK3ujDD/7wp6bnBSGlOvb8cpNBfV8erXNioXYxvF3i3MHfSe4aAB6+7QCQKLIdHbQK/Z81LzruSg59pESrTAVgMcxpoeXkoih1wNimnLC23/Qa2hd638TOaMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rN2pf0YXetb7f+YvektRRwd8Z4ck5VwIrOxi7s6v6M4=; b=toilljz0M7iTjUf/4N/dKvod/i2o14svYgTlpm6F3qmf12AfH9WkXW1OINtbeMueTv2W7Ru3ImCJj/kcczUoDKkBWMTYc4LSmKg4oh/wMiRrl9XmX5/ZHYWULfYLh1531PUTU7wRVXuPwUDenVYbV61t1/Uz0vrW4333sFCIySdroakqatZPL+qHz5crYOaqH5s8ehG6aZCHLbtLoBPoQbbvOAVFA5469pZ7SeK6I6E5/HQ0HRKXJ7853hDzOQ8O0s7IOKhzlCh2OmGgTkgZwYKhEGwgqcItC4I7qf/+PO5mvmnIrXrgu6VUBU/sjkkJhFJoGKkoyr8yLD58hsVdqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=gcc.gnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rN2pf0YXetb7f+YvektRRwd8Z4ck5VwIrOxi7s6v6M4=; b=NYTHws3CQkOGXk7EFEnaeFuFwYRDkTlaexxwf3YcmseyyRbXvICYaaIzf8Zj3SiY/OJ7diGNReKX9qe93nSVZjnmWxRSSRrMsPEn04V2p+D9szoIdhG2TyRcqQHO6WusY5819mOLK5CLO+CfyeTmgjWmvPQV9Fq8aG72JiRQGPtSa6104c2hPadABeRpxkli4QWILthOw0xfe2fnGB8OTB9zJK9oiR60AGeKx+ILfnTlLUlAwOM3+Iv9UYxQ+tghDCehe22cW+xNFZ9b9u/Sb7JfH4oJ91bUgqu10BhKBVoeyaQg9tTG+AfQTtGiLx0ptC7ZjGQIITJ8oSAdbA7/7w== Received: from BN1PR13CA0015.namprd13.prod.outlook.com (2603:10b6:408:e2::20) by IA0PR12MB7554.namprd12.prod.outlook.com (2603:10b6:208:43e::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.292.15; Tue, 4 Aug 2026 09:40:55 +0000 Received: from BN3PEPF0000B070.namprd21.prod.outlook.com (2603:10b6:408:e2:cafe::82) by BN1PR13CA0015.outlook.office365.com (2603:10b6:408:e2::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.292.15 via Frontend Transport; Tue, 4 Aug 2026 09:40:55 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN3PEPF0000B070.mail.protection.outlook.com (10.167.243.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.315.0 via Frontend Transport; Tue, 4 Aug 2026 09:40:55 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.45; Tue, 4 Aug 2026 02:40:39 -0700 Received: from ktkachov-mlt.nvidia.com (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.20; Tue, 4 Aug 2026 02:40:38 -0700 From: To: CC: Kyrylo Tkachov Subject: [PATCH] match.pd: turn a product of two quotients into one division Date: Tue, 4 Aug 2026 11:40:28 +0200 Message-ID: <20260804094028.54327-1-ktkachov@nvidia.com> X-Mailer: git-send-email 2.50.1 MIME-Version: 1.0 X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B070:EE_|IA0PR12MB7554:EE_ X-MS-Office365-Filtering-Correlation-Id: a3d58a5e-f701-465d-8d54-08def20c7b7d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|36860700016|23010399003|376014|82310400026|1800799024|13003099007|6133799003|56012099006|10067099003|11063799006|5023799004|18002099003; X-Microsoft-Antispam-Message-Info: bvCv9AmzUj5vygTcDcHE4DM1YjIIawpbJB7K02hGIpbO8ZovbFfpAHT6pQnsZlD/oN3x0apO9cylWEbAJfPcGMDgPRs+cSuwDTMxShkH1kaeOS3nN31ntmYCwDEho0B1tk4EOSwAwaHD8vuririGXno0agTJD5xEn60EUfRdwLyksKNZEuaWcUm/3fBlmyXJ7+cDshn/nu8Z77sA8u24OTL905xQNrPB524RIKdb+nqleiw2Eq4TmQMc/2xMSzFwfUPzmydnws8G8KEy69haG/EKn/+/BzZv9iKzFpHU2qA6ADHf0n3E3ERLOysXHHu1lcTshiurApG1RGDo3k/rdohRTVCsSgk+Zn7QIjlu0tWcEJHLp6fy0YHGJ614Mfu01pnLYKpGvqkwwaHhn23tuAIz6gKxSeIoTS62p9grUSeO/1idVaSpNME/XUs0bO6Dn64e9Eb+LhC0GWFUvP4oclIDIb10dEuF0YCnuEJhw1y1h/2qbk3cSXrKhl0zDkjFIzrWd6jEP+VyLfxWobrOW0RSSlw/aOzFFg2M/ZPrk7H6y546G4pbcp9YC2xfW9Ps3YgWt5bSVeJrps5AAbKN1hWqyB0RGpKmHMaIWenQsBum2Km7yG2ekGIHQ4Rtarq499lCmiu1Ye/9z11rSQPX25tamLazrv7F5ox8PvM3H+qs4cy8UwtASHrVnFauzhkg7AYfl5gvLEPrkxx4Ecdiig== X-Forefront-Antispam-Report: CIP:216.228.117.160; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc6edge1.nvidia.com; CAT:NONE; SFS:(13230040)(36860700016)(23010399003)(376014)(82310400026)(1800799024)(13003099007)(6133799003)(56012099006)(10067099003)(11063799006)(5023799004)(18002099003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Xm+o4zj3aeo7SarfQWc68MxZn3qibgPKARE5odyJEYeNnsSovLASM0E+6k3Vk84v8bfwjcRiP07bI6OkkP2EKZGDi13wt8VMuOi0hii2txsj1fzTZT2VyrsI6+n9Ls6SImCgZP2VvvT38NaFSTDll7FbsJLabcJMNsA/NBm/AiKcvcIZhHwRNTLIRqGEtGheHsL4IE1wGFabugYPU/paxTD/pP97wQXnc2o0sTma2eftvz3Cb5Jpqi+XgvO1LFksXaDqVTApkoHbNvfSE4/kGOL4cCLhevUGDndxLemmnmM5+yDVBdujOxAs6d1GkPonhUOeVudyaJMwlXunV6VdBTbADuekKLx7TWiZnjOi3Qx6qBBa5B+j50Sk3vx4Xs7n0xSYAdT2rAFOCx8nymCKbi7ytzYuic6X+UgEcdg6k9ADZsBNQMWck3tthv26cIcZ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Aug 2026 09:40:55.6760 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a3d58a5e-f701-465d-8d54-08def20c7b7d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.117.160]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B070.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7554 X-Spam-Status: No, score=-8.8 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, LOCAL_AUTHENTICATION_FAIL_SPF, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Kyrylo Tkachov (A / B) * (C / D) is (A * C) / (B * D), which replaces one of the two divisions with a multiply. The operation count is unchanged and a division costs several multiplies on every target. double f (double a, double b, double c) { return (a / b) * (1.0 / c); } aarch64 -Ofast before: fdiv d0, d0, d1 fdiv d0, d0, d2 after: fmul d1, d1, d2 fdiv d0, d0, d1 The reciprocal spelling is what appears in source that has been hand-tuned for -freciprocal-math: the reciprocal is folded into a quotient by the existing rules, but the resulting division of a division was never revisited because the multiply had already consumed it. The existing (A/B)/C rule therefore only caught the case where the second division was written out. The rule also needs infinities and NaNs excluded. It forms two products, and each is a new place for the exponent to leave the range: with both divisors large B * D is an infinity and the quotient becomes inf / inf, with both small it is a zero and the quotient becomes 0 / 0, and either turns a finite result into a NaN. The cancellation rules in the same block carry the same test for the same reason. Both quotients are required to be singly used, as a hard condition rather than a :s marker. :s only forbids emitting new statements, so it lets the rule fire whenever the two products happen to fold away, which is exactly what X * X does for one reciprocal square root: there the rule added a division and hid the pattern the recip pass looks for. Requiring two singly used quotients also excludes that case, since a value feeding both operands of a product has two uses. Complex values also require signed zeros to be ignored, because reassociation can change the sign of an imaginary zero. Bootstrapped and tested on aarch64-none-linux-gnu. Ok for trunk? Thanks, Kyrill gcc/ChangeLog: * match.pd ((A / B) * (C / D)): New simplification. gcc/testsuite/ChangeLog: * gcc.dg/tree-ssa/recip-mult-div-1.c: New test. * gcc.dg/tree-ssa/recip-mult-div-2.c: New test. Signed-off-by: Kyrylo Tkachov --- gcc/match.pd | 27 ++++++++++++++ .../gcc.dg/tree-ssa/recip-mult-div-1.c | 35 +++++++++++++++++++ .../gcc.dg/tree-ssa/recip-mult-div-2.c | 15 ++++++++ 3 files changed, 77 insertions(+) create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/recip-mult-div-1.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/recip-mult-div-2.c diff --git a/gcc/match.pd b/gcc/match.pd index 7c93eff6e47..be792e30f22 100644 --- a/gcc/match.pd +++ b/gcc/match.pd @@ -819,6 +819,33 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT) || !HONOR_SIGNED_ZEROS (type))) (rdiv @0 (mult @1 @2)))) + /* Convert (A/B) * (C/D) to (A*C) / (B*D). Two divisions become one + multiply and one division, and a division costs several multiplies on + every target. + + The two products are new places for the exponent to leave the range, so + the rule needs more than the rounding licence: with B and D both large + B * D is an infinity and the quotient becomes inf / inf, and with both + small it is a zero and the quotient becomes 0 / 0. Either way a finite + result turns into a NaN, so the rule is restricted to the case where + infinities and NaNs are excluded, as the cancellation rules above are. + + Both quotients have to be dead outside the product, otherwise a division + would be added rather than removed, and a shared reciprocal is better + left alone for the multiplications to reuse. That is a hard requirement + rather than a :s marker, because :s only forbids emitting new statements + and both products can fold away to nothing, as they do for X * X where X + is one reciprocal square root. Requiring two singly used quotients also + excludes that case, since a value feeding both operands of the product + has two uses. */ + (simplify + (mult (rdiv@4 @0 @1) (rdiv@5 @2 @3)) + (if (!HONOR_NANS (type) && !HONOR_INFINITIES (type) + && (TREE_CODE (type) != COMPLEX_TYPE + || !HONOR_SIGNED_ZEROS (type)) + && single_use (@4) && single_use (@5)) + (rdiv (mult @0 @2) (mult @1 @3)))) + /* Canonicalize x / (C1 * y) to (x * C2) / y. */ (simplify (rdiv @0 (mult:s @1 REAL_CST@2)) diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-mult-div-1.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-mult-div-1.c new file mode 100644 index 00000000000..40d36bd3e57 --- /dev/null +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-mult-div-1.c @@ -0,0 +1,35 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -freciprocal-math -ffinite-math-only -fdump-tree-optimized" } */ + +/* (A / B) * (C / D) is (A * C) / (B * D): one of the two divisions becomes + a multiply. The rule also needs infinities and NaNs excluded, because the + two products it forms can leave the range of the type. */ + +double f1 (double a, double b, double c) +{ + return (a / b) * (1.0 / c); +} + +double f2 (double a, double b, double c, double d) +{ + return (a / b) * (c / d); +} + +float f3 (float a, float b, float c, float d) +{ + return (a / b) * (c / d); +} + +/* Must not fold: the reciprocal is shared, so the multiplications should + reuse it rather than pay for a second division. */ +double keep (double a, double b, double c, double *r) +{ + double t = 1.0 / c; + r[0] = (a / b) * t; + r[1] = t; + return t; +} + +/* Must not fold without -ffinite-math-only: see recip-mult-div-2.c. */ + +/* { dg-final { scan-tree-dump-times " / " 5 "optimized" } } */ diff --git a/gcc/testsuite/gcc.dg/tree-ssa/recip-mult-div-2.c b/gcc/testsuite/gcc.dg/tree-ssa/recip-mult-div-2.c new file mode 100644 index 00000000000..e866d217ee7 --- /dev/null +++ b/gcc/testsuite/gcc.dg/tree-ssa/recip-mult-div-2.c @@ -0,0 +1,15 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -freciprocal-math -fdump-tree-optimized" } */ + +/* (A / B) * (C / D) -> (A * C) / (B * D) forms two products that can leave + the range of the type. With B and D both large B * D is an infinity and + the quotient becomes inf / inf; with both small it is a zero and the + quotient becomes 0 / 0. Either turns a finite result into a NaN, so + -freciprocal-math on its own must not enable the rule. */ + +double f (double a, double b, double c, double d) +{ + return (a / b) * (c / d); +} + +/* { dg-final { scan-tree-dump-times " / " 2 "optimized" } } */