From patchwork Tue Nov 30 13:04:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wilco Dijkstra X-Patchwork-Id: 48279 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 9C9763857C64 for ; Tue, 30 Nov 2021 13:05:05 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9C9763857C64 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1638277505; bh=ba/mqGObzM3oBB6ukWS95nMHpKycgKGTgzhorCD9jAg=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=hKMOBKGiQH5CTpsksrGSu/I1pXHgMO6xbrqlOqi0/2g8tT5Y2U2FCCGDp1A6vatXM ihXfHXgMrWPQIUf8YkQDug8ckk5Tbwj910xM6BcLezMJoDD4TqewlplOu5JXKb0lny lx95lXhbyG15lR7eAXHrSAc5c4Tf6g392LmdZD+M= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2074.outbound.protection.outlook.com [40.107.21.74]) by sourceware.org (Postfix) with ESMTPS id A5BB53858D28 for ; Tue, 30 Nov 2021 13:04:41 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org A5BB53858D28 Received: from AS9PR06CA0308.eurprd06.prod.outlook.com (2603:10a6:20b:45b::28) by AM9PR08MB6740.eurprd08.prod.outlook.com (2603:10a6:20b:305::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.23; Tue, 30 Nov 2021 13:04:38 +0000 Received: from AM5EUR03FT021.eop-EUR03.prod.protection.outlook.com (2603:10a6:20b:45b:cafe::75) by AS9PR06CA0308.outlook.office365.com (2603:10a6:20b:45b::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.22 via Frontend Transport; Tue, 30 Nov 2021 13:04:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM5EUR03FT021.mail.protection.outlook.com (10.152.16.105) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.20 via Frontend Transport; Tue, 30 Nov 2021 13:04:38 +0000 Received: ("Tessian outbound c61f076cbd30:v110"); Tue, 30 Nov 2021 13:04:38 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: d5f0c40b4b3fce5b X-CR-MTA-TID: 64aa7808 Received: from 4e6990516b17.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 17F279AF-CA06-429A-94D3-9EED61DCC407.1; Tue, 30 Nov 2021 13:04:31 +0000 Received: from EUR04-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 4e6990516b17.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Tue, 30 Nov 2021 13:04:31 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iNixQYtLvrIK1sY/UMwb3gLfAX0y0X36svqbA9vXHcEKnCjz0gGoEI27bVKHfu7lPWKKsptJRm64UbS8RnQZbY1yY3K+mJmWVgrcggM3ElYo3sOMWlK2nzO7qDeNFodlyYaVrkt0fYcugrZKmA5YwQsx+hHpv0KNGDA1cYDXPH4MQv1ftXGvKM4dENjmwlRn2oe5+do1QE331cf4E/GkC3iSkwSDEmASKMPz+yff93EVXiEVXBC3aIpFvnopHYhOSsUJ3YWxiwj7v3vnuWAvhCyG+BIWNkQyAgfBklAZSOy9JHL7nrX3aqy8RajKNPB/dBNE8R7eeCr/KzYBSHI/1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ba/mqGObzM3oBB6ukWS95nMHpKycgKGTgzhorCD9jAg=; b=XM/rMaeljKUoIqp9eg8edooQtlBzu48qFf2XULo/HUg+8epof9WciLXErfKtMnyhU/kZIX4PLUWK4zviKGAk81sc5AH/XodXpbf2i2nvjXiWRrXN9fAi0anA4017fFWobYSYMl+SWFsJ+/aV5zhnLyIjCVOY2yylw/tNDwFsOrupH3u2l7P5rvK0NiX+LMpE699AIBrsa8s/C43jT1/7rPUR8QeOCddAOU7z6hpNVkJCAUJQ8D4QmCV47xR0qQvyYoESpbAoeUipeJkDOGl5HLVyB3CiSpn4edlhYT1Hp4AZ3A+/s/57n2sHRHd7IHd3jZJVXfVLl9vMEPN8+6DFIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none Received: from VE1PR08MB5599.eurprd08.prod.outlook.com (2603:10a6:800:1a1::12) by VE1PR08MB5168.eurprd08.prod.outlook.com (2603:10a6:803:106::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.22; Tue, 30 Nov 2021 13:04:27 +0000 Received: from VE1PR08MB5599.eurprd08.prod.outlook.com ([fe80::f5b6:af99:142b:7a12]) by VE1PR08MB5599.eurprd08.prod.outlook.com ([fe80::f5b6:af99:142b:7a12%4]) with mapi id 15.20.4734.024; Tue, 30 Nov 2021 13:04:27 +0000 To: 'GNU C Library' Subject: [PATCH] Improve hypot performance Thread-Topic: [PATCH] Improve hypot performance Thread-Index: AQHX5eo7NT0d69TYpkGyTv+B5cMK3A== Date: Tue, 30 Nov 2021 13:04:27 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: f93fc1b7-7b75-e5c3-298e-3e5b06cb08ee Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-publictraffictype: Email X-MS-Office365-Filtering-Correlation-Id: 580ddf55-83b0-42d5-f6a1-08d9b401f74f x-ms-traffictypediagnostic: VE1PR08MB5168:|AM9PR08MB6740: X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true nodisclaimer: true x-ms-oob-tlc-oobclassifiers: OLM:8882;OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: imBSXZuGwzzT1/u33zA3wYYTft0dXFvO8XeI47eNkY8h8UJIoXHWO5QLTNZ35M+hepqvN4PeBiTUojuroRfTKBqBlyu7/ovn967n4XfvBp9dnp52sQR8PwpV4PHGBIfPOH+vIZtjxV8PmsZb5+ZA6AfOVNgYpfRbF58qkbRIhVNhW50utB9163jBz0q1EavSI3hdqQraHNKoyDNSsRZOhte2akZ/8+5lIxTmERcIZOsVukhrHj37a6jxKKokvmNm9D3ouwlP2kGs1kuvQwZSAFAXpYuoEE9o0289laWdYYrcZ6cs5UY0JC8J2v3dcsg0TICSeTTdR2fPMlXmzYHGd4TMWZfkylRlTReRwuznDE0wfDod8t+Ulxe9fontnyvtHQoHXqdSp97K2qfxyxgmk8qI/AO6ijRrSsqN96frT5sgCYwZCi2MNv+QD1uJ7EBTRKItSIdPV3qkUhf8IXpuBkkaDsLw1LHHBWQ4nPwgnrKQVnzQKpyyfR3B0DgLPlMFInlZpN7HqlDTKpkocZc3KaN1XEDNCPTjcJOY7UedXogtY0C3yJJxpOubbq+zsbsu3NidDZDlIeuioOHc1yz/iqR7li91Xc+b6DuD1/5NWxo2bWv/3uxiQhIsuw7NzcMNea5P9YMILRaEWColyoe3kTzCkkI4d3U9cDgdlBpT+9YvD+tJ1LUvY0ZioA+7ebpRVWtwSVRmKHKSem6rI3UFEzGVLZvOaNHooS1z6OY3GttOf/LhhSjo9+fnLm3QwZAksSq8YrQGV6DsRtUXni7DmaJ3YD9ZV3StbDA2bhJxyoqtEdX5/pAgZlHifQWXIYTChKUTqBf6woV4qE+k/FXVdZWwtoaQ5AK73dt+yLGZNLQ= X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VE1PR08MB5599.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(8676002)(316002)(66446008)(38100700002)(122000001)(71200400001)(33656002)(5660300002)(86362001)(9686003)(186003)(38070700005)(66476007)(83380400001)(52536014)(66556008)(55016003)(2906002)(966005)(6916009)(64756008)(76116006)(508600001)(6506007)(26005)(7696005)(8936002)(66946007)(2004002); DIR:OUT; SFP:1101; MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VE1PR08MB5168 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM5EUR03FT021.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: 2e82b004-9125-4c95-f6b0-08d9b401f0ba X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: swITduZFg8Z7sEJ41kMfvIRaNxZRWgreYAY0/7FJaakibq+c53CNQmahrZESk7sY9D+RKlX7i+Ngc1l6K4t0cceUFkhSUZuhR5KSPDnUQr3UtkWg4gUwFuju8nKMAbOvgrPNPqQMZ4UVzUD51BBxOsvzIiWRTbKT9jDlS5f9RRB+pfhs1lpEwj+GxSBdU7YgeKUoh/I3Qbhz67FbD9WBG9aYY3wDNcICm+RmnYWLOoVvVr7zS+99dCo2HQzY0FVmlmJW/rB14LfBPM0anEnn0NJWiCFN3m+vKo9cp7iXDNdn/Bjfn0QDTmOavKqP3uJ2lf766rjApovLg0te1sSFsLKPRK2AvzNm3OS78muRYc49PZKILtq3jfNMJ/QjnHM6WG9TD5lyA6iseE9RXv+M5mofTAR+bgA9M2+uU6rlelikjE1pZgkNMby1CDQlPYithKFqFpIqiod8+k9rTJBjQbyWkenf4NGj9/ExIIoAQ81K9VAa6rnThUt8HqCPOCsy/3K/wLrHk5xZiCcUMcrO+4ZOMzjvP98cITeWIwlRcraVU5qcNfJR05wNpTDwSRIb9uk2QmASUy/xH641f9wXF7pFXqiAuyT1Hr8+NUq/5oFcWQ0qb9GnbB7cGSmbHlvUF4mOyIn9S8qRE1Lkq3w4rdNE5lWIqR+6PRbK2bvkwcYde2q/3Ozqt78qB+6hLOEHeSK+rbIiNKOEAbk2V0A126IysqetxsPigSehTaAgUFUM6s3Ys1A/U1pQ9Au+Uf+QyYqsC86OHBRetCxssSDEuURLO7n/Mj+Dwsq3wGwYZYfUGzk9enL02vxH1FldNVtzqMi42+4fAtLX2AO+BpP/4g== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(8676002)(2906002)(8936002)(47076005)(36860700001)(7696005)(82310400004)(26005)(336012)(83380400001)(5660300002)(81166007)(508600001)(966005)(6506007)(6916009)(33656002)(52536014)(70586007)(70206006)(316002)(186003)(356005)(55016003)(9686003)(86362001)(2004002); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Nov 2021 13:04:38.5586 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 580ddf55-83b0-42d5-f6a1-08d9b401f74f X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM5EUR03FT021.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9PR08MB6740 X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_ASCII_DIVIDERS, KAM_SHORT, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Wilco Dijkstra via Libc-alpha From: Wilco Dijkstra Reply-To: Wilco Dijkstra Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Improve hypot performance significantly by using fma when available. The fma version has twice the throughput of the previous version and 70% of the latency. The non-fma version has 30% higher throughput and 10% higher latency. Max ULP error is 0.949 with fma and 0.792 without fma. Passes GLIBC testsuite. diff --git a/sysdeps/ieee754/dbl-64/e_hypot.c b/sysdeps/ieee754/dbl-64/e_hypot.c index 9ec4c1ced096160d7360afe2e002b8da81e0cca2..820da4ca6f87f7718510731318a8060705b61ea3 100644 --- a/sysdeps/ieee754/dbl-64/e_hypot.c +++ b/sysdeps/ieee754/dbl-64/e_hypot.c @@ -1,164 +1,147 @@ -/* @(#)e_hypot.c 5.1 93/09/24 */ -/* - * ==================================================== - * Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved. - * - * Developed at SunPro, a Sun Microsystems, Inc. business. - * Permission to use, copy, modify, and distribute this - * software is freely granted, provided that this notice - * is preserved. - * ==================================================== - */ - -/* __ieee754_hypot(x,y) - * - * Method : - * If (assume round-to-nearest) z=x*x+y*y - * has error less than sqrt(2)/2 ulp, than - * sqrt(z) has error less than 1 ulp (exercise). - * - * So, compute sqrt(x*x+y*y) with some care as - * follows to get the error below 1 ulp: - * - * Assume x>y>0; - * (if possible, set rounding to round-to-nearest) - * 1. if x > 2y use - * x1*x1+(y*y+(x2*(x+x1))) for x*x+y*y - * where x1 = x with lower 32 bits cleared, x2 = x-x1; else - * 2. if x <= 2y use - * t1*y1+((x-y)*(x-y)+(t1*y2+t2*y)) - * where t1 = 2x with lower 32 bits cleared, t2 = 2x-t1, - * y1= y with lower 32 bits chopped, y2 = y-y1. - * - * NOTE: scaling may be necessary if some argument is too - * large or too tiny - * - * Special cases: - * hypot(x,y) is INF if x or y is +INF or -INF; else - * hypot(x,y) is NAN if x or y is NAN. - * - * Accuracy: - * hypot(x,y) returns sqrt(x^2+y^2) with error less - * than 1 ulps (units in the last place) - */ +/* Euclidean distance function. Double/Binary64 version. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* Fast ~0.949 ULP hypot implementation using fma and fmin/fmax. + + The error of sqrt (x^2 + y^2) is below 1 ULP if x^2 + y^2 is computed + with less than 0.707 ULP error. If |x| >= |2y|, fma (x, x, y^2) has + ~0.625 ULP. If |x| < |2y|, fma (|2x|, |y|, (x - y)^2) has ~0.625 ULP. + + If fma is not available, the implementation uses a correction based on + 'An Improved Algorithm for hypot(a,b)' by Carlos F. Borges [1] using + the MyHypot3 with the following changes: + + - Handle qNaN and sNaN. + - Tune the 'widely varying operands' to avoid spurious underflow + due the multiplication and fix the return value for upwards + rounding mode. + - Handle required underflow exception for subnormal results. + + [1] https://arxiv.org/pdf/1904.09481.pdf */ #include #include #include +#include #include +#include +#include +#include +#include -double -__ieee754_hypot (double x, double y) +#define SCALE 0x1p-600 +#define LARGE_VAL 0x1p+511 +#define TINY_VAL 0x1p-459 +#define EPS 0x1p-54 + +#ifndef FAST_FMINMAX +# define FAST_FMINMAX 0 +#endif + +static inline double +handle_errno (double r) { - double a, b, t1, t2, y1, y2, w; - int32_t j, k, ha, hb; - - GET_HIGH_WORD (ha, x); - ha &= 0x7fffffff; - GET_HIGH_WORD (hb, y); - hb &= 0x7fffffff; - if (hb > ha) - { - a = y; b = x; j = ha; ha = hb; hb = j; - } + r = math_narrow_eval (r); + if (isinf (r)) + __set_errno (ERANGE); + return r; +} + +/* Hypot kernel. The inputs must be adjusted so that ax >= ay >= 0 + and squaring ax, ay and (ax - ay) does not overflow or underflow. */ +static inline double +kernel (double ax, double ay) +{ + double t1, t2; +#ifdef __FP_FAST_FMA + t1 = ay + ay; + t2 = ax - ay; + + if (t1 >= ax) + return sqrt (fma (t1, ax, t2 * t2)); else + return sqrt (fma (ax, ax, ay * ay)); + +#else + double h = sqrt (ax * ax + ay * ay); + + if (h <= 2.0 * ay) { - a = x; b = y; - } - SET_HIGH_WORD (a, ha); /* a <- |a| */ - SET_HIGH_WORD (b, hb); /* b <- |b| */ - if ((ha - hb) > 0x3c00000) - { - return a + b; - } /* x/y > 2**60 */ - k = 0; - if (__glibc_unlikely (ha > 0x5f300000)) /* a>2**500 */ - { - if (ha >= 0x7ff00000) /* Inf or NaN */ - { - uint32_t low; - w = a + b; /* for sNaN */ - if (issignaling (a) || issignaling (b)) - return w; - GET_LOW_WORD (low, a); - if (((ha & 0xfffff) | low) == 0) - w = a; - GET_LOW_WORD (low, b); - if (((hb ^ 0x7ff00000) | low) == 0) - w = b; - return w; - } - /* scale a and b by 2**-600 */ - ha -= 0x25800000; hb -= 0x25800000; k += 600; - SET_HIGH_WORD (a, ha); - SET_HIGH_WORD (b, hb); + double delta = h - ay; + t1 = ax * (2.0 * delta - ax); + t2 = (delta - 2.0 * (ax - ay)) * delta; } - if (__builtin_expect (hb < 0x23d00000, 0)) /* b < 2**-450 */ + else { - if (hb <= 0x000fffff) /* subnormal b or 0 */ - { - uint32_t low; - GET_LOW_WORD (low, b); - if ((hb | low) == 0) - return a; - t1 = 0; - SET_HIGH_WORD (t1, 0x7fd00000); /* t1=2^1022 */ - b *= t1; - a *= t1; - k -= 1022; - GET_HIGH_WORD (ha, a); - GET_HIGH_WORD (hb, b); - if (hb > ha) - { - t1 = a; - a = b; - b = t1; - j = ha; - ha = hb; - hb = j; - } - } - else /* scale a and b by 2^600 */ - { - ha += 0x25800000; /* a *= 2^600 */ - hb += 0x25800000; /* b *= 2^600 */ - k -= 600; - SET_HIGH_WORD (a, ha); - SET_HIGH_WORD (b, hb); - } + double delta = h - ax; + t1 = 2.0 * delta * (ax - 2.0 * ay); + t2 = (4.0 * delta - ay) * ay + delta * delta; } - /* medium size a and b */ - w = a - b; - if (w > b) + + h -= (t1 + t2) / (2.0 * h); + return h; +#endif +} + + +double +__ieee754_hypot (double x, double y) +{ + if (!isfinite (x) || !isfinite (y)) { - t1 = 0; - SET_HIGH_WORD (t1, ha); - t2 = a - t1; - w = sqrt (t1 * t1 - (b * (-b) - t2 * (a + t1))); + if ((isinf (x) || isinf (y)) + && !issignaling_inline (x) && !issignaling_inline (y)) + return INFINITY; + return x + y; } - else + + x = fabs (x); + y = fabs (y); + + /* Ensure ax >= ay >= 0 without branches if possible. */ + double ax = FAST_FMINMAX ? fmax (x, y) : (x < y ? y : x); + double ay = FAST_FMINMAX ? fmin (x, y) : (x < y ? x : y); + + /* If ax is huge, scale both inputs down. */ + if (__glibc_unlikely (ax > LARGE_VAL)) { - a = a + a; - y1 = 0; - SET_HIGH_WORD (y1, hb); - y2 = b - y1; - t1 = 0; - SET_HIGH_WORD (t1, ha + 0x00100000); - t2 = a - t1; - w = sqrt (t1 * y1 - (w * (-w) - (t1 * y2 + t2 * b))); + if (__glibc_unlikely (ay <= ax * EPS)) + return handle_errno (ax + ay); + + return handle_errno (kernel (ax * SCALE, ay * SCALE) / SCALE); } - if (k != 0) + + /* If ay is tiny, scale both inputs up. */ + if (__glibc_unlikely (ay < TINY_VAL)) { - uint32_t high; - t1 = 1.0; - GET_HIGH_WORD (high, t1); - SET_HIGH_WORD (t1, high + (k << 20)); - w *= t1; - math_check_force_underflow_nonneg (w); - return w; + if (__glibc_unlikely (ax >= ay / EPS)) + return math_narrow_eval (ax + ay); + + ax = math_narrow_eval (kernel (ax / SCALE, ay / SCALE) * SCALE); + math_check_force_underflow_nonneg (ax); + return ax; } - else - return w; + + /* Common case: ax is not huge and ay is not tiny. */ + if (__glibc_unlikely (ay <= ax * EPS)) + return math_narrow_eval (ax + ay); + + return math_narrow_eval (kernel (ax, ay)); } #ifndef __ieee754_hypot libm_alias_finite (__ieee754_hypot, __hypot)