diff mbox

Optimized generic expf and exp2f

Message ID 59AED6DA.6000509@arm.com
State New, archived
Headers show

Commit Message

Szabolcs Nagy Sept. 5, 2017, 4:54 p.m. UTC
Based on new expf and exp2f code from
https://github.com/ARM-software/optimized-routines/

with https://sourceware.org/ml/libc-alpha/2017-08/msg01126.html
expf reciprocal-throughput: 3.3x faster
expf latency: 1.7x faster
libm.so size: -8byte data, -2140byte text
expf/exp2f worst case nearest rounding ulp error: 0.502

Error checks are inline and actual error handling is
in separate functions that are tail called and expected
to be reusable when other math functions are implemented
without wrappers. (expf, __expf, __ieee754_expf symbols
are aliases, _LIB_VERSION is not checked, errno is set
unconditionally according to POSIX rules.)

Double precision arithmetics is used which is expected
to be faster on most targets (including soft-float) than
using single precision and it is easier to get good
precision result with it.

Const data is kept in a separate translation unit which
complicates maintenance a bit, but is expected to give
good code for literal loads on most targets and allows
sharing data across expf, exp2f and powf.

Some configuration is in a new math_config.h which is
currently kept similar to the one in optimized-routines
repo which had to be self-contained, I expect that this
will need more adjustments for glibc.

Some details may need target specific tweaks:
- best convert and round to int operation in the arg
reduction may be different across targets.
- code was optimized on fma target, optimal polynomial
eval may be different without fma.
- gcc does not always generate good code for fp bit
representation access via unions or it may be inherently
slow on some target.

One issue with the argument reduction is that it only
works right with nearest rounded rint, otherwise the
interval is [0,2c] or [-2c,0] instead of [-c,c]. The
polynomial is optimized for [-c,c] but it has sufficent
extra precision that it gives acceptable results on
[-2c,2c] too assuming users are less interested in
non-nearest rounded precision, however this means some
glibc ulp error limits will need adjustment.

Another issue is with the polynomial evaluation which
causes 1 ulp errors for tiny inputs in some non-nearest
rounding, but i think it is worth the trade off for
better pipelined polynomial.


2017-09-05  Szabolcs Nagy  <szabolcs.nagy@arm.com>

	* math/Makefile (type-float-routines): Add math_errf and e_exp2f_data.
	* sysdeps/ieee754/flt-32/w_expf_compat.c: Move to...
	* math/w_expf_compat.c: ... here.
	* sysdeps/aarch64/fpu/math_private.h (TOINT_INTRINSICS): Define.
	(roundtoint, converttoint): Likewise.
	* sysdeps/ieee754/flt-32/e_expf.c: New implementation.
	* sysdeps/ieee754/flt-32/e_exp2f.c: New implementation.
	* sysdeps/ieee754/flt-32/e_exp2f_data.c: New file.
	* sysdeps/ieee754/flt-32/math_config.h: New file.
	* sysdeps/ieee754/flt-32/math_errf.c: New file.
	* sysdeps/ieee754/flt-32/w_exp2f_compat.c: New file.
	* sysdeps/x86_64/fpu/w_expf_compat.c: New file.

Comments

Szabolcs Nagy Sept. 5, 2017, 5:17 p.m. UTC | #1
On 05/09/17 17:54, Szabolcs Nagy wrote:
> Based on new expf and exp2f code from
> https://github.com/ARM-software/optimized-routines/
> 
> with https://sourceware.org/ml/libc-alpha/2017-08/msg01126.html
> expf reciprocal-throughput: 3.3x faster
> expf latency: 1.7x faster

forgot to mention that this is on an aarch64 cpu
on an x86_64 cpu i measure

expf reciprocal-throughput: 1.7x faster than current asm
expf latency: 1.5x faster than current asm
Arjan van de Ven Sept. 5, 2017, 7:32 p.m. UTC | #2
On 9/5/2017 9:54 AM, Szabolcs Nagy wrote:
> Based on new expf and exp2f code from
> https://github.com/ARM-software/optimized-routines/
> 
> with https://sourceware.org/ml/libc-alpha/2017-08/msg01126.html
> expf reciprocal-throughput: 3.3x faster
> expf latency: 1.7x faster
> libm.so size: -8byte data, -2140byte text
> expf/exp2f worst case nearest rounding ulp error: 0.502
> 


you mentioned x86 data.. is that based on current git after
the recent optimizations (on a cpu with fma)?
Joseph Myers Sept. 5, 2017, 8:45 p.m. UTC | #3
On Tue, 5 Sep 2017, Szabolcs Nagy wrote:

> Error checks are inline and actual error handling is
> in separate functions that are tail called and expected
> to be reusable when other math functions are implemented
> without wrappers. (expf, __expf, __ieee754_expf symbols
> are aliases, _LIB_VERSION is not checked, errno is set
> unconditionally according to POSIX rules.)

The _LIB_VERSION and matherr handling is part of the ABI for the existing 
symbol versions of expf and exp2f.  Thus, if you wish to use integrated 
error handling that does not handle _LIB_VERSION and matherr the same way 
as the existing code, you need a new symbol version (with the old version 
then probably keeping the existing wrapper but having it wrap the new 
implementation - duplicate errno setting for existing binaries should be 
fine).

> One issue with the argument reduction is that it only
> works right with nearest rounded rint, otherwise the
> interval is [0,2c] or [-2c,0] instead of [-c,c]. The
> polynomial is optimized for [-c,c] but it has sufficent
> extra precision that it gives acceptable results on
> [-2c,2c] too assuming users are less interested in
> non-nearest rounded precision, however this means some
> glibc ulp error limits will need adjustment.

By ulp error limits do you mean the entries in libm-test-ulps, or the 
global max_valid_error limit in libm-test-support.c which no 
libm-test-ulps entry is allowed to exceed?

> 	* sysdeps/ieee754/flt-32/w_expf_compat.c: Move to...
> 	* math/w_expf_compat.c: ... here.

I'd expect all the w_exp{,f,l}_compat.c files to move at the same time 
(modulo probably needing to add an ldbl-opt version to deal with the long 
double versioning in the ldbl-64-128 and ldbl-128ibm versions).  I think 
they are all in fact generic and are only in sysdeps directories because 
older versions of them used to hardcode bounds on arguments to the exp 
functions that gave finite nonzero results for a particular format.

> 	* sysdeps/x86_64/fpu/w_expf_compat.c: New file.

Is this something to do with x86_64 having its own expf implementations?

It seems i386, ia64, m68k, powerpc64 and x86_64 all have their own 
implementations of expf, exp2f or both (sometimes multiarch, sometimes the 
multiarch variants may have a fallback to using the generic C version, or 
using it built with particular options).  I'd expect an expf replacement 
patch like this one to explain explicitly how it affects those 
architectures.  If on any architecture the answer is that the new C 
versions are not used at all, then I'd expect that architecture to get its 
own dummy versions of math_errf.c and e_exp2f_data.c to avoid building 
unused code into libm.  Of course, if an architecture uses either the 
generic expf or the generic exp2f under any circumstances, it needs the 
relevant support code built into libm when there is code referencing it in 
libm.

Now, I do not make assertions about the performance merits of any of those 
architecture-specific variants; it's entirely possible that some of them 
should be removed if their performance is inferior to the performance of 
this C version (once the C version has been appropriately configured for 
that architecture) (or replaced by building it with specific options to 
generate multiarch variants, if that is beneficial on a particular 
architecture).

> +  if (__builtin_expect (abstop >= top12 (128.0f), 0))

We use __glibc_unlikely in glibc.

> +  if (__builtin_expect (abstop >= top12 (88.0f), 0))

Likewise.

> +#ifndef WANT_ROUNDING
> +/* Correct special case results in non-nearest rounding modes.  */
> +#define WANT_ROUNDING 1
> +#endif

glibc style uses "# define" inside #if, etc.

> +static inline int
> +ieee_2008_issignaling (float x)
> +{
> +  uint32_t ix = asuint (x);
> +  ix ^= 0x00400000; /* IEEE 754-2008 snan bit.  */
> +  return 2 * ix > 2u * 0x7fc00000;
> +}

This doesn't seem to be used, but if you need issignaling tests in future 
functions (powf?), you need to respect HIGH_ORDER_BIT_IS_SET_FOR_SNAN from 
nan-high-order-bit.h.

> +#ifdef __GNUC__
> +#define HIDDEN __attribute__ ((__visibility__ ("hidden")))
> +#define NOINLINE __attribute__ ((noinline))

We don't generally want __GNUC__ conditionals in glibc code, unless it's 
actually shared verbatim with an external repository such as gnulib, or 
it's an installed header.  So attribute_hidden can be used 
unconditionally.
Joseph Myers Sept. 5, 2017, 8:58 p.m. UTC | #4
On Tue, 5 Sep 2017, Arjan van de Ven wrote:

> you mentioned x86 data.. is that based on current git after
> the recent optimizations (on a cpu with fma)?

Really you need to compare with both the fma and non-fma versions (and 
compare the C version built both with and without fma, since one 
possibility would be that the C version can replace the x86_64 ones but 
should be built twice, with and without fma, to achieve that replacement).
Szabolcs Nagy Sept. 6, 2017, 9:23 a.m. UTC | #5
On 05/09/17 21:58, Joseph Myers wrote:
> On Tue, 5 Sep 2017, Arjan van de Ven wrote:
> 
>> you mentioned x86 data.. is that based on current git after
>> the recent optimizations (on a cpu with fma)?
> 
> Really you need to compare with both the fma and non-fma versions (and 
> compare the C version built both with and without fma, since one 
> possibility would be that the C version can replace the x86_64 ones but 
> should be built twice, with and without fma, to achieve that replacement).
> 

i don't have a machine with fma, i tested it on
an older cpu (but i did test against git tip,
using gcc-7).
Gabriel F. T. Gomes Sept. 7, 2017, 1:03 p.m. UTC | #6
On Tue, 5 Sep 2017 20:45:44 +0000
Joseph Myers <joseph@codesourcery.com> wrote:

>On Tue, 5 Sep 2017, Szabolcs Nagy wrote:
>
>> +static inline int
>> +ieee_2008_issignaling (float x)
>> +{
>> +  uint32_t ix = asuint (x);
>> +  ix ^= 0x00400000; /* IEEE 754-2008 snan bit.  */
>> +  return 2 * ix > 2u * 0x7fc00000;
>> +}  
>
>This doesn't seem to be used, but if you need issignaling tests in future 
>functions (powf?), you need to respect HIGH_ORDER_BIT_IS_SET_FOR_SNAN from 
>nan-high-order-bit.h.

Is that also valid for _Float128 (meaning that test-math-issignaling.cc
needs to check for that, as well)?  It was my understanding that it isn't.
Joseph Myers Sept. 7, 2017, 2:48 p.m. UTC | #7
On Thu, 7 Sep 2017, Gabriel F. T. Gomes wrote:

> On Tue, 5 Sep 2017 20:45:44 +0000
> Joseph Myers <joseph@codesourcery.com> wrote:
> 
> >On Tue, 5 Sep 2017, Szabolcs Nagy wrote:
> >
> >> +static inline int
> >> +ieee_2008_issignaling (float x)
> >> +{
> >> +  uint32_t ix = asuint (x);
> >> +  ix ^= 0x00400000; /* IEEE 754-2008 snan bit.  */
> >> +  return 2 * ix > 2u * 0x7fc00000;
> >> +}  
> >
> >This doesn't seem to be used, but if you need issignaling tests in future 
> >functions (powf?), you need to respect HIGH_ORDER_BIT_IS_SET_FOR_SNAN from 
> >nan-high-order-bit.h.
> 
> Is that also valid for _Float128 (meaning that test-math-issignaling.cc
> needs to check for that, as well)?  It was my understanding that it isn't.

This applies just as much to _Float128.  test-math-issignaling.cc only 
does _Float128 tests in the __HAVE_DISTINCT_FLOAT128 case, which does not 
apply to any architectures using the other NaN convention, so there is no 
problem with that test, but once we support _FloatN/_FloatNx types that 
are ABI-aliases of other types, MIPS64 _Float128 / _Float64x will follow 
the same NaN convention as long double that those types alias (with which 
convention that is depending on the compiler configuration).
Szabolcs Nagy Sept. 11, 2017, 5:25 p.m. UTC | #8
On 05/09/17 21:45, Joseph Myers wrote:
> On Tue, 5 Sep 2017, Szabolcs Nagy wrote:
> 
>> Error checks are inline and actual error handling is
>> in separate functions that are tail called and expected
>> to be reusable when other math functions are implemented
>> without wrappers. (expf, __expf, __ieee754_expf symbols
>> are aliases, _LIB_VERSION is not checked, errno is set
>> unconditionally according to POSIX rules.)
> 
> The _LIB_VERSION and matherr handling is part of the ABI for the existing 
> symbol versions of expf and exp2f.  Thus, if you wish to use integrated 
> error handling that does not handle _LIB_VERSION and matherr the same way 
> as the existing code, you need a new symbol version (with the old version 
> then probably keeping the existing wrapper but having it wrap the new 
> implementation - duplicate errno setting for existing binaries should be 
> fine).
> 

i tried to do this but it became complicated dealing with
targets that have their own implementation.

so my plan now is to first post the new code with the
wrapper around it (so there is redundant errno setting),
then in a separate patch try to bump the symbol version
and move to errno-only-wrapper (on all targets for expf
and exp2f) and then finally remove the wrapper for the
new generic code.

it is not yet clear to me how to do the errno-only-wrapper,
since the existing wrapper-template machinery does not
work on a per function basis, but that code would be nice
to reuse, some guidance on that would be helpful.

>> One issue with the argument reduction is that it only
>> works right with nearest rounded rint, otherwise the
>> interval is [0,2c] or [-2c,0] instead of [-c,c]. The
>> polynomial is optimized for [-c,c] but it has sufficent
>> extra precision that it gives acceptable results on
>> [-2c,2c] too assuming users are less interested in
>> non-nearest rounded precision, however this means some
>> glibc ulp error limits will need adjustment.
> 
> By ulp error limits do you mean the entries in libm-test-ulps, or the 
> global max_valid_error limit in libm-test-support.c which no 
> libm-test-ulps entry is allowed to exceed?
> 

limb-test-ulps.

expf failures on x86_64 (no-fma, lrint using +-shift):

testing float (without inline functions)
Failure: Test: exp_downward (-0x1p-20)
Result:
 is:          9.99998986e-01   0x1.ffffdep-1
 should be:   9.99999046e-01   0x1.ffffe0p-1
 difference:  5.96046447e-08   0x1.000000p-24
 ulp       :  1.0000
 max.ulp   :  0.0000
Failure: Test: exp_downward (0x5.8b90b8p+4)
Result:
 is:          3.40279831e+38   0x1.ffff06p+127
 should be:   3.40279851e+38   0x1.ffff08p+127
 difference:  2.02824096e+31   0x1.000000p+104
 ulp       :  1.0000
 max.ulp   :  0.0000
Maximal error of `exp_downward'
 is      : 1 ulp
 accepted: 0 ulp
Failure: Test: exp_towardzero (-0x1p-20)
Result:
 is:          9.99998986e-01   0x1.ffffdep-1
 should be:   9.99999046e-01   0x1.ffffe0p-1
 difference:  5.96046447e-08   0x1.000000p-24
 ulp       :  1.0000
 max.ulp   :  0.0000
Failure: Test: exp_towardzero (0x5.8b90b8p+4)
Result:
 is:          3.40279831e+38   0x1.ffff06p+127
 should be:   3.40279851e+38   0x1.ffff08p+127
 difference:  2.02824096e+31   0x1.000000p+104
 ulp       :  1.0000
 max.ulp   :  0.0000
Maximal error of `exp_towardzero'
 is      : 1 ulp
 accepted: 0 ulp
Joseph Myers Sept. 11, 2017, 6:05 p.m. UTC | #9
On Mon, 11 Sep 2017, Szabolcs Nagy wrote:

> it is not yet clear to me how to do the errno-only-wrapper,
> since the existing wrapper-template machinery does not
> work on a per function basis, but that code would be nice
> to reuse, some guidance on that would be helpful.

I'd expect something like

#include <math-type-macros-float.h>
#undef __USE_WRAPPER_TEMPLATE
#define __USE_WRAPPER_TEMPLATE 1
#include <w_exp_template.c>

should work.  (Modulo possibly needing to do something extra to give expf 
the right symbol version.)  A sysdeps w_expf.c should always override the 
default automatic generation based on the template.
diff mbox

Patch

diff --git a/math/Makefile b/math/Makefile
index 0601f3ac43..04586156f8 100644
--- a/math/Makefile
+++ b/math/Makefile
@@ -115,7 +115,7 @@  type-double-routines := branred doasin dosincos halfulp mpa mpatan2	\
 
 # float support
 type-float-suffix := f
-type-float-routines := k_rem_pio2f
+type-float-routines := k_rem_pio2f math_errf e_exp2f_data
 
 # _Float128 support
 type-float128-suffix := f128
diff --git a/math/w_expf_compat.c b/math/w_expf_compat.c
new file mode 100644
index 0000000000..8a1fa51e46
--- /dev/null
+++ b/math/w_expf_compat.c
@@ -0,0 +1,35 @@ 
+/* Copyright (C) 2011-2017 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+   Contributed by Ulrich Drepper <drepper@gmail.com>, 2011.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#include <math.h>
+#include <math_private.h>
+#include <math-svid-compat.h>
+
+/* wrapper expf */
+float
+__expf (float x)
+{
+  float z = __ieee754_expf (x);
+  if (__builtin_expect (!isfinite (z) || z == 0, 0)
+      && isfinite (x) && _LIB_VERSION != _IEEE_)
+    return __kernel_standard_f (x, x, 106 + !!signbit (x));
+
+  return z;
+}
+hidden_def (__expf)
+weak_alias (__expf, expf)
diff --git a/sysdeps/aarch64/fpu/math_private.h b/sysdeps/aarch64/fpu/math_private.h
index 807111ea5a..775237ffb2 100644
--- a/sysdeps/aarch64/fpu/math_private.h
+++ b/sysdeps/aarch64/fpu/math_private.h
@@ -319,6 +319,26 @@  libc_feresetround_noex_aarch64_ctx (struct rm_ctx *ctx)
 #define libc_feresetround_noexf_ctx	libc_feresetround_noex_aarch64_ctx
 #define libc_feresetround_noexl_ctx	libc_feresetround_noex_aarch64_ctx
 
+/* Hack: only include the large arm_neon.h when needed.  */
+#ifdef _MATH_CONFIG_H
+#include <arm_neon.h>
+
+/* ACLE intrinsics for frintn and fcvtns instructions.  */
+#define TOINT_INTRINSICS 1
+
+static inline double_t
+roundtoint (double_t x)
+{
+  return vget_lane_f64 (vrndn_f64 (vld1_f64 (&x)), 0);
+}
+
+static inline uint64_t
+converttoint (double_t x)
+{
+  return vcvtnd_s64_f64 (x);
+}
+#endif
+
 #include_next <math_private.h>
 
 #endif
diff --git a/sysdeps/ieee754/flt-32/e_exp2f.c b/sysdeps/ieee754/flt-32/e_exp2f.c
index 567d3ff6d0..dfadd85f93 100644
--- a/sysdeps/ieee754/flt-32/e_exp2f.c
+++ b/sysdeps/ieee754/flt-32/e_exp2f.c
@@ -1,7 +1,6 @@ 
-/* Single-precision floating point 2^x.
-   Copyright (C) 1997-2017 Free Software Foundation, Inc.
+/* Single-precision 2^x function.
+   Copyright (C) 2017 Free Software Foundation, Inc.
    This file is part of the GNU C Library.
-   Contributed by Geoffrey Keating <geoffk@ozemail.com.au>
 
    The GNU C Library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Lesser General Public
@@ -17,116 +16,75 @@ 
    License along with the GNU C Library; if not, see
    <http://www.gnu.org/licenses/>.  */
 
-/* The basic design here is from
-   Shmuel Gal and Boris Bachelis, "An Accurate Elementary Mathematical
-   Library for the IEEE Floating Point Standard", ACM Trans. Math. Soft.,
-   17 (1), March 1991, pp. 26-45.
-   It has been slightly modified to compute 2^x instead of e^x, and for
-   single-precision.
-   */
-#ifndef _GNU_SOURCE
-# define _GNU_SOURCE
-#endif
-#include <stdlib.h>
-#include <float.h>
-#include <ieee754.h>
 #include <math.h>
-#include <fenv.h>
-#include <inttypes.h>
-#include <math_private.h>
-
-#include "t_exp2f.h"
-
-static const float TWOM100 = 7.88860905e-31;
-static const float TWO127 = 1.7014118346e+38;
+#include <stdint.h>
+#include "math_config.h"
+
+/*
+EXP2F_TABLE_BITS = 5
+EXP2F_POLY_ORDER = 3
+
+ULP error: 0.502 (nearest rounding.)
+Relative error: 1.69 * 2^-34 in [-1/64, 1/64] (before rounding.)
+Wrong count: 168353 (all nearest rounding wrong results with fma.)
+Non-nearest ULP error: 1 (rounded ULP error)
+*/
+
+#define N (1 << EXP2F_TABLE_BITS)
+#define T __exp2f_data.tab
+#define C __exp2f_data.poly
+#define SHIFT __exp2f_data.shift_scaled
+
+static inline uint32_t
+top12 (float x)
+{
+  return asuint (x) >> 20;
+}
 
 float
-__ieee754_exp2f (float x)
+__exp2f (float x)
 {
-  static const float himark = (float) FLT_MAX_EXP;
-  static const float lomark = (float) (FLT_MIN_EXP - FLT_MANT_DIG - 1);
-
-  /* Check for usual case.  */
-  if (isless (x, himark) && isgreaterequal (x, lomark))
+  uint32_t abstop;
+  uint64_t ki, t;
+  /* double_t for better performance on targets with FLT_EVAL_METHOD==2.  */
+  double_t kd, xd, z, r, r2, y, s;
+
+  xd = (double_t) x;
+  abstop = top12 (x) & 0x7ff;
+  if (__builtin_expect (abstop >= top12 (128.0f), 0))
     {
-      static const float THREEp14 = 49152.0;
-      int tval, unsafe;
-      float rx, x22, result;
-      union ieee754_float ex2_u, scale_u;
-
-      if (fabsf (x) < FLT_EPSILON / 4.0f)
-	return 1.0f + x;
-
-      {
-	SET_RESTORE_ROUND_NOEXF (FE_TONEAREST);
-
-	/* 1. Argument reduction.
-	   Choose integers ex, -128 <= t < 128, and some real
-	   -1/512 <= x1 <= 1/512 so that
-	   x = ex + t/512 + x1.
-
-	   First, calculate rx = ex + t/256.  */
-	rx = x + THREEp14;
-	rx -= THREEp14;
-	x -= rx;  /* Compute x=x1. */
-	/* Compute tval = (ex*256 + t)+128.
-	   Now, t = (tval mod 256)-128 and ex=tval/256  [that's mod, NOT %;
-	   and /-round-to-nearest not the usual c integer /].  */
-	tval = (int) (rx * 256.0f + 128.0f);
-
-	/* 2. Adjust for accurate table entry.
-	   Find e so that
-	   x = ex + t/256 + e + x2
-	   where -7e-4 < e < 7e-4, and
-	   (float)(2^(t/256+e))
-	   is accurate to one part in 2^-64.  */
-
-	/* 'tval & 255' is the same as 'tval%256' except that it's always
-	   positive.
-	   Compute x = x2.  */
-	x -= __exp2f_deltatable[tval & 255];
-
-	/* 3. Compute ex2 = 2^(t/255+e+ex).  */
-	ex2_u.f = __exp2f_atable[tval & 255];
-	tval >>= 8;
-	/* x2 is an integer multiple of 2^-30; avoid intermediate
-	   underflow from the calculation of x22 * x.  */
-	unsafe = abs(tval) >= -FLT_MIN_EXP - 32;
-	ex2_u.ieee.exponent += tval >> unsafe;
-	scale_u.f = 1.0;
-	scale_u.ieee.exponent += tval - (tval >> unsafe);
-
-	/* 4. Approximate 2^x2 - 1, using a second-degree polynomial,
-	   with maximum error in [-2^-9 - 2^-14, 2^-9 + 2^-14]
-	   less than 1.3e-10.  */
-
-	x22 = (.24022656679f * x + .69314736128f) * ex2_u.f;
-      }
-
-      /* 5. Return (2^x2-1) * 2^(t/512+e+ex) + 2^(t/512+e+ex).  */
-      result = x22 * x + ex2_u.f;
-
-      if (!unsafe)
-	return result;
-      else
-	{
-	  result *= scale_u.f;
-	  math_check_force_underflow_nonneg (result);
-	  return result;
-	}
-    }
-  /* Exceptional cases:  */
-  else if (isless (x, himark))
-    {
-      if (isinf (x))
-	/* e^-inf == 0, with no error.  */
-	return 0;
-      else
-	/* Underflow */
-	return TWOM100 * TWOM100;
+      /* |x| >= 128 or x is nan.  */
+      if (asuint (x) == asuint (-INFINITY))
+	return 0.0f;
+      if (abstop >= top12 (INFINITY))
+	return x + x;
+      if (x > 0.0f)
+	return __math_oflowf (0);
+      if (x <= -150.0f)
+	return __math_uflowf (0);
+#if WANT_ERRNO_UFLOW
+      if (x < -149.0f)
+	return __math_may_uflowf (0);
+#endif
     }
-  else
-    /* Return x, if x is a NaN or Inf; or overflow, otherwise.  */
-    return TWO127*x;
+
+  /* x = k/N + r with r in [-1/(2N), 1/(2N)] and int k.  */
+  kd = (double) (xd + SHIFT); /* Rounding to double precision is required.  */
+  ki = asuint64 (kd);
+  kd -= SHIFT; /* k/N for int k.  */
+  r = xd - kd;
+
+  /* exp2(x) = 2^(k/N) * 2^r ~= s * (C0*r^3 + C1*r^2 + C2*r + 1) */
+  t = T[ki % N];
+  t += ki << (52 - EXP2F_TABLE_BITS);
+  s = asdouble (t);
+  z = C[0] * r + C[1];
+  r2 = r * r;
+  y = C[2] * r + 1;
+  y = z * r2 + y;
+  y = y * s;
+  return (float) y;
 }
-strong_alias (__ieee754_exp2f, __exp2f_finite)
+weak_alias (__exp2f, exp2f)
+strong_alias (__exp2f, __ieee754_exp2f)
+strong_alias (__exp2f, __exp2f_finite)
diff --git a/sysdeps/ieee754/flt-32/e_exp2f_data.c b/sysdeps/ieee754/flt-32/e_exp2f_data.c
new file mode 100644
index 0000000000..390dcae333
--- /dev/null
+++ b/sysdeps/ieee754/flt-32/e_exp2f_data.c
@@ -0,0 +1,44 @@ 
+/* Shared data between expf, exp2f and powf.
+   Copyright (C) 2017 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#include "math_config.h"
+
+#define N (1 << EXP2F_TABLE_BITS)
+
+const struct exp2f_data __exp2f_data = {
+  /* tab[i] = uint(2^(i/N)) - (i << 52-BITS)
+     used for computing 2^(k/N) for an int |k| < 150 N as
+     double(tab[k%N] + (k << 52-BITS)) */
+  .tab = {
+0x3ff0000000000000, 0x3fefd9b0d3158574, 0x3fefb5586cf9890f, 0x3fef9301d0125b51,
+0x3fef72b83c7d517b, 0x3fef54873168b9aa, 0x3fef387a6e756238, 0x3fef1e9df51fdee1,
+0x3fef06fe0a31b715, 0x3feef1a7373aa9cb, 0x3feedea64c123422, 0x3feece086061892d,
+0x3feebfdad5362a27, 0x3feeb42b569d4f82, 0x3feeab07dd485429, 0x3feea47eb03a5585,
+0x3feea09e667f3bcd, 0x3fee9f75e8ec5f74, 0x3feea11473eb0187, 0x3feea589994cce13,
+0x3feeace5422aa0db, 0x3feeb737b0cdc5e5, 0x3feec49182a3f090, 0x3feed503b23e255d,
+0x3feee89f995ad3ad, 0x3feeff76f2fb5e47, 0x3fef199bdd85529c, 0x3fef3720dcef9069,
+0x3fef5818dcfba487, 0x3fef7c97337b9b5f, 0x3fefa4afa2a490da, 0x3fefd0765b6e4540,
+  },
+  .shift_scaled = 0x1.8p+52 / N,
+  .poly = { 0x1.c6af84b912394p-5, 0x1.ebfce50fac4f3p-3, 0x1.62e42ff0c52d6p-1 },
+  .shift = 0x1.8p+52,
+  .invln2_scaled = 0x1.71547652b82fep+0 * N,
+  .poly_scaled = {
+0x1.c6af84b912394p-5/N/N/N, 0x1.ebfce50fac4f3p-3/N/N, 0x1.62e42ff0c52d6p-1/N
+  },
+};
diff --git a/sysdeps/ieee754/flt-32/e_expf.c b/sysdeps/ieee754/flt-32/e_expf.c
index 782072f213..d1eabdae08 100644
--- a/sysdeps/ieee754/flt-32/e_expf.c
+++ b/sysdeps/ieee754/flt-32/e_expf.c
@@ -1,7 +1,6 @@ 
-/* Single-precision floating point e^x.
-   Copyright (C) 1997-2017 Free Software Foundation, Inc.
+/* Single-precision e^x function.
+   Copyright (C) 2017 Free Software Foundation, Inc.
    This file is part of the GNU C Library.
-   Contributed by Geoffrey Keating <geoffk@ozemail.com.au>
 
    The GNU C Library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Lesser General Public
@@ -17,117 +16,90 @@ 
    License along with the GNU C Library; if not, see
    <http://www.gnu.org/licenses/>.  */
 
-/* How this works:
-
-   The input value, x, is written as
-
-   x = n * ln(2) + t/512 + delta[t] + x;
-
-   where:
-   - n is an integer, 127 >= n >= -150;
-   - t is an integer, 177 >= t >= -177
-   - delta is based on a table entry, delta[t] < 2^-28
-   - x is whatever is left, |x| < 2^-10
-
-   Then e^x is approximated as
-
-   e^x = 2^n ( e^(t/512 + delta[t])
-	       + ( e^(t/512 + delta[t])
-		   * ( p(x + delta[t] + n * ln(2)) - delta ) ) )
-
-   where
-   - p(x) is a polynomial approximating e(x)-1;
-   - e^(t/512 + delta[t]) is obtained from a table.
-
-   The table used is the same one as for the double precision version;
-   since we have the table, we might as well use it.
-
-   It turns out to be faster to do calculations in double precision than
-   to perform an 'accurate table method' expf, because of the range reduction
-   overhead (compare exp2f).
-   */
-#include <float.h>
-#include <ieee754.h>
 #include <math.h>
-#include <fenv.h>
-#include <inttypes.h>
-#include <math_private.h>
-
-extern const float __exp_deltatable[178];
-extern const double __exp_atable[355] /* __attribute__((mode(DF))) */;
-
-static const float TWOM100 = 7.88860905e-31;
-static const float TWO127 = 1.7014118346e+38;
+#include <stdint.h>
+#include "math_config.h"
+
+/*
+EXP2F_TABLE_BITS = 5
+EXP2F_POLY_ORDER = 3
+
+ULP error: 0.502 (nearest rounding.)
+Relative error: 1.69 * 2^-34 in [-ln2/64, ln2/64] (before rounding.)
+Wrong count: 170635 (all nearest rounding wrong results with fma.)
+Non-nearest ULP error: 1 (rounded ULP error)
+*/
+
+#define N (1 << EXP2F_TABLE_BITS)
+#define InvLn2N __exp2f_data.invln2_scaled
+#define T __exp2f_data.tab
+#define C __exp2f_data.poly_scaled
+
+static inline uint32_t
+top12 (float x)
+{
+  return asuint (x) >> 20;
+}
 
 float
-__ieee754_expf (float x)
+__expf (float x)
 {
-  static const float himark = 88.72283935546875;
-  static const float lomark = -103.972084045410;
-  /* Check for usual case.  */
-  if (isless (x, himark) && isgreater (x, lomark))
+  uint32_t abstop;
+  uint64_t ki, t;
+  /* double_t for better performance on targets with FLT_EVAL_METHOD==2.  */
+  double_t kd, xd, z, r, r2, y, s;
+
+  xd = (double_t) x;
+  abstop = top12 (x) & 0x7ff;
+  if (__builtin_expect (abstop >= top12 (88.0f), 0))
     {
-      static const float THREEp42 = 13194139533312.0;
-      static const float THREEp22 = 12582912.0;
-      /* 1/ln(2).  */
-#undef M_1_LN2
-      static const float M_1_LN2 = 1.44269502163f;
-      /* ln(2) */
-#undef M_LN2
-      static const double M_LN2 = .6931471805599452862;
-
-      int tval;
-      double x22, t, result, dx;
-      float n, delta;
-      union ieee754_double ex2_u;
-
-      {
-	SET_RESTORE_ROUND_NOEXF (FE_TONEAREST);
-
-	/* Calculate n.  */
-	n = x * M_1_LN2 + THREEp22;
-	n -= THREEp22;
-	dx = x - n*M_LN2;
-
-	/* Calculate t/512.  */
-	t = dx + THREEp42;
-	t -= THREEp42;
-	dx -= t;
-
-	/* Compute tval = t.  */
-	tval = (int) (t * 512.0);
-
-	if (t >= 0)
-	  delta = - __exp_deltatable[tval];
-	else
-	  delta = __exp_deltatable[-tval];
-
-	/* Compute ex2 = 2^n e^(t/512+delta[t]).  */
-	ex2_u.d = __exp_atable[tval+177];
-	ex2_u.ieee.exponent += (int) n;
-
-	/* Approximate e^(dx+delta) - 1, using a second-degree polynomial,
-	   with maximum error in [-2^-10-2^-28,2^-10+2^-28]
-	   less than 5e-11.  */
-	x22 = (0.5000000496709180453 * dx + 1.0000001192102037084) * dx + delta;
-      }
-
-      /* Return result.  */
-      result = x22 * ex2_u.d + ex2_u.d;
-      return (float) result;
+      /* |x| >= 88 or x is nan.  */
+      if (asuint (x) == asuint (-INFINITY))
+	return 0.0f;
+      if (abstop >= top12 (INFINITY))
+	return x + x;
+      if (x > 0x1.62e42ep6f) /* x > log(0x1p128) ~= 88.72 */
+	return __math_oflowf (0);
+      if (x < -0x1.9fe368p6f) /* x < log(0x1p-150) ~= -103.97 */
+	return __math_uflowf (0);
+#if WANT_ERRNO_UFLOW
+      if (x < -0x1.9d1d9ep6f) /* x < log(0x1p-149) ~= -103.28 */
+	return __math_may_uflowf (0);
+#endif
     }
-  /* Exceptional cases:  */
-  else if (isless (x, himark))
-    {
-      if (isinf (x))
-	/* e^-inf == 0, with no error.  */
-	return 0;
-      else
-	/* Underflow */
-	return TWOM100 * TWOM100;
-    }
-  else
-    /* Return x, if x is a NaN or Inf; or overflow, otherwise.  */
-    return TWO127*x;
+
+  /* x*N/Ln2 = k + r with r in [-1/2, 1/2] and int k.  */
+  z = InvLn2N * xd;
+
+  /* Round and convert z to int, the result is in [-150*N, 128*N] and
+     ideally ties-to-even rule is used, otherwise the magnitude of r
+     can be bigger which gives larger approximation error.  */
+#if TOINT_INTRINSICS
+  kd = roundtoint (z);
+  ki = converttoint (z);
+#elif TOINT_RINT
+  kd = rint (z);
+  ki = (long) kd;
+#elif TOINT_SHIFT
+# define SHIFT __exp2f_data.shift
+  kd = (double) (z + SHIFT); /* Rounding to double precision is required.  */
+  ki = asuint64 (kd);
+  kd -= SHIFT;
+#endif
+  r = z - kd;
+
+  /* exp(x) = 2^(k/N) * 2^(r/N) ~= s * (C0*r^3 + C1*r^2 + C2*r + 1) */
+  t = T[ki % N];
+  t += ki << (52 - EXP2F_TABLE_BITS);
+  s = asdouble (t);
+  z = C[0] * r + C[1];
+  r2 = r * r;
+  y = C[2] * r + 1;
+  y = z * r2 + y;
+  y = y * s;
+  return (float) y;
 }
-strong_alias (__ieee754_expf, __expf_finite)
+hidden_def (__expf)
+weak_alias (__expf, expf)
+strong_alias (__expf, __ieee754_expf)
+strong_alias (__expf, __expf_finite)
diff --git a/sysdeps/ieee754/flt-32/math_config.h b/sysdeps/ieee754/flt-32/math_config.h
new file mode 100644
index 0000000000..aaaa90067d
--- /dev/null
+++ b/sysdeps/ieee754/flt-32/math_config.h
@@ -0,0 +1,128 @@ 
+/* Configuration for math routines.
+   Copyright (C) 2017 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#ifndef _MATH_CONFIG_H
+#define _MATH_CONFIG_H
+
+#include <math.h>
+#include <math_private.h>
+#include <stdint.h>
+
+#ifndef WANT_ROUNDING
+/* Correct special case results in non-nearest rounding modes.  */
+#define WANT_ROUNDING 1
+#endif
+#ifndef WANT_ERRNO
+/* Set errno according to ISO C with (math_errhandling & MATH_ERRNO) != 0.  */
+#define WANT_ERRNO 1
+#endif
+#ifndef WANT_ERRNO_UFLOW
+/* Set errno to ERANGE if result underflows to 0 (in all rounding modes).  */
+#define WANT_ERRNO_UFLOW (WANT_ROUNDING && WANT_ERRNO)
+#endif
+
+#ifndef TOINT_INTRINSICS
+#define TOINT_INTRINSICS 0
+#endif
+#ifndef TOINT_RINT
+#define TOINT_RINT 0
+#endif
+#ifndef TOINT_SHIFT
+#define TOINT_SHIFT 1
+#endif
+
+static inline uint32_t
+asuint (float f)
+{
+  union
+  {
+    float f;
+    uint32_t i;
+  } u = {f};
+  return u.i;
+}
+
+static inline float
+asfloat (uint32_t i)
+{
+  union
+  {
+    uint32_t i;
+    float f;
+  } u = {i};
+  return u.f;
+}
+
+static inline uint64_t
+asuint64 (double f)
+{
+  union
+  {
+    double f;
+    uint64_t i;
+  } u = {f};
+  return u.i;
+}
+
+static inline double
+asdouble (uint64_t i)
+{
+  union
+  {
+    uint64_t i;
+    double f;
+  } u = {i};
+  return u.f;
+}
+
+static inline int
+ieee_2008_issignaling (float x)
+{
+  uint32_t ix = asuint (x);
+  ix ^= 0x00400000; /* IEEE 754-2008 snan bit.  */
+  return 2 * ix > 2u * 0x7fc00000;
+}
+
+#ifdef __GNUC__
+#define HIDDEN __attribute__ ((__visibility__ ("hidden")))
+#define NOINLINE __attribute__ ((noinline))
+#else
+#define HIDDEN
+#define NOINLINE
+#endif
+
+HIDDEN float __math_oflowf (unsigned long);
+HIDDEN float __math_uflowf (unsigned long);
+HIDDEN float __math_may_uflowf (unsigned long);
+HIDDEN float __math_divzerof (unsigned long);
+HIDDEN float __math_invalidf (float);
+
+/* Shared between expf, exp2f and powf.  */
+#define EXP2F_TABLE_BITS 5
+#define EXP2F_POLY_ORDER 3
+extern const struct exp2f_data
+{
+  uint64_t tab[1 << EXP2F_TABLE_BITS];
+  double shift_scaled;
+  double poly[EXP2F_POLY_ORDER];
+  double shift;
+  double invln2_scaled;
+  double poly_scaled[EXP2F_POLY_ORDER];
+} __exp2f_data;
+
+#endif
diff --git a/sysdeps/ieee754/flt-32/math_errf.c b/sysdeps/ieee754/flt-32/math_errf.c
new file mode 100644
index 0000000000..11fbdc0b92
--- /dev/null
+++ b/sysdeps/ieee754/flt-32/math_errf.c
@@ -0,0 +1,77 @@ 
+/* Single-precision math error handling.
+   Copyright (C) 2017 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#include "math_config.h"
+
+#if WANT_ERRNO
+#include <errno.h>
+/* NOINLINE reduces code size and avoids making math functions non-leaf
+   when the error handling is inlined.  */
+NOINLINE static float
+with_errnof (float y, int e)
+{
+  errno = e;
+  return y;
+}
+#else
+#define with_errnof(x, e) (x)
+#endif
+
+/* NOINLINE prevents fenv semantics breaking optimizations.  */
+NOINLINE static float
+xflowf (unsigned long sign, float y)
+{
+  y = (sign ? -y : y) * y;
+  return with_errnof (y, ERANGE);
+}
+
+HIDDEN float
+__math_uflowf (unsigned long sign)
+{
+  return xflowf (sign, 0x1p-95f);
+}
+
+#if WANT_ERRNO_UFLOW
+/* Underflows to zero in some non-nearest rounding mode, setting errno
+   is valid even if the result is non-zero, but in the subnormal range.  */
+HIDDEN float
+__math_may_uflowf (unsigned long sign)
+{
+  return xflowf (sign, 0x1.4p-75f);
+}
+#endif
+
+HIDDEN float
+__math_oflowf (unsigned long sign)
+{
+  return xflowf (sign, 0x1p97f);
+}
+
+HIDDEN float
+__math_divzerof (unsigned long sign)
+{
+  float y = 0;
+  return with_errnof ((sign ? -1 : 1) / y, ERANGE);
+}
+
+HIDDEN float
+__math_invalidf (float x)
+{
+  float y = (x - x) / (x - x);
+  return isnan (x) ? y : with_errnof (y, EDOM);
+}
diff --git a/sysdeps/ieee754/flt-32/w_exp2f_compat.c b/sysdeps/ieee754/flt-32/w_exp2f_compat.c
new file mode 100644
index 0000000000..c37de4df43
--- /dev/null
+++ b/sysdeps/ieee754/flt-32/w_exp2f_compat.c
@@ -0,0 +1 @@ 
+/* Empty file to disable the exp2f wrapper.  */
diff --git a/sysdeps/ieee754/flt-32/w_expf_compat.c b/sysdeps/ieee754/flt-32/w_expf_compat.c
index 8a1fa51e46..578cf4d4a7 100644
--- a/sysdeps/ieee754/flt-32/w_expf_compat.c
+++ b/sysdeps/ieee754/flt-32/w_expf_compat.c
@@ -1,35 +1 @@ 
-/* Copyright (C) 2011-2017 Free Software Foundation, Inc.
-   This file is part of the GNU C Library.
-   Contributed by Ulrich Drepper <drepper@gmail.com>, 2011.
-
-   The GNU C Library is free software; you can redistribute it and/or
-   modify it under the terms of the GNU Lesser General Public
-   License as published by the Free Software Foundation; either
-   version 2.1 of the License, or (at your option) any later version.
-
-   The GNU C Library is distributed in the hope that it will be useful,
-   but WITHOUT ANY WARRANTY; without even the implied warranty of
-   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-   Lesser General Public License for more details.
-
-   You should have received a copy of the GNU Lesser General Public
-   License along with the GNU C Library; if not, see
-   <http://www.gnu.org/licenses/>.  */
-
-#include <math.h>
-#include <math_private.h>
-#include <math-svid-compat.h>
-
-/* wrapper expf */
-float
-__expf (float x)
-{
-  float z = __ieee754_expf (x);
-  if (__builtin_expect (!isfinite (z) || z == 0, 0)
-      && isfinite (x) && _LIB_VERSION != _IEEE_)
-    return __kernel_standard_f (x, x, 106 + !!signbit (x));
-
-  return z;
-}
-hidden_def (__expf)
-weak_alias (__expf, expf)
+/* Empty file to disable the expf wrapper.  */
diff --git a/sysdeps/x86_64/fpu/w_expf_compat.c b/sysdeps/x86_64/fpu/w_expf_compat.c
new file mode 100644
index 0000000000..dc454638c3
--- /dev/null
+++ b/sysdeps/x86_64/fpu/w_expf_compat.c
@@ -0,0 +1 @@ 
+#include <math/w_expf_compat.c>