[v4,0/2] std-simd-complex

  Hi gcc-patches mailing list,
Matthias Kretz via Sourceware Forge <forge-bot+mkretz@forge-stage.sourceware.org> has requested that the following forgejo pull request
be published on the mailing list.

Created on: 2026-03-21 13:52:32+00:00
Latest update: 2026-06-03 05:42:52+00:00
Changes: 25 changed files, 3589 additions, 120 deletions
Head revision: mkretz/gcc ref std-simd-complex commit c7901188bf9230a9515487dd4fe682fa91f16518
Base revision: gcc/gcc-TEST ref trunk commit 2ba702c966e3772d3d4e29e4b25beb7ef4dabf73 r17-638-g2ba702c966e377
Merge base: 2ba702c966e3772d3d4e29e4b25beb7ef4dabf73
Full diff url: https://forge.sourceware.org/gcc/gcc-TEST/pulls/145.diff
Discussion:  https://forge.sourceware.org/gcc/gcc-TEST/pulls/145
Requested Reviewers: tkaminsk

This patch set
- refactors std::simd's ABI tags to completely drop `_ScalarAbi<N>`
- adds complex to the vectorizable types
- implements [simd.bit]

Tested on x86_64-linux-gnu, also with unix/-m32/-march=i686


Changed files:
- A: libstdc++-v3/include/bits/simd_bit.h
- A: libstdc++-v3/include/bits/simd_complex.h
- A: libstdc++-v3/include/bits/simd_math.h
- A: libstdc++-v3/testsuite/std/simd/complex_init.h
- A: libstdc++-v3/testsuite/std/simd/simd_bit.cc
- A: libstdc++-v3/testsuite/std/simd/simd_bit_expensive.cc
- M: libstdc++-v3/include/Makefile.am
- M: libstdc++-v3/include/Makefile.in
- M: libstdc++-v3/include/bits/simd_details.h
- M: libstdc++-v3/include/bits/simd_loadstore.h
- M: libstdc++-v3/include/bits/simd_mask.h
- M: libstdc++-v3/include/bits/simd_mask_reductions.h
- M: libstdc++-v3/include/bits/simd_vec.h
- M: libstdc++-v3/include/bits/simd_x86.h
- M: libstdc++-v3/include/bits/vec_ops.h
- M: libstdc++-v3/include/bits/version.def
- M: libstdc++-v3/include/bits/version.h
- M: libstdc++-v3/include/std/simd
- M: libstdc++-v3/testsuite/std/simd/arithmetic.cc
- M: libstdc++-v3/testsuite/std/simd/create_tests.h
- M: libstdc++-v3/testsuite/std/simd/mask2.cc
- M: libstdc++-v3/testsuite/std/simd/stores.cc
- M: libstdc++-v3/testsuite/std/simd/test_setup.h
- M: libstdc++-v3/testsuite/std/simd/traits_common.cc
- M: libstdc++-v3/testsuite/std/simd/traits_impl.cc


Matthias Kretz (2):
  libstdc++: Refactor _ScalarAbi<N> into _Abi<N, N>
  libstdc++: Add std::complex to the [simd] vectorizable types

 libstdc++-v3/include/Makefile.am              |    3 +
 libstdc++-v3/include/Makefile.in              |    3 +
 libstdc++-v3/include/bits/simd_bit.h          |  192 ++
 libstdc++-v3/include/bits/simd_complex.h      | 2321 +++++++++++++++++
 libstdc++-v3/include/bits/simd_details.h      |  222 +-
 libstdc++-v3/include/bits/simd_loadstore.h    |    4 +
 libstdc++-v3/include/bits/simd_mask.h         |  200 +-
 .../include/bits/simd_mask_reductions.h       |    2 +-
 libstdc++-v3/include/bits/simd_math.h         |  125 +
 libstdc++-v3/include/bits/simd_vec.h          |   25 +-
 libstdc++-v3/include/bits/simd_x86.h          |   46 +
 libstdc++-v3/include/bits/vec_ops.h           |   64 +-
 libstdc++-v3/include/bits/version.def         |   11 +
 libstdc++-v3/include/bits/version.h           |    9 +
 libstdc++-v3/include/std/simd                 |    6 +
 libstdc++-v3/testsuite/std/simd/arithmetic.cc |    3 +-
 .../testsuite/std/simd/complex_init.h         |   17 +
 .../testsuite/std/simd/create_tests.h         |    6 +-
 libstdc++-v3/testsuite/std/simd/mask2.cc      |    3 +-
 libstdc++-v3/testsuite/std/simd/simd_bit.cc   |  203 ++
 .../testsuite/std/simd/simd_bit_expensive.cc  |    7 +
 libstdc++-v3/testsuite/std/simd/stores.cc     |    4 +-
 libstdc++-v3/testsuite/std/simd/test_setup.h  |   45 +-
 .../testsuite/std/simd/traits_common.cc       |  129 +
 .../testsuite/std/simd/traits_impl.cc         |   59 +-
 25 files changed, 3589 insertions(+), 120 deletions(-)
 create mode 100644 libstdc++-v3/include/bits/simd_bit.h
 create mode 100644 libstdc++-v3/include/bits/simd_complex.h
 create mode 100644 libstdc++-v3/include/bits/simd_math.h
 create mode 100644 libstdc++-v3/testsuite/std/simd/complex_init.h
 create mode 100644 libstdc++-v3/testsuite/std/simd/simd_bit.cc
 create mode 100644 libstdc++-v3/testsuite/std/simd/simd_bit_expensive.cc

Range-diff against v3:
1:  b5bab8901212 = 1:  b5bab8901212 libstdc++: Refactor _ScalarAbi<N> into _Abi<N, N>
2:  db5398e40d7d ! 2:  c7901188bf92 libstdc++: Add std::complex to the [simd] vectorizable types
    @@ Commit message
                 account.
                 (__value_preserving_convertible_to): Also allow conversion to
                 complex.
    +            (__simd_unsigned_integer): New.
                 (__simd_complex_value_type, __simd_complex): New.
                 * include/bits/simd_loadstore.h (unchecked_load): Use a cast to
                 the complex's value_type for converting loads to __complex_like.
    @@ Commit message
                 * testsuite/std/simd/simd_bit_expensive.cc: New test.
                 * testsuite/std/simd/stores.cc: Guard converting stores from
                 complex.
    -            * testsuite/std/simd/test_setup.h (complex_like): New.
    +            * testsuite/std/simd/test_setup.h (any_type_of): New.
    +            (complex_like): New.
                 (bit_equal): Handle multi-reg and complex arguments.
                 (cx_isinf): New.
                 (equal_with_nan_and_inf_fixup): Handle complex types.
    @@ libstdc++-v3/include/bits/simd_bit.h (new)
     +_GLIBCXX_BEGIN_NAMESPACE_VERSION
     +namespace simd
     +{
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_integral _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr _Vp
     +    byteswap(const _Vp& __v) noexcept
    -+    { return _Vp([&](int __i) { return std::byteswap(__v[__i]); }); }
    ++    {
    ++      if constexpr (sizeof(typename _Vp::value_type) == 1)
    ++	return __v;
    ++      else
    ++	return _Vp([&](int __i) { return std::byteswap(__v[__i]); });
    ++    }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr _Vp
     +    bit_ceil(const _Vp& __v)
    @@ libstdc++-v3/include/bits/simd_bit.h (new)
     +      return _Vp([&](int __i) { return std::bit_ceil(__v[__i]); });
     +    }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr _Vp
     +    bit_floor(const _Vp& __v) noexcept
     +    { return _Vp([&](int __i) { return std::bit_floor(__v[__i]); }); }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr typename _Vp::mask_type
     +    has_single_bit(const _Vp& __v) noexcept
     +    { return typename _Vp::mask_type([&](int __i) { return std::has_single_bit(__v[__i]); }); }
     +
    -+  template<__simd_vec_type _V0, __simd_vec_type _V1>
    ++  template<__simd_unsigned_integer _V0, __simd_integral _V1>
    ++    requires (_V0::size() == _V1::size())
    ++      && (sizeof(typename _V0::value_type) == sizeof(typename _V1::value_type))
     +    [[__gnu__::__always_inline__]]
     +    constexpr _V0
     +    rotl(const _V0& __v, const _V1& __s) noexcept
     +    { return _V0([&](int __i) { return std::rotl(__v[__i], __s[__i]); }); }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr _Vp
     +    rotl(const _Vp& __v, int __s) noexcept
     +    { return _Vp([&](int __i) { return std::rotl(__v[__i], __s); }); }
     +
    -+  template<__simd_vec_type _V0, __simd_vec_type _V1>
    ++  template<__simd_unsigned_integer _V0, __simd_integral _V1>
    ++    requires (_V0::size() == _V1::size())
    ++      && (sizeof(typename _V0::value_type) == sizeof(typename _V1::value_type))
     +    [[__gnu__::__always_inline__]]
     +    constexpr _V0
     +    rotr(const _V0& __v, const _V1& __s) noexcept
     +    { return _V0([&](int __i) { return std::rotr(__v[__i], __s[__i]); }); }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr _Vp
     +    rotr(const _Vp& __v, int __s) noexcept
     +    { return _Vp([&](int __i) { return std::rotr(__v[__i], __s); }); }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>
     +    bit_width(const _Vp& __v) noexcept
     +    {
    -+      return rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>([&](int __i) {
    -+	       return std::bit_width(__v[__i]);
    ++      using _Ip = make_signed_t<typename _Vp::value_type>;
    ++      return rebind_t<_Ip, _Vp>([&](int __i) {
    ++	       return static_cast<_Ip>(std::bit_width(__v[__i]));
     +	     });
     +    }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>
     +    countl_zero(const _Vp& __v) noexcept
     +    {
    -+      return rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>([&](int __i) {
    -+	       return std::countl_zero(__v[__i]);
    ++      using _Ip = make_signed_t<typename _Vp::value_type>;
    ++      return rebind_t<_Ip, _Vp>([&](int __i) {
    ++	       return static_cast<_Ip>(std::countl_zero(__v[__i]));
     +	     });
     +    }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>
     +    countl_one(const _Vp& __v) noexcept
     +    {
    -+      return rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>([&](int __i) {
    -+	       return std::countl_one(__v[__i]);
    ++      using _Ip = make_signed_t<typename _Vp::value_type>;
    ++      return rebind_t<_Ip, _Vp>([&](int __i) {
    ++	       return static_cast<_Ip>(std::countl_one(__v[__i]));
     +	     });
     +    }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>
     +    countr_zero(const _Vp& __v) noexcept
     +    {
    -+      return rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>([&](int __i) {
    -+	       return std::countr_zero(__v[__i]);
    ++      using _Ip = make_signed_t<typename _Vp::value_type>;
    ++      return rebind_t<_Ip, _Vp>([&](int __i) {
    ++	       return static_cast<_Ip>(std::countr_zero(__v[__i]));
     +	     });
     +    }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>
     +    countr_one(const _Vp& __v) noexcept
     +    {
    -+      return rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>([&](int __i) {
    -+	       return std::countr_one(__v[__i]);
    ++      using _Ip = make_signed_t<typename _Vp::value_type>;
    ++      return rebind_t<_Ip, _Vp>([&](int __i) {
    ++	       return static_cast<_Ip>(std::countr_one(__v[__i]));
     +	     });
     +    }
     +
    -+  template<__simd_vec_type _Vp>
    ++  template<__simd_unsigned_integer _Vp>
     +    [[__gnu__::__always_inline__]]
     +    constexpr rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>
     +    popcount(const _Vp& __v) noexcept
     +    {
    -+      return rebind_t<make_signed_t<typename _Vp::value_type>, _Vp>([&](int __i) {
    -+	       return std::popcount(__v[__i]);
    ++      using _Ip = make_signed_t<typename _Vp::value_type>;
    ++      return rebind_t<_Ip, _Vp>([&](int __i) {
    ++	       return static_cast<_Ip>(std::popcount(__v[__i]));
     +	     });
     +    }
     +} // namespace simd
    @@ libstdc++-v3/include/bits/simd_details.h: namespace simd
          concept __simd_integral
            = __simd_vec_type<_Vp> && integral<typename _Vp::value_type>;
      
    ++  template <typename _Vp>
    ++    concept __simd_unsigned_integer
    ++      = __simd_vec_type<_Vp> && __unsigned_integer<typename _Vp::value_type>;
    ++
     +  template <typename _Vp>
     +    using __simd_complex_value_type = typename _Vp::value_type::value_type;
     +
    @@ libstdc++-v3/testsuite/std/simd/simd_bit.cc (new)
     +#include <climits>
     +
     +template <typename V>
    ++  struct CheckInvocable
    ++  {
    ++    using T = typename V::value_type;
    ++    static constexpr bool unsigned_integer
    ++      = any_type_of<T, unsigned char, unsigned short, unsigned int, unsigned long,
    ++		    unsigned long long>;
    ++    static_assert(std::integral<T> == requires(V x) { std::byteswap(x); });
    ++    static_assert(unsigned_integer == requires(V x) { std::bit_ceil(x); });
    ++    static_assert(unsigned_integer == requires(V x) { std::bit_floor(x); });
    ++    static_assert(unsigned_integer == requires(V x) { std::has_single_bit(x); });
    ++    static_assert(unsigned_integer == requires(V x, V y) { std::rotl(x, y); });
    ++    static_assert(unsigned_integer == requires(V x, int y) { std::rotl(x, y); });
    ++    static_assert(unsigned_integer == requires(V x, V y) { std::rotr(x, y); });
    ++    static_assert(unsigned_integer == requires(V x, int y) { std::rotr(x, y); });
    ++    static_assert(unsigned_integer == requires(V x) { std::bit_width(x); });
    ++    static_assert(unsigned_integer == requires(V x) { std::countl_zero(x); });
    ++    static_assert(unsigned_integer == requires(V x) { std::countl_one(x); });
    ++    static_assert(unsigned_integer == requires(V x) { std::countr_zero(x); });
    ++    static_assert(unsigned_integer == requires(V x) { std::countr_one(x); });
    ++    static_assert(unsigned_integer == requires(V x) { std::popcount(x); });
    ++  };
    ++
    ++template <typename V>
     +  requires std::integral<typename V::value_type>
    -+  struct Tests<V>
    ++  struct Tests<V> : CheckInvocable<V>
     +  {
     +    using T = typename V::value_type;
     +    using M = typename V::mask_type;
    @@ libstdc++-v3/testsuite/std/simd/simd_bit.cc (new)
     +	t.verify_equal(bit_ceil(b), select(b == T(), T(1), b));
     +	t.verify_equal(std::bit_ceil(a), bit_ceil(a));
     +	t.verify_equal(std::simd::bit_ceil(a), bit_ceil(a));
    -+	t.verify_equal(bit_ceil(a),
    -+		       V([](T i) {
    -+			 if (i > msb)
    -+			   i -= msb + 1;
    -+			 while (!std::has_single_bit(i))
    -+			   i = (i | (i >> 1)) + 1;
    -+			 return T(i);
    -+		       }));
    ++	t.verify_equal(bit_ceil(a), V([&](int i) { return std::bit_ceil(a[i]); }));
     +      }
     +    };
     +
    @@ libstdc++-v3/testsuite/std/simd/simd_bit.cc (new)
     +	t.verify_equal(bit_floor(b), b);
     +	t.verify_equal(std::bit_floor(a), bit_floor(a));
     +	t.verify_equal(std::simd::bit_floor(a), bit_floor(a));
    -+	t.verify_equal(bit_floor(a),
    -+		       V([](T i) -> T {
    -+			 if (i == 0)
    -+			   return 0;
    -+			 int shift = 0;
    -+			 while ((i >> shift) > 1)
    -+			   ++shift;
    -+			 return T(1) << shift;
    -+		       }));
    ++	t.verify_equal(bit_floor(a), V([&](int i) { return std::bit_floor(a[i]); }));
     +      }
     +    };
     +
    @@ libstdc++-v3/testsuite/std/simd/simd_bit.cc (new)
     +      }
     +    };
     +
    -+    ADD_TEST(RotateLeft, std::__unsigned_integer<T>) {
    ++    ADD_TEST(FullRotate, std::__unsigned_integer<T>) {
     +      std::tuple {test_iota<V, 0, 0>},
     +      [](auto& t, const V a) {
    -+	t.verify_equal(rotl(a, sizeof(T) * CHAR_BIT), a);
    -+	t.verify_equal(std::rotl(a, sizeof(T) * CHAR_BIT), a);
    -+	t.verify_equal(std::simd::rotl(a, sizeof(T) * CHAR_BIT), a);
    ++	constexpr int digits = std::numeric_limits<T>::digits;
    ++	template for (int n : {0, digits, 5 * digits})
    ++	  {
    ++	    t.verify_equal(rotl(a, n), a);
    ++	    t.verify_equal(std::rotl(a, n), a);
    ++	    t.verify_equal(std::simd::rotl(a, n), a);
    ++	    t.verify_equal(rotr(a, n), a);
    ++	    t.verify_equal(std::rotr(a, n), a);
    ++	    t.verify_equal(std::simd::rotr(a, n), a);
    ++	  }
     +      }
     +    };
     +
    @@ libstdc++-v3/testsuite/std/simd/simd_bit.cc (new)
     +	t.verify_equal(rotr(x, I(sizeof(T) * CHAR_BIT) - vshiftx), refx);
     +      }
     +    };
    ++
    ++    // The value-type of reference is always going to be 'int', forcing a conversion in verify_equal
    ++    // (unless V::value_type is 'unsigned int'). That's intentional, since we thus can find
    ++    // (hypothetical) cases of value-changing conversions in the implementation.
    ++#define REFERENCE(x, fun) simd::rebind_t<decltype(fun(x[0])), V>([&](int i) { return fun(x[i]); })
    ++
    ++    ADD_TEST(BitWidth, std::__unsigned_integer<T>) {
    ++      std::tuple {test_iota<V>, msb - test_iota<V>},
    ++      [](auto& t, const V x, const V y) {
    ++	t.verify_equal(std::bit_width(x), REFERENCE(x, std::bit_width));
    ++	t.verify_equal(simd::bit_width(x), REFERENCE(x, std::bit_width));
    ++	t.verify_equal(std::bit_width(y), REFERENCE(y, std::bit_width));
    ++	t.verify_equal(simd::bit_width(y), REFERENCE(y, std::bit_width));
    ++      }
    ++    };
    ++
    ++    ADD_TEST(CountLZero, std::__unsigned_integer<T>) {
    ++      std::tuple {test_iota<V>, msb - test_iota<V>},
    ++      [](auto& t, const V x, const V y) {
    ++	t.verify_equal(std::countl_zero(x), REFERENCE(x, std::countl_zero));
    ++	t.verify_equal(simd::countl_zero(x), REFERENCE(x, std::countl_zero));
    ++	t.verify_equal(std::countl_zero(y), REFERENCE(y, std::countl_zero));
    ++	t.verify_equal(simd::countl_zero(y), REFERENCE(y, std::countl_zero));
    ++      }
    ++    };
    ++
    ++    ADD_TEST(CountLOne, std::__unsigned_integer<T>) {
    ++      std::tuple {test_iota<V>, msb - test_iota<V>},
    ++      [](auto& t, const V x, const V y) {
    ++	t.verify_equal(std::countl_one(x), REFERENCE(x, std::countl_one));
    ++	t.verify_equal(simd::countl_one(x), REFERENCE(x, std::countl_one));
    ++	t.verify_equal(std::countl_one(y), REFERENCE(y, std::countl_one));
    ++	t.verify_equal(simd::countl_one(y), REFERENCE(y, std::countl_one));
    ++      }
    ++    };
    ++
    ++    ADD_TEST(CountRZero, std::__unsigned_integer<T>) {
    ++      std::tuple {test_iota<V>, msb - test_iota<V>},
    ++      [](auto& t, const V x, const V y) {
    ++	t.verify_equal(std::countr_zero(x), REFERENCE(x, std::countr_zero));
    ++	t.verify_equal(simd::countr_zero(x), REFERENCE(x, std::countr_zero));
    ++	t.verify_equal(std::countr_zero(y), REFERENCE(y, std::countr_zero));
    ++	t.verify_equal(simd::countr_zero(y), REFERENCE(y, std::countr_zero));
    ++      }
    ++    };
    ++
    ++    ADD_TEST(CountROne, std::__unsigned_integer<T>) {
    ++      std::tuple {test_iota<V>, msb - test_iota<V>},
    ++      [](auto& t, const V x, const V y) {
    ++	t.verify_equal(std::countr_one(x), REFERENCE(x, std::countr_one));
    ++	t.verify_equal(simd::countr_one(x), REFERENCE(x, std::countr_one));
    ++	t.verify_equal(std::countr_one(y), REFERENCE(y, std::countr_one));
    ++	t.verify_equal(simd::countr_one(y), REFERENCE(y, std::countr_one));
    ++      }
    ++    };
    ++
    ++    ADD_TEST(PopCount, std::__unsigned_integer<T>) {
    ++      std::tuple {test_iota<V>, msb - test_iota<V>},
    ++      [](auto& t, const V x, const V y) {
    ++	t.verify_equal(std::popcount(x), REFERENCE(x, std::popcount));
    ++	t.verify_equal(simd::popcount(x), REFERENCE(x, std::popcount));
    ++	t.verify_equal(std::popcount(y), REFERENCE(y, std::popcount));
    ++	t.verify_equal(simd::popcount(y), REFERENCE(y, std::popcount));
    ++      }
    ++    };
     +  };
     +
     +template <typename V>
    -+  struct Tests
    ++  struct Tests : CheckInvocable<V>
     +  {};
     +
     +#include "create_tests.h"
    @@ libstdc++-v3/testsuite/std/simd/test_setup.h: static std::string_view test_name
      
      namespace simd = std::simd;
      
    ++template <typename T, typename... Us>
    ++  concept any_type_of = (std::same_as<T, Us> || ...);
    ++
     +template <typename T>
     +  concept complex_like = std::simd::__complex_like<T>;
     +
3:  b851112b9a09 < -:  ------------ libstdc++: Fix [simd.bit] constraints and implementation
  

Jonathan Wakely [Wednesday, 1 July 2026, 14:38:53 CEST]:
> The patch is OK with the new calls in this function qualified. Thanks.

Merged with the qualification change and I adjusted the patch for 
ranges::__static_sized_range, after it was moved from std::simd.

(I force-pushed everything to the forge PR as well, which then marked it as 
manually merged.)