From patchwork Fri Jan 10 21:23:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 104535 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 04439385828B for ; Fri, 10 Jan 2025 21:31:51 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 04439385828B Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=AizuQOEv X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id 68CEE3858290 for ; Fri, 10 Jan 2025 21:28:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 68CEE3858290 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 68CEE3858290 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544502; cv=none; b=nXRmu8YM8LdBSmQM4Fu+JkzLmexsSUD4srqLkzUnV+AEy3ls11hq/x0T/amLZzbjy6CE4iU+tDBHQgRbPW0K+NEmxKwsLNEouu6PFbNxf/IEjm0Wr/sNB+INCPdwy9bj3FwbcaGOzq+THqdJ6mdmf+3lgQbNj/xB70A1zr38QgM= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544502; c=relaxed/simple; bh=Hwqsn2oGsw8Qq8BehIa3LWY79z94Cvug/0DnhfYIj4Q=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=ctlHXlShu/vpJd3sUDa5Dbhf3GcIF+xoa2AIUbzXLdys95i1Dfkj+81WT6G1fQGq47IS0BW2PepzzWX63hoSKsK2fQa/oDFxOH5n2EOgaNn4TswTNWEJugHRAb1qAUF86pJOQu5luqBQP4r1ToY3K7kDM3UIAsru0F6452UUvt8= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 68CEE3858290 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736544502; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bc0+8667fGJw3p1lgxqo7WzZyYLmXoSl/n4EhVWfccI=; b=AizuQOEviUxEv+9tzy7WohOxxjeMsrkcqhzGMtC2KUK5xwCWl/hhAL/Tz54SJi9GRCLFfy vAsr14zQd1XuAUTyAxXyYkMc987Nag2P5NpyiDtm/SQxY0VE+JCSui1zVEkLhPzCCOH2/u SlO7clNF2yjLgGIODnYy47IjOY+R5kk= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-651-k7ppBui0P8-HVpF8fyqapw-1; Fri, 10 Jan 2025 16:28:16 -0500 X-MC-Unique: k7ppBui0P8-HVpF8fyqapw-1 X-Mimecast-MFC-AGG-ID: k7ppBui0P8-HVpF8fyqapw Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7C5AD19560BB; Fri, 10 Jan 2025 21:28:15 +0000 (UTC) Received: from localhost (unknown [10.42.28.9]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3E64D195E3D9; Fri, 10 Jan 2025 21:28:14 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH 01/10] libstdc++: Atomic wait/notify ABI stabilization Date: Fri, 10 Jan 2025 21:23:04 +0000 Message-ID: <20250110212810.832494-2-jwakely@redhat.com> In-Reply-To: <20250110212810.832494-1-jwakely@redhat.com> References: <20250110212810.832494-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: RLdfoRHRS1VhGeGxT5kfck45csvPVaPpWXHttjwsYtM_1736544495 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP, URIBL_DBL_SPAM, URI_HEX autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Thomas Rodgers This represents a major refactoring of the previous atomic::wait and atomic::notify implementation detail. The aim of this change is to simplify the implementation details and position the resulting implementation so that much of the current header-only detail can be moved into the shared library, while also accounting for anticipated changes to wait/notify functionality for C++26. The previous implementation implemented spin logic in terms of the types __default_spin_policy, __timed_backoff_spin_policy, and the free function __atomic_spin. These are replaced in favor of two new free functions; __spin_impl and __spin_until_impl. These currently inline free functions are expected to be moved into the libstdc++ shared library in a future commit. The previous implementation derived untimed and timed wait implementation detail from __detail::__waiter_pool_base. This is-a relationship is removed in the new version and the previous implementation detail is renamed to reflect this change. The static _S_for member has been renamed as well to indicate that it returns the __waiter_pool_impl entry in the static 'side table' for a given awaited address. This new implementation replaces all of the non-templated waiting detail of __waiter_base, __waiter_pool, __waiter, __enters_wait, and __bare_wait with the __wait_impl free function, and the supporting __wait_flags enum and __wait_args struct. This currenly inline free function is expected to be moved into the libstdc++ shared library in a future commit. This new implementation replaces all of the non-templated notifying detail of __waiter_base, __waiter_pool, and __waiter with the __notify_impl free function. This currently inline free function is expected to be moved into the libstdc++ shared library in a future commit. The __atomic_wait_address template function is updated to account for the above changes and to support the expected C++26 change to pass the most recent observed value to the caller supplied predicate. A new non-templated __atomic_wait_address_v free function is added that only works for atomic types that operate only on __platform_wait_t and requires the caller to supply a memory order. This is intended to be the simplest code path for such types. The __atomic_wait_address_v template function is now implemented in terms of new __atomic_wait_address template and continues to accept a user supplied "value function" to retrieve the current value of the atomic. The __atomic_notify_address template function is updated to account for the above changes. The template __platform_wait_until_impl is renamed to __wait_clock_t. The previous __platform_wait_until template is deleted and the functionality previously provided is moved t the new tempalate function __wait_until. A similar change is made to the __cond_wait_until_impl/__cond_wait_until implementation. This new implementation similarly replaces all of the non-templated waiting detail of __timed_waiter_pool, __timed_waiter, etc. with the new __wait_until_impl free function. This currently inline free function is expected to be moved into the libstdc++ shared library in a future commit. This implementation replaces all templated waiting functions that manage clock conversion as well as relative waiting (wait_for) with the new template functions __wait_until and __wait_for. Similarly the previous implementation detail for the various __atomic_wait_address_Xxx templates is adjusted to account for the implementation changes outlined above. All of the "bare wait" versions of __atomic_wait_Xxx have been removed and replaced with a defaulted boolean __bare_wait parameter on the new version of these templates. libstdc++-v3/ChangeLog: * include/bits/atomic_timed_wait.h: (__detail::__platform_wait_until_impl): Rename to __platform_wait_until. (__detail::__platform_wait_until): Remove previous definition. (__detail::__cond_wait_until_impl): Rename to __cond_wait_until. (__detail::__cond_wait_until): Remove previous definition. (__detail::__spin_until_impl): New function. (__detail::__wait_until_impl): New function. (__detail::__wait_until): New function. (__detail::__wait_for): New function. (__detail::__timed_waiter_pool): Remove type. (__detail::__timed_backoff_spin_policy): Remove type. (__detail::__timed_waiter): Remove type. (__detail::__enters_timed_wait): Remove type alias. (__detail::__bare_timed_wait): Remove type alias. (__atomic_wait_address_until): Adjust to new implementation detail. (__atomic_wait_address_until_v): Likewise. (__atomic_wait_address_bare): Remove. (__atomic_wait_address_for): Adjust to new implementation detail. (__atomic_wait_address_for_v): Likewise. (__atomic_wait_address_for_bare): Remove. * include/bits/atomic_wait.h: Include bits/stl_pair.h. (__detail::__default_spin_policy): Remove type. (__detail::__atomic_spin): Remove function. (__detail::__waiter_pool_base): Rename to __waiter_pool_impl. Remove _M_notify. Rename _S_for to _S_impl_for. (__detail::__waiter_base): Remove type. (__detail::__waiter_pool): Remove type. (__detail::__waiter): Remove type. (__detail::__enters_wait): Remove type alias. (__detail::__bare_wait): Remove type alias. (__detail::__wait_flags): New enum. (__detail::__wait_args): New struct. (__detail::__wait_result_type): New type alias. (__detail::__spin_impl): New function. (__detail::__wait_impl): New function. (__atomic_wait_address): Adjust to new implementation detail. (__atomic_wait_address_v): Likewise. (__atomic_notify_address): Likewise. (__atomic_wait_address_bare): Delete. (__atomic_notify_address_bare): Likewise. * include/bits/semaphore_base.h: Adjust implementation to use new __atomic_wait_address_v contract. * include/std/barrier: Adjust implementation to use new __atomic_wait contract. * include/std/latch: Adjust implementation to use new __atomic_wait contract. * testsuite/29_atomics/atomic/wait_notify/100334.cc (main): Adjust to for __detail::__waiter_pool_base renaming. --- libstdc++-v3/include/bits/atomic_timed_wait.h | 549 ++++++++---------- libstdc++-v3/include/bits/atomic_wait.h | 486 ++++++++-------- libstdc++-v3/include/bits/semaphore_base.h | 53 +- libstdc++-v3/include/std/barrier | 6 +- libstdc++-v3/include/std/latch | 5 +- .../29_atomics/atomic/wait_notify/100334.cc | 4 +- 6 files changed, 514 insertions(+), 589 deletions(-) diff --git a/libstdc++-v3/include/bits/atomic_timed_wait.h b/libstdc++-v3/include/bits/atomic_timed_wait.h index 9a6ac95b7d0e..196548484024 100644 --- a/libstdc++-v3/include/bits/atomic_timed_wait.h +++ b/libstdc++-v3/include/bits/atomic_timed_wait.h @@ -76,62 +76,32 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION #ifdef _GLIBCXX_HAVE_LINUX_FUTEX #define _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT // returns true if wait ended before timeout - template - bool - __platform_wait_until_impl(const __platform_wait_t* __addr, - __platform_wait_t __old, - const chrono::time_point<__wait_clock_t, _Dur>& - __atime) noexcept - { - auto __s = chrono::time_point_cast(__atime); - auto __ns = chrono::duration_cast(__atime - __s); + bool + __platform_wait_until(const __platform_wait_t* __addr, + __platform_wait_t __old, + const __wait_clock_t::time_point& __atime) noexcept + { + auto __s = chrono::time_point_cast(__atime); + auto __ns = chrono::duration_cast(__atime - __s); - struct timespec __rt = + struct timespec __rt = { static_cast(__s.time_since_epoch().count()), static_cast(__ns.count()) }; - auto __e = syscall (SYS_futex, __addr, - static_cast(__futex_wait_flags:: - __wait_bitset_private), - __old, &__rt, nullptr, - static_cast(__futex_wait_flags:: - __bitset_match_any)); - - if (__e) - { - if (errno == ETIMEDOUT) - return false; - if (errno != EINTR && errno != EAGAIN) - __throw_system_error(errno); - } - return true; - } - - // returns true if wait ended before timeout - template - bool - __platform_wait_until(const __platform_wait_t* __addr, __platform_wait_t __old, - const chrono::time_point<_Clock, _Dur>& __atime) - { - if constexpr (is_same_v<__wait_clock_t, _Clock>) - { - return __platform_wait_until_impl(__addr, __old, __atime); - } - else - { - if (!__platform_wait_until_impl(__addr, __old, - __to_wait_clock(__atime))) - { - // We got a timeout when measured against __clock_t but - // we need to check against the caller-supplied clock - // to tell whether we should return a timeout. - if (_Clock::now() < __atime) - return true; - } + auto __e = syscall (SYS_futex, __addr, + static_cast(__futex_wait_flags::__wait_bitset_private), + __old, &__rt, nullptr, + static_cast(__futex_wait_flags::__bitset_match_any)); + if (__e) + { + if (errno == ETIMEDOUT) return false; - } + if (errno != EINTR && errno != EAGAIN) + __throw_system_error(errno); + } + return true; } #else // define _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT and implement __platform_wait_until() @@ -141,15 +111,10 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION #ifdef _GLIBCXX_HAS_GTHREADS // Returns true if wait ended before timeout. - // _Clock must be either steady_clock or system_clock. - template - bool - __cond_wait_until_impl(__condvar& __cv, mutex& __mx, - const chrono::time_point<_Clock, _Dur>& __atime) - { - static_assert(std::__is_one_of<_Clock, chrono::steady_clock, - chrono::system_clock>::value); - + bool + __cond_wait_until(__condvar& __cv, mutex& __mx, + const __wait_clock_t::time_point& __atime) + { auto __s = chrono::time_point_cast(__atime); auto __ns = chrono::duration_cast(__atime - __s); @@ -160,293 +125,261 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION }; #ifdef _GLIBCXX_USE_PTHREAD_COND_CLOCKWAIT - if constexpr (is_same_v) + if constexpr (is_same_v) __cv.wait_until(__mx, CLOCK_MONOTONIC, __ts); else #endif __cv.wait_until(__mx, __ts); - return _Clock::now() < __atime; - } - - // returns true if wait ended before timeout - template - bool - __cond_wait_until(__condvar& __cv, mutex& __mx, - const chrono::time_point<_Clock, _Dur>& __atime) - { -#ifdef _GLIBCXX_USE_PTHREAD_COND_CLOCKWAIT - if constexpr (is_same_v<_Clock, chrono::steady_clock>) - return __detail::__cond_wait_until_impl(__cv, __mx, __atime); - else -#endif - if constexpr (is_same_v<_Clock, chrono::system_clock>) - return __detail::__cond_wait_until_impl(__cv, __mx, __atime); - else - { - if (__cond_wait_until_impl(__cv, __mx, - __to_wait_clock(__atime))) - { - // We got a timeout when measured against __clock_t but - // we need to check against the caller-supplied clock - // to tell whether we should return a timeout. - if (_Clock::now() < __atime) - return true; - } - return false; - } + return __wait_clock_t::now() < __atime; } #endif // _GLIBCXX_HAS_GTHREADS - struct __timed_waiter_pool : __waiter_pool_base + inline __wait_result_type + __spin_until_impl(const __platform_wait_t* __addr, __wait_args __args, + const __wait_clock_t::time_point& __deadline) { - // returns true if wait ended before timeout - template - bool - _M_do_wait_until(__platform_wait_t* __addr, __platform_wait_t __old, - const chrono::time_point<_Clock, _Dur>& __atime) + auto __t0 = __wait_clock_t::now(); + using namespace literals::chrono_literals; + + __platform_wait_t __val; + auto __now = __wait_clock_t::now(); + for (; __now < __deadline; __now = __wait_clock_t::now()) + { + auto __elapsed = __now - __t0; +#ifndef _GLIBCXX_NO_SLEEP + if (__elapsed > 128ms) + { + this_thread::sleep_for(64ms); + } + else if (__elapsed > 64us) + { + this_thread::sleep_for(__elapsed / 2); + } + else +#endif + if (__elapsed > 4us) + { + __thread_yield(); + } + else + { + auto __res = __detail::__spin_impl(__addr, __args); + if (__res.first) + return __res; + } + + __atomic_load(__addr, &__val, __args._M_order); + if (__val != __args._M_old) + return make_pair(true, __val); + } + return make_pair(false, __val); + } + + inline __wait_result_type + __wait_until_impl(const __platform_wait_t* __addr, __wait_args __args, + const __wait_clock_t::time_point& __atime) + { +#ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT + __waiter_pool_impl* __pool = nullptr; +#else + // if we don't have __platform_wait, we always need the side-table + __waiter_pool_impl* __pool = &__waiter_pool_impl::_S_impl_for(__addr); +#endif + + __platform_wait_t* __wait_addr; + if (__args & __wait_flags::__proxy_wait) { #ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT - return __platform_wait_until(__addr, __old, __atime); -#else - __platform_wait_t __val; - __atomic_load(__addr, &__val, __ATOMIC_RELAXED); - if (__val == __old) - { - lock_guard __l(_M_mtx); - return __cond_wait_until(_M_cv, _M_mtx, __atime); - } - else - return true; -#endif // _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT + __pool = &__waiter_pool_impl::_S_impl_for(__addr); +#endif + __wait_addr = &__pool->_M_ver; + __atomic_load(__wait_addr, &__args._M_old, __args._M_order); } - }; + else + __wait_addr = const_cast<__platform_wait_t*>(__addr); - struct __timed_backoff_spin_policy - { - __wait_clock_t::time_point _M_deadline; - __wait_clock_t::time_point _M_t0; + if (__args & __wait_flags::__do_spin) + { + auto __res = __detail::__spin_until_impl(__wait_addr, __args, __atime); + if (__res.first) + return __res; + if (__args & __wait_flags::__spin_only) + return __res; + } - template - __timed_backoff_spin_policy(chrono::time_point<_Clock, _Dur> - __deadline = _Clock::time_point::max(), - chrono::time_point<_Clock, _Dur> - __t0 = _Clock::now()) noexcept - : _M_deadline(__to_wait_clock(__deadline)) - , _M_t0(__to_wait_clock(__t0)) - { } - - bool - operator()() const noexcept + if (!(__args & __wait_flags::__track_contention)) { - using namespace literals::chrono_literals; - auto __now = __wait_clock_t::now(); - if (_M_deadline <= __now) - return false; - - // FIXME: this_thread::sleep_for not available #ifdef _GLIBCXX_NO_SLEEP - - auto __elapsed = __now - _M_t0; - if (__elapsed > 128ms) - { - this_thread::sleep_for(64ms); - } - else if (__elapsed > 64us) - { - this_thread::sleep_for(__elapsed / 2); - } - else if (__elapsed > 4us) - { - __thread_yield(); - } - else - return false; - return true; + // caller does not externally track contention +#ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT + __pool = (__pool == nullptr) ? &__waiter_pool_impl::_S_impl_for(__addr) + : __pool; +#endif + __pool->_M_enter_wait(); } - }; - template - struct __timed_waiter : __waiter_base<__timed_waiter_pool> + __wait_result_type __res; +#ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT + if (__platform_wait_until(__wait_addr, __args._M_old, __atime)) + __res = make_pair(true, __args._M_old); + else + __res = make_pair(false, __args._M_old); +#else + __platform_wait_t __val; + __atomic_load(__wait_addr, &__val, __args._M_order); + if (__val == __args._M_old) + { + lock_guard __l{ __pool->_M_mtx }; + __atomic_load(__wait_addr, &__val, __args._M_order); + if (__val == __args._M_old && + __cond_wait_until(__pool->_M_cv, __pool->_M_mtx, __atime)) + __res = make_pair(true, __val); + } + else + __res = make_pair(false, __val); +#endif + + if (!(__args & __wait_flags::__track_contention)) + // caller does not externally track contention + __pool->_M_leave_wait(); + return __res; + } + + template + __wait_result_type + __wait_until(const __platform_wait_t* __addr, __wait_args __args, + const chrono::time_point<_Clock, _Dur>& __atime) noexcept { - using __base_type = __waiter_base<__timed_waiter_pool>; - - template - __timed_waiter(const _Tp* __addr) noexcept - : __base_type(__addr) - { - if constexpr (_EntersWait::value) - _M_w._M_enter_wait(); - } - - ~__timed_waiter() - { - if constexpr (_EntersWait::value) - _M_w._M_leave_wait(); - } - - // returns true if wait ended before timeout - template - bool - _M_do_wait_until_v(_Tp __old, _ValFn __vfn, - const chrono::time_point<_Clock, _Dur>& - __atime) noexcept + if constexpr (is_same_v<__wait_clock_t, _Clock>) + return __detail::__wait_until_impl(__addr, __args, __atime); + else { - __platform_wait_t __val; - if (_M_do_spin(__old, std::move(__vfn), __val, - __timed_backoff_spin_policy(__atime))) - return true; - return __base_type::_M_w._M_do_wait_until(__base_type::_M_addr, __val, __atime); - } + auto __res = __detail::__wait_until_impl(__addr, __args, + __to_wait_clock(__atime)); + if (!__res.first) + { + // We got a timeout when measured against __clock_t but + // we need to check against the caller-supplied clock + // to tell whether we should return a timeout. + if (_Clock::now() < __atime) + return make_pair(true, __res.second); + } + return __res; + } + } - // returns true if wait ended before timeout - template - bool - _M_do_wait_until(_Pred __pred, __platform_wait_t __val, - const chrono::time_point<_Clock, _Dur>& - __atime) noexcept - { - for (auto __now = _Clock::now(); __now < __atime; - __now = _Clock::now()) - { - if (__base_type::_M_w._M_do_wait_until( - __base_type::_M_addr, __val, __atime) - && __pred()) - return true; - - if (__base_type::_M_do_spin(__pred, __val, - __timed_backoff_spin_policy(__atime, __now))) - return true; - } - return false; - } - - // returns true if wait ended before timeout - template - bool - _M_do_wait_until(_Pred __pred, - const chrono::time_point<_Clock, _Dur>& - __atime) noexcept - { - __platform_wait_t __val; - if (__base_type::_M_do_spin(__pred, __val, - __timed_backoff_spin_policy(__atime))) - return true; - return _M_do_wait_until(__pred, __val, __atime); - } - - template - bool - _M_do_wait_for_v(_Tp __old, _ValFn __vfn, - const chrono::duration<_Rep, _Period>& - __rtime) noexcept - { - __platform_wait_t __val; - if (_M_do_spin_v(__old, std::move(__vfn), __val)) - return true; - - if (!__rtime.count()) - return false; // no rtime supplied, and spin did not acquire - - auto __reltime = chrono::ceil<__wait_clock_t::duration>(__rtime); - - return __base_type::_M_w._M_do_wait_until( - __base_type::_M_addr, - __val, - chrono::steady_clock::now() + __reltime); - } - - template - bool - _M_do_wait_for(_Pred __pred, - const chrono::duration<_Rep, _Period>& __rtime) noexcept - { - __platform_wait_t __val; - if (__base_type::_M_do_spin(__pred, __val)) - return true; - - if (!__rtime.count()) - return false; // no rtime supplied, and spin did not acquire - - auto __reltime = chrono::ceil<__wait_clock_t::duration>(__rtime); - - return _M_do_wait_until(__pred, __val, - chrono::steady_clock::now() + __reltime); - } - }; - - using __enters_timed_wait = __timed_waiter; - using __bare_timed_wait = __timed_waiter; + template + __wait_result_type + __wait_for(const __platform_wait_t* __addr, __wait_args __args, + const chrono::duration<_Rep, _Period>& __rtime) noexcept + { + if (!__rtime.count()) + // no rtime supplied, just spin a bit + return __detail::__wait_impl(__addr, __args | __wait_flags::__spin_only); + auto const __reltime = chrono::ceil<__wait_clock_t::duration>(__rtime); + auto const __atime = chrono::steady_clock::now() + __reltime; + return __detail::__wait_until(__addr, __args, __atime); + } } // namespace __detail // returns true if wait ended before timeout + template + bool + __atomic_wait_address_until(const _Tp* __addr, _Pred&& __pred, + _ValFn&& __vfn, + const chrono::time_point<_Clock, _Dur>& __atime, + bool __bare_wait = false) noexcept + { + const auto __wait_addr = + reinterpret_cast(__addr); + __detail::__wait_args __args{ __addr, __bare_wait }; + _Tp __val = __vfn(); + while (!__pred(__val)) + { + auto __res = __detail::__wait_until(__wait_addr, __args, __atime); + if (!__res.first) + // timed out + return __res.first; // C++26 will also return last observed __val + __val = __vfn(); + } + return true; // C++26 will also return last observed __val + } + + template + bool + __atomic_wait_address_until_v(const __detail::__platform_wait_t* __addr, + __detail::__platform_wait_t __old, + int __order, + const chrono::time_point<_Clock, _Dur>& __atime, + bool __bare_wait = false) noexcept + { + __detail::__wait_args __args{ __addr, __old, __order, __bare_wait }; + auto __res = __detail::__wait_until(__addr, __args, __atime); + return __res.first; // C++26 will also return last observed __val + } + template bool __atomic_wait_address_until_v(const _Tp* __addr, _Tp&& __old, _ValFn&& __vfn, - const chrono::time_point<_Clock, _Dur>& - __atime) noexcept + const chrono::time_point<_Clock, _Dur>& __atime, + bool __bare_wait = false) noexcept { - __detail::__enters_timed_wait __w{__addr}; - return __w._M_do_wait_until_v(__old, __vfn, __atime); + auto __pfn = [&](const _Tp& __val) + { return !__detail::__atomic_compare(__old, __val); }; + return __atomic_wait_address_until(__addr, __pfn, forward<_ValFn>(__vfn), + __atime, __bare_wait); } - template - bool - __atomic_wait_address_until(const _Tp* __addr, _Pred __pred, - const chrono::time_point<_Clock, _Dur>& - __atime) noexcept - { - __detail::__enters_timed_wait __w{__addr}; - return __w._M_do_wait_until(__pred, __atime); - } + template + bool + __atomic_wait_address_for(const _Tp* __addr, _Pred&& __pred, + _ValFn&& __vfn, + const chrono::duration<_Rep, _Period>& __rtime, + bool __bare_wait = false) noexcept + { + const auto __wait_addr = + reinterpret_cast(__addr); + __detail::__wait_args __args{ __addr, __bare_wait }; + _Tp __val = __vfn(); + while (!__pred(__val)) + { + auto __res = __detail::__wait_for(__wait_addr, __args, __rtime); + if (!__res.first) + // timed out + return __res.first; // C++26 will also return last observed __val + __val = __vfn(); + } + return true; // C++26 will also return last observed __val + } - template + template bool - __atomic_wait_address_until_bare(const __detail::__platform_wait_t* __addr, - _Pred __pred, - const chrono::time_point<_Clock, _Dur>& - __atime) noexcept - { - __detail::__bare_timed_wait __w{__addr}; - return __w._M_do_wait_until(__pred, __atime); - } + __atomic_wait_address_for_v(const __detail::__platform_wait_t* __addr, + __detail::__platform_wait_t __old, + int __order, + const chrono::time_point<_Rep, _Period>& __rtime, + bool __bare_wait = false) noexcept + { + __detail::__wait_args __args{ __addr, __old, __order, __bare_wait }; + auto __res = __detail::__wait_for(__addr, __args, __rtime); + return __res.first; // C++26 will also return last observed __Val + } template bool __atomic_wait_address_for_v(const _Tp* __addr, _Tp&& __old, _ValFn&& __vfn, - const chrono::duration<_Rep, _Period>& __rtime) noexcept + const chrono::duration<_Rep, _Period>& __rtime, + bool __bare_wait = false) noexcept { - __detail::__enters_timed_wait __w{__addr}; - return __w._M_do_wait_for_v(__old, __vfn, __rtime); - } - - template - bool - __atomic_wait_address_for(const _Tp* __addr, _Pred __pred, - const chrono::duration<_Rep, _Period>& __rtime) noexcept - { - - __detail::__enters_timed_wait __w{__addr}; - return __w._M_do_wait_for(__pred, __rtime); - } - - template - bool - __atomic_wait_address_for_bare(const __detail::__platform_wait_t* __addr, - _Pred __pred, - const chrono::duration<_Rep, _Period>& __rtime) noexcept - { - __detail::__bare_timed_wait __w{__addr}; - return __w._M_do_wait_for(__pred, __rtime); + auto __pfn = [&](const _Tp& __val) + { return !__detail::__atomic_compare(__old, __val); }; + return __atomic_wait_address_for(__addr, __pfn, forward<_ValFn>(__vfn), + __rtime, __bare_wait); } _GLIBCXX_END_NAMESPACE_VERSION } // namespace std diff --git a/libstdc++-v3/include/bits/atomic_wait.h b/libstdc++-v3/include/bits/atomic_wait.h index 6d1554f68a56..18cfc2ef7bd2 100644 --- a/libstdc++-v3/include/bits/atomic_wait.h +++ b/libstdc++-v3/include/bits/atomic_wait.h @@ -50,7 +50,8 @@ # include #endif -# include // std::mutex, std::__condvar +#include +#include // std::mutex, std::__condvar namespace std _GLIBCXX_VISIBILITY(default) { @@ -134,7 +135,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __thread_yield() noexcept { #if defined _GLIBCXX_HAS_GTHREADS && defined _GLIBCXX_USE_SCHED_YIELD - __gthread_yield(); + __gthread_yield(); #endif } @@ -151,38 +152,6 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION inline constexpr auto __atomic_spin_count_relax = 12; inline constexpr auto __atomic_spin_count = 16; - struct __default_spin_policy - { - bool - operator()() const noexcept - { return false; } - }; - - template - bool - __atomic_spin(_Pred& __pred, _Spin __spin = _Spin{ }) noexcept - { - for (auto __i = 0; __i < __atomic_spin_count; ++__i) - { - if (__pred()) - return true; - - if (__i < __atomic_spin_count_relax) - __detail::__thread_relax(); - else - __detail::__thread_yield(); - } - - while (__spin()) - { - if (__pred()) - return true; - } - - return false; - } - // return true if equal template bool __atomic_compare(const _Tp& __a, const _Tp& __b) @@ -191,7 +160,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION return __builtin_memcmp(&__a, &__b, sizeof(_Tp)) == 0; } - struct __waiter_pool_base + struct __waiter_pool_impl { // Don't use std::hardware_destructive_interference_size here because we // don't want the layout of library types to depend on compiler options. @@ -208,7 +177,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION #ifndef _GLIBCXX_HAVE_PLATFORM_WAIT __condvar _M_cv; #endif - __waiter_pool_base() = default; + __waiter_pool_impl() = default; void _M_enter_wait() noexcept @@ -226,256 +195,271 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION return __res != 0; } - void - _M_notify(__platform_wait_t* __addr, [[maybe_unused]] bool __all, - bool __bare) noexcept - { -#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT - if (__addr == &_M_ver) - { - __atomic_fetch_add(__addr, 1, __ATOMIC_SEQ_CST); - __all = true; - } - - if (__bare || _M_waiting()) - __platform_notify(__addr, __all); -#else - { - lock_guard __l(_M_mtx); - __atomic_fetch_add(__addr, 1, __ATOMIC_RELAXED); - } - if (__bare || _M_waiting()) - _M_cv.notify_all(); -#endif - } - - static __waiter_pool_base& - _S_for(const void* __addr) noexcept + static __waiter_pool_impl& + _S_impl_for(const void* __addr) noexcept { constexpr __UINTPTR_TYPE__ __ct = 16; - static __waiter_pool_base __w[__ct]; + static __waiter_pool_impl __w[__ct]; auto __key = ((__UINTPTR_TYPE__)__addr >> 2) % __ct; return __w[__key]; } }; - struct __waiter_pool : __waiter_pool_base + enum class __wait_flags : __UINT_LEAST32_TYPE__ { - void - _M_do_wait(const __platform_wait_t* __addr, __platform_wait_t __old) noexcept - { -#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT - __platform_wait(__addr, __old); -#else - __platform_wait_t __val; - __atomic_load(__addr, &__val, __ATOMIC_SEQ_CST); - if (__val == __old) - { - lock_guard __l(_M_mtx); - __atomic_load(__addr, &__val, __ATOMIC_RELAXED); - if (__val == __old) - _M_cv.wait(_M_mtx); - } -#endif // __GLIBCXX_HAVE_PLATFORM_WAIT - } + __abi_version = 0, + __proxy_wait = 1, + __track_contention = 2, + __do_spin = 4, + __spin_only = 8 | __do_spin, // implies __do_spin + __abi_version_mask = 0xffff0000, }; - template - struct __waiter_base + struct __wait_args + { + __platform_wait_t _M_old; + int _M_order = __ATOMIC_ACQUIRE; + __wait_flags _M_flags; + + template + explicit __wait_args(const _Tp* __addr, + bool __bare_wait = false) noexcept + : _M_flags{ _S_flags_for(__addr, __bare_wait) } + { } + + __wait_args(const __platform_wait_t* __addr, __platform_wait_t __old, + int __order, bool __bare_wait = false) noexcept + : _M_old{ __old } + , _M_order{ __order } + , _M_flags{ _S_flags_for(__addr, __bare_wait) } + { } + + __wait_args(const __wait_args&) noexcept = default; + __wait_args& + operator=(const __wait_args&) noexcept = default; + + bool + operator&(__wait_flags __flag) const noexcept { - using __waiter_type = _Tp; + using __t = underlying_type_t<__wait_flags>; + return static_cast<__t>(_M_flags) + & static_cast<__t>(__flag); + } - __waiter_type& _M_w; - __platform_wait_t* _M_addr; + __wait_args + operator|(__wait_flags __flag) const noexcept + { + using __t = underlying_type_t<__wait_flags>; + __wait_args __res{ *this }; + const auto __flags = static_cast<__t>(__res._M_flags) + | static_cast<__t>(__flag); + __res._M_flags = __wait_flags{ __flags }; + return __res; + } - template - static __platform_wait_t* - _S_wait_addr(const _Up* __a, __platform_wait_t* __b) - { - if constexpr (__platform_wait_uses_type<_Up>) - return reinterpret_cast<__platform_wait_t*>(const_cast<_Up*>(__a)); - else - return __b; - } + private: + static int + constexpr _S_default_flags() noexcept + { + using __t = underlying_type_t<__wait_flags>; + return static_cast<__t>(__wait_flags::__abi_version) + | static_cast<__t>(__wait_flags::__do_spin); + } - static __waiter_type& - _S_for(const void* __addr) noexcept + template + static int + constexpr _S_flags_for(const _Tp*, bool __bare_wait) noexcept { - static_assert(sizeof(__waiter_type) == sizeof(__waiter_pool_base)); - auto& res = __waiter_pool_base::_S_for(__addr); - return reinterpret_cast<__waiter_type&>(res); + auto __res = _S_default_flags(); + if (!__bare_wait) + __res |= static_cast(__wait_flags::__track_contention); + if constexpr (!__platform_wait_uses_type<_Tp>) + __res |= static_cast(__wait_flags::__proxy_wait); + return __res; } - template - explicit __waiter_base(const _Up* __addr) noexcept - : _M_w(_S_for(__addr)) - , _M_addr(_S_wait_addr(__addr, &_M_w._M_ver)) - { } - - void - _M_notify(bool __all, bool __bare = false) noexcept - { _M_w._M_notify(_M_addr, __all, __bare); } - - template - static bool - _S_do_spin_v(__platform_wait_t* __addr, - const _Up& __old, _ValFn __vfn, - __platform_wait_t& __val, - _Spin __spin = _Spin{ }) - { - auto const __pred = [=] - { return !__detail::__atomic_compare(__old, __vfn()); }; - - if constexpr (__platform_wait_uses_type<_Up>) - { - __builtin_memcpy(&__val, &__old, sizeof(__val)); - } - else - { - __atomic_load(__addr, &__val, __ATOMIC_ACQUIRE); - } - return __atomic_spin(__pred, __spin); - } - - template - bool - _M_do_spin_v(const _Up& __old, _ValFn __vfn, - __platform_wait_t& __val, - _Spin __spin = _Spin{ }) - { return _S_do_spin_v(_M_addr, __old, __vfn, __val, __spin); } - - template - static bool - _S_do_spin(const __platform_wait_t* __addr, - _Pred __pred, - __platform_wait_t& __val, - _Spin __spin = _Spin{ }) - { - __atomic_load(__addr, &__val, __ATOMIC_ACQUIRE); - return __atomic_spin(__pred, __spin); - } - - template - bool - _M_do_spin(_Pred __pred, __platform_wait_t& __val, - _Spin __spin = _Spin{ }) - { return _S_do_spin(_M_addr, __pred, __val, __spin); } - }; - - template - struct __waiter : __waiter_base<__waiter_pool> - { - using __base_type = __waiter_base<__waiter_pool>; - - template - explicit __waiter(const _Tp* __addr) noexcept - : __base_type(__addr) - { - if constexpr (_EntersWait::value) - _M_w._M_enter_wait(); - } - - ~__waiter() + template + static int + _S_memory_order_for(const _Tp*, int __order) noexcept { - if constexpr (_EntersWait::value) - _M_w._M_leave_wait(); + if constexpr (__platform_wait_uses_type<_Tp>) + return __order; + return __ATOMIC_ACQUIRE; + } + }; + + using __wait_result_type = pair; + inline __wait_result_type + __spin_impl(const __platform_wait_t* __addr, __wait_args __args) + { + __platform_wait_t __val; + for (auto __i = 0; __i < __atomic_spin_count; ++__i) + { + __atomic_load(__addr, &__val, __args._M_order); + if (__val != __args._M_old) + return make_pair(true, __val); + if (__i < __atomic_spin_count_relax) + __detail::__thread_relax(); + else + __detail::__thread_yield(); + } + return make_pair(false, __val); + } + + inline __wait_result_type + __wait_impl(const __platform_wait_t* __addr, __wait_args __args) + { +#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT + __waiter_pool_impl* __pool = nullptr; +#else + // if we don't have __platform_wait, we always need the side-table + __waiter_pool_impl* __pool = &__waiter_pool_impl::_S_impl_for(__addr); +#endif + + __platform_wait_t* __wait_addr; + if (__args & __wait_flags::__proxy_wait) + { +#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT + __pool = &__waiter_pool_impl::_S_impl_for(__addr); +#endif + __wait_addr = &__pool->_M_ver; + __atomic_load(__wait_addr, &__args._M_old, __args._M_order); + } + else + __wait_addr = const_cast<__platform_wait_t*>(__addr); + + if (__args & __wait_flags::__do_spin) + { + auto __res = __detail::__spin_impl(__wait_addr, __args); + if (__res.first) + return __res; + if (__args & __wait_flags::__spin_only) + return __res; } - template - void - _M_do_wait_v(_Tp __old, _ValFn __vfn) - { - do - { - __platform_wait_t __val; - if (__base_type::_M_do_spin_v(__old, __vfn, __val)) - return; - __base_type::_M_w._M_do_wait(__base_type::_M_addr, __val); - } - while (__detail::__atomic_compare(__old, __vfn())); - } + if (!(__args & __wait_flags::__track_contention)) + { + // caller does not externally track contention +#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT + __pool = (__pool == nullptr) ? &__waiter_pool_impl::_S_impl_for(__addr) + : __pool; +#endif + __pool->_M_enter_wait(); + } - template - void - _M_do_wait(_Pred __pred) noexcept - { - do - { - __platform_wait_t __val; - if (__base_type::_M_do_spin(__pred, __val)) - return; - __base_type::_M_w._M_do_wait(__base_type::_M_addr, __val); - } - while (!__pred()); - } - }; + __wait_result_type __res; +#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT + __platform_wait(__wait_addr, __args._M_old); + __res = make_pair(false, __args._M_old); +#else + __platform_wait_t __val; + __atomic_load(__wait_addr, &__val, __args._M_order); + if (__val == __args._M_old) + { + lock_guard __l{ __pool->_M_mtx }; + __atomic_load(__wait_addr, &__val, __args._M_order); + if (__val == __args._M_old) + __pool->_M_cv.wait(__pool->_M_mtx); + } + __res = make_pair(false, __val); +#endif - using __enters_wait = __waiter; - using __bare_wait = __waiter; + if (!(__args & __wait_flags::__track_contention)) + // caller does not externally track contention + __pool->_M_leave_wait(); + return __res; + } + + inline void + __notify_impl(const __platform_wait_t* __addr, [[maybe_unused]] bool __all, + __wait_args __args) + { +#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT + __waiter_pool_impl* __pool = nullptr; +#else + // if we don't have __platform_notify, we always need the side-table + __waiter_pool_impl* __pool = &__waiter_pool_impl::_S_impl_for(__addr); +#endif + + if (!(__args & __wait_flags::__track_contention)) + { +#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT + __pool = &__waiter_pool_impl::_S_impl_for(__addr); +#endif + if (!__pool->_M_waiting()) + return; + } + + __platform_wait_t* __wait_addr; + if (__args & __wait_flags::__proxy_wait) + { +#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT + __pool = (__pool == nullptr) ? &__waiter_pool_impl::_S_impl_for(__addr) + : __pool; +#endif + __wait_addr = &__pool->_M_ver; + __atomic_fetch_add(__wait_addr, 1, __ATOMIC_RELAXED); + __all = true; + } + +#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT + __platform_notify(__wait_addr, __all); +#else + lock_guard __l{ __pool->_M_mtx }; + __pool->_M_cv.notify_all(); +#endif + } } // namespace __detail + template + void + __atomic_wait_address(const _Tp* __addr, + _Pred&& __pred, _ValFn&& __vfn, + bool __bare_wait = false) noexcept + { + const auto __wait_addr = + reinterpret_cast(__addr); + __detail::__wait_args __args{ __addr, __bare_wait }; + _Tp __val = __vfn(); + while (!__pred(__val)) + { + __detail::__wait_impl(__wait_addr, __args); + __val = __vfn(); + } + // C++26 will return __val + } + + inline void + __atomic_wait_address_v(const __detail::__platform_wait_t* __addr, + __detail::__platform_wait_t __old, + int __order) + { + __detail::__wait_args __args{ __addr, __old, __order }; + // C++26 will not ignore the return value here + __detail::__wait_impl(__addr, __args); + } + template void __atomic_wait_address_v(const _Tp* __addr, _Tp __old, _ValFn __vfn) noexcept { - __detail::__enters_wait __w(__addr); - __w._M_do_wait_v(__old, __vfn); - } - - template - void - __atomic_wait_address(const _Tp* __addr, _Pred __pred) noexcept - { - __detail::__enters_wait __w(__addr); - __w._M_do_wait(__pred); - } - - // This call is to be used by atomic types which track contention externally - template - void - __atomic_wait_address_bare(const __detail::__platform_wait_t* __addr, - _Pred __pred) noexcept - { -#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT - do - { - __detail::__platform_wait_t __val; - if (__detail::__bare_wait::_S_do_spin(__addr, __pred, __val)) - return; - __detail::__platform_wait(__addr, __val); - } - while (!__pred()); -#else // !_GLIBCXX_HAVE_PLATFORM_WAIT - __detail::__bare_wait __w(__addr); - __w._M_do_wait(__pred); -#endif + auto __pfn = [&](const _Tp& __val) + { return !__detail::__atomic_compare(__old, __val); }; + __atomic_wait_address(__addr, __pfn, forward<_ValFn>(__vfn)); } template void - __atomic_notify_address(const _Tp* __addr, bool __all) noexcept + __atomic_notify_address(const _Tp* __addr, bool __all, + bool __bare_wait = false) noexcept { - __detail::__bare_wait __w(__addr); - __w._M_notify(__all); + const auto __wait_addr = + reinterpret_cast(__addr); + __detail::__wait_args __args{ __addr, __bare_wait }; + __detail::__notify_impl(__wait_addr, __all, __args); } - - // This call is to be used by atomic types which track contention externally - inline void - __atomic_notify_address_bare(const __detail::__platform_wait_t* __addr, - bool __all) noexcept - { -#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT - __detail::__platform_notify(__addr, __all); -#else - __detail::__bare_wait __w(__addr); - __w._M_notify(__all, true); -#endif - } _GLIBCXX_END_NAMESPACE_VERSION } // namespace std #endif // __glibcxx_atomic_wait diff --git a/libstdc++-v3/include/bits/semaphore_base.h b/libstdc++-v3/include/bits/semaphore_base.h index d8f9bd8982bf..444a1589fb5a 100644 --- a/libstdc++-v3/include/bits/semaphore_base.h +++ b/libstdc++-v3/include/bits/semaphore_base.h @@ -181,10 +181,16 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __atomic_semaphore(const __atomic_semaphore&) = delete; __atomic_semaphore& operator=(const __atomic_semaphore&) = delete; - static _GLIBCXX_ALWAYS_INLINE bool - _S_do_try_acquire(__detail::__platform_wait_t* __counter) noexcept + static _GLIBCXX_ALWAYS_INLINE __detail::__platform_wait_t + _S_get_current(__detail::__platform_wait_t* __counter) noexcept + { + return __atomic_impl::load(__counter, memory_order::acquire); + } + + static _GLIBCXX_ALWAYS_INLINE bool + _S_do_try_acquire(__detail::__platform_wait_t* __counter, + __detail::__platform_wait_t __old) noexcept { - auto __old = __atomic_impl::load(__counter, memory_order::acquire); if (__old == 0) return false; @@ -197,17 +203,21 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION _GLIBCXX_ALWAYS_INLINE void _M_acquire() noexcept { - auto const __pred = - [this] { return _S_do_try_acquire(&this->_M_counter); }; - std::__atomic_wait_address_bare(&_M_counter, __pred); + auto const __vfn = [this]{ return _S_get_current(&this->_M_counter); }; + auto const __pred = [this](__detail::__platform_wait_t __cur) + { return _S_do_try_acquire(&this->_M_counter, __cur); }; + std::__atomic_wait_address(&_M_counter, __pred, __vfn, true); } bool _M_try_acquire() noexcept { - auto const __pred = - [this] { return _S_do_try_acquire(&this->_M_counter); }; - return std::__detail::__atomic_spin(__pred); + auto const __vfn = [this]{ return _S_get_current(&this->_M_counter); }; + auto const __pred = [this](__detail::__platform_wait_t __cur) + { return _S_do_try_acquire(&this->_M_counter, __cur); }; + return __atomic_wait_address_for(&_M_counter, __pred, __vfn, + __detail::__wait_clock_t::duration(), + true); } template @@ -215,21 +225,22 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION _M_try_acquire_until(const chrono::time_point<_Clock, _Duration>& __atime) noexcept { - auto const __pred = - [this] { return _S_do_try_acquire(&this->_M_counter); }; - - return __atomic_wait_address_until_bare(&_M_counter, __pred, __atime); + auto const __vfn = [this]{ return _S_get_current(&this->_M_counter); }; + auto const __pred = [this](__detail::__platform_wait_t __cur) + { return _S_do_try_acquire(&this->_M_counter, __cur); }; + return std::__atomic_wait_address_until(&_M_counter, + __pred, __vfn, __atime, true); } template _GLIBCXX_ALWAYS_INLINE bool - _M_try_acquire_for(const chrono::duration<_Rep, _Period>& __rtime) - noexcept + _M_try_acquire_for(const chrono::duration<_Rep, _Period>& __rtime) noexcept { - auto const __pred = - [this] { return _S_do_try_acquire(&this->_M_counter); }; - - return __atomic_wait_address_for_bare(&_M_counter, __pred, __rtime); + auto const __vfn = [this]{ return _S_get_current(&this->_M_counter); }; + auto const __pred = [this](__detail::__platform_wait_t __cur) + { return _S_do_try_acquire(&this->_M_counter, __cur); }; + return std::__atomic_wait_address_for(&_M_counter, + __pred, __vfn, __rtime, true); } _GLIBCXX_ALWAYS_INLINE void @@ -238,9 +249,9 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION if (0 < __atomic_impl::fetch_add(&_M_counter, __update, memory_order_release)) return; if (__update > 1) - __atomic_notify_address_bare(&_M_counter, true); + __atomic_notify_address(&_M_counter, true, true); else - __atomic_notify_address_bare(&_M_counter, true); + __atomic_notify_address(&_M_counter, true, true); // FIXME - Figure out why this does not wake a waiting thread // __atomic_notify_address_bare(&_M_counter, false); } diff --git a/libstdc++-v3/include/std/barrier b/libstdc++-v3/include/std/barrier index 6c3cfd44697c..62b03d0223f4 100644 --- a/libstdc++-v3/include/std/barrier +++ b/libstdc++-v3/include/std/barrier @@ -194,11 +194,7 @@ It looks different from literature pseudocode for two main reasons: wait(arrival_token&& __old_phase) const { __atomic_phase_const_ref_t __phase(_M_phase); - auto const __test_fn = [=] - { - return __phase.load(memory_order_acquire) != __old_phase; - }; - std::__atomic_wait_address(&_M_phase, __test_fn); + __phase.wait(__old_phase, memory_order_acquire); } void diff --git a/libstdc++-v3/include/std/latch b/libstdc++-v3/include/std/latch index 9220580613d2..de0afd8989bf 100644 --- a/libstdc++-v3/include/std/latch +++ b/libstdc++-v3/include/std/latch @@ -78,8 +78,9 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION _GLIBCXX_ALWAYS_INLINE void wait() const noexcept { - auto const __pred = [this] { return this->try_wait(); }; - std::__atomic_wait_address(&_M_a, __pred); + auto const __vfn = [this] { return this->try_wait(); }; + auto const __pred = [this](bool __b) { return __b; }; + std::__atomic_wait_address(&_M_a, __pred, __vfn); } _GLIBCXX_ALWAYS_INLINE void diff --git a/libstdc++-v3/testsuite/29_atomics/atomic/wait_notify/100334.cc b/libstdc++-v3/testsuite/29_atomics/atomic/wait_notify/100334.cc index 018c0c98d0ec..ec596e316500 100644 --- a/libstdc++-v3/testsuite/29_atomics/atomic/wait_notify/100334.cc +++ b/libstdc++-v3/testsuite/29_atomics/atomic/wait_notify/100334.cc @@ -54,8 +54,8 @@ main() atom->store(0); } - auto a = &std::__detail::__waiter_pool_base::_S_for(reinterpret_cast(atomics.a[0])); - auto b = &std::__detail::__waiter_pool_base::_S_for(reinterpret_cast(atomics.a[1])); + auto a = &std::__detail::__waiter_pool_impl::_S_impl_for(reinterpret_cast(atomics.a[0])); + auto b = &std::__detail::__waiter_pool_impl::_S_impl_for(reinterpret_cast(atomics.a[1])); VERIFY( a == b ); auto fut0 = std::async(std::launch::async, [&] { atomics.a[0]->wait(0); }); From patchwork Fri Jan 10 21:23:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 104537 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 2A7BB385828E for ; Fri, 10 Jan 2025 21:36:37 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2A7BB385828E Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=D3eysbgM X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id 6C4533858294 for ; Fri, 10 Jan 2025 21:28:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6C4533858294 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 6C4533858294 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544502; cv=none; b=LjOuNi0vy85l32P/TXTpBaBLtyyZSfJfDqcq14urrZxUWCSiWPS8TR5Hc0nAsT+Mzf6pX4VzNxAyRTPt3kA2CvCq1e98ljqdf9KENMqwpZk+03yMk5Ml/SvT84CBw2Myng/g8oOExC88JqFG9KEomITtA8vSIgF69sMKa+wQZQM= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544502; c=relaxed/simple; bh=GRykXzW9Dg1dH3puH66mHby/45RogDJPTo1Vh7ZGCos=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=eF9Pk7iPKUlkvRcAN+nlzrbXuuvBiuojKqI+00KDUj40SA/GsEQilnfQ/gLn68M6wTeZpQrkb2QL3LgRU3O1ly72nCLXB4I041qUAhRqm+oBeFIOUe/VvHACLWg8F6ciYRy/Z6vte1lzcbwY8xX8tHQoeCHyVhBns9ncd/3qYJo= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 6C4533858294 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736544502; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mg6QvtvV0KtXwZPV1okS6yPdDJ0J3AWvwsjeDcv4F8o=; b=D3eysbgM4iF9K/ajvQgYx6l1UJtjk4qmEuzAFam0gYgt3WSsoteLF2Flknm/hDRRttzslo i2xkLuErKqLfve1bnEdoo5Te3l4T6LcEMqFpg3rZLWGFVqpD77DD3bLOGEDKqVot/yJRBa RrghmuCUDq7MTmA3PxbdBR2T3xgCMWc= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-569-QnOsY_34MPCRTM1e702Cpg-1; Fri, 10 Jan 2025 16:28:18 -0500 X-MC-Unique: QnOsY_34MPCRTM1e702Cpg-1 X-Mimecast-MFC-AGG-ID: QnOsY_34MPCRTM1e702Cpg Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EB8191956064; Fri, 10 Jan 2025 21:28:17 +0000 (UTC) Received: from localhost (unknown [10.42.28.9]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 50A5A30001BE; Fri, 10 Jan 2025 21:28:17 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH 02/10] libstdc++: Pass __wait_args to internal API by const pointer Date: Fri, 10 Jan 2025 21:23:05 +0000 Message-ID: <20250110212810.832494-3-jwakely@redhat.com> In-Reply-To: <20250110212810.832494-1-jwakely@redhat.com> References: <20250110212810.832494-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: CBBdGCAdGzuPLMxdz-B6Ort367MM2R5mm54x7nwwT8M_1736544498 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org From: Thomas Rodgers This change splits the __wait_args data members to a new struct __wait_args_base and then passes that type by const pointer to the low level implementation functions. libstdc++-v3/ChangeLog: * include/bits/atomic_timed_wait.h (__spin_until_impl): Accept __wait_args as const __wait_args_base*. (__wait_until_impl): Likewise. (__wait_until): Likewise. (__wait_for): Likewise. (__atomic_wait_address_until): Pass __wait_args by address. (__atomic_wait_address_for): Likewise. * include/bits/atomic_wait.h (__wait_args_base): New struct. (__wait_args): Derive from __wait_args_base. (__wait_args::__wait_args()): Adjust ctors to call call base ctor. (__wait_args::__wait_args(const __wait_args_base&)): New ctor. (__wait_args::operator|=): New method. (__wait_args::_S_flags_for): Change return type to __wait_flags. (__spin_impl): Accept __wait_args as const __wait_args_base*. (__wait_impl): Likewise. (__notify_impl): Likewise. (__atomic_wait_address): Pass __wait_args by address. (__atomic_wait_address_v): Likewise. (__atomic_notify_address): Likewise. --- libstdc++-v3/include/bits/atomic_timed_wait.h | 35 +++++++---- libstdc++-v3/include/bits/atomic_wait.h | 62 +++++++++++++------ 2 files changed, 64 insertions(+), 33 deletions(-) diff --git a/libstdc++-v3/include/bits/atomic_timed_wait.h b/libstdc++-v3/include/bits/atomic_timed_wait.h index 196548484024..4504b1b84bb8 100644 --- a/libstdc++-v3/include/bits/atomic_timed_wait.h +++ b/libstdc++-v3/include/bits/atomic_timed_wait.h @@ -135,9 +135,11 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION #endif // _GLIBCXX_HAS_GTHREADS inline __wait_result_type - __spin_until_impl(const __platform_wait_t* __addr, __wait_args __args, + __spin_until_impl(const __platform_wait_t* __addr, + const __wait_args_base* __a, const __wait_clock_t::time_point& __deadline) { + __wait_args __args{ *__a }; auto __t0 = __wait_clock_t::now(); using namespace literals::chrono_literals; @@ -163,7 +165,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION } else { - auto __res = __detail::__spin_impl(__addr, __args); + auto __res = __detail::__spin_impl(__addr, __a); if (__res.first) return __res; } @@ -176,9 +178,11 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION } inline __wait_result_type - __wait_until_impl(const __platform_wait_t* __addr, __wait_args __args, + __wait_until_impl(const __platform_wait_t* __addr, + const __wait_args_base* __a, const __wait_clock_t::time_point& __atime) { + __wait_args __args{ *__a }; #ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT __waiter_pool_impl* __pool = nullptr; #else @@ -200,7 +204,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION if (__args & __wait_flags::__do_spin) { - auto __res = __detail::__spin_until_impl(__wait_addr, __args, __atime); + auto __res = __detail::__spin_until_impl(__wait_addr, __a, __atime); if (__res.first) return __res; if (__args & __wait_flags::__spin_only) @@ -246,7 +250,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION template __wait_result_type - __wait_until(const __platform_wait_t* __addr, __wait_args __args, + __wait_until(const __platform_wait_t* __addr, const __wait_args* __args, const chrono::time_point<_Clock, _Dur>& __atime) noexcept { if constexpr (is_same_v<__wait_clock_t, _Clock>) @@ -269,15 +273,20 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION template __wait_result_type - __wait_for(const __platform_wait_t* __addr, __wait_args __args, + __wait_for(const __platform_wait_t* __addr, const __wait_args_base* __a, const chrono::duration<_Rep, _Period>& __rtime) noexcept { + __wait_args __args{ *__a }; if (!__rtime.count()) - // no rtime supplied, just spin a bit - return __detail::__wait_impl(__addr, __args | __wait_flags::__spin_only); + { + // no rtime supplied, just spin a bit + __args |= __wait_flags::__spin_only; + return __detail::__wait_impl(__addr, &__args); + } + auto const __reltime = chrono::ceil<__wait_clock_t::duration>(__rtime); auto const __atime = chrono::steady_clock::now() + __reltime; - return __detail::__wait_until(__addr, __args, __atime); + return __detail::__wait_until(__addr, &__args, __atime); } } // namespace __detail @@ -297,7 +306,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION _Tp __val = __vfn(); while (!__pred(__val)) { - auto __res = __detail::__wait_until(__wait_addr, __args, __atime); + auto __res = __detail::__wait_until(__wait_addr, &__args, __atime); if (!__res.first) // timed out return __res.first; // C++26 will also return last observed __val @@ -315,7 +324,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION bool __bare_wait = false) noexcept { __detail::__wait_args __args{ __addr, __old, __order, __bare_wait }; - auto __res = __detail::__wait_until(__addr, __args, __atime); + auto __res = __detail::__wait_until(__addr, &__args, __atime); return __res.first; // C++26 will also return last observed __val } @@ -347,7 +356,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION _Tp __val = __vfn(); while (!__pred(__val)) { - auto __res = __detail::__wait_for(__wait_addr, __args, __rtime); + auto __res = __detail::__wait_for(__wait_addr, &__args, __rtime); if (!__res.first) // timed out return __res.first; // C++26 will also return last observed __val @@ -365,7 +374,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION bool __bare_wait = false) noexcept { __detail::__wait_args __args{ __addr, __old, __order, __bare_wait }; - auto __res = __detail::__wait_for(__addr, __args, __rtime); + auto __res = __detail::__wait_for(__addr, &__args, __rtime); return __res.first; // C++26 will also return last observed __Val } diff --git a/libstdc++-v3/include/bits/atomic_wait.h b/libstdc++-v3/include/bits/atomic_wait.h index 18cfc2ef7bd2..cb246ed616d8 100644 --- a/libstdc++-v3/include/bits/atomic_wait.h +++ b/libstdc++-v3/include/bits/atomic_wait.h @@ -215,23 +215,28 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __abi_version_mask = 0xffff0000, }; - struct __wait_args + struct __wait_args_base { - __platform_wait_t _M_old; - int _M_order = __ATOMIC_ACQUIRE; __wait_flags _M_flags; + int _M_order = __ATOMIC_ACQUIRE; + __platform_wait_t _M_old = 0; + }; + struct __wait_args : __wait_args_base + { template explicit __wait_args(const _Tp* __addr, bool __bare_wait = false) noexcept - : _M_flags{ _S_flags_for(__addr, __bare_wait) } + : __wait_args_base{ _S_flags_for(__addr, __bare_wait) } { } __wait_args(const __platform_wait_t* __addr, __platform_wait_t __old, int __order, bool __bare_wait = false) noexcept - : _M_old{ __old } - , _M_order{ __order } - , _M_flags{ _S_flags_for(__addr, __bare_wait) } + : __wait_args_base{ _S_flags_for(__addr, __bare_wait), __order, __old } + { } + + explicit __wait_args(const __wait_args_base& __base) + : __wait_args_base{ __base } { } __wait_args(const __wait_args&) noexcept = default; @@ -257,6 +262,16 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION return __res; } + __wait_args& + operator|=(__wait_flags __flag) noexcept + { + using __t = underlying_type_t<__wait_flags>; + const auto __flags = static_cast<__t>(_M_flags) + | static_cast<__t>(__flag); + _M_flags = __wait_flags{ __flags }; + return *this; + } + private: static int constexpr _S_default_flags() noexcept @@ -267,7 +282,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION } template - static int + static __wait_flags constexpr _S_flags_for(const _Tp*, bool __bare_wait) noexcept { auto __res = _S_default_flags(); @@ -275,7 +290,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __res |= static_cast(__wait_flags::__track_contention); if constexpr (!__platform_wait_uses_type<_Tp>) __res |= static_cast(__wait_flags::__proxy_wait); - return __res; + return static_cast<__wait_flags>(__res); } template @@ -290,13 +305,13 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION using __wait_result_type = pair; inline __wait_result_type - __spin_impl(const __platform_wait_t* __addr, __wait_args __args) + __spin_impl(const __platform_wait_t* __addr, const __wait_args_base* __args) { __platform_wait_t __val; for (auto __i = 0; __i < __atomic_spin_count; ++__i) { - __atomic_load(__addr, &__val, __args._M_order); - if (__val != __args._M_old) + __atomic_load(__addr, &__val, __args->_M_order); + if (__val != __args->_M_old) return make_pair(true, __val); if (__i < __atomic_spin_count_relax) __detail::__thread_relax(); @@ -307,8 +322,9 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION } inline __wait_result_type - __wait_impl(const __platform_wait_t* __addr, __wait_args __args) + __wait_impl(const __platform_wait_t* __addr, const __wait_args_base* __a) { + __wait_args __args{ *__a }; #ifdef _GLIBCXX_HAVE_PLATFORM_WAIT __waiter_pool_impl* __pool = nullptr; #else @@ -317,20 +333,25 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION #endif __platform_wait_t* __wait_addr; + __platform_wait_t __old; if (__args & __wait_flags::__proxy_wait) { #ifdef _GLIBCXX_HAVE_PLATFORM_WAIT __pool = &__waiter_pool_impl::_S_impl_for(__addr); #endif __wait_addr = &__pool->_M_ver; - __atomic_load(__wait_addr, &__args._M_old, __args._M_order); + __atomic_load(__wait_addr, &__old, __args._M_order); } else - __wait_addr = const_cast<__platform_wait_t*>(__addr); + { + __wait_addr = const_cast<__platform_wait_t*>(__addr); + __old = __args._M_old; + } + if (__args & __wait_flags::__do_spin) { - auto __res = __detail::__spin_impl(__wait_addr, __args); + auto __res = __detail::__spin_impl(__wait_addr, __a); if (__res.first) return __res; if (__args & __wait_flags::__spin_only) @@ -372,8 +393,9 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION inline void __notify_impl(const __platform_wait_t* __addr, [[maybe_unused]] bool __all, - __wait_args __args) + const __wait_args_base* __a) { + __wait_args __args{ __a }; #ifdef _GLIBCXX_HAVE_PLATFORM_WAIT __waiter_pool_impl* __pool = nullptr; #else @@ -424,7 +446,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION _Tp __val = __vfn(); while (!__pred(__val)) { - __detail::__wait_impl(__wait_addr, __args); + __detail::__wait_impl(__wait_addr, &__args); __val = __vfn(); } // C++26 will return __val @@ -437,7 +459,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION { __detail::__wait_args __args{ __addr, __old, __order }; // C++26 will not ignore the return value here - __detail::__wait_impl(__addr, __args); + __detail::__wait_impl(__addr, &__args); } template @@ -458,7 +480,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION const auto __wait_addr = reinterpret_cast(__addr); __detail::__wait_args __args{ __addr, __bare_wait }; - __detail::__notify_impl(__wait_addr, __all, __args); + __detail::__notify_impl(__wait_addr, __all, &__args); } _GLIBCXX_END_NAMESPACE_VERSION } // namespace std From patchwork Fri Jan 10 21:23:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 104539 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B99083858039 for ; Fri, 10 Jan 2025 21:42:10 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B99083858039 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=epgGx/3G X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id 275AE3858031 for ; Fri, 10 Jan 2025 21:28:25 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 275AE3858031 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 275AE3858031 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544505; cv=none; b=cm96WKA8RT4ds4Cx5gG3Aq/bPIwTj8gG0kPKcclQuTPcZ3i2iiPmpkLOJO0SkvcM2Mg9Kkhs605KwYD3mb8Rj+OHfUZ3RV4AeMkJXbaNqXpczbMhSzsRfI9XqrOTmVICmQPuzK5DEdKhUeM03g0Xjmq21LkNNM27RQa05w8Ac04= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544505; c=relaxed/simple; bh=0Okhpvzo5rbop8EEV2VjbQxBLdw37+UBVb1hmz/Yv6w=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=j9groAs7Mj0EVSbMZinRcDKQR9Icph7cPJA3Lha+QZUC0msYy1QoO14MSNmDfNpbK2pXastgv6NjeW2YWnJlq5cfWJazFeGSFQxd8vxB8IXUIYrVuV5PEVyCwD2YSiEb+FxIbPpd1kexFdCsV859H4udQEyyWdEE5pdhEe80HmY= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 275AE3858031 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736544504; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ykb8vT7eoduCXn+OBGZK1f8rWe5rzFY3khguuarU9W4=; b=epgGx/3GVYUL2zeVYrcKFsDHWRIXn+GdEoHvJziJ2J5Z5V4rPp6HCTin4D3VUDGCkkDkWa 4X5WhLVmnvOjGsEWbPvsazLITMM0frDhYZ+7VaG9eqVhF3HYxVI7FLAEaYL1tI1AiY8zjB +KU7NDAQBeLMjzr6Jfrl7VgbTNjbOtc= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-675-ZrAvYdLaMyqzxKYC4FgGFg-1; Fri, 10 Jan 2025 16:28:21 -0500 X-MC-Unique: ZrAvYdLaMyqzxKYC4FgGFg-1 X-Mimecast-MFC-AGG-ID: ZrAvYdLaMyqzxKYC4FgGFg Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 672C01956055; Fri, 10 Jan 2025 21:28:20 +0000 (UTC) Received: from localhost (unknown [10.42.28.9]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id BEEF71944D08; Fri, 10 Jan 2025 21:28:19 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH 03/10] libstdc++: Whitespace fixes in atomic wait/notify code Date: Fri, 10 Jan 2025 21:23:06 +0000 Message-ID: <20250110212810.832494-4-jwakely@redhat.com> In-Reply-To: <20250110212810.832494-1-jwakely@redhat.com> References: <20250110212810.832494-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Q0lzcWKLDt8lDAlVsDzdzmGaxmHZBclkGyRPHUcRx0U_1736544500 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org libstdc++-v3/ChangeLog: * include/bits/atomic_timed_wait.h: Whitespace fixes. * include/bits/atomic_wait.h: Likewise. --- libstdc++-v3/include/bits/atomic_timed_wait.h | 198 +++++++++--------- libstdc++-v3/include/bits/atomic_wait.h | 8 +- 2 files changed, 100 insertions(+), 106 deletions(-) diff --git a/libstdc++-v3/include/bits/atomic_timed_wait.h b/libstdc++-v3/include/bits/atomic_timed_wait.h index 4504b1b84bb8..73acea939504 100644 --- a/libstdc++-v3/include/bits/atomic_timed_wait.h +++ b/libstdc++-v3/include/bits/atomic_timed_wait.h @@ -101,8 +101,8 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION if (errno != EINTR && errno != EAGAIN) __throw_system_error(errno); } - return true; - } + return true; + } #else // define _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT and implement __platform_wait_until() // if there is a more efficient primitive supported by the platform @@ -115,23 +115,23 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __cond_wait_until(__condvar& __cv, mutex& __mx, const __wait_clock_t::time_point& __atime) { - auto __s = chrono::time_point_cast(__atime); - auto __ns = chrono::duration_cast(__atime - __s); + auto __s = chrono::time_point_cast(__atime); + auto __ns = chrono::duration_cast(__atime - __s); - __gthread_time_t __ts = - { - static_cast(__s.time_since_epoch().count()), - static_cast(__ns.count()) - }; + __gthread_time_t __ts = + { + static_cast(__s.time_since_epoch().count()), + static_cast(__ns.count()) + }; #ifdef _GLIBCXX_USE_PTHREAD_COND_CLOCKWAIT - if constexpr (is_same_v) - __cv.wait_until(__mx, CLOCK_MONOTONIC, __ts); - else + if constexpr (is_same_v) + __cv.wait_until(__mx, CLOCK_MONOTONIC, __ts); + else #endif - __cv.wait_until(__mx, __ts); - return __wait_clock_t::now() < __atime; - } + __cv.wait_until(__mx, __ts); + return __wait_clock_t::now() < __atime; + } #endif // _GLIBCXX_HAS_GTHREADS inline __wait_result_type @@ -146,34 +146,28 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __platform_wait_t __val; auto __now = __wait_clock_t::now(); for (; __now < __deadline; __now = __wait_clock_t::now()) - { - auto __elapsed = __now - __t0; + { + auto __elapsed = __now - __t0; #ifndef _GLIBCXX_NO_SLEEP - if (__elapsed > 128ms) - { - this_thread::sleep_for(64ms); - } - else if (__elapsed > 64us) - { - this_thread::sleep_for(__elapsed / 2); - } - else + if (__elapsed > 128ms) + this_thread::sleep_for(64ms); + else if (__elapsed > 64us) + this_thread::sleep_for(__elapsed / 2); + else #endif - if (__elapsed > 4us) - { - __thread_yield(); - } - else - { - auto __res = __detail::__spin_impl(__addr, __a); - if (__res.first) - return __res; - } + if (__elapsed > 4us) + __thread_yield(); + else + { + auto __res = __detail::__spin_impl(__addr, __a); + if (__res.first) + return __res; + } - __atomic_load(__addr, &__val, __args._M_order); - if (__val != __args._M_old) - return make_pair(true, __val); - } + __atomic_load(__addr, &__val, __args._M_order); + if (__val != __args._M_old) + return make_pair(true, __val); + } return make_pair(false, __val); } @@ -212,14 +206,14 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION } if (!(__args & __wait_flags::__track_contention)) - { - // caller does not externally track contention + { + // caller does not externally track contention #ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT - __pool = (__pool == nullptr) ? &__waiter_pool_impl::_S_impl_for(__addr) - : __pool; + __pool = (__pool == nullptr) ? &__waiter_pool_impl::_S_impl_for(__addr) + : __pool; #endif - __pool->_M_enter_wait(); - } + __pool->_M_enter_wait(); + } __wait_result_type __res; #ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT @@ -232,19 +226,19 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __atomic_load(__wait_addr, &__val, __args._M_order); if (__val == __args._M_old) { - lock_guard __l{ __pool->_M_mtx }; - __atomic_load(__wait_addr, &__val, __args._M_order); - if (__val == __args._M_old && - __cond_wait_until(__pool->_M_cv, __pool->_M_mtx, __atime)) - __res = make_pair(true, __val); + lock_guard __l{ __pool->_M_mtx }; + __atomic_load(__wait_addr, &__val, __args._M_order); + if (__val == __args._M_old && + __cond_wait_until(__pool->_M_cv, __pool->_M_mtx, __atime)) + __res = make_pair(true, __val); } else __res = make_pair(false, __val); #endif if (!(__args & __wait_flags::__track_contention)) - // caller does not externally track contention - __pool->_M_leave_wait(); + // caller does not externally track contention + __pool->_M_leave_wait(); return __res; } @@ -257,37 +251,37 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION return __detail::__wait_until_impl(__addr, __args, __atime); else { - auto __res = __detail::__wait_until_impl(__addr, __args, - __to_wait_clock(__atime)); - if (!__res.first) - { - // We got a timeout when measured against __clock_t but - // we need to check against the caller-supplied clock - // to tell whether we should return a timeout. - if (_Clock::now() < __atime) - return make_pair(true, __res.second); - } - return __res; - } + auto __res = __detail::__wait_until_impl(__addr, __args, + __to_wait_clock(__atime)); + if (!__res.first) + { + // We got a timeout when measured against __clock_t but + // we need to check against the caller-supplied clock + // to tell whether we should return a timeout. + if (_Clock::now() < __atime) + return make_pair(true, __res.second); + } + return __res; + } } template __wait_result_type __wait_for(const __platform_wait_t* __addr, const __wait_args_base* __a, const chrono::duration<_Rep, _Period>& __rtime) noexcept - { - __wait_args __args{ *__a }; - if (!__rtime.count()) - { - // no rtime supplied, just spin a bit - __args |= __wait_flags::__spin_only; - return __detail::__wait_impl(__addr, &__args); - } + { + __wait_args __args{ *__a }; + if (!__rtime.count()) + { + // no rtime supplied, just spin a bit + __args |= __wait_flags::__spin_only; + return __detail::__wait_impl(__addr, &__args); + } - auto const __reltime = chrono::ceil<__wait_clock_t::duration>(__rtime); - auto const __atime = chrono::steady_clock::now() + __reltime; - return __detail::__wait_until(__addr, &__args, __atime); - } + auto const __reltime = chrono::ceil<__wait_clock_t::duration>(__rtime); + auto const __atime = chrono::steady_clock::now() + __reltime; + return __detail::__wait_until(__addr, &__args, __atime); + } } // namespace __detail // returns true if wait ended before timeout @@ -300,19 +294,19 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION const chrono::time_point<_Clock, _Dur>& __atime, bool __bare_wait = false) noexcept { - const auto __wait_addr = - reinterpret_cast(__addr); - __detail::__wait_args __args{ __addr, __bare_wait }; - _Tp __val = __vfn(); - while (!__pred(__val)) - { - auto __res = __detail::__wait_until(__wait_addr, &__args, __atime); - if (!__res.first) - // timed out - return __res.first; // C++26 will also return last observed __val - __val = __vfn(); - } - return true; // C++26 will also return last observed __val + const auto __wait_addr = + reinterpret_cast(__addr); + __detail::__wait_args __args{ __addr, __bare_wait }; + _Tp __val = __vfn(); + while (!__pred(__val)) + { + auto __res = __detail::__wait_until(__wait_addr, &__args, __atime); + if (!__res.first) + // timed out + return __res.first; // C++26 will also return last observed __val + __val = __vfn(); + } + return true; // C++26 will also return last observed __val } template @@ -344,12 +338,12 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION template - bool - __atomic_wait_address_for(const _Tp* __addr, _Pred&& __pred, - _ValFn&& __vfn, - const chrono::duration<_Rep, _Period>& __rtime, - bool __bare_wait = false) noexcept - { + bool + __atomic_wait_address_for(const _Tp* __addr, _Pred&& __pred, + _ValFn&& __vfn, + const chrono::duration<_Rep, _Period>& __rtime, + bool __bare_wait = false) noexcept + { const auto __wait_addr = reinterpret_cast(__addr); __detail::__wait_args __args{ __addr, __bare_wait }; @@ -363,7 +357,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __val = __vfn(); } return true; // C++26 will also return last observed __val - } + } template bool @@ -372,11 +366,11 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION int __order, const chrono::time_point<_Rep, _Period>& __rtime, bool __bare_wait = false) noexcept - { - __detail::__wait_args __args{ __addr, __old, __order, __bare_wait }; - auto __res = __detail::__wait_for(__addr, &__args, __rtime); - return __res.first; // C++26 will also return last observed __Val - } + { + __detail::__wait_args __args{ __addr, __old, __order, __bare_wait }; + auto __res = __detail::__wait_for(__addr, &__args, __rtime); + return __res.first; // C++26 will also return last observed __Val + } template diff --git a/libstdc++-v3/include/bits/atomic_wait.h b/libstdc++-v3/include/bits/atomic_wait.h index cb246ed616d8..ebab4b099e66 100644 --- a/libstdc++-v3/include/bits/atomic_wait.h +++ b/libstdc++-v3/include/bits/atomic_wait.h @@ -377,10 +377,10 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __atomic_load(__wait_addr, &__val, __args._M_order); if (__val == __args._M_old) { - lock_guard __l{ __pool->_M_mtx }; - __atomic_load(__wait_addr, &__val, __args._M_order); - if (__val == __args._M_old) - __pool->_M_cv.wait(__pool->_M_mtx); + lock_guard __l{ __pool->_M_mtx }; + __atomic_load(__wait_addr, &__val, __args._M_order); + if (__val == __args._M_old) + __pool->_M_cv.wait(__pool->_M_mtx); } __res = make_pair(false, __val); #endif From patchwork Fri Jan 10 21:23:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 104534 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id BF44E3857B8F for ; Fri, 10 Jan 2025 21:31:28 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BF44E3857B8F Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Z5ThhA2/ X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id F23C63857C58 for ; Fri, 10 Jan 2025 21:28:26 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org F23C63857C58 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org F23C63857C58 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544507; cv=none; b=JW9/9cmf5puRFvI84uyQiJtinahAwunHawpqpvCgrJ9OePOOa7T6pNS1+ZBjESakNc5VhyipdgONAgKSqhbmnLX5vCBdq0p3FHOC2+avqyX962/edyO2ib567/5/mzb6Hu85uWfuUjVYYzDb2SFEA+jM8553iE1Fr50yN+2PaIc= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544507; c=relaxed/simple; bh=bnAC3wvMaDtNmG/mdQNe0hNLVaoGmnMZHOvh7JreRVo=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=YuJXnggX/1QJhjksAdtdphRJ5WcT6dvLZ5Gs/0fElUv6UmFMpbep+Aq5vetq7zltWegRDN05m2BjyIrXzMTcly2Wpl96WDN9LBan5qbLkQGmYrXBWd0kbv673Yya8dthaSBpLv11GrPK34NLKszcMiHs8A4JZ7kop1uf3+hBnHc= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org F23C63857C58 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736544506; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d2SQIAwFx6G1VfOmYWck2YVCV5gHMnAffFlLkUrPByg=; b=Z5ThhA2/Y7/fPr6iV62Gc/25vAMNXinEjg36r8ohM+I6xPIYVBawxTsdDXqouWd2Kp3gQZ IHo0VCb5pkmNj56HyMEELMw1wgOOrO1dDpxKuuAwjf8w9xNtUPKs4tT89dTHvLxoujvMOw 8GiCjzaQQYXHJpLr36O/YwP0r/Mym04= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-635-6-3WLqcwO_ulum8uhoBLTQ-1; Fri, 10 Jan 2025 16:28:23 -0500 X-MC-Unique: 6-3WLqcwO_ulum8uhoBLTQ-1 X-Mimecast-MFC-AGG-ID: 6-3WLqcwO_ulum8uhoBLTQ Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A60851955D8A; Fri, 10 Jan 2025 21:28:22 +0000 (UTC) Received: from localhost (unknown [10.42.28.9]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0951719560AD; Fri, 10 Jan 2025 21:28:21 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH 04/10] libstdc++: Various fixes for atomic wait/notify code Date: Fri, 10 Jan 2025 21:23:07 +0000 Message-ID: <20250110212810.832494-5-jwakely@redhat.com> In-Reply-To: <20250110212810.832494-1-jwakely@redhat.com> References: <20250110212810.832494-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Oc-BqSm2CaO0UGXsLOmi0sXJ1aRs7zffVz17-yNUS60_1736544502 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Remove the unnecessary __wait_args_base base class, which seems to serve no purpose. Everywhere it's used we actually want __wait_args instead. The code using the __wait_flags bitmask type is broken, because the __spin_only constant includes the __do_spin element. This means that testing (__args & __wait_flags::__spin_only) will be inadvertently true when only __do_spin is set. This causes the __wait_until_impl function to never actually wait on the futex (or condition variable), turning all uses of that function into expensive busy spins. Change __spin_only to be a single bit (i.e. a bitmask element) and adjust the places where that bit is set so that they also use the __do_spin element. Update the __args._M_old value when looping in __atomic_wait_address, so that the next wait doesn't fail spuriously. With the new __atomic_wait_address logic, the value function needs to return the correct type, not just a bool. Without that change, the boolean value returned by the value function is used as the value passed to the futex wait, but that mean we're comparing (_M_a == 0) to _M_a and so can block on the futex when we shouldn't, and then never wake up. libstdc++-v3/ChangeLog: * include/bits/atomic_timed_wait.h (__cond_wait_impl): Add missing inline keyword. (__spin_until_impl): Change parameter from __wait_args_base* to __wait_args*. Replace make_pair with list-initialization. Initialize variable for return value. (__wait_until_impl): Likewise. Remove some preprocessor conditional logic. Use _M_tracker() for contention tracking. Avoid unnecessary const_cast. (__wait_until): Replace make_pair with list-initialization. (__wait_for): Change parameter from __wait_args_base* to __wait_args*. Add __do_spin flag to args. * include/bits/atomic_wait.h (__wait_flags): Do not set the __do_spin flag in the __spin_only enumerator. Comment out the unused __abi_version_mask enumerator. Define operator| and operator|= overloads. (__wait_args_base): Remove. (__wait_args): Move __wait_args_base members here and remove the base class. Constrain constructor to avoid accidents. (__wait_args::_M_tracker): Helper function to create an RAII object to track contention. (__spin_impl): Change parameter from __wait_args_base* to __wait_args*. Replace make_pair call with list-initialization. (__wait_impl): Likewise. Remove some preprocessor conditional logic. Always store old value in __args._M_old. Avoid unnecessary const_cast. Use _M_tracker. (__notify_impl): Change parameter to __wait_args*. Remove some preprocessor conditional logic. (__atomic_wait_address): Add comment. Update __args._M_old on each iteration. (__atomic_wait_address_v): Add comment. * include/std/latch (latch::wait): Adjust predicates for new logic. * testsuite/29_atomics/atomic_integral/wait_notify.cc: Improve test. --- libstdc++-v3/include/bits/atomic_timed_wait.h | 73 ++---- libstdc++-v3/include/bits/atomic_wait.h | 221 +++++++++--------- libstdc++-v3/include/std/latch | 8 +- .../29_atomics/atomic_integral/wait_notify.cc | 4 + 4 files changed, 147 insertions(+), 159 deletions(-) diff --git a/libstdc++-v3/include/bits/atomic_timed_wait.h b/libstdc++-v3/include/bits/atomic_timed_wait.h index 73acea939504..7e2017f2f515 100644 --- a/libstdc++-v3/include/bits/atomic_timed_wait.h +++ b/libstdc++-v3/include/bits/atomic_timed_wait.h @@ -111,7 +111,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION #ifdef _GLIBCXX_HAS_GTHREADS // Returns true if wait ended before timeout. - bool + inline bool __cond_wait_until(__condvar& __cv, mutex& __mx, const __wait_clock_t::time_point& __atime) { @@ -136,14 +136,14 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION inline __wait_result_type __spin_until_impl(const __platform_wait_t* __addr, - const __wait_args_base* __a, + const __wait_args* __a, const __wait_clock_t::time_point& __deadline) { __wait_args __args{ *__a }; auto __t0 = __wait_clock_t::now(); using namespace literals::chrono_literals; - __platform_wait_t __val; + __platform_wait_t __val{}; auto __now = __wait_clock_t::now(); for (; __now < __deadline; __now = __wait_clock_t::now()) { @@ -157,44 +157,32 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION #endif if (__elapsed > 4us) __thread_yield(); - else - { - auto __res = __detail::__spin_impl(__addr, __a); - if (__res.first) - return __res; - } + else if (auto __res = __detail::__spin_impl(__addr, __a); __res.first) + return __res; __atomic_load(__addr, &__val, __args._M_order); if (__val != __args._M_old) - return make_pair(true, __val); + return { true, __val }; } - return make_pair(false, __val); + return { false, __val }; } inline __wait_result_type __wait_until_impl(const __platform_wait_t* __addr, - const __wait_args_base* __a, + const __wait_args* __a, const __wait_clock_t::time_point& __atime) { __wait_args __args{ *__a }; -#ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT __waiter_pool_impl* __pool = nullptr; -#else - // if we don't have __platform_wait, we always need the side-table - __waiter_pool_impl* __pool = &__waiter_pool_impl::_S_impl_for(__addr); -#endif - - __platform_wait_t* __wait_addr; + const __platform_wait_t* __wait_addr; if (__args & __wait_flags::__proxy_wait) { -#ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT __pool = &__waiter_pool_impl::_S_impl_for(__addr); -#endif __wait_addr = &__pool->_M_ver; __atomic_load(__wait_addr, &__args._M_old, __args._M_order); } else - __wait_addr = const_cast<__platform_wait_t*>(__addr); + __wait_addr = __addr; if (__args & __wait_flags::__do_spin) { @@ -205,43 +193,31 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION return __res; } - if (!(__args & __wait_flags::__track_contention)) - { - // caller does not externally track contention -#ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT - __pool = (__pool == nullptr) ? &__waiter_pool_impl::_S_impl_for(__addr) - : __pool; -#endif - __pool->_M_enter_wait(); - } + auto __tracker = __args._M_tracker(__pool, __addr); - __wait_result_type __res; #ifdef _GLIBCXX_HAVE_PLATFORM_TIMED_WAIT if (__platform_wait_until(__wait_addr, __args._M_old, __atime)) - __res = make_pair(true, __args._M_old); + return { true, __args._M_old }; else - __res = make_pair(false, __args._M_old); + return { false, __args._M_old }; #else - __platform_wait_t __val; + __platform_wait_t __val{}; __atomic_load(__wait_addr, &__val, __args._M_order); if (__val == __args._M_old) { + if (!__pool) + __pool = &__waiter_pool_impl::_S_impl_for(__addr); lock_guard __l{ __pool->_M_mtx }; __atomic_load(__wait_addr, &__val, __args._M_order); - if (__val == __args._M_old && - __cond_wait_until(__pool->_M_cv, __pool->_M_mtx, __atime)) - __res = make_pair(true, __val); + if (__val == __args._M_old + && __cond_wait_until(__pool->_M_cv, __pool->_M_mtx, __atime)) + return { true, __val }; } - else - __res = make_pair(false, __val); + return { false, __val }; #endif - - if (!(__args & __wait_flags::__track_contention)) - // caller does not externally track contention - __pool->_M_leave_wait(); - return __res; } + // Returns {true, val} if wait ended before a timeout. template __wait_result_type __wait_until(const __platform_wait_t* __addr, const __wait_args* __args, @@ -259,22 +235,23 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION // we need to check against the caller-supplied clock // to tell whether we should return a timeout. if (_Clock::now() < __atime) - return make_pair(true, __res.second); + __res.first = true; } return __res; } } + // Returns {true, val} if wait ended before a timeout. template __wait_result_type - __wait_for(const __platform_wait_t* __addr, const __wait_args_base* __a, + __wait_for(const __platform_wait_t* __addr, const __wait_args* __a, const chrono::duration<_Rep, _Period>& __rtime) noexcept { __wait_args __args{ *__a }; if (!__rtime.count()) { // no rtime supplied, just spin a bit - __args |= __wait_flags::__spin_only; + __args |= __wait_flags::__do_spin | __wait_flags::__spin_only; return __detail::__wait_impl(__addr, &__args); } diff --git a/libstdc++-v3/include/bits/atomic_wait.h b/libstdc++-v3/include/bits/atomic_wait.h index ebab4b099e66..29b83cad6e6c 100644 --- a/libstdc++-v3/include/bits/atomic_wait.h +++ b/libstdc++-v3/include/bits/atomic_wait.h @@ -177,6 +177,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION #ifndef _GLIBCXX_HAVE_PLATFORM_WAIT __condvar _M_cv; #endif + __waiter_pool_impl() = default; void @@ -211,88 +212,104 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __proxy_wait = 1, __track_contention = 2, __do_spin = 4, - __spin_only = 8 | __do_spin, // implies __do_spin - __abi_version_mask = 0xffff0000, + __spin_only = 8, // Ignored unless __do_spin is also set. + // __abi_version_mask = 0xffff0000, }; - struct __wait_args_base + [[__gnu__::__always_inline__]] + constexpr __wait_flags + operator|(__wait_flags __l, __wait_flags __r) noexcept + { + using _Ut = underlying_type_t<__wait_flags>; + return static_cast<__wait_flags>(static_cast<_Ut>(__l) + | static_cast<_Ut>(__r)); + } + + [[__gnu__::__always_inline__]] + constexpr __wait_flags& + operator|=(__wait_flags& __l, __wait_flags __r) noexcept + { return __l = __l | __r; } + + struct __wait_args { __wait_flags _M_flags; int _M_order = __ATOMIC_ACQUIRE; __platform_wait_t _M_old = 0; - }; - struct __wait_args : __wait_args_base - { - template - explicit __wait_args(const _Tp* __addr, - bool __bare_wait = false) noexcept - : __wait_args_base{ _S_flags_for(__addr, __bare_wait) } + template requires (!is_same_v<_Tp, __wait_args>) + explicit + __wait_args(const _Tp* __addr, bool __bare_wait = false) noexcept + : _M_flags{ _S_flags_for(__addr, __bare_wait) } { } __wait_args(const __platform_wait_t* __addr, __platform_wait_t __old, int __order, bool __bare_wait = false) noexcept - : __wait_args_base{ _S_flags_for(__addr, __bare_wait), __order, __old } - { } - - explicit __wait_args(const __wait_args_base& __base) - : __wait_args_base{ __base } - { } + : _M_flags{ _S_flags_for(__addr, __bare_wait)}, + _M_order{__order}, _M_old{__old} + { } __wait_args(const __wait_args&) noexcept = default; - __wait_args& - operator=(const __wait_args&) noexcept = default; + __wait_args& operator=(const __wait_args&) noexcept = default; + // Test whether _M_flags & __flag is non-zero. bool - operator&(__wait_flags __flag) const noexcept + operator&(__wait_flags __flags) const noexcept { - using __t = underlying_type_t<__wait_flags>; - return static_cast<__t>(_M_flags) - & static_cast<__t>(__flag); - } - - __wait_args - operator|(__wait_flags __flag) const noexcept - { - using __t = underlying_type_t<__wait_flags>; - __wait_args __res{ *this }; - const auto __flags = static_cast<__t>(__res._M_flags) - | static_cast<__t>(__flag); - __res._M_flags = __wait_flags{ __flags }; - return __res; + using _Ut = underlying_type_t<__wait_flags>; + return static_cast<_Ut>(_M_flags) & static_cast<_Ut>(__flags); } + // Set __flags in _M_flags. __wait_args& - operator|=(__wait_flags __flag) noexcept + operator|=(__wait_flags __flags) noexcept { - using __t = underlying_type_t<__wait_flags>; - const auto __flags = static_cast<__t>(_M_flags) - | static_cast<__t>(__flag); - _M_flags = __wait_flags{ __flags }; + _M_flags |= __flags; return *this; } - private: - static int - constexpr _S_default_flags() noexcept + // Return an RAII type that calls __pool->_M_leave_wait() on destruction. + auto + _M_tracker(__waiter_pool_impl* __pool, const void* __addr) const { - using __t = underlying_type_t<__wait_flags>; - return static_cast<__t>(__wait_flags::__abi_version) - | static_cast<__t>(__wait_flags::__do_spin); + struct _Guard + { + explicit _Guard(__waiter_pool_impl* __p) : _M_pool(__p) { } + _Guard(const _Guard&) = delete; + ~_Guard() { if (_M_pool) _M_pool->_M_leave_wait(); } + __waiter_pool_impl* _M_pool; + }; + + if (*this & __wait_flags::__track_contention) + { + // Caller does not externally track contention, + // so we want to increment+decrement __pool->_M_waiters + + // First make sure we have a pool for the address. + if (!__pool) + __pool = &__waiter_pool_impl::_S_impl_for(__addr); + // Increment number of waiters: + __pool->_M_enter_wait(); + // Returned _Guard will decrement it again on destruction. + return _Guard{__pool}; + } + return _Guard{nullptr}; // For bare waits caller tracks waiters. } + private: template - static __wait_flags - constexpr _S_flags_for(const _Tp*, bool __bare_wait) noexcept + static constexpr __wait_flags + _S_flags_for(const _Tp*, bool __bare_wait) noexcept { - auto __res = _S_default_flags(); + using enum __wait_flags; + __wait_flags __res = __abi_version | __do_spin; if (!__bare_wait) - __res |= static_cast(__wait_flags::__track_contention); + __res |= __track_contention; if constexpr (!__platform_wait_uses_type<_Tp>) - __res |= static_cast(__wait_flags::__proxy_wait); - return static_cast<__wait_flags>(__res); + __res |= __proxy_wait; + return __res; } + // XXX what is this for? It's never used. template static int _S_memory_order_for(const _Tp*, int __order) noexcept @@ -304,50 +321,39 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION }; using __wait_result_type = pair; + inline __wait_result_type - __spin_impl(const __platform_wait_t* __addr, const __wait_args_base* __args) + __spin_impl(const __platform_wait_t* __addr, const __wait_args* __args) { __platform_wait_t __val; for (auto __i = 0; __i < __atomic_spin_count; ++__i) { __atomic_load(__addr, &__val, __args->_M_order); if (__val != __args->_M_old) - return make_pair(true, __val); + return { true, __val }; if (__i < __atomic_spin_count_relax) __detail::__thread_relax(); else __detail::__thread_yield(); } - return make_pair(false, __val); + return { false, __val }; } inline __wait_result_type - __wait_impl(const __platform_wait_t* __addr, const __wait_args_base* __a) + __wait_impl(const __platform_wait_t* __addr, const __wait_args* __a) { __wait_args __args{ *__a }; -#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT __waiter_pool_impl* __pool = nullptr; -#else - // if we don't have __platform_wait, we always need the side-table - __waiter_pool_impl* __pool = &__waiter_pool_impl::_S_impl_for(__addr); -#endif - __platform_wait_t* __wait_addr; - __platform_wait_t __old; + const __platform_wait_t* __wait_addr; if (__args & __wait_flags::__proxy_wait) { -#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT __pool = &__waiter_pool_impl::_S_impl_for(__addr); -#endif __wait_addr = &__pool->_M_ver; - __atomic_load(__wait_addr, &__old, __args._M_order); + __atomic_load(__wait_addr, &__args._M_old, __args._M_order); } else - { - __wait_addr = const_cast<__platform_wait_t*>(__addr); - __old = __args._M_old; - } - + __wait_addr = __addr; if (__args & __wait_flags::__do_spin) { @@ -358,86 +364,75 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION return __res; } - if (!(__args & __wait_flags::__track_contention)) - { - // caller does not externally track contention -#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT - __pool = (__pool == nullptr) ? &__waiter_pool_impl::_S_impl_for(__addr) - : __pool; -#endif - __pool->_M_enter_wait(); - } + auto __tracker = __args._M_tracker(__pool, __addr); - __wait_result_type __res; #ifdef _GLIBCXX_HAVE_PLATFORM_WAIT __platform_wait(__wait_addr, __args._M_old); - __res = make_pair(false, __args._M_old); + return { false, __args._M_old }; #else __platform_wait_t __val; __atomic_load(__wait_addr, &__val, __args._M_order); if (__val == __args._M_old) { + if (!__pool) + __pool = &__waiter_pool_impl::_S_impl_for(__addr); lock_guard __l{ __pool->_M_mtx }; __atomic_load(__wait_addr, &__val, __args._M_order); if (__val == __args._M_old) __pool->_M_cv.wait(__pool->_M_mtx); } - __res = make_pair(false, __val); + return { false, __val }; #endif - - if (!(__args & __wait_flags::__track_contention)) - // caller does not externally track contention - __pool->_M_leave_wait(); - return __res; } inline void __notify_impl(const __platform_wait_t* __addr, [[maybe_unused]] bool __all, - const __wait_args_base* __a) + const __wait_args* __a) { - __wait_args __args{ __a }; -#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT + __wait_args __args{ *__a }; __waiter_pool_impl* __pool = nullptr; -#else - // if we don't have __platform_notify, we always need the side-table - __waiter_pool_impl* __pool = &__waiter_pool_impl::_S_impl_for(__addr); -#endif - if (!(__args & __wait_flags::__track_contention)) + if (__args & __wait_flags::__track_contention) { -#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT __pool = &__waiter_pool_impl::_S_impl_for(__addr); -#endif if (!__pool->_M_waiting()) return; } - __platform_wait_t* __wait_addr; + const __platform_wait_t* __wait_addr; if (__args & __wait_flags::__proxy_wait) { -#ifdef _GLIBCXX_HAVE_PLATFORM_WAIT - __pool = (__pool == nullptr) ? &__waiter_pool_impl::_S_impl_for(__addr) - : __pool; -#endif - __wait_addr = &__pool->_M_ver; - __atomic_fetch_add(__wait_addr, 1, __ATOMIC_RELAXED); - __all = true; - } + if (!__pool) + __pool = &__waiter_pool_impl::_S_impl_for(__addr); + // Waiting for *__addr is actually done on the proxy's _M_ver. + __wait_addr = &__pool->_M_ver; + __atomic_fetch_add(&__pool->_M_ver, 1, __ATOMIC_RELAXED); + // Because the proxy might be shared by several waiters waiting + // on different atomic variables, we need to wake them all so + // they can re-evaluate their conditions to see if they should + // stop waiting or should wait again. + __all = true; + } + else // Use the atomic variable's own address. + __wait_addr = __addr; #ifdef _GLIBCXX_HAVE_PLATFORM_WAIT __platform_notify(__wait_addr, __all); #else + if (!__pool) + __pool = &__waiter_pool_impl::_S_impl_for(__addr); lock_guard __l{ __pool->_M_mtx }; __pool->_M_cv.notify_all(); #endif } } // namespace __detail - template + // Wait on __addr while __pred(__vfn()) is false. + // If __bare_wait is false, increment a counter while waiting. + // For callers that keep their own count of waiters, use __bare_wait=true. + template void - __atomic_wait_address(const _Tp* __addr, - _Pred&& __pred, _ValFn&& __vfn, + __atomic_wait_address(const _Tp* __addr, _Pred&& __pred, _ValFn&& __vfn, bool __bare_wait = false) noexcept { const auto __wait_addr = @@ -446,6 +441,13 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION _Tp __val = __vfn(); while (!__pred(__val)) { + // If the wait is not proxied, set the value that we're waiting + // to change. + if constexpr (__platform_wait_uses_type<_Tp>) + __args._M_old = __builtin_bit_cast(__detail::__platform_wait_t, + __val); + // Otherwise, it's a proxy wait and the proxy's _M_ver is used. + __detail::__wait_impl(__wait_addr, &__args); __val = __vfn(); } @@ -462,6 +464,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __detail::__wait_impl(__addr, &__args); } + // Wait on __addr while __vfn() == __old is true. template void __atomic_wait_address_v(const _Tp* __addr, _Tp __old, diff --git a/libstdc++-v3/include/std/latch b/libstdc++-v3/include/std/latch index de0afd8989bf..c81a6631d53f 100644 --- a/libstdc++-v3/include/std/latch +++ b/libstdc++-v3/include/std/latch @@ -78,8 +78,12 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION _GLIBCXX_ALWAYS_INLINE void wait() const noexcept { - auto const __vfn = [this] { return this->try_wait(); }; - auto const __pred = [this](bool __b) { return __b; }; + auto const __vfn = [this] { + return __atomic_impl::load(&_M_a, memory_order::acquire); + }; + auto const __pred = [](__detail::__platform_wait_t __v) { + return __v == 0; + }; std::__atomic_wait_address(&_M_a, __pred, __vfn); } diff --git a/libstdc++-v3/testsuite/29_atomics/atomic_integral/wait_notify.cc b/libstdc++-v3/testsuite/29_atomics/atomic_integral/wait_notify.cc index c7f8779e4fb2..c6b7f637a2b4 100644 --- a/libstdc++-v3/testsuite/29_atomics/atomic_integral/wait_notify.cc +++ b/libstdc++-v3/testsuite/29_atomics/atomic_integral/wait_notify.cc @@ -33,12 +33,16 @@ template std::atomic a{ Tp(1) }; VERIFY( a.load() == Tp(1) ); a.wait( Tp(0) ); + std::atomic b{false}; std::thread t([&] { + b.store(true, std::memory_order_relaxed); a.store(Tp(0)); a.notify_one(); }); a.wait(Tp(1)); + // Ensure we actually waited until a.store(0) happened: + VERIFY( b.load(std::memory_order_relaxed) ); t.join(); } From patchwork Fri Jan 10 21:23:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 104544 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 819E6385828E for ; Fri, 10 Jan 2025 21:44:59 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 819E6385828E Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YoVKWdb/ X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id DC01A3858427 for ; Fri, 10 Jan 2025 21:28:27 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org DC01A3858427 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org DC01A3858427 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544508; cv=none; b=Z8yVUAMzrmYuHRep0M9Pox2xwY47Vcu2JV3tZECfp4teHUQpvYqN33a//DSCUvl1c22OAFTo2jnotUAU7RxtoESFJj4EenpO6Px0+TxyILe/TKikmgQIdWYHTZm1uJSuapKv+RjUHiD8P2T9SdspWSLrJPl0OtTZVACH2mdSeHg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544508; c=relaxed/simple; bh=/zSTl9J+VkQiLx/nJniKDmx33mK/zT3rtJi0IUgA3LI=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=O6NVIpwbtclfvoU0VbFyG3n9V8LV0xD9/8ySt0WrKKvMOQzN01l1w6IV72UkePFyU6tMlscbjzxKiura0REHXC4wmkxnl7hZeGAokEA/83TtsO8B3g7UAZfYQ+sJ2KtViHefKjkeKAQpIYDfo7wFs5KbnfOttroEKhSOm0drRFE= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org DC01A3858427 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736544507; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SL6TjAxboyXln06LecGsXNYSMzDpPsgTIEIUtFy6bUA=; b=YoVKWdb/+rk6pLs+tOBZZAxyFIEZSARwT+OSFTGYgKaYGMN6S48L2XeVKbYGE/FBfLZaaV jyDFSxxnWcYWyYizfPv6Rod5Z7v0eKf6K3+V6b3CrUx7WWxBCic7d0koQ3UDUwxDISe0xT LN+OKeTKmuXzZmF+gXRQIXoxNnIPEQA= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-225-g8RoXtAuOqe7M2bePM4mpw-1; Fri, 10 Jan 2025 16:28:26 -0500 X-MC-Unique: g8RoXtAuOqe7M2bePM4mpw-1 X-Mimecast-MFC-AGG-ID: g8RoXtAuOqe7M2bePM4mpw Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1ED911956070; Fri, 10 Jan 2025 21:28:25 +0000 (UTC) Received: from localhost (unknown [10.42.28.9]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7FF8A1944D10; Fri, 10 Jan 2025 21:28:24 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH 05/10] libstdc++: Fix race condition in new atomic notify code Date: Fri, 10 Jan 2025 21:23:08 +0000 Message-ID: <20250110212810.832494-6-jwakely@redhat.com> In-Reply-To: <20250110212810.832494-1-jwakely@redhat.com> References: <20250110212810.832494-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: hvR5yaov8cvPDWvOhzVXyJt94ROQry4V0HaPp1e4Cs0_1736544505 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org When using a proxy object for atomic waiting and notifying operations, we need to ensure that the _M_ver value is always incremented by a notifying operation, even if we return early without doing the futex wake syscall. Otherwise we get missed wake-ups because the notifying thread doesn't modify the value that other threads are doing a futex wait on. libstdc++-v3/ChangeLog: * include/bits/atomic_wait.h (__notify_impl): Increment the proxy value before returning early for the uncontended case. --- libstdc++-v3/include/bits/atomic_wait.h | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/libstdc++-v3/include/bits/atomic_wait.h b/libstdc++-v3/include/bits/atomic_wait.h index 29b83cad6e6c..4a9652ed8f1d 100644 --- a/libstdc++-v3/include/bits/atomic_wait.h +++ b/libstdc++-v3/include/bits/atomic_wait.h @@ -392,13 +392,6 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __wait_args __args{ *__a }; __waiter_pool_impl* __pool = nullptr; - if (__args & __wait_flags::__track_contention) - { - __pool = &__waiter_pool_impl::_S_impl_for(__addr); - if (!__pool->_M_waiting()) - return; - } - const __platform_wait_t* __wait_addr; if (__args & __wait_flags::__proxy_wait) { @@ -416,6 +409,13 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION else // Use the atomic variable's own address. __wait_addr = __addr; + if (__args & __wait_flags::__track_contention) + { + __pool = &__waiter_pool_impl::_S_impl_for(__addr); + if (!__pool->_M_waiting()) + return; + } + #ifdef _GLIBCXX_HAVE_PLATFORM_WAIT __platform_notify(__wait_addr, __all); #else From patchwork Fri Jan 10 21:23:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 104540 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 394133857833 for ; Fri, 10 Jan 2025 21:42:40 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id A59023857C7B for ; Fri, 10 Jan 2025 21:28:31 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A59023857C7B Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org A59023857C7B Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544511; cv=none; b=ngNf8Ocm4BRSJjK2m9z4thppJYAncuIkdqtLO16CO3XEYW8542hh00wl9k0ZuI/sEATatHcJZ6qQmTFydf9MuOsFdq5nxpVKwViaaHokNkVKp3SeK9bHkyPUnzEAevsnOhSKhCA/dsWMuwY2Ntvms9RfaWVjsY57h/FoBgpBJbk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544511; c=relaxed/simple; bh=Cb7cG0TqDDRtFUFHZsQFsj4Nj3bKyl/6o8sat3EZhH0=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=mOGo1A1nOVjRZkV5PE4a7Wd+N7FhoJbsS2JMFyzl383xA9QJZ70ZnE4AsmhC7VCZEEZZMvuPUAaMk+q7/SJpNjo/1GJL7kK+4SY2zzFPAutwpudZmiA1bpBJvUrrQw+Tr/8N68wEzU8m1vqNZiud1Q/u7TKJOZJ2qzCktoN9DX8= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A59023857C7B Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=XALLzTj1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736544511; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3cmjYn62FkLNBlU80CXyBHsFHm4QR+Yo/IDVwRamRuQ=; b=XALLzTj1o58cTnl32OWK8S52KeygtfebnCtfFv3W0K6SoLveQwiLUZpXSxx+FLn3SUdAsl UR6+5C3ov6oUaIy+N7uxU8qAHw77QHwXmBdRJUm6pNVbWObDRn/a16GFkG5DaTvBbmas80 NwVLjL65DLq7v9j+0TZdMc2pZrPTHZQ= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-407-LywjJI7xOZeMPT4z1f7vxA-1; Fri, 10 Jan 2025 16:28:28 -0500 X-MC-Unique: LywjJI7xOZeMPT4z1f7vxA-1 X-Mimecast-MFC-AGG-ID: LywjJI7xOZeMPT4z1f7vxA Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5C8D119560B3; Fri, 10 Jan 2025 21:28:27 +0000 (UTC) Received: from localhost (unknown [10.42.28.9]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B6F8119560AD; Fri, 10 Jan 2025 21:28:26 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH 06/10] libstdc++: Simplify futex wrapper functions for atomic wait/notify Date: Fri, 10 Jan 2025 21:23:09 +0000 Message-ID: <20250110212810.832494-7-jwakely@redhat.com> In-Reply-To: <20250110212810.832494-1-jwakely@redhat.com> References: <20250110212810.832494-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: ndr6Tm3tzJpJYtkx8zJQmEKE4as4WH7UEU0qlozJhdc_1736544507 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Making these non-templates will allow them to be moved into the library at some point. libstdc++-v3/ChangeLog: * include/bits/atomic_wait.h (__platform_wait): Change function template to a normal function. The parameter is always __platform_wait_t* which is just int* for this implementation of the function. (__platform_notify): Likewise. --- libstdc++-v3/include/bits/atomic_wait.h | 40 ++++++++++++------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/libstdc++-v3/include/bits/atomic_wait.h b/libstdc++-v3/include/bits/atomic_wait.h index 4a9652ed8f1d..38a2bd3f95f2 100644 --- a/libstdc++-v3/include/bits/atomic_wait.h +++ b/libstdc++-v3/include/bits/atomic_wait.h @@ -108,27 +108,27 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __bitset_match_any = -1 }; - template - void - __platform_wait(const _Tp* __addr, __platform_wait_t __val) noexcept - { - auto __e = syscall (SYS_futex, static_cast(__addr), - static_cast(__futex_wait_flags::__wait_private), - __val, nullptr); - if (!__e || errno == EAGAIN) - return; - if (errno != EINTR) - __throw_system_error(errno); - } + // If the futex *__addr is equal to __val, wait on the futex until woken. + inline void + __platform_wait(const int* __addr, int __val) noexcept + { + auto __e = syscall (SYS_futex, __addr, + static_cast(__futex_wait_flags::__wait_private), + __val, nullptr); + if (!__e || errno == EAGAIN) + return; + if (errno != EINTR) + __throw_system_error(errno); + } - template - void - __platform_notify(const _Tp* __addr, bool __all) noexcept - { - syscall (SYS_futex, static_cast(__addr), - static_cast(__futex_wait_flags::__wake_private), - __all ? INT_MAX : 1); - } + // Wake threads waiting on the futex *__addr. + inline void + __platform_notify(const int* __addr, bool __all) noexcept + { + syscall (SYS_futex, __addr, + static_cast(__futex_wait_flags::__wake_private), + __all ? INT_MAX : 1); + } #endif inline void From patchwork Fri Jan 10 21:23:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 104538 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D364E3858401 for ; Fri, 10 Jan 2025 21:38:35 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D364E3858401 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=dkxydjWd X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id DC4C23857C63 for ; Fri, 10 Jan 2025 21:28:33 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org DC4C23857C63 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org DC4C23857C63 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544514; cv=none; b=tXmruH+cJiW1I1vyrLIckuqSWgrC7XbAKtCbvaWamsvSTVRkdIteNMHQUeCHT/CIMeIX374g9kj+giAdiDJl/cdNkhY/bNEidxVPhcQvzIvfnpUWsZXd0BOguLRdLshBGiWyzbsB9n825OFnwQKem4CoEiX931p1QDFdQgF9nVE= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544514; c=relaxed/simple; bh=Q6Cr0uN1inSwG1FZX+4GsSl+xNMrwkSFYvHi1c2viVY=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=Ml2eAei0/+B+QrYMYGy+KYmQpCNAB7L87Lb599o8F3rLb/wYbvaGQNA/f/5DMIWab4yV7TZTZsFffi+qvzQ2CAPXaLau+1SZ5tK4GCBDrLtDCmwpar1rIk7tC/tIkKrsEwA/gLqYF6cw8KDo6xJ4VVGH/xtIyyple1GgHwDMk5w= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org DC4C23857C63 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736544513; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5hPBs8LXLk87p8EBrjHlD7MxBRUm9HUwtWM1PKMUaeE=; b=dkxydjWdayzfvpbVJI//U/XinIjlDEnZKKXWqt2HnpNxAG/wMi9353wasWJyN7QnZOUeow AEzEgLHMo/z2pGZa5eddjFUr20cI9laOLpragYKe4dgnga5yv855noxrq96BlMkIrs2z2P mGYXO8SrOyowEI9djJoXxe7Sy2MtboA= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-39-0IJ9Bih9OaCdqmJ5xDuoiQ-1; Fri, 10 Jan 2025 16:28:30 -0500 X-MC-Unique: 0IJ9Bih9OaCdqmJ5xDuoiQ-1 X-Mimecast-MFC-AGG-ID: 0IJ9Bih9OaCdqmJ5xDuoiQ Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 9A1851955F79; Fri, 10 Jan 2025 21:28:29 +0000 (UTC) Received: from localhost (unknown [10.42.28.9]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3167319560AD; Fri, 10 Jan 2025 21:28:28 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH 07/10] libstdc++: Remove reinterpret_cast uses in atomic wait/notify Date: Fri, 10 Jan 2025 21:23:10 +0000 Message-ID: <20250110212810.832494-8-jwakely@redhat.com> In-Reply-To: <20250110212810.832494-1-jwakely@redhat.com> References: <20250110212810.832494-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: LsTg2ME5AxT61LOVyi3Kjbii7GzruZLWTyAkSeeWF_4_1736544509 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org We can pass around void* instead of casting incompatible pointers to __platform_wait_t*, and then only static_cast to __platform_wait_t* when we know that's valid. libstdc++-v3/ChangeLog: * include/bits/atomic_timed_wait.h (__atomic_wait_address_until): Remove reinterpret_cast and allow address to implicitly convert to const void* instead. (__atomic_wait_address_for): Likewise. * include/bits/atomic_wait.h: (__wait_impl, __notify_impl): Change first parameter to const void* and then static_cast to const __platform_wait_t* when not using proxied wait. (__atomic_wait_address, __atomic_notify_address) Remove reinterpret_cast and allow address to implicitly convert to const void* instead. --- libstdc++-v3/include/bits/atomic_timed_wait.h | 8 ++------ libstdc++-v3/include/bits/atomic_wait.h | 16 ++++++---------- 2 files changed, 8 insertions(+), 16 deletions(-) diff --git a/libstdc++-v3/include/bits/atomic_timed_wait.h b/libstdc++-v3/include/bits/atomic_timed_wait.h index 7e2017f2f515..645b8cfc4a8b 100644 --- a/libstdc++-v3/include/bits/atomic_timed_wait.h +++ b/libstdc++-v3/include/bits/atomic_timed_wait.h @@ -271,13 +271,11 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION const chrono::time_point<_Clock, _Dur>& __atime, bool __bare_wait = false) noexcept { - const auto __wait_addr = - reinterpret_cast(__addr); __detail::__wait_args __args{ __addr, __bare_wait }; _Tp __val = __vfn(); while (!__pred(__val)) { - auto __res = __detail::__wait_until(__wait_addr, &__args, __atime); + auto __res = __detail::__wait_until(__addr, &__args, __atime); if (!__res.first) // timed out return __res.first; // C++26 will also return last observed __val @@ -321,13 +319,11 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION const chrono::duration<_Rep, _Period>& __rtime, bool __bare_wait = false) noexcept { - const auto __wait_addr = - reinterpret_cast(__addr); __detail::__wait_args __args{ __addr, __bare_wait }; _Tp __val = __vfn(); while (!__pred(__val)) { - auto __res = __detail::__wait_for(__wait_addr, &__args, __rtime); + auto __res = __detail::__wait_for(__addr, &__args, __rtime); if (!__res.first) // timed out return __res.first; // C++26 will also return last observed __val diff --git a/libstdc++-v3/include/bits/atomic_wait.h b/libstdc++-v3/include/bits/atomic_wait.h index 38a2bd3f95f2..db4fa031d2cf 100644 --- a/libstdc++-v3/include/bits/atomic_wait.h +++ b/libstdc++-v3/include/bits/atomic_wait.h @@ -340,7 +340,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION } inline __wait_result_type - __wait_impl(const __platform_wait_t* __addr, const __wait_args* __a) + __wait_impl(const void* __addr, const __wait_args* __a) { __wait_args __args{ *__a }; __waiter_pool_impl* __pool = nullptr; @@ -353,7 +353,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __atomic_load(__wait_addr, &__args._M_old, __args._M_order); } else - __wait_addr = __addr; + __wait_addr = static_cast(__addr); if (__args & __wait_flags::__do_spin) { @@ -386,7 +386,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION } inline void - __notify_impl(const __platform_wait_t* __addr, [[maybe_unused]] bool __all, + __notify_impl(const void* __addr, [[maybe_unused]] bool __all, const __wait_args* __a) { __wait_args __args{ *__a }; @@ -407,7 +407,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __all = true; } else // Use the atomic variable's own address. - __wait_addr = __addr; + __wait_addr = static_cast(__addr); if (__args & __wait_flags::__track_contention) { @@ -435,8 +435,6 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __atomic_wait_address(const _Tp* __addr, _Pred&& __pred, _ValFn&& __vfn, bool __bare_wait = false) noexcept { - const auto __wait_addr = - reinterpret_cast(__addr); __detail::__wait_args __args{ __addr, __bare_wait }; _Tp __val = __vfn(); while (!__pred(__val)) @@ -448,7 +446,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __val); // Otherwise, it's a proxy wait and the proxy's _M_ver is used. - __detail::__wait_impl(__wait_addr, &__args); + __detail::__wait_impl(__addr, &__args); __val = __vfn(); } // C++26 will return __val @@ -480,10 +478,8 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION __atomic_notify_address(const _Tp* __addr, bool __all, bool __bare_wait = false) noexcept { - const auto __wait_addr = - reinterpret_cast(__addr); __detail::__wait_args __args{ __addr, __bare_wait }; - __detail::__notify_impl(__wait_addr, __all, &__args); + __detail::__notify_impl(__addr, __all, &__args); } _GLIBCXX_END_NAMESPACE_VERSION } // namespace std From patchwork Fri Jan 10 21:23:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 104550 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 97BBE3858C35 for ; Fri, 10 Jan 2025 21:51:40 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 97BBE3858C35 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=T+8D2NfG X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTP id 96CB13857C7F for ; Fri, 10 Jan 2025 21:28:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 96CB13857C7F Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 96CB13857C7F Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544514; cv=none; b=olWFPy58XuzwAkQEq2V/KVF0JmOsa/wq7OLt+1W4RoWE4B+9yngPtWCntHUsPYgCEGXf9giQzdO2ZyyXUoJMBfbozdHMxrR64oR5B/TCxAg68pR7+aO9ymnGdtk+XlUqodtNkcUhR0222ltCejCzl3q8YeiFfS7euZwU9gfjDQk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544514; c=relaxed/simple; bh=QI+bcmzHxkDyoirNxOPWGRg897yVFcisdiK/SNFXeu0=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=YOU6760imWwVXk24GO6fGkqKFj1CUwAUOXzXrHvwl9AssFXczbXeGbFgy/Pmgcr0YCdALgScLVby/v+bIgh9ZzDklAJL3FPC34By2RvQuN+ePu3ZryqCUzirG1dmuh1Qk4LkQHbEFMakHJk4iEhV13CwWij9mW7YAB9Mi0ZMCrM= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 96CB13857C7F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736544514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CtteGef7OhFZ7IFssNGFZ1j2jR0IMwxVc+s/O55Y99E=; b=T+8D2NfGyc+ZM4wHJz2GQ+hiluwkRi3WMp+EjfUesuJryBSpmc1yQ54g30xJdPXSsMRPS8 m6BD2QBRmC+pP9A87DoXAnUjLQxQLTmEnzcJVVMOjdTYt8you7s4aMh64j0imaByeFlxx2 HK0rKAzPOkN3K4GifpMQalf6G6TSxc4= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-540-9z17TTKNPua3nKNbiVi1SQ-1; Fri, 10 Jan 2025 16:28:32 -0500 X-MC-Unique: 9z17TTKNPua3nKNbiVi1SQ-1 X-Mimecast-MFC-AGG-ID: 9z17TTKNPua3nKNbiVi1SQ Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id DA9C81955D86; Fri, 10 Jan 2025 21:28:31 +0000 (UTC) Received: from localhost (unknown [10.42.28.9]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3D55E19560AB; Fri, 10 Jan 2025 21:28:30 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH 08/10] libstdc++: Rename __atomic_compare to __atomic_eq Date: Fri, 10 Jan 2025 21:23:11 +0000 Message-ID: <20250110212810.832494-9-jwakely@redhat.com> In-Reply-To: <20250110212810.832494-1-jwakely@redhat.com> References: <20250110212810.832494-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: MCFztfwrDZ8R8GpZhOhL4W8mYKf53io_mwNwR1Hq7is_1736544512 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org This is an equality comparison rather than a three-way comparison like memcmp and <=>, so name it more precisely. libstdc++-v3/ChangeLog: * include/bits/atomic_timed_wait.h (__atomic_wait_address_until_v): Replace __atomic_compare with __atomic_eq. (__atomic_wait_address_for_v): Likewise. * include/bits/atomic_wait.h (__atomic_compare): Rename to __atomic_eq. (__atomic_wait_address_v): Replace __atomic_compare with __atomic_eq. --- libstdc++-v3/include/bits/atomic_timed_wait.h | 10 ++++++---- libstdc++-v3/include/bits/atomic_wait.h | 5 +++-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/libstdc++-v3/include/bits/atomic_timed_wait.h b/libstdc++-v3/include/bits/atomic_timed_wait.h index 645b8cfc4a8b..9a60f34c130d 100644 --- a/libstdc++-v3/include/bits/atomic_timed_wait.h +++ b/libstdc++-v3/include/bits/atomic_timed_wait.h @@ -304,8 +304,9 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION const chrono::time_point<_Clock, _Dur>& __atime, bool __bare_wait = false) noexcept { - auto __pfn = [&](const _Tp& __val) - { return !__detail::__atomic_compare(__old, __val); }; + auto __pfn = [&](const _Tp& __val) { + return !__detail::__atomic_eq(__old, __val); + }; return __atomic_wait_address_until(__addr, __pfn, forward<_ValFn>(__vfn), __atime, __bare_wait); } @@ -352,8 +353,9 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION const chrono::duration<_Rep, _Period>& __rtime, bool __bare_wait = false) noexcept { - auto __pfn = [&](const _Tp& __val) - { return !__detail::__atomic_compare(__old, __val); }; + auto __pfn = [&](const _Tp& __val) { + return !__detail::__atomic_eq(__old, __val); + }; return __atomic_wait_address_for(__addr, __pfn, forward<_ValFn>(__vfn), __rtime, __bare_wait); } diff --git a/libstdc++-v3/include/bits/atomic_wait.h b/libstdc++-v3/include/bits/atomic_wait.h index db4fa031d2cf..0b29b17178e9 100644 --- a/libstdc++-v3/include/bits/atomic_wait.h +++ b/libstdc++-v3/include/bits/atomic_wait.h @@ -154,7 +154,8 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION // return true if equal template - bool __atomic_compare(const _Tp& __a, const _Tp& __b) + inline bool + __atomic_eq(const _Tp& __a, const _Tp& __b) { // TODO make this do the correct padding bit ignoring comparison return __builtin_memcmp(&__a, &__b, sizeof(_Tp)) == 0; @@ -469,7 +470,7 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION _ValFn __vfn) noexcept { auto __pfn = [&](const _Tp& __val) - { return !__detail::__atomic_compare(__old, __val); }; + { return !__detail::__atomic_eq(__old, __val); }; __atomic_wait_address(__addr, __pfn, forward<_ValFn>(__vfn)); } From patchwork Fri Jan 10 21:23:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 104548 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 66C723858433 for ; Fri, 10 Jan 2025 21:48:46 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 66C723858433 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=fAYNifqs X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTP id D78D33857B98 for ; Fri, 10 Jan 2025 21:28:38 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D78D33857B98 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org D78D33857B98 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544518; cv=none; b=SuNVq4oZliIZJnQd0Iwl6/Tb7GpAwYtmFVD6YGaudehojseCyvhUGV9ZqDJnICcXyjdIkVckSSo6EWakklWJonmm/NKVpqdAjTREFLm6RNblQVd0bLUS5+Y+fSKz+E4LF1YB7yZ0NZeMck6dqozASi1VlE2YgJvPEsDmOI4FWPo= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544518; c=relaxed/simple; bh=k8kUw+NDidBbbIQ9oMMwAIjg5FySq/yeFK/wXVYQ1Os=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=K8ypalmiyU40RAnNv57eJdZEnzEz+lzdFaV8sphWPTs9EoRMJ42MvM4eE8SKAyLybvPSVIxc2jJQTeeH4ILi7ROtlNjBUuY+AHya1KJlO4ZzVmaBz6NPlsZnzE8yAiQ1ZJHUbW66Igp/kLUGPVyT5Tw7nAYUKXqe6tin4DB2fWw= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D78D33857B98 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736544518; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s+/EWYIJVh6ZsxxIIhfG8/7AKh8QFfTKMDAf06T/H6s=; b=fAYNifqs+196Ba720UoZqW0fiWNRxKTxIIE5xTV/OL9H4GC9z8Fe5R72nLmVFWJR6i2riR jBtFajdfNZ8/liSqpUGPW3WNjj0PSXNAS/Y3YGCeF21cvOyBV40xW0AmMUBlhdBjYmsBvw qn49RSTd72hTZSAltk27sLoOdxa8GuE= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-662-pTwNVItDOtWqdcq_kus4Dg-1; Fri, 10 Jan 2025 16:28:35 -0500 X-MC-Unique: pTwNVItDOtWqdcq_kus4Dg-1 X-Mimecast-MFC-AGG-ID: pTwNVItDOtWqdcq_kus4Dg Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2303F1955DCC; Fri, 10 Jan 2025 21:28:34 +0000 (UTC) Received: from localhost (unknown [10.42.28.9]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 93F7C1955BE3; Fri, 10 Jan 2025 21:28:33 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH 09/10] libstdc++: Use safe integer comparisons in std::latch [PR98749] Date: Fri, 10 Jan 2025 21:23:12 +0000 Message-ID: <20250110212810.832494-10-jwakely@redhat.com> In-Reply-To: <20250110212810.832494-1-jwakely@redhat.com> References: <20250110212810.832494-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: KPrFl3GwoCXL7uzSqYY3xK8Xg5vDKFpeskuODQ6MAGg_1736544514 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Also add missing precondition check to constructor and fix existing check in count_down which was duplicated by mistake. libstdc++-v3/ChangeLog: PR libstdc++/98749 * include/std/latch (latch::max()): Use std::cmp_less to handle the case where __platform_wait_t is wider than ptrdiff_t or is unsigned. (latch::latch(ptrdiff_t)): Add assertion. (latch::count_down): Fix copy & pasted duplicate assertion. Use std::cmp_equal to compare __platform_wait_t and ptrdiff_t values. --- libstdc++-v3/include/std/latch | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/libstdc++-v3/include/std/latch b/libstdc++-v3/include/std/latch index c81a6631d53f..8bdf68f3390a 100644 --- a/libstdc++-v3/include/std/latch +++ b/libstdc++-v3/include/std/latch @@ -41,6 +41,7 @@ #ifdef __cpp_lib_latch // C++ >= 20 && atomic_wait #include #include +#include // cmp_equal, cmp_less_equal, etc. namespace std _GLIBCXX_VISIBILITY(default) { @@ -51,24 +52,34 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION public: static constexpr ptrdiff_t max() noexcept - { return __gnu_cxx::__int_traits<__detail::__platform_wait_t>::__max; } + { + using __gnu_cxx::__int_traits; + constexpr auto __max = __int_traits<__detail::__platform_wait_t>::__max; + if constexpr (std::cmp_less(__max, __PTRDIFF_MAX__)) + return __max; + return __PTRDIFF_MAX__; + } - constexpr explicit latch(ptrdiff_t __expected) noexcept - : _M_a(__expected) { } + constexpr explicit + latch(ptrdiff_t __expected) noexcept + : _M_a(__expected) + { __glibcxx_assert(__expected >= 0 && __expected <= max()); } ~latch() = default; + latch(const latch&) = delete; latch& operator=(const latch&) = delete; _GLIBCXX_ALWAYS_INLINE void count_down(ptrdiff_t __update = 1) { - __glibcxx_assert(__update >= 0); - auto const __old = __atomic_impl::fetch_sub(&_M_a, - __update, memory_order::release); - __glibcxx_assert(__update >= 0); - if (__old == static_cast<__detail::__platform_wait_t>(__update)) + __glibcxx_assert(__update >= 0 && __update <= max()); + auto const __old = __atomic_impl::fetch_sub(&_M_a, __update, + memory_order::release); + if (std::cmp_equal(__old, __update)) __atomic_impl::notify_all(&_M_a); + else + __glibcxx_assert(std::cmp_less(__update, __old)); } _GLIBCXX_ALWAYS_INLINE bool @@ -97,7 +108,8 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION private: // This alignas is not redundant, it increases the alignment for // long long on x86. - alignas(__alignof__(__detail::__platform_wait_t)) __detail::__platform_wait_t _M_a; + alignas(__alignof__(__detail::__platform_wait_t)) + __detail::__platform_wait_t _M_a; }; _GLIBCXX_END_NAMESPACE_VERSION } // namespace From patchwork Fri Jan 10 21:23:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Wakely X-Patchwork-Id: 104543 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 370173857BB3 for ; Fri, 10 Jan 2025 21:44:16 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTP id 0D05C3857C68 for ; Fri, 10 Jan 2025 21:28:41 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 0D05C3857C68 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 0D05C3857C68 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544521; cv=none; b=VD+8vGUXYGLcSbHwF659M4XLlGVFthvWzresTOsM2Y3YhVK2akaZ8ypsuZ2faufPLUUDWNsGaruCB2hysQIh/54wYWne4zLne+YminCZUa+QXSDmOIwEwd/rUOTCkXb2Ih7y3G8q5G7TBInhkCr6rrAWmslFsE33FSE+zySLQXA= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1736544521; c=relaxed/simple; bh=QytCtcI2tharPH3OzB1gQ1fm3ApzMDt9p/9JLl0UyTY=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=Wj5OpAv3W4Eow9xPJC0GED2sckxq/55rjYiS2AbtwhNax1onLCr6rbIS2R1kYnj+DmynN0iLbZBG9LpZXy/0RKJFoIQ/yjb3GeRxyeC9V1tHKKBa5Iu8bfU5/ABfozH1+J4RAyZTPKjhxfBgMe+xiVXXHLb+iC8Ft1Ip/ogOSIA= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 0D05C3857C68 Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=X0qQy2Bg DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736544520; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=omn9AOQLgykw+ADpOFRjJhS7k2a9ockKhjbv0f54+qw=; b=X0qQy2BgTln4N0Gm4qIvWK/oNZbqAHw0ZFO+qu1qSREu902DHCCQ7YKHPcq5xa57usizNg uq2/kP8WRaBqpxtM0qvhJU3EuRIivfkn6g4A0OVUSbtta6l3z/HNl1Vla6ljN7D0o35e0X 0lqiDgchYX7eECIs29IpOCUOseaoEF0= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-624-sTiJEerAOwK93pbcd3geUg-1; Fri, 10 Jan 2025 16:28:37 -0500 X-MC-Unique: sTiJEerAOwK93pbcd3geUg-1 X-Mimecast-MFC-AGG-ID: sTiJEerAOwK93pbcd3geUg Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7A94B19560A3; Fri, 10 Jan 2025 21:28:36 +0000 (UTC) Received: from localhost (unknown [10.42.28.9]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EAC5B19560AB; Fri, 10 Jan 2025 21:28:35 +0000 (UTC) From: Jonathan Wakely To: libstdc++@gcc.gnu.org, gcc-patches@gcc.gnu.org Subject: [PATCH 10/10] libstdc++: Optimise std::latch::arrive_and_wait Date: Fri, 10 Jan 2025 21:23:13 +0000 Message-ID: <20250110212810.832494-11-jwakely@redhat.com> In-Reply-To: <20250110212810.832494-1-jwakely@redhat.com> References: <20250110212810.832494-1-jwakely@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: VMAt2inhbHVHdjpIBIQWpE5AC0elhUArFSH6eooj8Bo_1736544516 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-13.8 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org We don't need to wait if we know the counter has reached zero. libstdc++-v3/ChangeLog: * include/std/latch (latch::arrive_and_wait): Optimise. --- This one's commented out for now, but sending for review anyway. libstdc++-v3/include/std/latch | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/libstdc++-v3/include/std/latch b/libstdc++-v3/include/std/latch index 8bdf68f3390a..af24dd081e04 100644 --- a/libstdc++-v3/include/std/latch +++ b/libstdc++-v3/include/std/latch @@ -101,8 +101,20 @@ _GLIBCXX_BEGIN_NAMESPACE_VERSION _GLIBCXX_ALWAYS_INLINE void arrive_and_wait(ptrdiff_t __update = 1) noexcept { +#if 0 + __glibcxx_assert(__update >= 0 && __update <= max()); + auto const __old = __atomic_impl::fetch_sub(&_M_a, __update, + memory_order::release); + if (std::cmp_equal(__old, __update)) + __atomic_impl::notify_all(&_M_a); + else if (std::cmp_greater(__old, __update)) + wait(); + else + __glibcxx_assert(std::cmp_less_equal(__update, __old)); +#else count_down(__update); wait(); +#endif } private: