From patchwork Wed Nov 10 00:16:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 47344 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B37143858018 for ; Wed, 10 Nov 2021 00:21:12 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B37143858018 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1636503672; bh=aVqux7lBP9ur7GWlQ+k+NpS2JY66dIN7KrJeBckVMZw=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=HfPpkhj3wnhImsPCQlFUszrkD7w/Mq90w9bH5gA9WcckclWngcuOGwS7IGoy2fg6h UMF5sRGhhAJd5SpGFxyk4grZLNI1kZHHyGWZ6g/u8+dYLPYt4flyrL0o/dXJrLuUni uwXlG9eEhr1zGM80FE35hEDsbFFPt5/lQnmRsWeM= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) by sourceware.org (Postfix) with ESMTPS id 1B31C3857C72 for ; Wed, 10 Nov 2021 00:16:17 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 1B31C3857C72 Received: by mail-pg1-x529.google.com with SMTP id b4so574039pgh.10 for ; Tue, 09 Nov 2021 16:16:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aVqux7lBP9ur7GWlQ+k+NpS2JY66dIN7KrJeBckVMZw=; b=QfCkcNVUToFqmXwmAFDpUSaUTHII7sotCM6rJSZod7yGnXYLdGTTeVbzro/QkRDnIZ XY08In8MPaz3YhA4rw499oHtjLFBw9vcLzKRmUHh1eVoUPYNZYI6nEtCeibFzW9fzBuT 3YUklwDtvyIlGpMzGliEEE9cZkTPvZ4HrH5rM3kp86EzZg/BpQqUIFUWT/vOSw34Flms kXDRd0r2LSYbYvYjJL/i+D+IzlLYK//XQtQglEHFldXTBZYUIXUf/xftkLvOyDOQCQMs DfmeNuVbdaArpH1BswCrMCFzyzgUJPRJeui3Bh8qtB558x5cQ0O7unSMpr6kfg6bAUAv k+3w== X-Gm-Message-State: AOAM533GkL5tH/qYkhzqAVIk4QoxHZ08Bg69XOnATn/c8mRAXXlOmRdl I5Yzm8SV6Q1M8DwJQFPWIGA= X-Google-Smtp-Source: ABdhPJzwbS6v1Jx/m7PtfAEckpmkQfHwGO5PghzaGcJM2p/w4mEFwPrKgZ7rMZ9aSe1LM8M0HjRGZA== X-Received: by 2002:a63:b252:: with SMTP id t18mr9252764pgo.19.1636503376226; Tue, 09 Nov 2021 16:16:16 -0800 (PST) Received: from gnu-cfl-2.localdomain ([172.58.35.133]) by smtp.gmail.com with ESMTPSA id 59sm3736928pjz.34.2021.11.09.16.16.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Nov 2021 16:16:15 -0800 (PST) Received: from gnu-cfl-2.lan (localhost [IPv6:::1]) by gnu-cfl-2.localdomain (Postfix) with ESMTP id 8A51F1A0987; Tue, 9 Nov 2021 16:16:14 -0800 (PST) To: libc-alpha@sourceware.org Subject: [PATCH v4 2/3] Reduce CAS in __pthread_mutex_lock_full [BZ #28537] Date: Tue, 9 Nov 2021 16:16:13 -0800 Message-Id: <20211110001614.2087610-3-hjl.tools@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211110001614.2087610-1-hjl.tools@gmail.com> References: <20211110001614.2087610-1-hjl.tools@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-3030.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Libc-alpha" From: "H.J. Lu" Reply-To: "H.J. Lu" Cc: Florian Weimer , Arjan van de Ven , Andreas Schwab Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Change __pthread_mutex_lock_full to do an atomic load and skip CAS if compare may fail to reduce cache line bouncing on contended locks. --- nptl/pthread_mutex_lock.c | 38 +++++++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/nptl/pthread_mutex_lock.c b/nptl/pthread_mutex_lock.c index 2bd41767e0..1126ecba95 100644 --- a/nptl/pthread_mutex_lock.c +++ b/nptl/pthread_mutex_lock.c @@ -204,6 +204,10 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) our TID | assume_other_futex_waiters. */ if (__glibc_likely (oldval == 0)) { + oldval = atomic_load_relaxed (&mutex->__data.__lock); + if (oldval != 0) + goto tid_failed; + oldval = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, id | assume_other_futex_waiters, 0); @@ -213,6 +217,13 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) if ((oldval & FUTEX_OWNER_DIED) != 0) { + int currval = atomic_load_relaxed (&mutex->__data.__lock); + if (currval != oldval) + { + oldval = currval; + continue; + } + /* The previous owner died. Try locking the mutex. */ int newval = id; #ifdef NO_INCR @@ -259,6 +270,7 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) return EOWNERDEAD; } + tid_failed: /* Check whether we already hold the mutex. */ if (__glibc_unlikely ((oldval & FUTEX_TID_MASK) == id)) { @@ -411,11 +423,15 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) # ifdef NO_INCR newval |= FUTEX_WAITERS; # endif + oldval = atomic_load_relaxed (&mutex->__data.__lock); + if (oldval != 0) + goto locked_mutex; oldval = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, newval, 0); if (oldval != 0) { + locked_mutex: /* The mutex is locked. The kernel will now take care of everything. */ int private = (robust @@ -554,6 +570,10 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) ceilval = ceiling << PTHREAD_MUTEX_PRIO_CEILING_SHIFT; oldprio = ceiling; + oldval = atomic_load_relaxed (&mutex->__data.__lock); + if (oldval != ceilval) + goto ceilval_failed; + oldval = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, #ifdef NO_INCR @@ -568,10 +588,13 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) do { - oldval - = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, - ceilval | 2, - ceilval | 1); + oldval = atomic_load_relaxed (&mutex->__data.__lock); + ceilval_failed: + if (oldval == (ceilval | 1)) + oldval + = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, + ceilval | 2, + ceilval | 1); if ((oldval & PTHREAD_MUTEX_PRIO_CEILING_MASK) != ceilval) break; @@ -581,9 +604,10 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) ceilval | 2, PTHREAD_MUTEX_PSHARED (mutex)); } - while (atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, - ceilval | 2, ceilval) - != ceilval); + while (atomic_load_relaxed (&mutex->__data.__lock) != ceilval + || (atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, + ceilval | 2, ceilval) + != ceilval)); } while ((oldval & PTHREAD_MUTEX_PRIO_CEILING_MASK) != ceilval);