From patchwork Wed Nov 10 18:41:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 47420 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B65F93857C60 for ; Wed, 10 Nov 2021 18:42:17 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B65F93857C60 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1636569737; bh=Te2kDB5bSGuJgqf1hGqPBvZBcSssSE6oJdEN0jJb8YE=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=e0bLVyX7+IyoCXs7M7thHbPp/8MJPQq1eAwixwxFO65Orq+cQgrUq9adRk3IKiFW/ u6mG6DS6VPW8ZxpB+eWxA2WLNUh8hAiKo2kKgN83F1B/aodKSRd/KSwr6N8i6yKDsE rJ9DAeS7JFFTM94NjGgnKARtU+XM2IzhKgbhp9Bs= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by sourceware.org (Postfix) with ESMTPS id 95F113858400 for ; Wed, 10 Nov 2021 18:41:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 95F113858400 Received: by mail-pf1-x42d.google.com with SMTP id c4so3471520pfj.2 for ; Wed, 10 Nov 2021 10:41:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Te2kDB5bSGuJgqf1hGqPBvZBcSssSE6oJdEN0jJb8YE=; b=M6p3yQePo5sghAPN+KOa+NhaoHWbjXcOCgXxgSw56UhaPic7OtkgmuRyXFEaEjy/8O X3KXVRqaqACSgpMCTmMOJdwTEasfgzFCh/E0djT6goaDTN58ERkUPZOMvFLypr+QeVWN tWcCoFyBCGzWnyUVelB0YSaoAN/bpjhpvxqYTKlHhCWSxUCFlmZeplzuOqumqKgCXZ/1 8YqIGGnSJn0qYZmUrBPvHDG/29zp8b+v1C1znjmd0FIYNbc62gzaBeJuaNFBpnIhbHWa 1NjwyQTakX7bUQdp9o76oHrHEnCRVdJZrwqYmAS7VdVKRbODHUHi8gUxim8Ew4fcB24x 9X1w== X-Gm-Message-State: AOAM531OdJMtoucEKAwzoRlsdpgZExxAmf5gjC7V62uLQSctkVF9mMmQ k3IcEKFL4om/TfoUqhtbno+LdoAZJ5M= X-Google-Smtp-Source: ABdhPJyxQnG6diryNi2dEIfZY95/G4R8ocqIWm0RmL/ngnnKPrQydAcO0qN8O+bMLeMaZRaiNUUKLg== X-Received: by 2002:aa7:83c9:0:b0:481:1d47:3362 with SMTP id j9-20020aa783c9000000b004811d473362mr1180589pfn.5.1636569714737; Wed, 10 Nov 2021 10:41:54 -0800 (PST) Received: from gnu-cfl-2.localdomain ([2607:fb90:a63f:468b:b937:402b:0:c66]) by smtp.gmail.com with ESMTPSA id x20sm301430pjp.48.2021.11.10.10.41.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Nov 2021 10:41:54 -0800 (PST) Received: from gnu-cfl-2.lan (localhost [IPv6:::1]) by gnu-cfl-2.localdomain (Postfix) with ESMTP id 516B61A0EC0; Wed, 10 Nov 2021 10:41:53 -0800 (PST) To: libc-alpha@sourceware.org Subject: [PATCH v5 2/3] Reduce CAS in __pthread_mutex_lock_full [BZ #28537] Date: Wed, 10 Nov 2021 10:41:52 -0800 Message-Id: <20211110184153.2269857-3-hjl.tools@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211110184153.2269857-1-hjl.tools@gmail.com> References: <20211110184153.2269857-1-hjl.tools@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-3031.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Libc-alpha" From: "H.J. Lu" Reply-To: "H.J. Lu" Cc: Florian Weimer , Andreas Schwab , "Paul A . Clarke" , Arjan van de Ven Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Change __pthread_mutex_lock_full to do an atomic load and skip CAS if compare may fail to reduce cache line bouncing on contended locks. --- nptl/pthread_mutex_lock.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/nptl/pthread_mutex_lock.c b/nptl/pthread_mutex_lock.c index 2bd41767e0..d7e8efedd2 100644 --- a/nptl/pthread_mutex_lock.c +++ b/nptl/pthread_mutex_lock.c @@ -223,13 +223,13 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) newval |= (oldval & FUTEX_WAITERS) | assume_other_futex_waiters; #endif - newval - = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, - newval, oldval); - - if (newval != oldval) + int val = atomic_load_relaxed (&mutex->__data.__lock); + if (val != oldval + || ((val = atomic_compare_and_exchange_val_acq + (&mutex->__data.__lock, newval, oldval)) + != oldval)) { - oldval = newval; + oldval = val; continue; } @@ -411,11 +411,15 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) # ifdef NO_INCR newval |= FUTEX_WAITERS; # endif + oldval = atomic_load_relaxed (&mutex->__data.__lock); + if (oldval != 0) + goto locked_mutex; oldval = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, newval, 0); if (oldval != 0) { + locked_mutex:; /* The mutex is locked. The kernel will now take care of everything. */ int private = (robust @@ -554,6 +558,10 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) ceilval = ceiling << PTHREAD_MUTEX_PRIO_CEILING_SHIFT; oldprio = ceiling; + oldval = atomic_load_relaxed (&mutex->__data.__lock); + if (oldval != ceilval) + goto ceilval_failed; + oldval = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, #ifdef NO_INCR @@ -568,10 +576,13 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) do { - oldval - = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, - ceilval | 2, - ceilval | 1); + oldval = atomic_load_relaxed (&mutex->__data.__lock); + ceilval_failed: + if (oldval == (ceilval | 1)) + oldval + = atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, + ceilval | 2, + ceilval | 1); if ((oldval & PTHREAD_MUTEX_PRIO_CEILING_MASK) != ceilval) break; @@ -581,9 +592,10 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) ceilval | 2, PTHREAD_MUTEX_PSHARED (mutex)); } - while (atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, - ceilval | 2, ceilval) - != ceilval); + while (atomic_load_relaxed (&mutex->__data.__lock) != ceilval + || (atomic_compare_and_exchange_val_acq (&mutex->__data.__lock, + ceilval | 2, ceilval) + != ceilval)); } while ((oldval & PTHREAD_MUTEX_PRIO_CEILING_MASK) != ceilval);