From patchwork Tue Mar 16 17:31:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 42623 X-Patchwork-Delegate: azanella@linux.vnet.ibm.com Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 57824384B110; Tue, 16 Mar 2021 17:31:25 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 57824384B110 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1615915885; bh=qNWMPH1ICvKPde2+chbDpsLIV8zNqmswUKXYIMp87Io=; h=To:Subject:In-Reply-To:References:Date:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=VA2JZbB5RZWsd36eHRX5SQc5BGBA/E4Z69Ij6wvInbMjdx3mJ7fd5vgSKCg5zqR8c 1J20cKJAuGWTRTxIIhd132s3Yod8r7riphU4SKoZXu+ZvciA0Jp13Gg3Ptkp0RYtbF 3JSyfYSSvPK5n+mGllNcvbmlM6jy79LhNUOR3++4= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by sourceware.org (Postfix) with ESMTP id 58920384B823 for ; Tue, 16 Mar 2021 17:31:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 58920384B823 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-237-207VzbjDN_uRuqfc34D22A-1; Tue, 16 Mar 2021 13:31:20 -0400 X-MC-Unique: 207VzbjDN_uRuqfc34D22A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 10025107ACCD for ; Tue, 16 Mar 2021 17:31:19 +0000 (UTC) Received: from oldenburg.str.redhat.com (ovpn-112-254.ams2.redhat.com [10.36.112.254]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 60E515C1A3 for ; Tue, 16 Mar 2021 17:31:18 +0000 (UTC) To: libc-alpha@sourceware.org Subject: [PATCH v3 32/37] nptl: pthread_mutex_lock, pthread_mutex_unock single-threaded optimization In-Reply-To: References: Message-Id: <782ad0d9a371fa66bd54df07413f3d15fba0cf5a.1615914632.git.fweimer@redhat.com> Date: Tue, 16 Mar 2021 18:31:26 +0100 User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Florian Weimer via Libc-alpha From: Florian Weimer Reply-To: Florian Weimer Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" This is optimization is similar in spirit to the SINGLE_THREAD_P check in the malloc implementation. Doing this in generic code allows us to prioritize those cases which are likely to occur in single-threaded programs (normal and recursive mutexes). Reviewed-by: Adhemerval Zanella --- nptl/pthread_mutex_cond_lock.c | 1 + nptl/pthread_mutex_lock.c | 25 ++++++++++++++++++++++--- nptl/pthread_mutex_unlock.c | 17 ++++++++++++++++- 3 files changed, 39 insertions(+), 4 deletions(-) diff --git a/nptl/pthread_mutex_cond_lock.c b/nptl/pthread_mutex_cond_lock.c index 2f0771302f..3386bd689b 100644 --- a/nptl/pthread_mutex_cond_lock.c +++ b/nptl/pthread_mutex_cond_lock.c @@ -2,6 +2,7 @@ #define LLL_MUTEX_LOCK(mutex) \ lll_cond_lock ((mutex)->__data.__lock, PTHREAD_MUTEX_PSHARED (mutex)) +#define LLL_MUTEX_LOCK_OPTIMIZED(mutex) LLL_MUTEX_LOCK (mutex) /* Not actually elided so far. Needed? */ #define LLL_MUTEX_LOCK_ELISION(mutex) \ diff --git a/nptl/pthread_mutex_lock.c b/nptl/pthread_mutex_lock.c index f0de7b7fd6..8649a92ffb 100644 --- a/nptl/pthread_mutex_lock.c +++ b/nptl/pthread_mutex_lock.c @@ -30,8 +30,27 @@ /* Some of the following definitions differ when pthread_mutex_cond_lock.c includes this file. */ #ifndef LLL_MUTEX_LOCK -# define LLL_MUTEX_LOCK(mutex) \ +/* lll_lock with single-thread optimization. */ +static inline void +lll_mutex_lock_optimized (pthread_mutex_t *mutex) +{ + /* The single-threaded optimization is only valid for private + mutexes. For process-shared mutexes, the mutex could be in a + shared mapping, so synchronization with another process is needed + even without any threads. If the lock is already marked as + acquired, POSIX requires that pthread_mutex_lock deadlocks for + normal mutexes, so skip the optimization in that case as + well. */ + int private = PTHREAD_MUTEX_PSHARED (mutex); + if (private == LLL_PRIVATE && SINGLE_THREAD_P && mutex->__data.__lock == 0) + mutex->__data.__lock = 1; + else + lll_lock (mutex->__data.__lock, private); +} + +# define LLL_MUTEX_LOCK(mutex) \ lll_lock ((mutex)->__data.__lock, PTHREAD_MUTEX_PSHARED (mutex)) +# define LLL_MUTEX_LOCK_OPTIMIZED(mutex) lll_mutex_lock_optimized (mutex) # define LLL_MUTEX_TRYLOCK(mutex) \ lll_trylock ((mutex)->__data.__lock) # define LLL_ROBUST_MUTEX_LOCK_MODIFIER 0 @@ -64,7 +83,7 @@ __pthread_mutex_lock (pthread_mutex_t *mutex) FORCE_ELISION (mutex, goto elision); simple: /* Normal mutex. */ - LLL_MUTEX_LOCK (mutex); + LLL_MUTEX_LOCK_OPTIMIZED (mutex); assert (mutex->__data.__owner == 0); } #if ENABLE_ELISION_SUPPORT @@ -99,7 +118,7 @@ __pthread_mutex_lock (pthread_mutex_t *mutex) } /* We have to get the mutex. */ - LLL_MUTEX_LOCK (mutex); + LLL_MUTEX_LOCK_OPTIMIZED (mutex); assert (mutex->__data.__owner == 0); mutex->__data.__count = 1; diff --git a/nptl/pthread_mutex_unlock.c b/nptl/pthread_mutex_unlock.c index 3b5ccdacf9..655093ee9a 100644 --- a/nptl/pthread_mutex_unlock.c +++ b/nptl/pthread_mutex_unlock.c @@ -28,6 +28,21 @@ static int __pthread_mutex_unlock_full (pthread_mutex_t *mutex, int decr) __attribute_noinline__; +/* lll_lock with single-thread optimization. */ +static inline void +lll_mutex_unlock_optimized (pthread_mutex_t *mutex) +{ + /* The single-threaded optimization is only valid for private + mutexes. For process-shared mutexes, the mutex could be in a + shared mapping, so synchronization with another process is needed + even without any threads. */ + int private = PTHREAD_MUTEX_PSHARED (mutex); + if (private == LLL_PRIVATE && SINGLE_THREAD_P) + mutex->__data.__lock = 0; + else + lll_unlock (mutex->__data.__lock, private); +} + int attribute_hidden __pthread_mutex_unlock_usercnt (pthread_mutex_t *mutex, int decr) @@ -51,7 +66,7 @@ __pthread_mutex_unlock_usercnt (pthread_mutex_t *mutex, int decr) --mutex->__data.__nusers; /* Unlock. */ - lll_unlock (mutex->__data.__lock, PTHREAD_MUTEX_PSHARED (mutex)); + lll_mutex_unlock_optimized (mutex); LIBC_PROBE (mutex_release, 1, mutex);