From patchwork Tue May 6 21:36:47 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andi Kleen X-Patchwork-Id: 823 X-Patchwork-Delegate: carlos@redhat.com Return-Path: X-Original-To: siddhesh@wilcox.dreamhost.com Delivered-To: siddhesh@wilcox.dreamhost.com Received: from homiemail-mx23.g.dreamhost.com (mx2.sub5.homie.mail.dreamhost.com [208.113.200.128]) by wilcox.dreamhost.com (Postfix) with ESMTP id 27E50360079 for ; Tue, 6 May 2014 14:37:40 -0700 (PDT) Received: by homiemail-mx23.g.dreamhost.com (Postfix, from userid 14307373) id C66D363A948CA; Tue, 6 May 2014 14:37:39 -0700 (PDT) X-Original-To: glibc@patchwork.siddhesh.in Delivered-To: x14307373@homiemail-mx23.g.dreamhost.com Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by homiemail-mx23.g.dreamhost.com (Postfix) with ESMTPS id 9A50463A948BE for ; Tue, 6 May 2014 14:37:39 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:subject:date:message-id:in-reply-to :references; q=dns; s=default; b=bMf/ZgcqFnSDeSN3vGZYgHDBfKNlXxz IoKWAUhuAWJiqJ8rAKuWTlylAdUoAK8HpvnhmXpbg/vC/6+/fzFyEv0pyTxmW+MH AVZ4r1YE2xbjmgATdgir5djNi929nkLftOIJGO1FxdIfi2s7+4/yf7Qvf6ezIs0i zJKEDJOx0LvQ= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:subject:date:message-id:in-reply-to :references; s=default; bh=UjMJxskmPxCvLoBIH0xMJdWA/Fo=; b=dUDUp Ix8fD6u9OrQNC6Xagw5N50OryPWaRS9zUwJ3GaOT/pIhhfZQ/fb20vkZd/aXWAM0 XjW5broWQtmNlD//jnmrFDQp+rhzdVAZ82ad8jbaSi+LIUERBttvsgUD6NsEOTgI FGW5JBoHhzp1OVBClYy05F5x4PsLbvXru9Y1Go= Received: (qmail 5462 invoked by alias); 6 May 2014 21:37:23 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 5306 invoked by uid 89); 6 May 2014 21:37:19 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-4.0 required=5.0 tests=AWL, BAYES_00, RP_MATCHES_RCVD autolearn=ham version=3.3.2 X-HELO: mga01.intel.com X-ExtLoop1: 1 From: Andi Kleen To: libc-alpha@sourceware.org Cc: Andi Kleen Subject: [PATCH 1/3] Add a fast path for C rd/wrlock v2 Date: Tue, 6 May 2014 14:36:47 -0700 Message-Id: <1399412209-28245-2-git-send-email-andi@firstfloor.org> In-Reply-To: <1399412209-28245-1-git-send-email-andi@firstfloor.org> References: <1399412209-28245-1-git-send-email-andi@firstfloor.org> X-DH-Original-To: glibc@patchwork.siddhesh.in From: Andi Kleen One difference of the C versions to the assembler wr/rdlock is that the C compiler saves some registers which are unnecessary for the fast path in the prologue of the functions. Split the uncontended fast path out into a separate function. Only when contention is detected is the full featured function called. This makes the fast path code (nearly) identical to the assembler version, and gives uncontended performance within a few cycles. nptl/: 2014-05-06 Andi Kleen * pthread_rwlock_rdlock (__pthread_rwlock_rdlock): Split into __do_pthread_rwlock_rdlock and __pthread_rwlock_rdlock. * pthread_rwlock_wrlock (__pthread_rwlock_wrlock): Split into __do_pthread_rwlock_wrlock and __pthread_wrlock_rdlock. v2: Rename some functions and add space. --- nptl/pthread_rwlock_rdlock.c | 88 ++++++++++++++++++++++++++++++-------------- nptl/pthread_rwlock_wrlock.c | 59 ++++++++++++++++++++--------- 2 files changed, 103 insertions(+), 44 deletions(-) diff --git a/nptl/pthread_rwlock_rdlock.c b/nptl/pthread_rwlock_rdlock.c index 3773f7d..1df0327 100644 --- a/nptl/pthread_rwlock_rdlock.c +++ b/nptl/pthread_rwlock_rdlock.c @@ -24,39 +24,16 @@ #include -/* Acquire read lock for RWLOCK. */ -int -__pthread_rwlock_rdlock (rwlock) - pthread_rwlock_t *rwlock; +/* Acquire read lock for RWLOCK. Slow path. */ +static int __attribute__((noinline)) +__pthread_rwlock_rdlock_slow (pthread_rwlock_t *rwlock) { int result = 0; - LIBC_PROBE (rdlock_entry, 1, rwlock); - - /* Make sure we are alone. */ - lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); + /* Lock is taken in caller. */ while (1) { - /* Get the rwlock if there is no writer... */ - if (rwlock->__data.__writer == 0 - /* ...and if either no writer is waiting or we prefer readers. */ - && (!rwlock->__data.__nr_writers_queued - || PTHREAD_RWLOCK_PREFER_READER_P (rwlock))) - { - /* Increment the reader counter. Avoid overflow. */ - if (__glibc_unlikely (++rwlock->__data.__nr_readers == 0)) - { - /* Overflow on number of readers. */ - --rwlock->__data.__nr_readers; - result = EAGAIN; - } - else - LIBC_PROBE (rdlock_acquire_read, 1, rwlock); - - break; - } - /* Make sure we are not holding the rwlock as a writer. This is a deadlock situation we recognize and report. */ if (__builtin_expect (rwlock->__data.__writer @@ -88,6 +65,25 @@ __pthread_rwlock_rdlock (rwlock) lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); --rwlock->__data.__nr_readers_queued; + + /* Get the rwlock if there is no writer... */ + if (rwlock->__data.__writer == 0 + /* ...and if either no writer is waiting or we prefer readers. */ + && (!rwlock->__data.__nr_writers_queued + || PTHREAD_RWLOCK_PREFER_READER_P (rwlock))) + { + /* Increment the reader counter. Avoid overflow. */ + if (__glibc_unlikely (++rwlock->__data.__nr_readers == 0)) + { + /* Overflow on number of readers. */ + --rwlock->__data.__nr_readers; + result = EAGAIN; + } + else + LIBC_PROBE (rdlock_acquire_read, 1, rwlock); + + break; + } } /* We are done, free the lock. */ @@ -96,5 +92,43 @@ __pthread_rwlock_rdlock (rwlock) return result; } + +/* Fast path of acquiring read lock on RWLOCK. */ + +int +__pthread_rwlock_rdlock (pthread_rwlock_t *rwlock) +{ + int result = 0; + + LIBC_PROBE (rdlock_entry, 1, rwlock); + + /* Make sure we are alone. */ + lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); + + /* Get the rwlock if there is no writer... */ + if (rwlock->__data.__writer == 0 + /* ...and if either no writer is waiting or we prefer readers. */ + && (!rwlock->__data.__nr_writers_queued + || PTHREAD_RWLOCK_PREFER_READER_P (rwlock))) + { + /* Increment the reader counter. Avoid overflow. */ + if (__glibc_unlikely (++rwlock->__data.__nr_readers == 0)) + { + /* Overflow on number of readers. */ + --rwlock->__data.__nr_readers; + result = EAGAIN; + } + else + LIBC_PROBE (rdlock_acquire_read, 1, rwlock); + + /* We are done, free the lock. */ + lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared); + + return result; + } + + return __pthread_rwlock_rdlock_slow (rwlock); +} + weak_alias (__pthread_rwlock_rdlock, pthread_rwlock_rdlock) hidden_def (__pthread_rwlock_rdlock) diff --git a/nptl/pthread_rwlock_wrlock.c b/nptl/pthread_rwlock_wrlock.c index 1613d45..de54e51 100644 --- a/nptl/pthread_rwlock_wrlock.c +++ b/nptl/pthread_rwlock_wrlock.c @@ -25,29 +25,15 @@ /* Acquire write lock for RWLOCK. */ -int -__pthread_rwlock_wrlock (rwlock) - pthread_rwlock_t *rwlock; +static int __attribute__((noinline)) +__pthread_rwlock_wrlock_slow (pthread_rwlock_t *rwlock) { int result = 0; - LIBC_PROBE (wrlock_entry, 1, rwlock); - - /* Make sure we are alone. */ - lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); + /* Caller has taken the lock. */ while (1) { - /* Get the rwlock if there is no writer and no reader. */ - if (rwlock->__data.__writer == 0 && rwlock->__data.__nr_readers == 0) - { - /* Mark self as writer. */ - rwlock->__data.__writer = THREAD_GETMEM (THREAD_SELF, tid); - - LIBC_PROBE (wrlock_acquire_write, 1, rwlock); - break; - } - /* Make sure we are not holding the rwlock as a writer. This is a deadlock situation we recognize and report. */ if (__builtin_expect (rwlock->__data.__writer @@ -80,6 +66,16 @@ __pthread_rwlock_wrlock (rwlock) /* To start over again, remove the thread from the writer list. */ --rwlock->__data.__nr_writers_queued; + + /* Get the rwlock if there is no writer and no reader. */ + if (rwlock->__data.__writer == 0 && rwlock->__data.__nr_readers == 0) + { + /* Mark self as writer. */ + rwlock->__data.__writer = THREAD_GETMEM (THREAD_SELF, tid); + + LIBC_PROBE (wrlock_acquire_write, 1, rwlock); + break; + } } /* We are done, free the lock. */ @@ -88,5 +84,34 @@ __pthread_rwlock_wrlock (rwlock) return result; } +/* Fast path of acquiring write lock for RWLOCK. */ + +int +__pthread_rwlock_wrlock (pthread_rwlock_t *rwlock) +{ + LIBC_PROBE (wrlock_entry, 1, rwlock); + + /* Make sure we are alone. */ + lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); + + /* Get the rwlock if there is no writer and no reader. */ + if (__glibc_likely((rwlock->__data.__writer | + rwlock->__data.__nr_readers) == 0)) + { + /* Mark self as writer. */ + rwlock->__data.__writer = THREAD_GETMEM (THREAD_SELF, tid); + + LIBC_PROBE (wrlock_acquire_write, 1, rwlock); + + /* We are done, free the lock. */ + lll_unlock (rwlock->__data.__lock, rwlock->__data.__shared); + + return 0; + } + + return __pthread_rwlock_wrlock_slow (rwlock); +} + + weak_alias (__pthread_rwlock_wrlock, pthread_rwlock_wrlock) hidden_def (__pthread_rwlock_wrlock)