From patchwork Fri Apr 4 22:54:43 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andi Kleen X-Patchwork-Id: 414 Return-Path: X-Original-To: siddhesh@wilcox.dreamhost.com Delivered-To: siddhesh@wilcox.dreamhost.com Received: from homiemail-mx23.g.dreamhost.com (mx2.sub5.homie.mail.dreamhost.com [208.113.200.128]) by wilcox.dreamhost.com (Postfix) with ESMTP id 7FAE536007C for ; Fri, 4 Apr 2014 15:54:53 -0700 (PDT) Received: by homiemail-mx23.g.dreamhost.com (Postfix, from userid 14307373) id 35A17624D9D6D; Fri, 4 Apr 2014 15:54:53 -0700 (PDT) X-Original-To: glibc@patchwork.siddhesh.in Delivered-To: x14307373@homiemail-mx23.g.dreamhost.com Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by homiemail-mx23.g.dreamhost.com (Postfix) with ESMTPS id 14535624D9D50 for ; Fri, 4 Apr 2014 15:54:53 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:subject:date:message-id; q=dns; s= default; b=sgu2CWyEhQxjG41+44ubMrH7R2QopTcAeJjD3HXB6yWFVqPmoL7bO KFy6OtVhgCPvSTzLw9cRk5LGejI0Kw6Gc+atQ+MHpI/UpUvUDrH9++fH2kpI7ivw h/Wn/FdaYAOQ/FkETlnuMPxOFGyergsljsMk5+zN4ftqxaid7W3G+Q= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:subject:date:message-id; s=default; bh=O9+wZLqgOdWPUnBEYXpH//MUH+I=; b=jaccEnW8JvNuKazMeXHbjifp7BvK gSn9CAq4tvuOqbQt4gJOMBjSbJQeBqhafWrdVkpHKLZKJRypG5+0cuaQRujqdX/N P9NAha49977AC7N7WRPsKZmwHyGsW0hvpngGjQU21Z1Z9bcnaW/X4ok0LH+RbYg7 eNXw1slX499QzzY= Received: (qmail 31297 invoked by alias); 4 Apr 2014 22:54:51 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 31287 invoked by uid 89); 4 Apr 2014 22:54:50 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-4.2 required=5.0 tests=AWL, BAYES_00, RP_MATCHES_RCVD autolearn=ham version=3.3.2 X-HELO: mga11.intel.com X-ExtLoop1: 1 From: Andi Kleen To: libc-alpha@sourceware.org Cc: Andi Kleen Subject: [PATCH] Add adaptive elision to rwlocks Date: Fri, 4 Apr 2014 15:54:43 -0700 Message-Id: <1396652083-18920-1-git-send-email-andi@firstfloor.org> X-DH-Original-To: glibc@patchwork.siddhesh.in From: Andi Kleen This patch relies on the C version of the rwlocks posted earlier. With C rwlocks it is very straight forward to do adaptive elision using TSX. It is based on the infrastructure added earlier for mutexes, but uses its own elision macros. The macros are fairly general purpose and could be used for other elision purposes too. This version is much cleaner than the earlier assembler based version, and in particular implements adaptation which makes it safer. I changed the behavior slightly to not require any changes in the test suite and fully conform to all expected behaviors (generally at the cost of not eliding in various situations). In particular this means the timedlock variants are not elided. Nested trylock aborts. nptl/: 2014-4-04 Andi Kleen * pthread_rwlock_rdlock.c: Include elide.h. (pthread_rwlock_rdlock): Add elision. * pthread_rwlock_wrlock.c: Include elide.h. (pthread_rwlock_wrlock): Add elision. * pthread_rwlock_trywrlock.c: Include elide.h. (pthread_rwlock_trywrlock): Add elision. * pthread_rwlock_tryrdlock.c: Include elide.h. (pthread_rwlock_tryrdlock): Add elision. * pthread_rwlock_unlock.c: Include elide.h. (pthread_rwlock_tryrdlock): Add elision unlock. * sysdeps/pthread/pthread.h: (__PTHREAD_RWLOCK_ELISION_EXTRA): Handle new define (PTHREAD_RWLOCK_INITIALIZER, PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP): Handle new elision field. * sysdeps/unix/sysv/linux/x86/bits/pthreadtypes.h: (pthread_rwlock_t): Change __pad1 to __rwelision. (__PTHREAD_RWLOCK_ELISION_EXTRA): Add. * sysdeps/unix/sysv/linux/x86/elide.h: New file. Add generic elision macros. * sysdeps/unix/sysv/linux/x86/elision-conf.c: (elision_init): Set try_xbegin to zero when no RTM. --- nptl/pthread_rwlock_rdlock.c | 8 ++- nptl/pthread_rwlock_tryrdlock.c | 8 ++- nptl/pthread_rwlock_trywrlock.c | 8 ++- nptl/pthread_rwlock_unlock.c | 5 ++ nptl/pthread_rwlock_wrlock.c | 8 ++- nptl/sysdeps/pthread/pthread.h | 10 ++- .../unix/sysv/linux/x86/bits/pthreadtypes.h | 6 +- nptl/sysdeps/unix/sysv/linux/x86/elide.h | 81 ++++++++++++++++++++++ nptl/sysdeps/unix/sysv/linux/x86/elision-conf.c | 2 + 9 files changed, 127 insertions(+), 9 deletions(-) create mode 100644 nptl/sysdeps/unix/sysv/linux/x86/elide.h diff --git a/nptl/pthread_rwlock_rdlock.c b/nptl/pthread_rwlock_rdlock.c index a4deed4..413aa76 100644 --- a/nptl/pthread_rwlock_rdlock.c +++ b/nptl/pthread_rwlock_rdlock.c @@ -22,7 +22,7 @@ #include #include #include - +#include "elide.h" /* Acquire read lock for RWLOCK. Slow path. */ static int __attribute__((noinline)) @@ -102,6 +102,12 @@ __pthread_rwlock_rdlock (pthread_rwlock_t *rwlock) LIBC_PROBE (rdlock_entry, 1, rwlock); + if (ELIDE_LOCK (rwlock->__data.__rwelision, + rwlock->__data.__lock == 0 + && rwlock->__data.__writer == 0 + && rwlock->__data.__nr_readers == 0)) + return 0; + /* Make sure we are alone. */ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); diff --git a/nptl/pthread_rwlock_tryrdlock.c b/nptl/pthread_rwlock_tryrdlock.c index f7b1e6b..fe615bc 100644 --- a/nptl/pthread_rwlock_tryrdlock.c +++ b/nptl/pthread_rwlock_tryrdlock.c @@ -19,13 +19,19 @@ #include #include "pthreadP.h" #include - +#include "elide.h" int __pthread_rwlock_tryrdlock (pthread_rwlock_t *rwlock) { int result = EBUSY; + if (ELIDE_TRYLOCK (rwlock->__data.__rwelision, + rwlock->__data.__lock == 0 + && rwlock->__data.__nr_readers == 0 + && rwlock->__data.__writer, 0)) + return 0; + lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); if (rwlock->__data.__writer == 0 diff --git a/nptl/pthread_rwlock_trywrlock.c b/nptl/pthread_rwlock_trywrlock.c index 106f157..9891c3d 100644 --- a/nptl/pthread_rwlock_trywrlock.c +++ b/nptl/pthread_rwlock_trywrlock.c @@ -19,13 +19,19 @@ #include #include "pthreadP.h" #include - +#include "elide.h" int __pthread_rwlock_trywrlock (pthread_rwlock_t *rwlock) { int result = EBUSY; + if (ELIDE_TRYLOCK (rwlock->__data.__rwelision, + rwlock->__data.__lock == 0 + && rwlock->__data.__nr_readers == 0 + && rwlock->__data.__writer, 1)) + return 0; + lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); if (rwlock->__data.__writer == 0 && rwlock->__data.__nr_readers == 0) diff --git a/nptl/pthread_rwlock_unlock.c b/nptl/pthread_rwlock_unlock.c index d492383..e3c63e4 100644 --- a/nptl/pthread_rwlock_unlock.c +++ b/nptl/pthread_rwlock_unlock.c @@ -22,6 +22,7 @@ #include #include #include +#include "elide.h" /* Unlock RWLOCK. */ int @@ -29,6 +30,10 @@ __pthread_rwlock_unlock (pthread_rwlock_t *rwlock) { LIBC_PROBE (rwlock_unlock, 1, rwlock); + if (ELIDE_UNLOCK (rwlock->__data.__writer == 0 + && rwlock->__data.__nr_readers == 0)) + return 0; + lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); if (rwlock->__data.__writer) rwlock->__data.__writer = 0; diff --git a/nptl/pthread_rwlock_wrlock.c b/nptl/pthread_rwlock_wrlock.c index 2907681..fc1217c 100644 --- a/nptl/pthread_rwlock_wrlock.c +++ b/nptl/pthread_rwlock_wrlock.c @@ -22,7 +22,7 @@ #include #include #include - +#include "elide.h" /* Acquire write lock for RWLOCK. */ static int __attribute__((noinline)) @@ -91,6 +91,12 @@ __pthread_rwlock_wrlock (pthread_rwlock_t *rwlock) { LIBC_PROBE (wrlock_entry, 1, rwlock); + if (ELIDE_LOCK (rwlock->__data.__rwelision, + rwlock->__data.__lock == 0 + && rwlock->__data.__writer == 0 + && rwlock->__data.__nr_readers == 0)) + return 0; + /* Make sure we are alone. */ lll_lock (rwlock->__data.__lock, rwlock->__data.__shared); diff --git a/nptl/sysdeps/pthread/pthread.h b/nptl/sysdeps/pthread/pthread.h index 1e0c5dc..6992e4a 100644 --- a/nptl/sysdeps/pthread/pthread.h +++ b/nptl/sysdeps/pthread/pthread.h @@ -139,19 +139,23 @@ enum # endif #endif +#ifndef __PTHREAD_RWLOCK_ELISION_EXTRA +# define __PTHREAD_RWLOCK_ELISION_EXTRA 0 +#endif + /* Read-write lock initializers. */ # define PTHREAD_RWLOCK_INITIALIZER \ - { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } } + { { 0, 0, 0, 0, 0, 0, 0, 0, __PTHREAD_RWLOCK_ELISION_EXTRA, 0, 0 } } # ifdef __USE_GNU # ifdef __PTHREAD_RWLOCK_INT_FLAGS_SHARED # define PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP \ - { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, \ + { { 0, 0, 0, 0, 0, 0, 0, 0, __PTHREAD_RWLOCK_ELISION_EXTRA, 0, \ PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP } } # else # if __BYTE_ORDER == __LITTLE_ENDIAN # define PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP \ { { 0, 0, 0, 0, 0, 0, PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP, \ - 0, 0, 0, 0 } } + 0, __PTHREAD_RWLOCK_ELISION_EXTRA, 0, 0 } } # else # define PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP \ { { 0, 0, 0, 0, 0, 0, 0, 0, 0, PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP,\ diff --git a/nptl/sysdeps/unix/sysv/linux/x86/bits/pthreadtypes.h b/nptl/sysdeps/unix/sysv/linux/x86/bits/pthreadtypes.h index 28e5144..6152bdf 100644 --- a/nptl/sysdeps/unix/sysv/linux/x86/bits/pthreadtypes.h +++ b/nptl/sysdeps/unix/sysv/linux/x86/bits/pthreadtypes.h @@ -183,11 +183,13 @@ typedef union unsigned int __nr_writers_queued; int __writer; int __shared; - unsigned long int __pad1; + unsigned char __rwelision; + unsigned char __pad1[7]; unsigned long int __pad2; /* FLAGS must stay at this position in the structure to maintain binary compatibility. */ unsigned int __flags; +# define __PTHREAD_RWLOCK_ELISION_EXTRA 0, {0, 0, 0, 0, 0, 0, 0 } # define __PTHREAD_RWLOCK_INT_FLAGS_SHARED 1 } __data; # else @@ -203,7 +205,7 @@ typedef union binary compatibility. */ unsigned char __flags; unsigned char __shared; - unsigned char __pad1; + unsigned char __rwelision; unsigned char __pad2; int __writer; } __data; diff --git a/nptl/sysdeps/unix/sysv/linux/x86/elide.h b/nptl/sysdeps/unix/sysv/linux/x86/elide.h new file mode 100644 index 0000000..cb56901 --- /dev/null +++ b/nptl/sysdeps/unix/sysv/linux/x86/elide.h @@ -0,0 +1,81 @@ +#ifndef ELIDE_H +#define ELIDE_H 1 + +#include "hle.h" +#include "elision-conf.h" + +/* Adapt elision with ADAPT_COUNT and STATUS and decide retries. */ + +static inline int elision_adapt(unsigned char *adapt_count, unsigned status) +{ + if (status & _XABORT_RETRY) + return 0; + if ((status & _XABORT_EXPLICIT) + && _XABORT_CODE (status) == _ABORT_LOCK_BUSY) + { + /* Right now we skip here. Better would be to wait a bit + and retry. This likely needs some spinning. */ + if (*adapt_count != __elision_aconf.skip_lock_busy) + *adapt_count = __elision_aconf.skip_lock_busy; + } + /* Internal abort. There is no chance for retry. + Use the normal locking and next time use lock. + Be careful to avoid writing to the lock. */ + else if (*adapt_count != __elision_aconf.skip_lock_internal_abort) + *adapt_count = __elision_aconf.skip_lock_internal_abort; + return 1; +} + +/* is_lock_free must be executed inside the transaction */ + +#define ELIDE_LOCK(adapt_count, is_lock_free) \ + ({ \ + int i; \ + unsigned status; \ + int ret = 0; \ + \ + if ((adapt_count) <= 0) \ + { \ + for (i = __elision_aconf.retry_try_xbegin; i > 0; i--) \ + { \ + if ((status = _xbegin ()) == _XBEGIN_STARTED) \ + { \ + if (is_lock_free) \ + { \ + ret = 1; \ + break; \ + } \ + _xabort (_ABORT_LOCK_BUSY); \ + } \ + if (!elision_adapt (&(adapt_count), status)) \ + break; \ + } \ + } \ + else \ + (adapt_count)--; /* missing updates ok */ \ + ret; \ + }) + +#define ELIDE_TRYLOCK(adapt_count, is_lock_free, write) ({ \ + int ret = 0; \ + if (__elision_aconf.retry_try_xbegin > 0) \ + { \ + if (write) \ + _xabort (_ABORT_NESTED_TRYLOCK); \ + ret = ELIDE_LOCK (adapt_count, is_lock_free); \ + } \ + ret; \ + }) + +#define ELIDE_UNLOCK(is_lock_free) \ + ({ \ + int ret = 0; \ + if (is_lock_free) \ + { \ + _xend (); \ + ret = 1; \ + } \ + ret; \ + }) + +#endif diff --git a/nptl/sysdeps/unix/sysv/linux/x86/elision-conf.c b/nptl/sysdeps/unix/sysv/linux/x86/elision-conf.c index e6f5d6d..28e48d9 100644 --- a/nptl/sysdeps/unix/sysv/linux/x86/elision-conf.c +++ b/nptl/sysdeps/unix/sysv/linux/x86/elision-conf.c @@ -66,6 +66,8 @@ elision_init (int argc __attribute__ ((unused)), #ifdef ENABLE_LOCK_ELISION __pthread_force_elision = __libc_enable_secure ? 0 : __elision_available; #endif + if (!HAS_RTM) + __elision_aconf.retry_try_xbegin = 0; /* Disable elision on rwlocks */ } #ifdef SHARED