From patchwork Wed Jul 2 23:14:04 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roland McGrath X-Patchwork-Id: 1861 Received: (qmail 12210 invoked by alias); 2 Jul 2014 23:14:24 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 12110 invoked by uid 89); 2 Jul 2014 23:14:18 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.2 required=5.0 tests=AWL, BAYES_00 autolearn=ham version=3.3.2 X-HELO: topped-with-meat.com MIME-Version: 1.0 From: Roland McGrath To: "GNU C. Library" CC: Bernard Ogden Subject: [PATCH roland/lll-futex] Separate Linuxisms from lowlevellock.h, make a generic one Message-Id: <20140702231404.F04F32C3978@topped-with-meat.com> Date: Wed, 2 Jul 2014 16:14:04 -0700 (PDT) X-CMAE-Score: 0 X-CMAE-Analysis: v=2.1 cv=SvUDtp+0 c=1 sm=1 tr=0 a=WkljmVdYkabdwxfqvArNOQ==:117 a=14OXPxybAAAA:8 a=rTK20uzY-okA:10 a=Z6MIti7PxpgA:10 a=kj9zAlcOel0A:10 a=hOe2yjtxAAAA:8 a=mDV3o1hIAAAA:8 a=QsdAfc806aeg_2HelVUA:9 a=23BPzig0Y2Jfn7c4:21 a=eRd8tEp-MAZf4KjR:21 a=CjuIK1q_8ugA:10 This makes lowlevellock.h generic, and splits out the Linux-specific bits into a new file lowlevellock-futex.h. Aside from the split, the generic file is very similar to Bernie's linux-generic version. (I actually started with the ARM file and worked from there, but then looked at the differences from Bernie's file.) I think Bernie's version introduced a regression that's hard to test for. In his file, all the macros return the "result" of the syscall, i.e. either a negated errno code or the non-error return value of the syscall. However, at least one or two places rely on the non-error return value of at least some of these macros being exactly zero. For example, lll_futex_requeue is expected to return 0, but the syscall when successful will usually return a positive number. The failure mode of this regression is that the code will fall back to thundering-herd wakeups, which is just a performance regression (but a very important one). So we need to take great care in looking for those kinds of changes to the compiled code, not simply verify that the test suite passes. Unfortunately, the status quo ante is rather haphazard treatment of the syscall return value in the different macros. So cleaning the macros up causes more changes to the generated code than I'd like. I'm still looking at the testing (on arm-linux-gnueabihf). There are some nontrivial differences I wasn't expecting, and I need to examine them all more thoroughly. Some are just mysterious, where the compiler made a different code-generation choice for no apparent reason (like swapping the order of operands in a comparison and inverting the associated conditional branch). Other are understandable but stupid, where it does more checking on the syscall return value than it actually cares about (an example of this costs three or four extra instructions and an extra branch, but that's after doing a blocking syscall, so it's lost deeply in the noise). Others are probably from the new uniformity of the syscall return value handling but should be harmless. I need to examine them all before claiming they don't matter, and I won't get to that until next week. Joseph is away and I don't expect him to approve the change for ARM very soon. After I've finished the testing and if Bernie and others are happy with the new code, I figure I'll commit the new files next week but not remove the ARM file. I note that the AAarch64 file is well nigh identical to the ARM file and several others probably are too. So those machines' maintainers could (and should) remove their files and test the new generic code as soon as they manage. 2014-07-02 Roland McGrath * sysdeps/nptl/lowlevellock.h: New file. * sysdeps/unix/sysv/linux/lowlevellock-futex.h: New file. * sysdeps/nptl/lowlevellock-futex.h: New file. * sysdeps/unix/sysv/linux/arm/lowlevellock.h: File removed. diff --git a/sysdeps/nptl/lowlevellock-futex.h b/sysdeps/nptl/lowlevellock-futex.h new file mode 100644 index 0000000..12f3876 --- /dev/null +++ b/sysdeps/nptl/lowlevellock-futex.h @@ -0,0 +1,86 @@ +/* Low-level locking access to futex facilities. Stub version. + Copyright (C) 2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#ifndef _LOWLEVELLOCK_FUTEX_H +#define _LOWLEVELLOCK_FUTEX_H 1 + +#include + + +/* Values for 'private' parameter of locking macros. Note pthreadP.h + optimizes for these exact values, though they are not required. */ +#define LLL_PRIVATE 0 +#define LLL_SHARED 128 + + +/* For most of these macros, the return value is never really used. + Nevertheless, the protocol is that each one returns a negated errno + code for failure or zero for success. (Note that the corresponding + Linux system calls can sometimes return positive values for success + cases too. We never use those values.) */ + + +/* Wait while *FUTEXP == VAL for an lll_futex_wake call on FUTEXP. */ +#define lll_futex_wait(futexp, val, private) \ + lll_futex_timed_wait (futexp, val, NULL, private) + +/* Wait until a lll_futex_wake call on FUTEXP, or TIMEOUT elapses. */ +#define lll_futex_timed_wait(futexp, val, timeout, private) \ + -ENOSYS + +/* This macro should be defined only if FUTEX_CLOCK_REALTIME is also defined. + If CLOCKBIT is zero, this is identical to lll_futex_timed_wait. + If CLOCKBIT has FUTEX_CLOCK_REALTIME set, then it's the same but + TIMEOUT is counted by CLOCK_REALTIME rather than CLOCK_MONOTONIC. */ +#define lll_futex_timed_wait_bitset(futexp, val, timeout, clockbit, private) \ + -ENOSYS + +/* Wake up up to NR waiters on FUTEXP. */ +#define lll_futex_wake(futexp, nr, private) \ + -ENOSYS + +/* Wake up up to NR_WAKE waiters on FUTEXP. Move up to NR_MOVE of the + rest from waiting on FUTEXP to waiting on MUTEX (a different futex). */ +#define lll_futex_requeue(futexp, nr_wake, nr_move, mutex, val, private) \ + -ENOSYS + +/* Wake up up to NR_WAKE waiters on FUTEXP and NR_WAKE2 on FUTEXP2. */ +#define lll_futex_wake_unlock(futexp, nr_wake, nr_wake2, futexp2, private) \ + -ENOSYS + + +/* Like lll_futex_wait (FUTEXP, VAL, PRIVATE) but with the expectation + that lll_futex_cmp_requeue_pi (FUTEXP, _, _, MUTEX, _, PRIVATE) will + be used to do the wakeup. Confers priority-inheritance behavior on + the waiter. */ +#define lll_futex_wait_requeue_pi(futexp, val, mutex, private) \ + lll_futex_timed_wait_requeue_pi (futexp, val, NULL, 0, mutex, private) + +/* Like lll_futex_wait_requeue_pi, but with a timeout. */ +#define lll_futex_timed_wait_requeue_pi(futexp, val, timeout, clockbit, \ + mutex, private) \ + -ENOSYS + +/* Like lll_futex_requeue, but pairs with lll_futex_wait_requeue_pi + and inherits priority from the waiter. */ +#define lll_futex_cmp_requeue_pi(futexp, nr_wake, nr_move, mutex, \ + val, private) \ + -ENOSYS + + +#endif /* lowlevellock-futex.h */ diff --git a/sysdeps/nptl/lowlevellock.h b/sysdeps/nptl/lowlevellock.h new file mode 100644 index 0000000..e440bb4 --- /dev/null +++ b/sysdeps/nptl/lowlevellock.h @@ -0,0 +1,201 @@ +/* Low-level lock implementation. Generic futex-based version. + Copyright (C) 2005-2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#ifndef _LOWLEVELLOCK_H +#define _LOWLEVELLOCK_H 1 + +#include +#include + +#define lll_robust_dead(futexv, private) \ + do \ + { \ + int *__futexp = &(futexv); \ + atomic_or (__futexp, FUTEX_OWNER_DIED); \ + lll_futex_wake (__futexp, 1, private); \ + } \ + while (0) + +#define lll_trylock(lock) \ + atomic_compare_and_exchange_bool_acq (&(lock), 1, 0) + +#define lll_cond_trylock(lock) \ + atomic_compare_and_exchange_bool_acq (&(lock), 2, 0) + +#define __lll_robust_trylock(futex, id) \ + (atomic_compare_and_exchange_val_acq (futex, id, 0) != 0) +#define lll_robust_trylock(lock, id) \ + __lll_robust_trylock (&(lock), id) + +extern void __lll_lock_wait_private (int *futex) attribute_hidden; +extern void __lll_lock_wait (int *futex, int private) attribute_hidden; +extern int __lll_robust_lock_wait (int *futex, int private) attribute_hidden; + +/* This is an expression rather than a statement even though its value is + void, so that it can be used in a comma expression or as an expression + that's cast to void. */ +#define __lll_lock(futex, private) \ + ((void) \ + ({ \ + int *__futex = (futex); \ + if (__glibc_unlikely \ + (atomic_compare_and_exchange_bool_acq (__futex, 1, 0))) \ + { \ + if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \ + __lll_lock_wait_private (__futex); \ + else \ + __lll_lock_wait (__futex, private); \ + } \ + })) +#define lll_lock(futex, private) \ + __lll_lock (&(futex), private) + + +#define __lll_robust_lock(futex, id, private) \ + ({ \ + int *__futex = (futex); \ + int __val = 0; \ + \ + if (__glibc_unlikely \ + (atomic_compare_and_exchange_bool_acq (__futex, id, 0))) \ + __val = __lll_robust_lock_wait (__futex, private); \ + __val; \ + }) +#define lll_robust_lock(futex, id, private) \ + __lll_robust_lock (&(futex), id, private) + + +/* This is an expression rather than a statement even though its value is + void, so that it can be used in a comma expression or as an expression + that's cast to void. */ +#define __lll_cond_lock(futex, private) \ + ((void) \ + ({ \ + int *__futex = (futex); \ + if (__glibc_unlikely (atomic_exchange_acq (__futex, 2))) \ + __lll_lock_wait (__futex, private); \ + })) +#define lll_cond_lock(futex, private) __lll_cond_lock (&(futex), private) + + +#define lll_robust_cond_lock(futex, id, private) \ + __lll_robust_lock (&(futex), (id) | FUTEX_WAITERS, private) + + +extern int __lll_timedlock_wait (int *futex, const struct timespec *, + int private) attribute_hidden; +extern int __lll_robust_timedlock_wait (int *futex, const struct timespec *, + int private) attribute_hidden; + +#define __lll_timedlock(futex, abstime, private) \ + ({ \ + int *__futex = (futex); \ + int __val = 0; \ + \ + if (__glibc_unlikely (atomic_exchange_acq (__futex, 1))) \ + __val = __lll_timedlock_wait (__futex, abstime, private); \ + __val; \ + }) +#define lll_timedlock(futex, abstime, private) \ + __lll_timedlock (&(futex), abstime, private) + + +#define __lll_robust_timedlock(futex, abstime, id, private) \ + ({ \ + int *__futex = (futex); \ + int __val = 0; \ + \ + if (__glibc_unlikely \ + (atomic_compare_and_exchange_bool_acq (__futex, id, 0))) \ + __val = __lll_robust_timedlock_wait (__futex, abstime, private); \ + __val; \ + }) +#define lll_robust_timedlock(futex, abstime, id, private) \ + __lll_robust_timedlock (&(futex), abstime, id, private) + + +/* This is an expression rather than a statement even though its value is + void, so that it can be used in a comma expression or as an expression + that's cast to void. */ +#define __lll_unlock(futex, private) \ + ((void) \ + ({ \ + int *__futex = (futex); \ + int __oldval = atomic_exchange_rel (__futex, 0); \ + if (__glibc_unlikely (__oldval > 1)) \ + lll_futex_wake (__futex, 1, private); \ + })) +#define lll_unlock(futex, private) \ + __lll_unlock (&(futex), private) + + +/* This is an expression rather than a statement even though its value is + void, so that it can be used in a comma expression or as an expression + that's cast to void. */ +#define __lll_robust_unlock(futex, private) \ + ((void) \ + ({ \ + int *__futex = (futex); \ + int __oldval = atomic_exchange_rel (__futex, 0); \ + if (__glibc_unlikely (__oldval & FUTEX_WAITERS)) \ + lll_futex_wake (__futex, 1, private); \ + })) +#define lll_robust_unlock(futex, private) \ + __lll_robust_unlock (&(futex), private) + + +#define lll_islocked(futex) \ + ((futex) != LLL_LOCK_INITIALIZER) + + +/* Our internal lock implementation is identical to the binary-compatible + mutex implementation. */ + +/* Initializers for lock. */ +#define LLL_LOCK_INITIALIZER (0) +#define LLL_LOCK_INITIALIZER_LOCKED (1) + +/* The states of a lock are: + 0 - untaken + 1 - taken by one user + >1 - taken by more users */ + +/* The kernel notifies a process which uses CLONE_CHILD_CLEARTID via futex + wakeup when the clone terminates. The memory location contains the + thread ID while the clone is running and is reset to zero + afterwards. */ +#define lll_wait_tid(tid) \ + do { \ + __typeof (tid) __tid; \ + while ((__tid = (tid)) != 0) \ + lll_futex_wait (&(tid), __tid, LLL_SHARED);\ + } while (0) + +extern int __lll_timedwait_tid (int *, const struct timespec *) + attribute_hidden; + +#define lll_timedwait_tid(tid, abstime) \ + ({ \ + int __res = 0; \ + if ((tid) != 0) \ + __res = __lll_timedwait_tid (&(tid), (abstime)); \ + __res; \ + }) + + +#endif /* lowlevellock.h */ diff --git a/sysdeps/unix/sysv/linux/arm/lowlevellock.h b/sysdeps/unix/sysv/linux/arm/lowlevellock.h deleted file mode 100644 index 5d19434..0000000 --- a/sysdeps/unix/sysv/linux/arm/lowlevellock.h +++ /dev/null @@ -1,321 +0,0 @@ -/* Copyright (C) 2005-2014 Free Software Foundation, Inc. - This file is part of the GNU C Library. - - The GNU C Library is free software; you can redistribute it and/or - modify it under the terms of the GNU Lesser General Public - License as published by the Free Software Foundation; either - version 2.1 of the License, or (at your option) any later version. - - The GNU C Library is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - Lesser General Public License for more details. - - You should have received a copy of the GNU Lesser General Public - License along with the GNU C Library. If not, see - . */ - -#ifndef _LOWLEVELLOCK_H -#define _LOWLEVELLOCK_H 1 - -#include -#include -#include -#include -#include -#include - -#define FUTEX_WAIT 0 -#define FUTEX_WAKE 1 -#define FUTEX_REQUEUE 3 -#define FUTEX_CMP_REQUEUE 4 -#define FUTEX_WAKE_OP 5 -#define FUTEX_OP_CLEAR_WAKE_IF_GT_ONE ((4 << 24) | 1) -#define FUTEX_LOCK_PI 6 -#define FUTEX_UNLOCK_PI 7 -#define FUTEX_TRYLOCK_PI 8 -#define FUTEX_WAIT_BITSET 9 -#define FUTEX_WAKE_BITSET 10 -#define FUTEX_WAIT_REQUEUE_PI 11 -#define FUTEX_CMP_REQUEUE_PI 12 -#define FUTEX_PRIVATE_FLAG 128 -#define FUTEX_CLOCK_REALTIME 256 - -#define FUTEX_BITSET_MATCH_ANY 0xffffffff - -/* Values for 'private' parameter of locking macros. Yes, the - definition seems to be backwards. But it is not. The bit will be - reversed before passing to the system call. */ -#define LLL_PRIVATE 0 -#define LLL_SHARED FUTEX_PRIVATE_FLAG - - -#if !defined NOT_IN_libc || defined IS_IN_rtld -/* In libc.so or ld.so all futexes are private. */ -# ifdef __ASSUME_PRIVATE_FUTEX -# define __lll_private_flag(fl, private) \ - ((fl) | FUTEX_PRIVATE_FLAG) -# else -# define __lll_private_flag(fl, private) \ - ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) -# endif -#else -# ifdef __ASSUME_PRIVATE_FUTEX -# define __lll_private_flag(fl, private) \ - (((fl) | FUTEX_PRIVATE_FLAG) ^ (private)) -# else -# define __lll_private_flag(fl, private) \ - (__builtin_constant_p (private) \ - ? ((private) == 0 \ - ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \ - : (fl)) \ - : ((fl) | (((private) ^ FUTEX_PRIVATE_FLAG) \ - & THREAD_GETMEM (THREAD_SELF, header.private_futex)))) -# endif -#endif - - -#define lll_futex_wait(futexp, val, private) \ - lll_futex_timed_wait(futexp, val, NULL, private) - -#define lll_futex_timed_wait(futexp, val, timespec, private) \ - ({ \ - INTERNAL_SYSCALL_DECL (__err); \ - long int __ret; \ - __ret = INTERNAL_SYSCALL (futex, __err, 4, (futexp), \ - __lll_private_flag (FUTEX_WAIT, private), \ - (val), (timespec)); \ - __ret; \ - }) - -#define lll_futex_timed_wait_bitset(futexp, val, timespec, clockbit, private) \ - ({ \ - INTERNAL_SYSCALL_DECL (__err); \ - long int __ret; \ - int __op = FUTEX_WAIT_BITSET | clockbit; \ - __ret = INTERNAL_SYSCALL (futex, __err, 6, (futexp), \ - __lll_private_flag (__op, private), \ - (val), (timespec), NULL /* Unused. */, \ - FUTEX_BITSET_MATCH_ANY); \ - __ret; \ - }) - -#define lll_futex_wake(futexp, nr, private) \ - ({ \ - INTERNAL_SYSCALL_DECL (__err); \ - long int __ret; \ - __ret = INTERNAL_SYSCALL (futex, __err, 4, (futexp), \ - __lll_private_flag (FUTEX_WAKE, private), \ - (nr), 0); \ - __ret; \ - }) - -#define lll_robust_dead(futexv, private) \ - do \ - { \ - int *__futexp = &(futexv); \ - atomic_or (__futexp, FUTEX_OWNER_DIED); \ - lll_futex_wake (__futexp, 1, private); \ - } \ - while (0) - -/* Returns non-zero if error happened, zero if success. */ -#define lll_futex_requeue(futexp, nr_wake, nr_move, mutex, val, private) \ - ({ \ - INTERNAL_SYSCALL_DECL (__err); \ - long int __ret; \ - __ret = INTERNAL_SYSCALL (futex, __err, 6, (futexp), \ - __lll_private_flag (FUTEX_CMP_REQUEUE, private),\ - (nr_wake), (nr_move), (mutex), (val)); \ - INTERNAL_SYSCALL_ERROR_P (__ret, __err); \ - }) - - -/* Returns non-zero if error happened, zero if success. */ -#define lll_futex_wake_unlock(futexp, nr_wake, nr_wake2, futexp2, private) \ - ({ \ - INTERNAL_SYSCALL_DECL (__err); \ - long int __ret; \ - __ret = INTERNAL_SYSCALL (futex, __err, 6, (futexp), \ - __lll_private_flag (FUTEX_WAKE_OP, private), \ - (nr_wake), (nr_wake2), (futexp2), \ - FUTEX_OP_CLEAR_WAKE_IF_GT_ONE); \ - INTERNAL_SYSCALL_ERROR_P (__ret, __err); \ - }) - -/* Priority Inheritance support. */ -#define lll_futex_wait_requeue_pi(futexp, val, mutex, private) \ - lll_futex_timed_wait_requeue_pi (futexp, val, NULL, 0, mutex, private) - -#define lll_futex_timed_wait_requeue_pi(futexp, val, timespec, clockbit, \ - mutex, private) \ - ({ \ - INTERNAL_SYSCALL_DECL (__err); \ - int __op = FUTEX_WAIT_REQUEUE_PI | clockbit; \ - \ - INTERNAL_SYSCALL (futex, __err, 5, (futexp), \ - __lll_private_flag (__op, private), \ - (val), (timespec), mutex); \ - }) - -#define lll_futex_cmp_requeue_pi(futexp, nr_wake, nr_move, mutex, val, priv) \ - ({ \ - INTERNAL_SYSCALL_DECL (__err); \ - long int __ret; \ - \ - __ret = INTERNAL_SYSCALL (futex, __err, 6, (futexp), \ - __lll_private_flag (FUTEX_CMP_REQUEUE_PI, priv),\ - (nr_wake), (nr_move), (mutex), (val)); \ - INTERNAL_SYSCALL_ERROR_P (__ret, __err); \ - }) - - -#define lll_trylock(lock) \ - atomic_compare_and_exchange_val_acq(&(lock), 1, 0) - -#define lll_cond_trylock(lock) \ - atomic_compare_and_exchange_val_acq(&(lock), 2, 0) - -#define __lll_robust_trylock(futex, id) \ - (atomic_compare_and_exchange_val_acq (futex, id, 0) != 0) -#define lll_robust_trylock(lock, id) \ - __lll_robust_trylock (&(lock), id) - -extern void __lll_lock_wait_private (int *futex) attribute_hidden; -extern void __lll_lock_wait (int *futex, int private) attribute_hidden; -extern int __lll_robust_lock_wait (int *futex, int private) attribute_hidden; - -#define __lll_lock(futex, private) \ - ((void) ({ \ - int *__futex = (futex); \ - if (__builtin_expect (atomic_compare_and_exchange_val_acq (__futex, \ - 1, 0), 0)) \ - { \ - if (__builtin_constant_p (private) && (private) == LLL_PRIVATE) \ - __lll_lock_wait_private (__futex); \ - else \ - __lll_lock_wait (__futex, private); \ - } \ - })) -#define lll_lock(futex, private) __lll_lock (&(futex), private) - - -#define __lll_robust_lock(futex, id, private) \ - ({ \ - int *__futex = (futex); \ - int __val = 0; \ - \ - if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, id, \ - 0), 0)) \ - __val = __lll_robust_lock_wait (__futex, private); \ - __val; \ - }) -#define lll_robust_lock(futex, id, private) \ - __lll_robust_lock (&(futex), id, private) - - -#define __lll_cond_lock(futex, private) \ - ((void) ({ \ - int *__futex = (futex); \ - if (__builtin_expect (atomic_exchange_acq (__futex, 2), 0)) \ - __lll_lock_wait (__futex, private); \ - })) -#define lll_cond_lock(futex, private) __lll_cond_lock (&(futex), private) - - -#define lll_robust_cond_lock(futex, id, private) \ - __lll_robust_lock (&(futex), (id) | FUTEX_WAITERS, private) - - -extern int __lll_timedlock_wait (int *futex, const struct timespec *, - int private) attribute_hidden; -extern int __lll_robust_timedlock_wait (int *futex, const struct timespec *, - int private) attribute_hidden; - -#define __lll_timedlock(futex, abstime, private) \ - ({ \ - int *__futex = (futex); \ - int __val = 0; \ - \ - if (__builtin_expect (atomic_exchange_acq (__futex, 1), 0)) \ - __val = __lll_timedlock_wait (__futex, abstime, private); \ - __val; \ - }) -#define lll_timedlock(futex, abstime, private) \ - __lll_timedlock (&(futex), abstime, private) - - -#define __lll_robust_timedlock(futex, abstime, id, private) \ - ({ \ - int *__futex = (futex); \ - int __val = 0; \ - \ - if (__builtin_expect (atomic_compare_and_exchange_bool_acq (__futex, id, \ - 0), 0)) \ - __val = __lll_robust_timedlock_wait (__futex, abstime, private); \ - __val; \ - }) -#define lll_robust_timedlock(futex, abstime, id, private) \ - __lll_robust_timedlock (&(futex), abstime, id, private) - - -#define __lll_unlock(futex, private) \ - (void) \ - ({ int *__futex = (futex); \ - int __oldval = atomic_exchange_rel (__futex, 0); \ - if (__builtin_expect (__oldval > 1, 0)) \ - lll_futex_wake (__futex, 1, private); \ - }) -#define lll_unlock(futex, private) __lll_unlock(&(futex), private) - - -#define __lll_robust_unlock(futex, private) \ - (void) \ - ({ int *__futex = (futex); \ - int __oldval = atomic_exchange_rel (__futex, 0); \ - if (__builtin_expect (__oldval & FUTEX_WAITERS, 0)) \ - lll_futex_wake (__futex, 1, private); \ - }) -#define lll_robust_unlock(futex, private) \ - __lll_robust_unlock(&(futex), private) - - -#define lll_islocked(futex) \ - (futex != 0) - - -/* Our internal lock implementation is identical to the binary-compatible - mutex implementation. */ - -/* Initializers for lock. */ -#define LLL_LOCK_INITIALIZER (0) -#define LLL_LOCK_INITIALIZER_LOCKED (1) - -/* The states of a lock are: - 0 - untaken - 1 - taken by one user - >1 - taken by more users */ - -/* The kernel notifies a process which uses CLONE_CHILD_CLEARTID via futex - wakeup when the clone terminates. The memory location contains the - thread ID while the clone is running and is reset to zero - afterwards. */ -#define lll_wait_tid(tid) \ - do { \ - __typeof (tid) __tid; \ - while ((__tid = (tid)) != 0) \ - lll_futex_wait (&(tid), __tid, LLL_SHARED);\ - } while (0) - -extern int __lll_timedwait_tid (int *, const struct timespec *) - attribute_hidden; - -#define lll_timedwait_tid(tid, abstime) \ - ({ \ - int __res = 0; \ - if ((tid) != 0) \ - __res = __lll_timedwait_tid (&(tid), (abstime)); \ - __res; \ - }) - -#endif /* lowlevellock.h */ diff --git a/sysdeps/unix/sysv/linux/lowlevellock-futex.h b/sysdeps/unix/sysv/linux/lowlevellock-futex.h new file mode 100644 index 0000000..343afd6 --- /dev/null +++ b/sysdeps/unix/sysv/linux/lowlevellock-futex.h @@ -0,0 +1,137 @@ +/* Low-level locking access to futex facilities. Linux version. + Copyright (C) 2005-2014 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#ifndef _LOWLEVELLOCK_FUTEX_H +#define _LOWLEVELLOCK_FUTEX_H 1 + +#include +#include +#include + + +#define FUTEX_WAIT 0 +#define FUTEX_WAKE 1 +#define FUTEX_REQUEUE 3 +#define FUTEX_CMP_REQUEUE 4 +#define FUTEX_WAKE_OP 5 +#define FUTEX_OP_CLEAR_WAKE_IF_GT_ONE ((4 << 24) | 1) +#define FUTEX_LOCK_PI 6 +#define FUTEX_UNLOCK_PI 7 +#define FUTEX_TRYLOCK_PI 8 +#define FUTEX_WAIT_BITSET 9 +#define FUTEX_WAKE_BITSET 10 +#define FUTEX_WAIT_REQUEUE_PI 11 +#define FUTEX_CMP_REQUEUE_PI 12 +#define FUTEX_PRIVATE_FLAG 128 +#define FUTEX_CLOCK_REALTIME 256 + +#define FUTEX_BITSET_MATCH_ANY 0xffffffff + +/* Values for 'private' parameter of locking macros. Yes, the + definition seems to be backwards. But it is not. The bit will be + reversed before passing to the system call. */ +#define LLL_PRIVATE 0 +#define LLL_SHARED FUTEX_PRIVATE_FLAG + + +#if !defined NOT_IN_libc || defined IS_IN_rtld +/* In libc.so or ld.so all futexes are private. */ +# ifdef __ASSUME_PRIVATE_FUTEX +# define __lll_private_flag(fl, private) \ + ((fl) | FUTEX_PRIVATE_FLAG) +# else +# define __lll_private_flag(fl, private) \ + ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) +# endif +#else +# ifdef __ASSUME_PRIVATE_FUTEX +# define __lll_private_flag(fl, private) \ + (((fl) | FUTEX_PRIVATE_FLAG) ^ (private)) +# else +# define __lll_private_flag(fl, private) \ + (__builtin_constant_p (private) \ + ? ((private) == 0 \ + ? ((fl) | THREAD_GETMEM (THREAD_SELF, header.private_futex)) \ + : (fl)) \ + : ((fl) | (((private) ^ FUTEX_PRIVATE_FLAG) \ + & THREAD_GETMEM (THREAD_SELF, header.private_futex)))) +# endif +#endif + +#define lll_futex_syscall(nargs, futexp, op, ...) \ + ({ \ + INTERNAL_SYSCALL_DECL (__err); \ + long int __ret = INTERNAL_SYSCALL (futex, __err, nargs, futexp, op, \ + __VA_ARGS__); \ + (__glibc_unlikely (INTERNAL_SYSCALL_ERROR_P (__ret, __err)) \ + ? -INTERNAL_SYSCALL_ERRNO (__ret, __err) : 0); \ + }) + +#define lll_futex_wait(futexp, val, private) \ + lll_futex_timed_wait (futexp, val, NULL, private) + +#define lll_futex_timed_wait(futexp, val, timeout, private) \ + lll_futex_syscall (4, futexp, \ + __lll_private_flag (FUTEX_WAIT, private), \ + val, timeout) + +#define lll_futex_timed_wait_bitset(futexp, val, timeout, clockbit, private) \ + lll_futex_syscall (6, futexp, \ + __lll_private_flag (FUTEX_WAIT_BITSET | (clockbit), \ + private), \ + val, timeout, NULL /* Unused. */, \ + FUTEX_BITSET_MATCH_ANY) + +#define lll_futex_wake(futexp, nr, private) \ + lll_futex_syscall (4, futexp, \ + __lll_private_flag (FUTEX_WAKE, private), nr, 0) + +/* Returns non-zero if error happened, zero if success. */ +#define lll_futex_requeue(futexp, nr_wake, nr_move, mutex, val, private) \ + lll_futex_syscall (6, futexp, \ + __lll_private_flag (FUTEX_CMP_REQUEUE, private), \ + nr_wake, nr_move, mutex, val) + +/* Returns non-zero if error happened, zero if success. */ +#define lll_futex_wake_unlock(futexp, nr_wake, nr_wake2, futexp2, private) \ + lll_futex_syscall (6, futexp, \ + __lll_private_flag (FUTEX_WAKE_OP, private), \ + nr_wake, nr_wake2, futexp2, \ + FUTEX_OP_CLEAR_WAKE_IF_GT_ONE) + +/* Priority Inheritance support. */ +#define lll_futex_wait_requeue_pi(futexp, val, mutex, private) \ + lll_futex_timed_wait_requeue_pi (futexp, val, NULL, 0, mutex, private) + +#define lll_futex_timed_wait_requeue_pi(futexp, val, timeout, clockbit, \ + mutex, private) \ + lll_futex_syscall (5, futexp, \ + __lll_private_flag (FUTEX_WAIT_REQUEUE_PI \ + | (clockbit), private), \ + val, timeout, mutex) + + +#define lll_futex_cmp_requeue_pi(futexp, nr_wake, nr_move, mutex, \ + val, private) \ + lll_futex_syscall (6, futexp, \ + __lll_private_flag (FUTEX_CMP_REQUEUE_PI, \ + private), \ + nr_wake, nr_move, mutex, val) + + +#endif /* lowlevellock-futex.h */