From patchwork Wed Aug 11 20:01:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 44646 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C3B07398583A for ; Wed, 11 Aug 2021 20:03:56 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org C3B07398583A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1628712236; bh=kfQhGgR1Vls9GSPgtYZUE99sqejSgjmBoY9T+zAHB0s=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=pt/Z+Pxcce1VmuL9X033trsfG8777YnKS1TMi6ZjUbPJmKV+OCjZDQSwP1HmhVY1P UafZr1zc1D9hj6jKQ8iTthiOkrCHOEIf0/lidJwErZhAUHsH+CG8lqSU3Zzk08ShUF 2jz9W0OiMc1WDkq76gbBzZUwXn8CYVCkwbSJqiDo= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by sourceware.org (Postfix) with ESMTPS id 52B73398582C for ; Wed, 11 Aug 2021 20:01:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 52B73398582C Received: by mail-pj1-x1036.google.com with SMTP id oa17so5330978pjb.1 for ; Wed, 11 Aug 2021 13:01:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kfQhGgR1Vls9GSPgtYZUE99sqejSgjmBoY9T+zAHB0s=; b=F6Yz6YBQvIn8kjycZLTpsILtChH98coYzMd61iHTippR0rQgJ9piZtWqNuJOg7/YUS Hx4AJnsn8lRoqQ4S48ouKVj0LBkXq6Q4q4PYZT11WW2K2XHsxvGfQ+Ot7lLaE2ogri8q wchOciICo3x7AmC6NyGURWLxyDJAk+f4Uu33xXld40ufGP4fAF4AkY6brXupeOX9w/PH NRUvvga6f88wV8kZj2PD71gdnQfrlXD6WuIWUNS9tjTjiEWCXi0efBU0oYeB5qGpBv2y Lh23/9x9yupEAt1XO0nsTqnZEyL1vzYV0dC594bd7jPn3dk6PUvNXAhsWZrLYXtCxUOR Swgw== X-Gm-Message-State: AOAM530bz/rvOMrusffP08hLyIxqsoB4GSRc+xTs972IsZlpoPz3/Lln Er2oLNXfSc90GIPNCxAe7Cr7lJhssqfw+Q== X-Google-Smtp-Source: ABdhPJx2CBAiZqef3RRDumrbt1j0d3R8Vb/DFoOUP9zSHtDU2bgXprWGdlsLZ3AadM9c5OIOm2jrog== X-Received: by 2002:a17:902:8d8f:b029:12d:3d2d:b043 with SMTP id v15-20020a1709028d8fb029012d3d2db043mr483340plo.77.1628712104225; Wed, 11 Aug 2021 13:01:44 -0700 (PDT) Received: from birita.. ([2804:431:c7cb:9dce:70f4:4f7a:652c:7fb]) by smtp.gmail.com with ESMTPSA id r15sm281370pjl.29.2021.08.11.13.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Aug 2021 13:01:43 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH 2/4] nptl: Use FUTEX_LOCK_PI2 when available Date: Wed, 11 Aug 2021 17:01:31 -0300 Message-Id: <20210811200133.3869287-3-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210811200133.3869287-1-adhemerval.zanella@linaro.org> References: <20210811200133.3869287-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Kurt Kanzenbach Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" This patch uses the new futex PI operation provided by Linux v5.14 when it is required. The futex_lock_pi64() is moved to futex-internal.c (since it used on two different places and its code size might be large depending of the kernel configuration) and clockid is added as an argument. Co-authored-by: Kurt Kanzenbach --- nptl/futex-internal.c | 63 +++++++++++++++++++++++++++++++ nptl/pthread_mutex_lock.c | 3 +- nptl/pthread_mutex_timedlock.c | 3 +- sysdeps/nptl/futex-internal.h | 54 +------------------------- sysdeps/nptl/lowlevellock-futex.h | 1 + 5 files changed, 70 insertions(+), 54 deletions(-) diff --git a/nptl/futex-internal.c b/nptl/futex-internal.c index e74647a9d4..58605b2fca 100644 --- a/nptl/futex-internal.c +++ b/nptl/futex-internal.c @@ -140,3 +140,66 @@ __futex_abstimed_wait_cancelable64 (unsigned int* futex_word, abstime, private, true); } libc_hidden_def (__futex_abstimed_wait_cancelable64) + +int +__futex_lock_pi64 (int *futex_word, clockid_t clockid, + const struct __timespec64 *abstime, int private) +{ + int err; + + unsigned int clockbit = clockid == CLOCK_REALTIME + ? FUTEX_CLOCK_REALTIME : 0; + int op_pi2 = __lll_private_flag (FUTEX_LOCK_PI2 | clockbit, private); +#if __ASSUME_FUTEX_LOCK_PI2 + /* Assume __ASSUME_TIME64_SYSCALLS since FUTEX_LOCK_PI2 was added later. */ + err = INTERNAL_SYSCALL_CALL (futex_time64, futex_word, op_pi2, 0, abstime); +#else + /* FUTEX_LOCK_PI does not support clock selection, so for CLOCK_MONOTONIC + the only option is to use FUTEX_LOCK_PI2. */ + int op_pi1 = __lll_private_flag (FUTEX_LOCK_PI, private); + int op_pi = abstime != NULL && clockid != CLOCK_REALTIME ? op_pi2 : op_pi1; + +# ifdef __ASSUME_TIME64_SYSCALLS + err = INTERNAL_SYSCALL_CALL (futex_time64, futex_word, op_pi, 0, abstime); +# else + bool need_time64 = abstime != NULL && !in_time_t_range (abstime->tv_sec); + if (need_time64) + err = INTERNAL_SYSCALL_CALL (futex_time64, futex_word, op_pi, 0, abstime); + else + { + struct timespec ts32, *pts32 = NULL; + if (abstime != NULL) + { + ts32 = valid_timespec64_to_timespec (*abstime); + pts32 = &ts32; + } + err = INTERNAL_SYSCALL_CALL (futex, futex_word, op_pi, 0, pts32); + } +# endif /* __ASSUME_TIME64_SYSCALLS */ + /* FUTEX_LOCK_PI2 is not available on this kernel. */ + if (err == -ENOSYS) + err = -EINVAL; +#endif /* __ASSUME_FUTEX_LOCK_PI2 */ + + switch (err) + { + case 0: + case -EAGAIN: + case -EINTR: + case -ETIMEDOUT: + case -ESRCH: + case -EDEADLK: + case -EINVAL: /* This indicates either state corruption or that the kernel + found a waiter on futex address which is waiting via + FUTEX_WAIT or FUTEX_WAIT_BITSET. This is reported on + some futex_lock_pi usage (pthread_mutex_timedlock for + instance). */ + return -err; + + case -EFAULT: /* Must have been caused by a glibc or application bug. */ + case -ENOSYS: /* Must have been caused by a glibc bug. */ + /* No other errors are documented at this time. */ + default: + futex_fatal_error (); + } +} diff --git a/nptl/pthread_mutex_lock.c b/nptl/pthread_mutex_lock.c index da624f322d..8e2ff2793f 100644 --- a/nptl/pthread_mutex_lock.c +++ b/nptl/pthread_mutex_lock.c @@ -422,7 +422,8 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) int private = (robust ? PTHREAD_ROBUST_MUTEX_PSHARED (mutex) : PTHREAD_MUTEX_PSHARED (mutex)); - int e = futex_lock_pi64 (&mutex->__data.__lock, NULL, private); + int e = __futex_lock_pi64 (&mutex->__data.__lock, 0 /* ununsed */, + NULL, private); if (e == ESRCH || e == EDEADLK) { assert (e != EDEADLK diff --git a/nptl/pthread_mutex_timedlock.c b/nptl/pthread_mutex_timedlock.c index 11ad7005d0..ca51da6f6c 100644 --- a/nptl/pthread_mutex_timedlock.c +++ b/nptl/pthread_mutex_timedlock.c @@ -370,7 +370,8 @@ __pthread_mutex_clocklock_common (pthread_mutex_t *mutex, int private = (robust ? PTHREAD_ROBUST_MUTEX_PSHARED (mutex) : PTHREAD_MUTEX_PSHARED (mutex)); - int e = futex_lock_pi64 (&mutex->__data.__lock, abstime, private); + int e = __futex_lock_pi64 (&mutex->__data.__lock, clockid, abstime, + private); if (e == ETIMEDOUT) return ETIMEDOUT; else if (e == ESRCH || e == EDEADLK) diff --git a/sysdeps/nptl/futex-internal.h b/sysdeps/nptl/futex-internal.h index 79a366604d..f0b7f814cc 100644 --- a/sysdeps/nptl/futex-internal.h +++ b/sysdeps/nptl/futex-internal.h @@ -250,58 +250,8 @@ futex_wake (unsigned int* futex_word, int processes_to_wake, int private) futex. - ETIMEDOUT if the ABSTIME expires. */ -static __always_inline int -futex_lock_pi64 (int *futex_word, const struct __timespec64 *abstime, - int private) -{ - int err; -#ifdef __ASSUME_TIME64_SYSCALLS - err = INTERNAL_SYSCALL_CALL (futex_time64, futex_word, - __lll_private_flag (FUTEX_LOCK_PI, private), 0, - abstime); -#else - bool need_time64 = abstime != NULL && !in_time_t_range (abstime->tv_sec); - if (need_time64) - { - err = INTERNAL_SYSCALL_CALL (futex_time64, futex_word, - __lll_private_flag (FUTEX_LOCK_PI, private), - 0, abstime); - if (err == -ENOSYS) - err = -EOVERFLOW; - } - else - { - struct timespec ts32; - if (abstime != NULL) - ts32 = valid_timespec64_to_timespec (*abstime); - - err = INTERNAL_SYSCALL_CALL (futex, futex_word, __lll_private_flag - (FUTEX_LOCK_PI, private), 0, - abstime != NULL ? &ts32 : NULL); - } -#endif - switch (err) - { - case 0: - case -EAGAIN: - case -EINTR: - case -ETIMEDOUT: - case -ESRCH: - case -EDEADLK: - case -EINVAL: /* This indicates either state corruption or that the kernel - found a waiter on futex address which is waiting via - FUTEX_WAIT or FUTEX_WAIT_BITSET. This is reported on - some futex_lock_pi usage (pthread_mutex_timedlock for - instance). */ - return -err; - - case -EFAULT: /* Must have been caused by a glibc or application bug. */ - case -ENOSYS: /* Must have been caused by a glibc bug. */ - /* No other errors are documented at this time. */ - default: - futex_fatal_error (); - } -} +int __futex_lock_pi64 (int *futex_word, clockid_t clockid, + const struct __timespec64 *abstime, int private); /* Wakes the top priority waiter that called a futex_lock_pi operation on the futex. diff --git a/sysdeps/nptl/lowlevellock-futex.h b/sysdeps/nptl/lowlevellock-futex.h index 66ebfe50f4..abda179e0d 100644 --- a/sysdeps/nptl/lowlevellock-futex.h +++ b/sysdeps/nptl/lowlevellock-futex.h @@ -38,6 +38,7 @@ #define FUTEX_WAKE_BITSET 10 #define FUTEX_WAIT_REQUEUE_PI 11 #define FUTEX_CMP_REQUEUE_PI 12 +#define FUTEX_LOCK_PI2 13 #define FUTEX_PRIVATE_FLAG 128 #define FUTEX_CLOCK_REALTIME 256