From patchwork Mon Sep 6 20:47:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 44875 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 76C18385C411 for ; Mon, 6 Sep 2021 20:49:38 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 76C18385C411 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1630961378; bh=JleYG/kTUpZ4H9nsTXpCvCrB15F/ZG0BxFjv/M5R7pE=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=Y9GfvawChCLZFwfPmsqqBVkFYJnO4+dM29XIVlmMIwmVFjWwMBwVedZHfIx6Nmt7s htoNCOESZtVNs6F2xufhF5kMKvF+s9hoWcEQCrurc5oZfm5gFxmw9IvbcGNbEvhQb5 AGtjra/dOeB7wyhjWvqO9+0iGh1Irnk5uwKhbHrk= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) by sourceware.org (Postfix) with ESMTPS id 13F1A385B83B for ; Mon, 6 Sep 2021 20:47:53 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 13F1A385B83B Received: by mail-qk1-x736.google.com with SMTP id c10so7958901qko.11 for ; Mon, 06 Sep 2021 13:47:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JleYG/kTUpZ4H9nsTXpCvCrB15F/ZG0BxFjv/M5R7pE=; b=sSIMlL6wpPSTZM7xCqD0HH1BpWbqXL9OmYLmqV3eFQ9KThnx7aTYemC/uMg/7Pyl3V SWxHr362cbEVmEW8VO4kpL0U4I4hkQOBI5MXy4z7UfUka4tI48eahxry1TalrylZ1Xwu Rp5uRdZ+nIE686p5B72qBEF4U9Bp0+uJgsHl07U99Alw+rn9nIKAd8nEB/9JFZxycM3b uBx0FEh7SqdGCY7h6d+ynamM+raA8uxW0yHKPQL40EqwqOFtM92vdayiUoovQSx4E3x6 YnBqmzzl2RLaMF2mtSmmyJTxsRjrMo9H62DhUy7L8qhJ9xejmqHQ9n2VYlwSTKMZOfi1 bMEg== X-Gm-Message-State: AOAM532vbb4Z0XVHkZPp6xjY5B+K/Bep/oM9s3EcwQj38wolKee+cpYI IOgT+azBh5OwM3+ihcfXx1T5pDEhOp/LIA== X-Google-Smtp-Source: ABdhPJx3IhIYWHZ4bvyaw7hAWXHJSbpexWAb3nqV1miJqSlwfcizBHAogIcK/Y3ajGYjvIOZxIvO8Q== X-Received: by 2002:a05:620a:1009:: with SMTP id z9mr12712565qkj.483.1630961272461; Mon, 06 Sep 2021 13:47:52 -0700 (PDT) Received: from birita.. ([2804:431:c7cb:733d:210:e147:8627:2d6]) by smtp.gmail.com with ESMTPSA id w129sm7468416qkb.61.2021.09.06.13.47.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Sep 2021 13:47:52 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH v3 2/4] nptl: Use FUTEX_LOCK_PI2 when available Date: Mon, 6 Sep 2021 17:47:38 -0300 Message-Id: <20210906204740.3988273-3-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210906204740.3988273-1-adhemerval.zanella@linaro.org> References: <20210906204740.3988273-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Kurt Kanzenbach Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" This patch uses the new futex PI operation provided by Linux v5.14 when it is required. The futex_lock_pi64() is moved to futex-internal.c (since it used on two different places and its code size might be large depending of the kernel configuration) and clockid is added as an argument. Co-authored-by: Kurt Kanzenbach --- nptl/futex-internal.c | 63 +++++++++++++++++++++++++++++++ nptl/pthread_mutex_lock.c | 3 +- nptl/pthread_mutex_timedlock.c | 3 +- sysdeps/nptl/futex-internal.h | 58 ++-------------------------- sysdeps/nptl/lowlevellock-futex.h | 1 + 5 files changed, 72 insertions(+), 56 deletions(-) diff --git a/nptl/futex-internal.c b/nptl/futex-internal.c index e74647a9d4..58605b2fca 100644 --- a/nptl/futex-internal.c +++ b/nptl/futex-internal.c @@ -140,3 +140,66 @@ __futex_abstimed_wait_cancelable64 (unsigned int* futex_word, abstime, private, true); } libc_hidden_def (__futex_abstimed_wait_cancelable64) + +int +__futex_lock_pi64 (int *futex_word, clockid_t clockid, + const struct __timespec64 *abstime, int private) +{ + int err; + + unsigned int clockbit = clockid == CLOCK_REALTIME + ? FUTEX_CLOCK_REALTIME : 0; + int op_pi2 = __lll_private_flag (FUTEX_LOCK_PI2 | clockbit, private); +#if __ASSUME_FUTEX_LOCK_PI2 + /* Assume __ASSUME_TIME64_SYSCALLS since FUTEX_LOCK_PI2 was added later. */ + err = INTERNAL_SYSCALL_CALL (futex_time64, futex_word, op_pi2, 0, abstime); +#else + /* FUTEX_LOCK_PI does not support clock selection, so for CLOCK_MONOTONIC + the only option is to use FUTEX_LOCK_PI2. */ + int op_pi1 = __lll_private_flag (FUTEX_LOCK_PI, private); + int op_pi = abstime != NULL && clockid != CLOCK_REALTIME ? op_pi2 : op_pi1; + +# ifdef __ASSUME_TIME64_SYSCALLS + err = INTERNAL_SYSCALL_CALL (futex_time64, futex_word, op_pi, 0, abstime); +# else + bool need_time64 = abstime != NULL && !in_time_t_range (abstime->tv_sec); + if (need_time64) + err = INTERNAL_SYSCALL_CALL (futex_time64, futex_word, op_pi, 0, abstime); + else + { + struct timespec ts32, *pts32 = NULL; + if (abstime != NULL) + { + ts32 = valid_timespec64_to_timespec (*abstime); + pts32 = &ts32; + } + err = INTERNAL_SYSCALL_CALL (futex, futex_word, op_pi, 0, pts32); + } +# endif /* __ASSUME_TIME64_SYSCALLS */ + /* FUTEX_LOCK_PI2 is not available on this kernel. */ + if (err == -ENOSYS) + err = -EINVAL; +#endif /* __ASSUME_FUTEX_LOCK_PI2 */ + + switch (err) + { + case 0: + case -EAGAIN: + case -EINTR: + case -ETIMEDOUT: + case -ESRCH: + case -EDEADLK: + case -EINVAL: /* This indicates either state corruption or that the kernel + found a waiter on futex address which is waiting via + FUTEX_WAIT or FUTEX_WAIT_BITSET. This is reported on + some futex_lock_pi usage (pthread_mutex_timedlock for + instance). */ + return -err; + + case -EFAULT: /* Must have been caused by a glibc or application bug. */ + case -ENOSYS: /* Must have been caused by a glibc bug. */ + /* No other errors are documented at this time. */ + default: + futex_fatal_error (); + } +} diff --git a/nptl/pthread_mutex_lock.c b/nptl/pthread_mutex_lock.c index fbe8a3cd21..2bd41767e0 100644 --- a/nptl/pthread_mutex_lock.c +++ b/nptl/pthread_mutex_lock.c @@ -421,7 +421,8 @@ __pthread_mutex_lock_full (pthread_mutex_t *mutex) int private = (robust ? PTHREAD_ROBUST_MUTEX_PSHARED (mutex) : PTHREAD_MUTEX_PSHARED (mutex)); - int e = futex_lock_pi64 (&mutex->__data.__lock, NULL, private); + int e = __futex_lock_pi64 (&mutex->__data.__lock, 0 /* ununsed */, + NULL, private); if (e == ESRCH || e == EDEADLK) { assert (e != EDEADLK diff --git a/nptl/pthread_mutex_timedlock.c b/nptl/pthread_mutex_timedlock.c index bf2af6b048..a695faeb7e 100644 --- a/nptl/pthread_mutex_timedlock.c +++ b/nptl/pthread_mutex_timedlock.c @@ -369,7 +369,8 @@ __pthread_mutex_clocklock_common (pthread_mutex_t *mutex, int private = (robust ? PTHREAD_ROBUST_MUTEX_PSHARED (mutex) : PTHREAD_MUTEX_PSHARED (mutex)); - int e = futex_lock_pi64 (&mutex->__data.__lock, abstime, private); + int e = __futex_lock_pi64 (&mutex->__data.__lock, clockid, abstime, + private); if (e == ETIMEDOUT) return ETIMEDOUT; else if (e == ESRCH || e == EDEADLK) diff --git a/sysdeps/nptl/futex-internal.h b/sysdeps/nptl/futex-internal.h index 79a366604d..a46730cad6 100644 --- a/sysdeps/nptl/futex-internal.h +++ b/sysdeps/nptl/futex-internal.h @@ -236,8 +236,8 @@ futex_wake (unsigned int* futex_word, int processes_to_wake, int private) are done in descending priority order. The ABSTIME arguments provides an absolute timeout (measured against the - CLOCK_REALTIME clock). If TIMEOUT is NULL, the operation will block - indefinitely. + CLOCK_REALTIME or CLOCK_MONOTONIC clock). If TIMEOUT is NULL, the operation + will block indefinitely. Returns: @@ -250,58 +250,8 @@ futex_wake (unsigned int* futex_word, int processes_to_wake, int private) futex. - ETIMEDOUT if the ABSTIME expires. */ -static __always_inline int -futex_lock_pi64 (int *futex_word, const struct __timespec64 *abstime, - int private) -{ - int err; -#ifdef __ASSUME_TIME64_SYSCALLS - err = INTERNAL_SYSCALL_CALL (futex_time64, futex_word, - __lll_private_flag (FUTEX_LOCK_PI, private), 0, - abstime); -#else - bool need_time64 = abstime != NULL && !in_time_t_range (abstime->tv_sec); - if (need_time64) - { - err = INTERNAL_SYSCALL_CALL (futex_time64, futex_word, - __lll_private_flag (FUTEX_LOCK_PI, private), - 0, abstime); - if (err == -ENOSYS) - err = -EOVERFLOW; - } - else - { - struct timespec ts32; - if (abstime != NULL) - ts32 = valid_timespec64_to_timespec (*abstime); - - err = INTERNAL_SYSCALL_CALL (futex, futex_word, __lll_private_flag - (FUTEX_LOCK_PI, private), 0, - abstime != NULL ? &ts32 : NULL); - } -#endif - switch (err) - { - case 0: - case -EAGAIN: - case -EINTR: - case -ETIMEDOUT: - case -ESRCH: - case -EDEADLK: - case -EINVAL: /* This indicates either state corruption or that the kernel - found a waiter on futex address which is waiting via - FUTEX_WAIT or FUTEX_WAIT_BITSET. This is reported on - some futex_lock_pi usage (pthread_mutex_timedlock for - instance). */ - return -err; - - case -EFAULT: /* Must have been caused by a glibc or application bug. */ - case -ENOSYS: /* Must have been caused by a glibc bug. */ - /* No other errors are documented at this time. */ - default: - futex_fatal_error (); - } -} +int __futex_lock_pi64 (int *futex_word, clockid_t clockid, + const struct __timespec64 *abstime, int private); /* Wakes the top priority waiter that called a futex_lock_pi operation on the futex. diff --git a/sysdeps/nptl/lowlevellock-futex.h b/sysdeps/nptl/lowlevellock-futex.h index 66ebfe50f4..abda179e0d 100644 --- a/sysdeps/nptl/lowlevellock-futex.h +++ b/sysdeps/nptl/lowlevellock-futex.h @@ -38,6 +38,7 @@ #define FUTEX_WAKE_BITSET 10 #define FUTEX_WAIT_REQUEUE_PI 11 #define FUTEX_CMP_REQUEUE_PI 12 +#define FUTEX_LOCK_PI2 13 #define FUTEX_PRIVATE_FLAG 128 #define FUTEX_CLOCK_REALTIME 256