[v3] nptl: Release robust mutex after trylock returns ENOTRECOVERABLE (BZ #27458)

  pthread_mutex_trylock on a not-recoverable robust non-PI mutex acquires
the lock word using a compare-and-exchange, then returns
ENOTRECOVERABLE without releasing it.  The CAS returns the old lock
word value, which is zero on success, so testing oldval against the
thread ID never unlocks the mutex.  A subsequent trylock returns EBUSY,
and a lock attempt can block indefinitely.

Release the lock when oldval is zero.  Add a regression test covering
normal, recursive, and error-checking robust mutexes with both
process-private and process-shared attributes, with and without PI.
Skip PI combinations on systems that do not support PI robust mutexes.

Tested on x86_64-linux-gnu (WSL2):
	nptl/tst-robust-trylock-notrecoverable (12 combinations, PASS).
	scripts/lint-makefiles.sh (188 Makefiles, no diagnostics).

Signed-off-by: Ryo Goto <ryou_gotou@aiphone.co.jp>
---
Copyright status: no FSF assignment; contributing under DCO.

Changes since v2:
- Sort tst-robust-trylock-notrecoverable before tst-robust12 in
  nptl/Makefile, which fixes the lint-makefiles regression reported
  by the pre-commit CI on patch 144821.

 nptl/Makefile                            |   1 +
 nptl/pthread_mutex_trylock.c             |   2 +-
 nptl/tst-robust-trylock-notrecoverable.c | 108 +++++++++++++++++++++++
 3 files changed, 110 insertions(+), 1 deletion(-)
 create mode 100644 nptl/tst-robust-trylock-notrecoverable.c
  

On 30/09/26 01:55, "後藤 遼(新製品開発3)" wrote:
> pthread_mutex_trylock on a not-recoverable robust non-PI mutex acquires
> the lock word using a compare-and-exchange, then returns
> ENOTRECOVERABLE without releasing it.  The CAS returns the old lock
> word value, which is zero on success, so testing oldval against the
> thread ID never unlocks the mutex.  A subsequent trylock returns EBUSY,
> and a lock attempt can block indefinitely.
> 
> Release the lock when oldval is zero.  Add a regression test covering
> normal, recursive, and error-checking robust mutexes with both
> process-private and process-shared attributes, with and without PI.
> Skip PI combinations on systems that do not support PI robust mutexes.
> 
> Tested on x86_64-linux-gnu (WSL2):
> 	nptl/tst-robust-trylock-notrecoverable (12 combinations, PASS).
> 	scripts/lint-makefiles.sh (188 Makefiles, no diagnostics).
> 
> Signed-off-by: Ryo Goto <ryou_gotou@aiphone.co.jp>

Hi Ryo, thanks for working on this.  The fix is correct, just minor nits
below.

With the nits fixed (I can fix them up before pushing if you prefer):

Reviewed-by: Adhemerval Zanella  <adhemerval.zanella@linaro.org>

> ---
> Copyright status: no FSF assignment; contributing under DCO.
> 
> Changes since v2:
> - Sort tst-robust-trylock-notrecoverable before tst-robust12 in
>   nptl/Makefile, which fixes the lint-makefiles regression reported
>   by the pre-commit CI on patch 144821.
> 
>  nptl/Makefile                            |   1 +
>  nptl/pthread_mutex_trylock.c             |   2 +-
>  nptl/tst-robust-trylock-notrecoverable.c | 108 +++++++++++++++++++++++
>  3 files changed, 110 insertions(+), 1 deletion(-)
>  create mode 100644 nptl/tst-robust-trylock-notrecoverable.c
> 
> diff --git a/nptl/Makefile b/nptl/Makefile
> index dfdfd261..ea1d337b 100644
> --- a/nptl/Makefile
> +++ b/nptl/Makefile
> @@ -328,6 +328,7 @@ tests = \
>    tst-pthread_exit-nothreads-static \
>    tst-pthread_gettid_np \
>    tst-robust-fork \
> +  tst-robust-trylock-notrecoverable \
>    tst-robust12 \
>    tst-robustpi1 \
>    tst-robustpi2 \
> diff --git a/nptl/pthread_mutex_trylock.c b/nptl/pthread_mutex_trylock.c
> index 15786fda..e1662d00 100644
> --- a/nptl/pthread_mutex_trylock.c
> +++ b/nptl/pthread_mutex_trylock.c
> @@ -170,7 +170,7 @@ ___pthread_mutex_trylock (pthread_mutex_t *mutex)
>  	    {
>  	      /* This mutex is now not recoverable.  */
>  	      mutex->__data.__count = 0;
> -	      if (oldval == id)
> +	      if (oldval == 0)
>  		lll_unlock (mutex->__data.__lock,
>  			    PTHREAD_ROBUST_MUTEX_PSHARED (mutex));
>  	      /* FIXME This violates the mutex destruction requirements.  See
> diff --git a/nptl/tst-robust-trylock-notrecoverable.c b/nptl/tst-robust-trylock-notrecoverable.c
> new file mode 100644
> index 00000000..f09c74bd
> --- /dev/null
> +++ b/nptl/tst-robust-trylock-notrecoverable.c
> @@ -0,0 +1,108 @@
> +/* Test that pthread_mutex_trylock releases an unrecoverable robust mutex.
> +   Copyright The GNU Toolchain Authors.
> +   This file is part of the GNU C Library.
> +
> +   The GNU C Library is free software; you can redistribute it and/or
> +   modify it under the terms of the GNU Lesser General Public
> +   License as published by the Free Software Foundation; either
> +   version 2.1 of the License, or (at your option) any later version.
> +
> +   The GNU C Library is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +   Lesser General Public License for more details.
> +
> +   You should have received a copy of the GNU Lesser General Public
> +   License along with the GNU C Library; if not, see
> +   <https://www.gnu.org/licenses/>.  */
> +
> +#include <errno.h>
> +#include <pthread.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <support/check.h>
> +#include <support/timespec.h>
> +#include <support/xthread.h>
> +#include <support/xtime.h>
> +
> +static pthread_mutex_t mutex;
> +
> +static void *
> +lock_and_exit (void *arg)
> +{
> +  (void) arg;

Not needed, glibc does not build with -Wunused-parameter.


> +  TEST_COMPARE (pthread_mutex_lock (&mutex), 0);
> +  return NULL;
> +}
> +
> +static void *
> +trylock_in_thread (void *arg)
> +{
> +  int *result = arg;
> +  *result = pthread_mutex_trylock (&mutex);
> +  return NULL;
> +}
> +
> +static void
> +run_test (bool pshared, bool pi, int type)
> +{
> +  printf ("info: pshared=%d pi=%d type=%d\n", pshared, pi, type);
> +
> +  pthread_mutexattr_t attr;
> +  xpthread_mutexattr_init (&attr);
> +  xpthread_mutexattr_setrobust (&attr, PTHREAD_MUTEX_ROBUST);
> +  xpthread_mutexattr_settype (&attr, type);
> +  if (pshared)
> +    xpthread_mutexattr_setpshared (&attr, PTHREAD_PROCESS_SHARED);
> +  if (pi)
> +    xpthread_mutexattr_setprotocol (&attr, PTHREAD_PRIO_INHERIT);
> +
> +  int init_result = pthread_mutex_init (&mutex, &attr);
> +  xpthread_mutexattr_destroy (&attr);
> +  if (pi && init_result == ENOTSUP)
> +    {
> +      puts ("info: PI robust mutexes not supported; skipping this combination");
> +      return;
> +    }
> +  TEST_COMPARE (init_result, 0);
> +  if (init_result != 0)
> +    return;
> +
> +  /* The owner dies, and its successor unlocks without making the mutex
> +     consistent.  All subsequent lock attempts must fail.  */
> +  xpthread_join (xpthread_create (NULL, lock_and_exit, NULL));
> +  TEST_COMPARE (pthread_mutex_lock (&mutex), EOWNERDEAD);
> +  TEST_COMPARE (pthread_mutex_unlock (&mutex), 0);
> +
> +  TEST_COMPARE (pthread_mutex_lock (&mutex), ENOTRECOVERABLE);
> +  TEST_COMPARE (pthread_mutex_trylock (&mutex), ENOTRECOVERABLE);
> +  /* With a recursive mutex, an incorrectly retained lock can return 0
> +     rather than EBUSY on a second trylock by the same thread.  */
> +  TEST_COMPARE (pthread_mutex_trylock (&mutex), ENOTRECOVERABLE);
> +
> +  int result = -1;
> +  xpthread_join (xpthread_create (NULL, trylock_in_thread, &result));
> +  TEST_COMPARE (result, ENOTRECOVERABLE);
> +
> +  /* A bounded lock attempt detects a lock word retained by trylock.  */
> +  struct timespec timeout = timespec_add (xclock_now (CLOCK_REALTIME),
> +					  make_timespec (1, 0));
> +  TEST_COMPARE (pthread_mutex_timedlock (&mutex, &timeout), ENOTRECOVERABLE);
> +  TEST_COMPARE (pthread_mutex_destroy (&mutex), 0);
> +}
> +
> +static int
> +do_test (void)
> +{
> +  static const int types[] = {
> +    PTHREAD_MUTEX_NORMAL, PTHREAD_MUTEX_RECURSIVE, PTHREAD_MUTEX_ERRORCHECK
> +  };
> +
> +  for (size_t index = 0; index < sizeof (types) / sizeof (types[0]); ++index)

Use array_length from <array_length.h>.


> +    for (int pshared = 0; pshared < 2; ++pshared)
> +      for (int pi = 0; pi < 2; ++pi)
> +	  run_test (pshared, pi, types[index]);
> +  return 0;
> +}
> +
> +#include <support/test-driver.c>
  

Hi Adhemerval,

Thanks for the review.  Please go ahead and fix up both nits
(drop the (void) arg cast and use array_length) before pushing.

Thanks,
Ryo

----- Original Message -----
Subject: Re: [PATCH v3] nptl: Release robust mutex after trylock returns ENOTRECOVERABLE (BZ #27458)
Date: Wed, 7 Oct 2026 10:47:30 -0300
From: Adhemerval Zanella Netto <adhemerval.zanella@linaro.org>
To: "後藤 遼(新製品開発3)"<ryou_gotou@aiphone.co.jp>, libc-alpha@sourceware.org
Adhemerval Zanella Netto wrote:
> 
> 
> On 30/09/26 01:55, "後藤 遼(新製品開発3)" wrote:
> > pthread_mutex_trylock on a not-recoverable robust non-PI mutex acquires
> > the lock word using a compare-and-exchange, then returns
> > ENOTRECOVERABLE without releasing it.  The CAS returns the old lock
> > word value, which is zero on success, so testing oldval against the
> > thread ID never unlocks the mutex.  A subsequent trylock returns EBUSY,
> > and a lock attempt can block indefinitely.
> > 
> > Release the lock when oldval is zero.  Add a regression test covering
> > normal, recursive, and error-checking robust mutexes with both
> > process-private and process-shared attributes, with and without PI.
> > Skip PI combinations on systems that do not support PI robust mutexes.
> > 
> > Tested on x86_64-linux-gnu (WSL2):
> > 	nptl/tst-robust-trylock-notrecoverable (12 combinations, PASS).
> > 	scripts/lint-makefiles.sh (188 Makefiles, no diagnostics).
> > 
> > Signed-off-by: Ryo Goto <ryou_gotou@aiphone.co.jp>
> 
> Hi Ryo, thanks for working on this.  The fix is correct, just minor nits
> below.
> 
> With the nits fixed (I can fix them up before pushing if you prefer):
> 
> Reviewed-by: Adhemerval Zanella  <adhemerval.zanella@linaro.org>
> 

---------------------------------------
新製品開発部　開発グループ３
後藤 遼（ゴトウ リョウ）
E-mail: ryou_gotou@aiphone.co.jp
---------------------------------------