gdb: fix an issue with thread list corruption

  When resuming a target in non-stop mode with 'c -a', the continue command
uses for_each_thread() to proceed each stopped thread individually.  This
uses an all_threads_safe() iteration.

If one of the stopped threads does an inline step-over, since the target
is non-stop, we stop_all_threads(), which involves update_thread_list(),
which, in turn, may delete_thread().

If this deleted the thread pointed to by the m_next safe iterator member,
the above all_threads_safe() iteration will be corrupted.

The thread we're proceeding is stopped and there is no reason to delete
it.  Consequently, there is no reason for all_threads_safe(), which isn't
that safe in this scenario.

Iterate using all_threads() and inline proceed_thread_callback().
---
 gdb/infcmd.c | 52 ++++++++++++++++++++++++++--------------------------
 1 file changed, 26 insertions(+), 26 deletions(-)
  

On 5/4/26 3:16 AM, Markus Metzger wrote:
> When resuming a target in non-stop mode with 'c -a', the continue command
> uses for_each_thread() to proceed each stopped thread individually.  This
> uses an all_threads_safe() iteration.
> 
> If one of the stopped threads does an inline step-over, since the target
> is non-stop, we stop_all_threads(), which involves update_thread_list(),
> which, in turn, may delete_thread().
> 
> If this deleted the thread pointed to by the m_next safe iterator member,
> the above all_threads_safe() iteration will be corrupted.
> 
> The thread we're proceeding is stopped and there is no reason to delete
> it.  Consequently, there is no reason for all_threads_safe(), which isn't
> that safe in this scenario.
> 
> Iterate using all_threads() and inline proceed_thread_callback().

Just to try to make sure I understand the circumstances that lead the
the failure correcty (and make sure this doesn't just cover up other
problems):

 - thread 1 is stopped on a breakpoint
 - thread 2 is executing
 - displaced stepping is disabled
 - the target doesn't report thread events to the core
 - you "continue -a"
 - thread 1 starts an inline step-over, which calls stop_all_threads,
   which calls update_thread_list
 - meanwhile, thread 2 exits
 - update_thread_list causes thread 2 to be deleted
 - the safe iterator's m_next field now points to a deleted thread info

I thought: don't we now ask the target to report thread exit events when
doing steps now (to handle step over exit)?  But we only ask for to
report the exit event for the stepping thread, so the target wouldn't
report the exit of thread 2.

If my understand above is corerct, then I agree with your reasoning.
The currently iterated on thread is stopped and should not disappear,
except maybe on a misbehaving target (in which case we'd fix the
target).  Maybe it could happen with a remote target where communication
breaks during this update_thread_list, but that is notoriously difficult
to handle correctly.

> @@ -762,7 +739,30 @@ continue_1 (int all_threads)
>        scoped_disable_commit_resumed disable_commit_resumed
>  	("continue all threads in non-stop");
>  
> -      for_each_thread (proceed_thread_callback);
> +      /* Do not use all_threads_safe in case threads get removed while
> +	 resuming THREAD.  */

I understand this comment because I just read your commit message, but I
don't think I would understand it in isolation.  Because
all_threads_safe is usually used to handle cases where threads (the
current one) get removed while iterating.  I suggest:

      /* Do not use all_threads_safe, because it's possible for the next thread
         to get removed while resuming THREAD.  We know that thread is stopped
	 and should not disappear under our feet.  */

Then if I wanted to understand more about the circumstances where this
check was added, I would do a git blame and find the commit message.

Simon
  

Hello Simon,

>-----Original Message-----
>From: Simon Marchi <simark@simark.ca>
>Sent: Monday, May 4, 2026 5:30 PM
>To: Metzger, Markus T <markus.t.metzger@intel.com>; gdb-
>patches@sourceware.org
>Subject: Re: [PATCH] gdb: fix an issue with thread list corruption
>
>On 5/4/26 3:16 AM, Markus Metzger wrote:
>> When resuming a target in non-stop mode with 'c -a', the continue
>command
>> uses for_each_thread() to proceed each stopped thread individually.  This
>> uses an all_threads_safe() iteration.
>>
>> If one of the stopped threads does an inline step-over, since the target
>> is non-stop, we stop_all_threads(), which involves update_thread_list(),
>> which, in turn, may delete_thread().
>>
>> If this deleted the thread pointed to by the m_next safe iterator member,
>> the above all_threads_safe() iteration will be corrupted.
>>
>> The thread we're proceeding is stopped and there is no reason to delete
>> it.  Consequently, there is no reason for all_threads_safe(), which isn't
>> that safe in this scenario.
>>
>> Iterate using all_threads() and inline proceed_thread_callback().
>
>Just to try to make sure I understand the circumstances that lead the
>the failure correcty (and make sure this doesn't just cover up other
>problems):
>
> - thread 1 is stopped on a breakpoint
> - thread 2 is executing
> - displaced stepping is disabled
> - the target doesn't report thread events to the core
> - you "continue -a"
> - thread 1 starts an inline step-over, which calls stop_all_threads,
>   which calls update_thread_list
> - meanwhile, thread 2 exits
> - update_thread_list causes thread 2 to be deleted
> - the safe iterator's m_next field now points to a deleted thread info
>
>I thought: don't we now ask the target to report thread exit events when
>doing steps now (to handle step over exit)?  But we only ask for to
>report the exit event for the stepping thread, so the target wouldn't
>report the exit of thread 2.

I saw this with our new GPU target.  Even if we asked for exit events,
our target would only be able to provide them when single-stepping
(i.e. for thread 1) and not for running threads (i.e. for thread 2, assuming
GDB even asked for it).
I believe that AMD GPU targets suffer from the same limitation.

>If my understand above is corerct, then I agree with your reasoning.
>The currently iterated on thread is stopped and should not disappear,
>except maybe on a misbehaving target (in which case we'd fix the
>target).  Maybe it could happen with a remote target where communication
>breaks during this update_thread_list, but that is notoriously difficult
>to handle correctly.

Our GPU target is a non-stop remote target.  I wouldn't rule out that it still
misbehaves somewhat, but the scenario described above is the normal behavior.
We will be able to step over breakpoints inside the target in a future version, so
stopping all threads for an inline step-over will not be necessary anymore. But
I'd argue that this will simply hide the bug.

>
>> @@ -762,7 +739,30 @@ continue_1 (int all_threads)
>>        scoped_disable_commit_resumed disable_commit_resumed
>>  	("continue all threads in non-stop");
>>
>> -      for_each_thread (proceed_thread_callback);
>> +      /* Do not use all_threads_safe in case threads get removed while
>> +	 resuming THREAD.  */
>
>I understand this comment because I just read your commit message, but I
>don't think I would understand it in isolation.  Because
>all_threads_safe is usually used to handle cases where threads (the
>current one) get removed while iterating.  I suggest:
>
>      /* Do not use all_threads_safe, because it's possible for the next thread
>         to get removed while resuming THREAD.  We know that thread is
>stopped
>	 and should not disappear under our feet.  */
>
>Then if I wanted to understand more about the circumstances where this
>check was added, I would do a git blame and find the commit message.

I added your comment to v2.

Thanks,
Markus.

Intel Deutschland GmbH
Registered Address: Dornacher Strasse 1, 85622 Feldkirchen, Germany
Tel: +49 89 991 430, www.intel.de
Managing Directors: Harry Demas, Jeffrey Schneiderman, Yin Chong Sorrell
Chairperson of the Supervisory Board: Nicole Lau
Registered Seat: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928
  

On 2026-05-05 01:04, Metzger, Markus T wrote:
> Hello Simon,
> 
>> -----Original Message-----
>> From: Simon Marchi <simark@simark.ca>
>> Sent: Monday, May 4, 2026 5:30 PM
>> To: Metzger, Markus T <markus.t.metzger@intel.com>; gdb-
>> patches@sourceware.org
>> Subject: Re: [PATCH] gdb: fix an issue with thread list corruption
>>
>> On 5/4/26 3:16 AM, Markus Metzger wrote:
>>> When resuming a target in non-stop mode with 'c -a', the continue
>> command
>>> uses for_each_thread() to proceed each stopped thread individually.  This
>>> uses an all_threads_safe() iteration.
>>>
>>> If one of the stopped threads does an inline step-over, since the target
>>> is non-stop, we stop_all_threads(), which involves update_thread_list(),
>>> which, in turn, may delete_thread().
>>>
>>> If this deleted the thread pointed to by the m_next safe iterator member,
>>> the above all_threads_safe() iteration will be corrupted.
>>>
>>> The thread we're proceeding is stopped and there is no reason to delete
>>> it.  Consequently, there is no reason for all_threads_safe(), which isn't
>>> that safe in this scenario.
>>>
>>> Iterate using all_threads() and inline proceed_thread_callback().
>>
>> Just to try to make sure I understand the circumstances that lead the
>> the failure correcty (and make sure this doesn't just cover up other
>> problems):
>>
>> - thread 1 is stopped on a breakpoint
>> - thread 2 is executing
>> - displaced stepping is disabled
>> - the target doesn't report thread events to the core
>> - you "continue -a"
>> - thread 1 starts an inline step-over, which calls stop_all_threads,
>>   which calls update_thread_list
>> - meanwhile, thread 2 exits
>> - update_thread_list causes thread 2 to be deleted
>> - the safe iterator's m_next field now points to a deleted thread info

Can you comment on whether the sequence of events above sounds correct?
If it is, I think it would be possible to write a CPU test to reproduce
it, that would be helpful along with the fix.

Simon
  

Hello Simon,

>>> On 5/4/26 3:16 AM, Markus Metzger wrote:
>>>> When resuming a target in non-stop mode with 'c -a', the continue
>>> command
>>>> uses for_each_thread() to proceed each stopped thread individually.  This
>>>> uses an all_threads_safe() iteration.
>>>>
>>>> If one of the stopped threads does an inline step-over, since the target
>>>> is non-stop, we stop_all_threads(), which involves update_thread_list(),
>>>> which, in turn, may delete_thread().
>>>>
>>>> If this deleted the thread pointed to by the m_next safe iterator member,
>>>> the above all_threads_safe() iteration will be corrupted.
>>>>
>>>> The thread we're proceeding is stopped and there is no reason to delete
>>>> it.  Consequently, there is no reason for all_threads_safe(), which isn't
>>>> that safe in this scenario.
>>>>
>>>> Iterate using all_threads() and inline proceed_thread_callback().
>>>
>>> Just to try to make sure I understand the circumstances that lead the
>>> the failure correcty (and make sure this doesn't just cover up other
>>> problems):
>>>
>>> - thread 1 is stopped on a breakpoint
>>> - thread 2 is executing
>>> - displaced stepping is disabled
>>> - the target doesn't report thread events to the core
>>> - you "continue -a"
>>> - thread 1 starts an inline step-over, which calls stop_all_threads,
>>>   which calls update_thread_list
>>> - meanwhile, thread 2 exits
>>> - update_thread_list causes thread 2 to be deleted
>>> - the safe iterator's m_next field now points to a deleted thread info
>
>Can you comment on whether the sequence of events above sounds correct?
>If it is, I think it would be possible to write a CPU test to reproduce
>it, that would be helpful along with the fix.

The sequence sounds good, but I wonder how we would ensure that thread 2
exits just when we need it to.  This is non-stop mode, so we should learn about
the exit immediately.  The test would also need to rely on the ordering of threads
in the thread list.  And detecting the issue requires the next pointer of the next
element to be overwritten in a way that causes a deterministic effect - like crashing
GDB.  We wouldn't want to add another test that fails sporadically.

When looking at the safe-iterator code, it is obvious that it allows deleting the
current element and it does not allow deleting the next element.  So, nesting
safe-iterator iterations of the same list is a bad idea.

Regards,
Markus.
Intel Deutschland GmbH
Registered Address: Dornacher Strasse 1, 85622 Feldkirchen, Germany
Tel: +49 89 991 430, www.intel.de
Managing Directors: Harry Demas, Jeffrey Schneiderman, Yin Chong Sorrell
Chairperson of the Supervisory Board: Nicole Lau
Registered Seat: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928
  

On 2026-05-06 01:43, Metzger, Markus T wrote:
> Hello Simon,
> 
>>>> On 5/4/26 3:16 AM, Markus Metzger wrote:
>>>>> When resuming a target in non-stop mode with 'c -a', the continue
>>>> command
>>>>> uses for_each_thread() to proceed each stopped thread individually.  This
>>>>> uses an all_threads_safe() iteration.
>>>>>
>>>>> If one of the stopped threads does an inline step-over, since the target
>>>>> is non-stop, we stop_all_threads(), which involves update_thread_list(),
>>>>> which, in turn, may delete_thread().
>>>>>
>>>>> If this deleted the thread pointed to by the m_next safe iterator member,
>>>>> the above all_threads_safe() iteration will be corrupted.
>>>>>
>>>>> The thread we're proceeding is stopped and there is no reason to delete
>>>>> it.  Consequently, there is no reason for all_threads_safe(), which isn't
>>>>> that safe in this scenario.
>>>>>
>>>>> Iterate using all_threads() and inline proceed_thread_callback().
>>>>
>>>> Just to try to make sure I understand the circumstances that lead the
>>>> the failure correcty (and make sure this doesn't just cover up other
>>>> problems):
>>>>
>>>> - thread 1 is stopped on a breakpoint
>>>> - thread 2 is executing
>>>> - displaced stepping is disabled
>>>> - the target doesn't report thread events to the core
>>>> - you "continue -a"
>>>> - thread 1 starts an inline step-over, which calls stop_all_threads,
>>>>   which calls update_thread_list
>>>> - meanwhile, thread 2 exits
>>>> - update_thread_list causes thread 2 to be deleted
>>>> - the safe iterator's m_next field now points to a deleted thread info
>>
>> Can you comment on whether the sequence of events above sounds correct?
>> If it is, I think it would be possible to write a CPU test to reproduce
>> it, that would be helpful along with the fix.
> 
> The sequence sounds good, but I wonder how we would ensure that thread 2
> exits just when we need it to.

Usually, by hammering.  But I think it's not as difficult as it sounds.
The state before the "continue -a" on thread N is that GDB knows about
thread N+1.  Assuming that thread N+1 is a short-lived thread (one that
does nothing but exit), it exits even before the "continue -a" is
issued, but GDB doesn't know that until the next thread list update.
When you "continue -a", GDB iterates using all_threads_safe, without
doing any thread list update.  When iteration reaches thread N, the next
thread is N+1, exited on the target but it still exists in GDB's mind.
Thread N starts its inline step-over, and etc etc until the crash.

> This is non-stop mode, so we should learn about the exit immediately.
> The test would also need to rely on the ordering of threads in the
> thread list.  And detecting the issue requires the next pointer of the
> next element to be overwritten in a way that causes a deterministic
> effect - like crashing GDB.  We wouldn't want to add another test that
> fails sporadically.

I wrote a test that fails consistently (at least with ASan) with
the board native-extended-gdbserver:

https://review.lttng.org/c/binutils-gdb/+/17704

    (gdb) PASS: gdb.threads/continue-a-step-over-other-thread-exit.exp: step_over_thread_slot=1: continue to break_here
    continue -a
    Continuing.
    =================================================================
    ==1250831==ERROR: AddressSanitizer: heap-use-after-free on address 0x7d0984eb4480 at pc 0x564f8b3da4dd bp 0x7ffdad6a63d0 sp 0x7ffdad6a63c0

With your fixes, it passes.

Feel free to check it out and add it to your patch.  It will need a bit
of cleanup, and there is a FIXME saying that it won't work with
native-gdbserver because it requires inferior args.  I have another
patch series that I need to send that add the ability to pass inferior
args when using native-gdbserver, I'll try to get that done.

Simon
  

Hello Simon,

>>>>> On 5/4/26 3:16 AM, Markus Metzger wrote:
>>>>>> When resuming a target in non-stop mode with 'c -a', the continue
>>>>> command
>>>>>> uses for_each_thread() to proceed each stopped thread individually.
>This
>>>>>> uses an all_threads_safe() iteration.
>>>>>>
>>>>>> If one of the stopped threads does an inline step-over, since the target
>>>>>> is non-stop, we stop_all_threads(), which involves update_thread_list(),
>>>>>> which, in turn, may delete_thread().
>>>>>>
>>>>>> If this deleted the thread pointed to by the m_next safe iterator
>member,
>>>>>> the above all_threads_safe() iteration will be corrupted.
>>>>>>
>>>>>> The thread we're proceeding is stopped and there is no reason to
>delete
>>>>>> it.  Consequently, there is no reason for all_threads_safe(), which isn't
>>>>>> that safe in this scenario.
>>>>>>
>>>>>> Iterate using all_threads() and inline proceed_thread_callback().
>>>>>
>>>>> Just to try to make sure I understand the circumstances that lead the
>>>>> the failure correcty (and make sure this doesn't just cover up other
>>>>> problems):
>>>>>
>>>>> - thread 1 is stopped on a breakpoint
>>>>> - thread 2 is executing
>>>>> - displaced stepping is disabled
>>>>> - the target doesn't report thread events to the core
>>>>> - you "continue -a"
>>>>> - thread 1 starts an inline step-over, which calls stop_all_threads,
>>>>>   which calls update_thread_list
>>>>> - meanwhile, thread 2 exits
>>>>> - update_thread_list causes thread 2 to be deleted
>>>>> - the safe iterator's m_next field now points to a deleted thread info
>>>
>>> Can you comment on whether the sequence of events above sounds
>correct?
>>> If it is, I think it would be possible to write a CPU test to reproduce
>>> it, that would be helpful along with the fix.
>>
>> The sequence sounds good, but I wonder how we would ensure that thread
>2
>> exits just when we need it to.
>
>Usually, by hammering.  But I think it's not as difficult as it sounds.
>The state before the "continue -a" on thread N is that GDB knows about
>thread N+1.  Assuming that thread N+1 is a short-lived thread (one that
>does nothing but exit), it exits even before the "continue -a" is
>issued, but GDB doesn't know that until the next thread list update.
>When you "continue -a", GDB iterates using all_threads_safe, without
>doing any thread list update.  When iteration reaches thread N, the next
>thread is N+1, exited on the target but it still exists in GDB's mind.
>Thread N starts its inline step-over, and etc etc until the crash.
>
>> This is non-stop mode, so we should learn about the exit immediately.
>> The test would also need to rely on the ordering of threads in the
>> thread list.  And detecting the issue requires the next pointer of the
>> next element to be overwritten in a way that causes a deterministic
>> effect - like crashing GDB.  We wouldn't want to add another test that
>> fails sporadically.
>
>I wrote a test that fails consistently (at least with ASan) with
>the board native-extended-gdbserver:
>
>https://review.lttng.org/c/binutils-gdb/+/17704
>
>    (gdb) PASS: gdb.threads/continue-a-step-over-other-thread-exit.exp:
>step_over_thread_slot=1: continue to break_here
>    continue -a
>    Continuing.
>
>=================================================================
>    ==1250831==ERROR: AddressSanitizer: heap-use-after-free on address
>0x7d0984eb4480 at pc 0x564f8b3da4dd bp 0x7ffdad6a63d0 sp
>0x7ffdad6a63c0
>
>With your fixes, it passes.

And what happens without ASan?  Do we introduce yet another sporadically failing test?

If we only created two additional threads, we could vary which thread we're stepping
in two different runs of the test program, which makes the test program fairly trivial.

We'd need to step from one pthread_create() to the next to guarantee the order in
which GDB adds the threads.  As long as GDB adds them to either the front or the
back of the thread list consistently, we would get the same order in the two runs,
and by varying the thread we're stepping, we would cover both possible orders.

But we wouldn't be able to guarantee that the other thread exited at just the right
moment, would we?  We try to mitigate that by repeating this 100 times, but this is
just asking for the test to fail sporadically at a random iteration - or pass.

We have too many such tests already.  They may indicate an actual problem, or maybe
the test itself is broken or inherently non-deterministic.  What such tests definitely do is
create noise when trying to test a completely unrelated patch, thus slowing down GDB
development.

To get this deterministic, one would need to mock the remote target to guarantee
the ordering of thread exit and step completion events.  Something like gdbreplay,
but with hand-written logs, which would need to be maintained whenever anything
in the remote communication changes.  For more substantial changes to the remote
communication, this could become a significant burden.

And without additional tooling, like ASan, we still wouldn't be able to fail deterministically.

>Feel free to check it out and add it to your patch.  It will need a bit
>of cleanup, and there is a FIXME saying that it won't work with
>native-gdbserver because it requires inferior args.  I have another
>patch series that I need to send that add the ability to pass inferior
>args when using native-gdbserver, I'll try to get that done.
>
>Simon
Intel Deutschland GmbH
Registered Address: Dornacher Strasse 1, 85622 Feldkirchen, Germany
Tel: +49 89 991 430, www.intel.de
Managing Directors: Harry Demas, Jeffrey Schneiderman, Yin Chong Sorrell
Chairperson of the Supervisory Board: Nicole Lau
Registered Seat: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928
  

On 5/7/26 3:36 AM, Metzger, Markus T wrote:
> And what happens without ASan?  Do we introduce yet another sporadically failing test?

Perhaps, but such is the nature of "use after free" bugs, and that's the
point of ASan, to make them fail predictably.  Some people (including
me) test with ASan enabled, so someone would certainly catch it.  In
2026 we can assume that ASan is a core component of testing the
software.

If a test is failing sporadically because of a real bug in GDB, then the
test is doing its job.  If a test fails sporadically but not because of
a real bug in GDB, then it's a bad test.

> If we only created two additional threads, we could vary which thread we're stepping
> in two different runs of the test program, which makes the test program fairly trivial.
> 
> We'd need to step from one pthread_create() to the next to guarantee the order in
> which GDB adds the threads.  As long as GDB adds them to either the front or the
> back of the thread list consistently, we would get the same order in the two runs,
> and by varying the thread we're stepping, we would cover both possible orders.

Not sure I understand, but if you have an idea to improve what I
propose, go for it.

> But we wouldn't be able to guarantee that the other thread exited at just the right
> moment, would we?  We try to mitigate that by repeating this 100 times, but this is
> just asking for the test to fail sporadically at a random iteration - or pass.

Like I explained in my previous message, I think that with the test I
propose, the thread doesn't have to exit in a very tiny window for the
test to catch the failure.  That's because when threads exit on the
remote target, GDB doesn't learn about it until the next thread list
update.  So the thread has to exit after GDB learned about the thread
(which is at the previous breakpoint hit I think), and before the next
"continue -a".  It's a big window, so it's almost guaranteed that it
will happen, since the thread is short-lived.  In my testing, it always
caught it at iteration 1.

I think that the crash could be observed with the linux-nat target, but
then the window would be much smaller.  With the linux-nat target, the
thread_info is deleted as soon as the thread exits (and linux-nat
processes the event).  So for the crash to be observed, the thread would
have to exit between the moment the safe iterator grabs a reference to
the thread_info and the moment it tries to use it, that is a small
window.

> We have too many such tests already.  They may indicate an actual problem, or maybe
> the test itself is broken or inherently non-deterministic.  What such tests definitely do is
> create noise when trying to test a completely unrelated patch, thus slowing down GDB
> development.

With your fix, the test shouldn't randomly fail.  If it does, then it
means there is a problem with the test or with GDB, which should be
fixed either way.  The solution to this is not to not have tests.

> To get this deterministic, one would need to mock the remote target to guarantee
> the ordering of thread exit and step completion events.  Something like gdbreplay,
> but with hand-written logs, which would need to be maintained whenever anything
> in the remote communication changes.  For more substantial changes to the remote
> communication, this could become a significant burden.

If you wanted to write a test using gdbreplay, I wouldn't be against it,
but I also think that the maintenance burden probably makes it not worth
it.  I think that gdbreplay is useful to mock a misbehaving remote, for
example.

I think that "live" debugging test still has value, because it tests
against a real system.  For example, perhaps that running this new test
on Windows will point out that something is broken in the Windows target
or the Windows gdbserver, who knows.

> And without additional tooling, like ASan, we still wouldn't be able to fail deterministically.

So we shouldn't ever write tests for use-after-free or buffer overflow
bugs?

Simon
  

Hello Simon,

>On 5/7/26 3:36 AM, Metzger, Markus T wrote:
>> And what happens without ASan?  Do we introduce yet another sporadically
>failing test?
>
>Perhaps, but such is the nature of "use after free" bugs, and that's the
>point of ASan, to make them fail predictably.  Some people (including
>me) test with ASan enabled, so someone would certainly catch it.  In
>2026 we can assume that ASan is a core component of testing the
>software.
>
>If a test is failing sporadically because of a real bug in GDB, then the
>test is doing its job.  If a test fails sporadically but not because of
>a real bug in GDB, then it's a bad test.

And what am I supposed to do about such sporadically failing tests when
I want to test a patch series to ensure that I'm not introducing regressions?

I created two empty patches and tested upstream master (of a few days ago)
and my two empty patches, then diff'ed the corresponding gdb.sum files
against the respective parent commit [1].

The noise caused by sporadically failing tests makes it difficult to judge
whether a given patch is OK.  Even if a patch would trigger a new fail in a
test that sporadically passes, it would not be noticed in all that noise.

>> If we only created two additional threads, we could vary which thread we're
>stepping
>> in two different runs of the test program, which makes the test program
>fairly trivial.
>>
>> We'd need to step from one pthread_create() to the next to guarantee the
>order in
>> which GDB adds the threads.  As long as GDB adds them to either the front
>or the
>> back of the thread list consistently, we would get the same order in the two
>runs,
>> and by varying the thread we're stepping, we would cover both possible
>orders.
>
>Not sure I understand, but if you have an idea to improve what I
>propose, go for it.

I sketched a simpler version to emphasize where it breaks.

>> But we wouldn't be able to guarantee that the other thread exited at just the
>right
>> moment, would we?  We try to mitigate that by repeating this 100 times, but
>this is
>> just asking for the test to fail sporadically at a random iteration - or pass.
>
>Like I explained in my previous message, I think that with the test I
>propose, the thread doesn't have to exit in a very tiny window for the
>test to catch the failure.  That's because when threads exit on the
>remote target, GDB doesn't learn about it until the next thread list
>update.  So the thread has to exit after GDB learned about the thread
>(which is at the previous breakpoint hit I think), and before the next
>"continue -a".  It's a big window, so it's almost guaranteed that it
>will happen, since the thread is short-lived.  In my testing, it always
>caught it at iteration 1.
>
>I think that the crash could be observed with the linux-nat target, but
>then the window would be much smaller.  With the linux-nat target, the
>thread_info is deleted as soon as the thread exits (and linux-nat
>processes the event).  So for the crash to be observed, the thread would
>have to exit between the moment the safe iterator grabs a reference to
>the thread_info and the moment it tries to use it, that is a small
>window.
>
>> We have too many such tests already.  They may indicate an actual problem,
>or maybe
>> the test itself is broken or inherently non-deterministic.  What such tests
>definitely do is
>> create noise when trying to test a completely unrelated patch, thus slowing
>down GDB
>> development.
>
>With your fix, the test shouldn't randomly fail.  If it does, then it
>means there is a problem with the test or with GDB, which should be
>fixed either way.  The solution to this is not to not have tests.

It would randomly fail on native targets should my fix be reverted or otherwise
modified to not be effective anymore.  Of course, as long as the fix is active,
the test should pass consistently.  But, because the test is probabilistic and it
requires extra tooling, it may not fail on the patch that introduces the bug.
That's the problem I'm trying to point out.

>> To get this deterministic, one would need to mock the remote target to
>guarantee
>> the ordering of thread exit and step completion events.  Something like
>gdbreplay,
>> but with hand-written logs, which would need to be maintained whenever
>anything
>> in the remote communication changes.  For more substantial changes to the
>remote
>> communication, this could become a significant burden.
>
>If you wanted to write a test using gdbreplay, I wouldn't be against it,
>but I also think that the maintenance burden probably makes it not worth
>it.  I think that gdbreplay is useful to mock a misbehaving remote, for
>example.
>
>I think that "live" debugging test still has value, because it tests
>against a real system.  For example, perhaps that running this new test
>on Windows will point out that something is broken in the Windows target
>or the Windows gdbserver, who knows.
>
>> And without additional tooling, like ASan, we still wouldn't be able to fail
>deterministically.
>
>So we shouldn't ever write tests for use-after-free or buffer overflow
>bugs?

If the test requires special tooling to function, let's require that tooling and
skip the test if that tooling is not available.  Maybe we can use valgrind, which
could be run from the test and does not require GDB to be built in a certain
way (e.g. with sanitizers enabled).

The test should still pass/fail deterministically.

There are two aspects to this IMHO:
  #1 we want to find bugs in GDB so we can fix them
  #2 we want to ensure that new patches do not break things

For #1, we're happy about any issue we can detect.  Some bugs only show in
very specific scenarios and appear random until we understand what's wrong.

For #2, we want a clear yes/no answer: is my patch OK?

I'd argue that #2 is the main purpose of a regression test suite.  Bugs are
typically found during normal use.  Once the bug is understood, a test can
often be written to exercise exactly the bug condition.

In this case, we understand the exact scenario but creating it reliably in a test
gets very complicated.

The general issue is that safe-iterations should not be nested.  They should only
be used with the intent of deleting the current item.  A comment may be more
effective at preventing mis-use.

Both for_each_thread() and find_thread() use all_threads_safe().  They probably
shouldn't.  So, instead of fixing this one case by not using for_each_thread(), we
should probably change for_each_thread() to use all_threads() and find those few
cases, if any, that do use for_each_thread() to delete threads and change them
to explicitly use all_threads_safe().

Regards,
Markus.

[1]:
comparing 1fe09bfa16d29ca9897d863ff02992190c961ab6 to 237eed6d652aa1b341c913d37142d060eb9481c4...
..comparing gcc-g++-gfortran...
....comparing native...failed.
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tld: get valueof "ref_val_struct_static_04_01" (timeout)
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: gdb-command<p/d check_arg_struct_05_04 (ref_val_struct_05_04)> (timeout)
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: get valueof "ref_val_struct_05_04"
+FAIL: gdb.multi/multi-target-no-resumed.exp: inf_A=5: inf_B=2: load file in inferior 5 (timeout)
+FAIL: gdb.multi/multi-target-no-resumed.exp: inf_A=5: inf_B=2: set remote-exec file in inferior 5
+FAIL: gdb.reverse/insn-reverse.exp: rdseed: wait for prompt (timeout)
+FAIL: gdb.server/attach-flag.exp: non-stop=1: target-non-stop=1: thread 11
+FAIL: gdb.server/ext-attach.exp: target_async=on: target_non_stop=off: to_disable=T: attach to remote program 1 (timeout)
+FAIL: gdb.server/ext-attach.exp: target_async=on: target_non_stop=on: to_disable=Tthread: attach to remote program 1 (timeout)
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argc
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[1]
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[2]
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[3]
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: continue to breakpoint: main (the program is no longer running)
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: start gdbserver over stdin (timeout)
+FAIL: gdb.server/stop-reply-no-thread.exp: to_disable=T,qfThreadInfo: continue to main (timeout)
+FAIL: gdb.server/target-exec-file.exp: after starting inferior for the second time: python print(global_exec_changed_event)
+FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event)
+FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.progspace.executable_filename)
+FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.reload)
+FAIL: gdb.server/target-exec-file.exp: start inferior a second time: stopped at main (timeout)
+FAIL: gdb.server/target-exec-file.exp: start inferior a third time: check symbol re-read behavior
+FAIL: gdb.server/target-exec-file.exp: start inferior a third time: stopped at main
+FAIL: gdb.testsuite/gdb-caching-proc-consistency.exp: gnat_runtime_has_debug_info: 6: runto: run to GNAT_Debug_Info_Test (timeout)
+FAIL: gdb.threads/access-mem-running-thread-exit.exp: all-stop: access mem (write to global_var again, inf=2, iter=497) (timeout)
+FAIL: gdb.threads/attach-into-signal.exp: nonthreaded: attach (pass 1), pending signal catch
+FAIL: gdb.threads/attach-into-signal.exp: nonthreaded: attach (pass 1), pending signal catch (timeout)
+FAIL: gdb.threads/attach-many-short-lived-threads.exp: iter 1: attach (timeout)
+FAIL: gdb.threads/attach-many-short-lived-threads.exp: iter 1: no new threads
+FAIL: gdb.threads/attach-many-short-lived-threads.exp: iter 3: attach (timeout)
+FAIL: gdb.threads/attach-many-short-lived-threads.exp: iter 3: no new threads
+FAIL: gdb.threads/attach-non-stop.exp: target-non-stop=off: non-stop=off: cmd=attach: attach (timeout)
+FAIL: gdb.threads/check-libthread-db.exp: automated load-time check: libpthread.so fully initialized: check debug libthread-db output (pattern 1) (timeout)
+FAIL: gdb.threads/clone-attach-detach.exp: fg attach 1: attach (timeout)
+FAIL: gdb.threads/continue-pending-status.exp: attempt 2: caught interrupt (timeout)
+FAIL: gdb.threads/continue-pending-status.exp: attempt 8: caught interrupt (timeout)
+FAIL: gdb.threads/foll-fork-other-thread.exp: fork_func=fork: follow=child: target-non-stop=auto: non-stop=off: displaced-stepping=auto: runto: run to main (timeout)
+FAIL: gdb.threads/foll-fork-other-thread.exp: fork_func=vfork: follow=child: target-non-stop=auto: non-stop=off: displaced-stepping=off: runto: run to main (timeout)
+FAIL: gdb.threads/fork-thread-pending.exp: 1, followed to the child, found one thread
+FAIL: gdb.threads/fork-thread-pending.exp: 1, refused to resume (timeout)
+FAIL: gdb.threads/infcall-from-bp-cond-simple.exp: target_async=on: target_non_stop=off: no thread is hit: runto: run to main (timeout)
+FAIL: gdb.threads/infcall-from-bp-cond-timeout.exp: target_async=on: target_non_stop=off: non_stop=off: unwind=off: other_thread_bp=true: expected timeout waiting for inferior call to complete (timeout)
+FAIL: gdb.threads/main-thread-exit-during-detach.exp: spawn_inferior=true: python set_and_detach() (timeout)
+FAIL: gdb.threads/next-fork-exec-other-thread.exp: fork_func=fork: target-non-stop=auto: non-stop=off: displaced-stepping=off: runto: run to main (timeout)
+FAIL: gdb.threads/pending-fork-event-detach-ns.exp: file exists after detach
+FAIL: gdb.threads/pthreads.exp: after continue: continue until common routine run 15 times (timeout)
+FAIL: gdb.threads/pthreads.exp: after startup: continue until common routine run 15 times (timeout)
+FAIL: gdb.threads/staticthreads.exp: (timeout) rerun to main
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tld: check return value struct_static_04_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: check return value struct_05_04
....comparing native-gdbserver...failed.
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_01_03
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_02_04
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_04_02
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_06_03
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_06_04
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_01_04 (ref_val_struct_01_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_02_01 (ref_val_struct_02_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_02_02 (ref_val_struct_02_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_02_03 (ref_val_struct_02_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_02_04 (ref_val_struct_02_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_04_01 (ref_val_struct_04_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_04_02 (ref_val_struct_04_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_04_03 (ref_val_struct_04_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_04_04 (ref_val_struct_04_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_05_01 (ref_val_struct_05_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_05_02 (ref_val_struct_05_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_05_03 (ref_val_struct_05_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_05_04 (ref_val_struct_05_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_02_01 (ref_val_struct_static_02_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_02_02 (ref_val_struct_static_02_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_02_03 (ref_val_struct_static_02_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_02_04 (ref_val_struct_static_02_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_04_01 (ref_val_struct_static_04_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_04_02 (ref_val_struct_static_04_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_04_03 (ref_val_struct_static_04_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_04_04 (ref_val_struct_static_04_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_06_01 (ref_val_struct_static_06_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_06_02 (ref_val_struct_static_06_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_06_03 (ref_val_struct_static_06_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: gdb-command<p/d check_arg_struct_static_06_04 (ref_val_struct_static_06_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_01_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_02_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_02_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_02_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_04_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_04_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_04_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_04_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_05_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_05_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_05_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_05_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_static_02_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_static_02_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_static_02_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_static_02_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_static_04_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_static_04_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_static_04_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_static_06_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "ref_val_struct_static_06_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "rtn_str_struct_01_03 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "rtn_str_struct_02_04 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "rtn_str_struct_static_04_02 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "rtn_str_struct_static_06_03 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: get valueof "rtn_str_struct_static_06_04 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_01_04
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_04_04
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_02_01 (ref_val_struct_02_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_02_02 (ref_val_struct_02_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_02_03 (ref_val_struct_02_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_02_04 (ref_val_struct_02_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_04_01 (ref_val_struct_04_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_04_02 (ref_val_struct_04_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_04_03 (ref_val_struct_04_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_04_04 (ref_val_struct_04_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_05_01 (ref_val_struct_05_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_05_02 (ref_val_struct_05_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_05_03 (ref_val_struct_05_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_05_04 (ref_val_struct_05_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_02_01 (ref_val_struct_static_02_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_02_02 (ref_val_struct_static_02_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_02_03 (ref_val_struct_static_02_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_02_04 (ref_val_struct_static_02_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_04_01 (ref_val_struct_static_04_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_04_02 (ref_val_struct_static_04_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_04_03 (ref_val_struct_static_04_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_04_04 (ref_val_struct_static_04_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_06_01 (ref_val_struct_static_06_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_06_02 (ref_val_struct_static_06_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_06_03 (ref_val_struct_static_06_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: gdb-command<p/d check_arg_struct_static_06_04 (ref_val_struct_static_06_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_02_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_02_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_02_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_02_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_04_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_04_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_04_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_05_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_05_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_05_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_05_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_02_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_02_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_02_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_02_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_04_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_04_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_04_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_04_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_06_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_06_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_06_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "ref_val_struct_static_06_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "rtn_str_struct_01_04 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: get valueof "rtn_str_struct_04_04 ()"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_04_02
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_04_03 (ref_val_struct_04_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_04_04 (ref_val_struct_04_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_05_01 (ref_val_struct_05_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_05_02 (ref_val_struct_05_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_05_03 (ref_val_struct_05_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_05_04 (ref_val_struct_05_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_02_01 (ref_val_struct_static_02_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_02_02 (ref_val_struct_static_02_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_02_03 (ref_val_struct_static_02_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_02_04 (ref_val_struct_static_02_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_04_01 (ref_val_struct_static_04_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_04_02 (ref_val_struct_static_04_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_04_03 (ref_val_struct_static_04_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_04_04 (ref_val_struct_static_04_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_06_01 (ref_val_struct_static_06_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_06_02 (ref_val_struct_static_06_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_06_03 (ref_val_struct_static_06_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_static_06_04 (ref_val_struct_static_06_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_04_02"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_04_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_04_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_05_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_05_02"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_05_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_05_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_02_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_02_02"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_02_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_02_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_04_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_04_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_04_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_06_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_06_02"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_06_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_06_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "rtn_str_struct_static_04_02 ()"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_01_02
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_02_04
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_04_02
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_06_03
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_06_04
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_01_03 (ref_val_struct_01_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_01_04 (ref_val_struct_01_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_02_01 (ref_val_struct_02_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_02_02 (ref_val_struct_02_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_02_03 (ref_val_struct_02_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_02_04 (ref_val_struct_02_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_04_01 (ref_val_struct_04_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_04_02 (ref_val_struct_04_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_04_03 (ref_val_struct_04_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_04_04 (ref_val_struct_04_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_05_01 (ref_val_struct_05_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_05_02 (ref_val_struct_05_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_05_03 (ref_val_struct_05_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_05_04 (ref_val_struct_05_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_02_01 (ref_val_struct_static_02_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_02_02 (ref_val_struct_static_02_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_02_03 (ref_val_struct_static_02_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_02_04 (ref_val_struct_static_02_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_04_01 (ref_val_struct_static_04_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_04_02 (ref_val_struct_static_04_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_04_03 (ref_val_struct_static_04_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_04_04 (ref_val_struct_static_04_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_06_01 (ref_val_struct_static_06_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_06_02 (ref_val_struct_static_06_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_06_03 (ref_val_struct_static_06_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_06_04 (ref_val_struct_static_06_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_01_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_01_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_02_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_02_02"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_02_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_04_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_04_02"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_04_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_04_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_05_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_05_02"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_05_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_05_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_02_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_02_02"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_02_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_02_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_04_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_04_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_04_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_06_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_06_02"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "rtn_str_struct_01_02 ()"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "rtn_str_struct_02_04 ()"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "rtn_str_struct_static_04_02 ()"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "rtn_str_struct_static_06_03 ()"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "rtn_str_struct_static_06_04 ()"
-FAIL: gdb.threads/thread-specific.exp: continue to breakpoint: all threads started (the program is no longer running)
-FAIL: gdb.threads/thread-specific.exp: get threads list (no threads)
-FAIL: gdb.threads/thread-specific.exp: runto: run to main
+FAIL: gdb.threads/continue-pending-after-query.exp: iter 6: runto: run to main
+FAIL: gdb.threads/infcall-from-bp-cond-other-thread-event.exp: target_async=on: target_non_stop=on: other_thread_signal=true: stop_at_cond=true: hit the conditional breakpoint
+FAIL: gdb.threads/infcall-from-bp-cond-other-thread-event.exp: target_async=on: target_non_stop=on: other_thread_signal=true: stop_at_cond=true: received signal in other thread (timeout)
+FAIL: gdb.threads/multiple-successive-infcall.exp: thread=1: call inferior function
+FAIL: gdb.threads/multiple-successive-infcall.exp: thread=1: thread 1
+FAIL: gdb.threads/multiple-successive-infcall.exp: thread=2: call inferior function
+FAIL: gdb.threads/process-exit-status-is-leader-exit-status.exp: iteration=9: runto: run to main
+FAIL: gdb.threads/stepi-over-clone.exp: third_thread=false: non-stop=on: displaced=off: i=1: $thread_count == 2
+FAIL: gdb.threads/stepi-over-clone.exp: third_thread=false: non-stop=on: displaced=off: i=1: runto: run to main
+FAIL: gdb.threads/stepi-over-clone.exp: third_thread=false: non-stop=on: displaced=off: i=1: stepi (the program is no longer running)
+FAIL: gdb.threads/threadapply.exp: thread apply all: runto: run to main
-ERROR: GDB process no longer exists
-ERROR: GDB process no longer exists
-ERROR: GDB process no longer exists
+ERROR: : spawn id exp9 not open
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_01_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_02_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_02_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_02_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_04_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_04_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_04_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_04_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_05_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_05_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_05_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_05_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_02_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_02_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_02_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_02_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_04_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_04_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_04_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_06_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tll: check return value struct_static_06_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_02_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_02_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_02_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_02_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_04_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_04_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_04_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_05_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_05_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_05_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_05_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_02_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_02_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_02_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_02_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_04_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_04_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_04_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_04_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_06_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_06_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_06_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-ts: check return value struct_static_06_04
-UNRESOLVED: gdb.threads/infcall-from-bp-cond-other-thread-event.exp: target_async=on: target_non_stop=on: other_thread_signal=false: stop_at_cond=true: hit the breakpoint in other thread
-UNRESOLVED: gdb.threads/infcall-from-bp-cond-other-thread-event.exp: target_async=on: target_non_stop=on: other_thread_signal=false: stop_at_cond=true: hit the conditional breakpoint
-UNRESOLVED: gdb.threads/infcall-from-bp-cond-other-thread-event.exp: target_async=on: target_non_stop=on: other_thread_signal=false: stop_at_cond=true: hit the final breakpoint
-UNRESOLVED: gdb.threads/infcall-from-bp-cond-other-thread-event.exp: target_async=on: target_non_stop=on: other_thread_signal=true: stop_at_cond=true: hit the conditional breakpoint
-UNRESOLVED: gdb.threads/infcall-from-bp-cond-other-thread-event.exp: target_async=on: target_non_stop=on: other_thread_signal=true: stop_at_cond=true: received signal in other thread
-UNRESOLVED: gdb.threads/infcall-from-bp-cond-simple.exp: target_async=on: target_non_stop=on: exactly two threads are hit: spot all breakpoint hits
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_04_02
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_04_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_04_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_05_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_05_02
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_05_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_05_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_02_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_02_02
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_02_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_02_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_04_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_04_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_04_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_06_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_06_02
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_06_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_06_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_01_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_01_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_02_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_02_02
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_02_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_04_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_04_02
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_04_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_04_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_05_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_05_02
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_05_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_05_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_02_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_02_02
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_02_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_02_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_04_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_04_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_04_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_06_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_06_02
+UNRESOLVED: gdb.server/build-id-seqno.exp: find debuginfo, first build-id file is bad: stat_pkt=off: connect to gdbserver
....comparing native-extended-gdbserver...failed.
-FAIL: gdb.server/build-id-seqno.exp: find debuginfo, first build-id file is bad: stat_pkt=off: debuginfo was read from remote target
-FAIL: gdb.server/build-id-seqno.exp: find debuginfo, first build-id file is bad: stat_pkt=off: debuginfo was read via build-id
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argc
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[1]
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[2]
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[3]
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: continue to breakpoint: main
+FAIL: gdb.threads/attach-non-stop.exp: target-non-stop=off: non-stop=on: cmd=attach: attach (GDB internal error)
-FAIL: gdb.threads/break-while-running.exp: w/ithr: always-inserted on: non-stop: runto: run to main
+FAIL: gdb.threads/break-while-running.exp: w/ithr: always-inserted on: non-stop: runto: run to main (timeout)
-FAIL: gdb.threads/fork-plus-threads.exp: detach-on-fork=off: inferior 1 exited (timeout)
-FAIL: gdb.threads/fork-plus-threads.exp: detach-on-fork=off: only inferior 1 left
-FAIL: gdb.threads/fork-plus-threads.exp: detach-on-fork=off: seen all expected inferior exits
+FAIL: gdb.threads/continue-pending-after-query.exp: iter 10: gdb_breakpoint: set breakpoint at main
+FAIL: gdb.threads/continue-pending-after-query.exp: iter 9: runto: run to main
-FAIL: gdb.threads/hand-call-in-threads.exp: all dummies popped
+FAIL: gdb.threads/hand-call-in-threads.exp: runto: run to main
+FAIL: gdb.threads/hand-call-new-thread.exp: runto: run to main
+FAIL: gdb.threads/infcall-from-bp-cond-simple.exp: target_async=on: target_non_stop=off: exactly one thread is hit: runto: run to main
+FAIL: gdb.threads/non-ldr-exc-4.exp: nonstop=off: schedlock=on: runto: run to main
+FAIL: gdb.threads/pending-fork-event-detach.exp: target-non-stop=auto: who_forks=main: fork_function=fork: stop_mode=stepi: could not run to main
+FAIL: gdb.threads/pending-fork-event-detach.exp: target-non-stop=auto: who_forks=main: fork_function=fork: stop_mode=stepi: runto: run to main
+ERROR: Could not resync from internal error (eof)
-ERROR: : spawn id exp9 not open
-UNRESOLVED: gdb.server/build-id-seqno.exp: find debuginfo, first build-id file is bad: stat_pkt=off: connect to gdbserver
....comparing m32...failed.
+FAIL: gdb.server/ext-run.exp: clear_sysroot=false: set_remote_exec=true: fetch_exec_and_args=off: continue to main
+FAIL: gdb.server/ext-run.exp: clear_sysroot=false: set_remote_exec=true: fetch_exec_and_args=off: get process list
+FAIL: gdb.server/ext-run.exp: clear_sysroot=false: set_remote_exec=true: fetch_exec_and_args=off: load new file without any gdbserver inferior
+FAIL: gdb.server/ext-run.exp: clear_sysroot=false: set_remote_exec=true: fetch_exec_and_args=off: monitor exit
-FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event)
-FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.progspace.executable_filename)
-FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.reload)
-FAIL: gdb.server/target-exec-file.exp: start inferior a second time: stopped at main
-FAIL: gdb.server/target-exec-file.exp: start inferior a third time: start command
-FAIL: gdb.server/target-exec-file.exp: start inferior the first time: stopped at main
-FAIL: gdb.threads/next-fork-other-thread.exp: fork_func=fork: target-non-stop=off: non-stop=off: displaced-stepping=auto: i=10: next to break here
-FAIL: gdb.threads/next-fork-other-thread.exp: fork_func=fork: target-non-stop=off: non-stop=off: displaced-stepping=off: i=8: next to for loop
+FAIL: gdb.threads/next-fork-other-thread.exp: fork_func=fork: target-non-stop=off: non-stop=off: displaced-stepping=auto: i=0: next to for loop
+FAIL: gdb.threads/next-fork-other-thread.exp: fork_func=fork: target-non-stop=off: non-stop=off: displaced-stepping=off: i=9: next to other line
+ERROR: can not find channel named "exp9"
+ERROR: can not find channel named "exp9"
+ERROR: GDB process no longer exists
+UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the first time: python print(global_exec_changed_event)
+UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the second time: python print(global_exec_changed_event)
+UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the third time: python global_exec_changed_event = None
+UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event)
+UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.progspace.executable_filename)
+UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.reload)
+UNRESOLVED: gdb.server/target-exec-file.exp: start inferior a second time: start command
+UNRESOLVED: gdb.server/target-exec-file.exp: start inferior a third time: start command
+UNRESOLVED: gdb.server/target-exec-file.exp: start inferior the first time: stopped at main
comparing f3275943453538d68af3bb10af17b95c03dd9f0c to 1fe09bfa16d29ca9897d863ff02992190c961ab6...
..comparing gcc-g++-gfortran...
....comparing native...failed.
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tld: get valueof "ref_val_struct_static_04_01" (timeout)
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: gdb-command<p/d check_arg_struct_05_04 (ref_val_struct_05_04)> (timeout)
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: get valueof "ref_val_struct_05_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: check return value struct_static_04_01
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: gdb-command<p/d check_arg_struct_static_04_02 (ref_val_struct_static_04_02)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: gdb-command<p/d check_arg_struct_static_04_03 (ref_val_struct_static_04_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: gdb-command<p/d check_arg_struct_static_04_04 (ref_val_struct_static_04_04)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: gdb-command<p/d check_arg_struct_static_06_01 (ref_val_struct_static_06_01)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: get valueof "ref_val_struct_static_04_02"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: get valueof "ref_val_struct_static_04_03"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: get valueof "ref_val_struct_static_04_04"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: get valueof "ref_val_struct_static_06_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: get valueof "rtn_str_struct_static_04_01 ()" (timeout)
-FAIL: gdb.multi/multi-target-no-resumed.exp: inf_A=5: inf_B=2: load file in inferior 5 (timeout)
-FAIL: gdb.multi/multi-target-no-resumed.exp: inf_A=5: inf_B=2: set remote-exec file in inferior 5
+FAIL: gdb.multi/multi-target-no-resumed.exp: inf_A=5: inf_B=1: runto: run to all_started (timeout)
-FAIL: gdb.reverse/insn-reverse.exp: rdseed: wait for prompt (timeout)
+FAIL: gdb.python/lotsa-lines.exp: runto: run to callee (timeout)
+FAIL: gdb.reverse/insn-reverse.exp: -1: disassemble -1
+FAIL: gdb.reverse/insn-reverse.exp: -1: disassemble -1
+FAIL: gdb.reverse/insn-reverse.exp: -1: find the last instruction of function -1
+FAIL: gdb.reverse/insn-reverse.exp: -1: find the last instruction of function -1
+FAIL: gdb.reverse/insn-reverse.exp: -1: turn on process record
+FAIL: gdb.reverse/insn-reverse.exp: -1: turn on process record
+FAIL: gdb.reverse/insn-reverse.exp: -1: wait for prompt (got interactive prompt)
+FAIL: gdb.reverse/insn-reverse.exp: -1: x/i $pc
+FAIL: gdb.reverse/insn-reverse.exp: -1: x/i $pc
+FAIL: gdb.reverse/insn-reverse.exp: read name of test case 1 (timeout)
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argc
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[1]
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[2]
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[3]
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: continue to breakpoint: main (the program is no longer running)
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: start gdbserver over stdin (timeout)
-FAIL: gdb.server/stop-reply-no-thread.exp: to_disable=T,qfThreadInfo: continue to main (timeout)
-FAIL: gdb.server/target-exec-file.exp: after starting inferior for the second time: python print(global_exec_changed_event)
-FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event)
-FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.progspace.executable_filename)
-FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.reload)
-FAIL: gdb.server/target-exec-file.exp: start inferior a second time: stopped at main (timeout)
-FAIL: gdb.server/target-exec-file.exp: start inferior a third time: check symbol re-read behavior
-FAIL: gdb.server/target-exec-file.exp: start inferior a third time: stopped at main
-FAIL: gdb.testsuite/gdb-caching-proc-consistency.exp: gnat_runtime_has_debug_info: 6: runto: run to GNAT_Debug_Info_Test (timeout)
-FAIL: gdb.threads/access-mem-running-thread-exit.exp: all-stop: access mem (write to global_var again, inf=2, iter=497) (timeout)
+FAIL: gdb.server/ext-run.exp: clear_sysroot=true: set_remote_exec=true: fetch_exec_and_args=on: kill (timeout)
+FAIL: gdb.threads/access-mem-running-thread-exit.exp: all-stop: access mem (print global_var after writing, inf=2, iter=453) (timeout)
-FAIL: gdb.threads/attach-many-short-lived-threads.exp: iter 3: attach (timeout)
-FAIL: gdb.threads/attach-many-short-lived-threads.exp: iter 3: no new threads
+FAIL: gdb.threads/attach-non-stop.exp: target-non-stop=off: non-stop=on: cmd=attach: attach (timeout)
+FAIL: gdb.threads/break-while-running.exp: wo/ithr: always-inserted off: all-stop: runto: run to main (timeout)
-FAIL: gdb.threads/foll-fork-other-thread.exp: fork_func=fork: follow=child: target-non-stop=auto: non-stop=off: displaced-stepping=auto: runto: run to main (timeout)
-FAIL: gdb.threads/foll-fork-other-thread.exp: fork_func=vfork: follow=child: target-non-stop=auto: non-stop=off: displaced-stepping=off: runto: run to main (timeout)
-FAIL: gdb.threads/fork-thread-pending.exp: 1, followed to the child, found one thread
-FAIL: gdb.threads/fork-thread-pending.exp: 1, refused to resume (timeout)
-FAIL: gdb.threads/infcall-from-bp-cond-simple.exp: target_async=on: target_non_stop=off: no thread is hit: runto: run to main (timeout)
-FAIL: gdb.threads/infcall-from-bp-cond-timeout.exp: target_async=on: target_non_stop=off: non_stop=off: unwind=off: other_thread_bp=true: expected timeout waiting for inferior call to complete (timeout)
+FAIL: gdb.threads/create-fail.exp: iteration 4: runto: run to main (timeout)
+FAIL: gdb.threads/dlopen-libpthread.exp: gdb_breakpoint: set breakpoint at main (timeout)
+FAIL: gdb.threads/fork-thread-pending.exp: runto: run to main (timeout)
+FAIL: gdb.threads/infcall-from-bp-cond-timeout.exp: target_async=on: target_non_stop=off: non_stop=off: unwind=off: other_thread_bp=true: runto: run to main (timeout)
-FAIL: gdb.threads/next-fork-exec-other-thread.exp: fork_func=fork: target-non-stop=auto: non-stop=off: displaced-stepping=off: runto: run to main (timeout)
+FAIL: gdb.threads/multiple-step-overs.exp: displaced=on: step: setup: runto: run to main (timeout)
+FAIL: gdb.threads/multiple-step-overs.exp: displaced=on: step: step
+FAIL: gdb.threads/multiple-step-overs.exp: displaced=on: step: thread 1
-FAIL: gdb.threads/staticthreads.exp: (timeout) rerun to main
+FAIL: gdb.threads/schedlock-new-thread.exp: non-stop=off: schedlock=off: gdb_breakpoint: set breakpoint at thread_func (timeout)
+FAIL: gdb.threads/schedlock-new-thread.exp: non-stop=off: schedlock=off: set scheduler-locking off
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tld: check return value struct_static_04_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: check return value struct_05_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: check return value struct_static_04_02
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: check return value struct_static_04_03
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: check return value struct_static_04_04
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-tld-ti: check return value struct_static_06_01
+UNTESTED: gdb.multi/multi-target-no-resumed.exp: inf_A=5: inf_B=1: setup failed
....comparing native-gdbserver...failed.
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_04_02
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: gdb-command<p/d check_arg_struct_04_03 (ref_val_struct_04_03)>
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_04_01
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_06_02
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_06_03
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_04_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_04_01"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_04_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_06_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "ref_val_struct_static_06_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "rtn_str_struct_static_04_02 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_01_02
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_02_04
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_04_02
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_06_03
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_06_04
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_01_03 (ref_val_struct_01_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_01_04 (ref_val_struct_01_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_02_01 (ref_val_struct_02_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_02_02 (ref_val_struct_02_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_02_03 (ref_val_struct_02_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_02_04 (ref_val_struct_02_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_04_01 (ref_val_struct_04_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_04_02 (ref_val_struct_04_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_04_03 (ref_val_struct_04_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_04_04 (ref_val_struct_04_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_05_01 (ref_val_struct_05_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_05_02 (ref_val_struct_05_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_05_03 (ref_val_struct_05_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_05_04 (ref_val_struct_05_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_02_01 (ref_val_struct_static_02_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_02_02 (ref_val_struct_static_02_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_02_03 (ref_val_struct_static_02_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_02_04 (ref_val_struct_static_02_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_04_01 (ref_val_struct_static_04_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_04_02 (ref_val_struct_static_04_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_04_03 (ref_val_struct_static_04_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_04_04 (ref_val_struct_static_04_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_06_01 (ref_val_struct_static_06_01)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_06_02 (ref_val_struct_static_06_02)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_06_03 (ref_val_struct_static_06_03)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: gdb-command<p/d check_arg_struct_static_06_04 (ref_val_struct_static_06_04)>
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_01_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_01_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_02_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_02_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_02_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_04_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_04_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_04_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_04_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_05_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_05_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_05_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_05_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_02_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_02_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_02_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_02_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_04_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_04_03"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_04_04"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_06_01"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "ref_val_struct_static_06_02"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "rtn_str_struct_01_02 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "rtn_str_struct_02_04 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "rtn_str_struct_static_04_02 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "rtn_str_struct_static_06_03 ()"
-FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: get valueof "rtn_str_struct_static_06_04 ()"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "rtn_str_struct_static_04_01 ()"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "rtn_str_struct_static_06_02 ()"
+FAIL: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: get valueof "rtn_str_struct_static_06_03 ()"
-FAIL: gdb.threads/continue-pending-after-query.exp: iter 6: runto: run to main
-FAIL: gdb.threads/infcall-from-bp-cond-other-thread-event.exp: target_async=on: target_non_stop=on: other_thread_signal=true: stop_at_cond=true: hit the conditional breakpoint
-FAIL: gdb.threads/infcall-from-bp-cond-other-thread-event.exp: target_async=on: target_non_stop=on: other_thread_signal=true: stop_at_cond=true: received signal in other thread (timeout)
-FAIL: gdb.threads/multiple-successive-infcall.exp: thread=1: call inferior function
-FAIL: gdb.threads/multiple-successive-infcall.exp: thread=1: thread 1
-FAIL: gdb.threads/multiple-successive-infcall.exp: thread=2: call inferior function
-FAIL: gdb.threads/process-exit-status-is-leader-exit-status.exp: iteration=9: runto: run to main
-FAIL: gdb.threads/stepi-over-clone.exp: third_thread=false: non-stop=on: displaced=off: i=1: $thread_count == 2
-FAIL: gdb.threads/stepi-over-clone.exp: third_thread=false: non-stop=on: displaced=off: i=1: runto: run to main
-FAIL: gdb.threads/stepi-over-clone.exp: third_thread=false: non-stop=on: displaced=off: i=1: stepi (the program is no longer running)
-FAIL: gdb.threads/threadapply.exp: thread apply all: runto: run to main
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=off: escape_p=no: one empty: check argc
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=off: escape_p=no: one empty: check argv[1]
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=off: escape_p=no: one empty: check argv[2]
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=off: escape_p=no: one empty: check argv[3]
+FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=off: escape_p=no: one empty: continue to breakpoint: main (the program is no longer running)
+FAIL: gdb.threads/continue-pending-after-query.exp: iter 5: runto: run to main
+FAIL: gdb.threads/infcall-from-bp-cond-simple.exp: target_async=on: target_non_stop=off: all three threads are hit: runto: run to main
+FAIL: gdb.threads/multiple-successive-infcall.exp: runto: run to main
+FAIL: gdb.threads/process-exit-status-is-leader-exit-status.exp: iteration=6: runto: run to main
+FAIL: gdb.threads/step-over-thread-exit.exp: step_over_mode=inline: non-stop=on: target-non-stop=on: schedlock=off: cmd=next: ns_stop_all=0: selected thread didn't change (timeout)
+FAIL: gdb.threads/threadapply.exp: runto: run to main
+FAIL: gdb.threads/thread-execl.exp: schedlock off: runto: run to main
+FAIL: gdb.threads/tls-core.exp: gcore
+FAIL: gdb.threads/tls-core.exp: runto: run to thread_proc
-ERROR: : spawn id exp9 not open
+ERROR: GDB process no longer exists
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_04_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_04_01
+UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_04_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_06_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tc: check return value struct_static_06_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_01_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_01_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_02_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_02_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_02_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_04_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_04_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_04_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_04_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_05_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_05_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_05_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_05_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_02_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_02_02
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_02_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_02_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_04_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_04_03
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_04_04
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_06_01
-UNRESOLVED: gdb.base/infcall-nested-structs-c++.exp: types-td-tl: check return value struct_static_06_02
-UNRESOLVED: gdb.server/build-id-seqno.exp: find debuginfo, first build-id file is bad: stat_pkt=off: connect to gdbserver
+UNRESOLVED: gdb.threads/infcall-from-bp-cond-other-thread-event.exp: target_async=on: target_non_stop=on: other_thread_signal=true: stop_at_cond=false: received signal in other thread
+UNSUPPORTED: gdb.threads/tls-core.exp: gcore: load core file
+UNSUPPORTED: gdb.threads/tls-core.exp: gcore: print thread-local storage variable
....comparing native-extended-gdbserver...failed.
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argc
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[1]
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[2]
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: check argv[3]
-FAIL: gdb.server/inferior-args.exp: extended_p=no: startup_with_shell=on: escape_p=no: basic: continue to breakpoint: main
-FAIL: gdb.threads/break-while-running.exp: w/ithr: always-inserted on: non-stop: runto: run to main (timeout)
+FAIL: gdb.threads/break-while-running.exp: w/ithr: always-inserted on: non-stop: runto: run to main
-FAIL: gdb.threads/continue-pending-after-query.exp: iter 10: gdb_breakpoint: set breakpoint at main
-FAIL: gdb.threads/continue-pending-after-query.exp: iter 9: runto: run to main
-FAIL: gdb.threads/hand-call-in-threads.exp: runto: run to main
-FAIL: gdb.threads/hand-call-new-thread.exp: runto: run to main
-FAIL: gdb.threads/infcall-from-bp-cond-simple.exp: target_async=on: target_non_stop=off: exactly one thread is hit: runto: run to main
-FAIL: gdb.threads/non-ldr-exc-4.exp: nonstop=off: schedlock=on: runto: run to main
+FAIL: gdb.threads/next-fork-exec-other-thread.exp: fork_func=fork: target-non-stop=auto: non-stop=off: displaced-stepping=on: runto: run to main
-FAIL: gdb.threads/pending-fork-event-detach.exp: target-non-stop=auto: who_forks=main: fork_function=fork: stop_mode=stepi: could not run to main
-FAIL: gdb.threads/pending-fork-event-detach.exp: target-non-stop=auto: who_forks=main: fork_function=fork: stop_mode=stepi: runto: run to main
....comparing m32...failed.
+FAIL: gdb.multi/multi-target-no-resumed.exp: inf_A=5: inf_B=1: all threads stopped (timeout)
+FAIL: gdb.multi/multi-target-no-resumed.exp: inf_A=5: inf_B=1: send_gdb control C (timeout)
+FAIL: gdb.multi/multi-target-no-resumed.exp: inf_A=5: inf_B=2: inferior 2 (timeout)
+FAIL: gdb.multi/multi-target-no-resumed.exp: inf_A=5: inf_B=2: inferior 5 (timeout)
-FAIL: gdb.server/ext-run.exp: clear_sysroot=false: set_remote_exec=true: fetch_exec_and_args=off: continue to main
-FAIL: gdb.server/ext-run.exp: clear_sysroot=false: set_remote_exec=true: fetch_exec_and_args=off: get process list
-FAIL: gdb.server/ext-run.exp: clear_sysroot=false: set_remote_exec=true: fetch_exec_and_args=off: load new file without any gdbserver inferior
-FAIL: gdb.server/ext-run.exp: clear_sysroot=false: set_remote_exec=true: fetch_exec_and_args=off: monitor exit
-FAIL: gdb.threads/next-fork-other-thread.exp: fork_func=fork: target-non-stop=off: non-stop=off: displaced-stepping=auto: i=0: next to for loop
-FAIL: gdb.threads/next-fork-other-thread.exp: fork_func=fork: target-non-stop=off: non-stop=off: displaced-stepping=off: i=9: next to other line
-FAIL: gdb.threads/next-fork-other-thread.exp: fork_func=fork: target-non-stop=off: non-stop=off: displaced-stepping=on: i=0: next to for loop
+FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event)
+FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.progspace.executable_filename)
+FAIL: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.reload)
+FAIL: gdb.server/target-exec-file.exp: start inferior a second time: stopped at main
+FAIL: gdb.server/target-exec-file.exp: start inferior a third time: start command
+FAIL: gdb.server/target-exec-file.exp: start inferior the first time: stopped at main
+FAIL: gdb.threads/next-fork-other-thread.exp: fork_func=fork: target-non-stop=off: non-stop=off: displaced-stepping=auto: i=1: next to for loop
+FAIL: gdb.threads/next-fork-other-thread.exp: fork_func=fork: target-non-stop=off: non-stop=off: displaced-stepping=off: i=1: next to other line
+FAIL: gdb.threads/next-fork-other-thread.exp: fork_func=fork: target-non-stop=off: non-stop=off: displaced-stepping=on: i=3: next to other line
-ERROR: can not find channel named "exp9"
-ERROR: can not find channel named "exp9"
-ERROR: GDB process no longer exists
-UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the first time: python print(global_exec_changed_event)
-UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the second time: python print(global_exec_changed_event)
-UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the third time: python global_exec_changed_event = None
-UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event)
-UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.progspace.executable_filename)
-UNRESOLVED: gdb.server/target-exec-file.exp: after starting inferior for the third time: python print(global_exec_changed_event.reload)
-UNRESOLVED: gdb.server/target-exec-file.exp: start inferior a second time: start command
-UNRESOLVED: gdb.server/target-exec-file.exp: start inferior a third time: start command
-UNRESOLVED: gdb.server/target-exec-file.exp: start inferior the first time: stopped at main
Intel Deutschland GmbH
Registered Address: Dornacher Strasse 1, 85622 Feldkirchen, Germany
Tel: +49 89 991 430, www.intel.de
Managing Directors: Harry Demas, Jeffrey Schneiderman, Yin Chong Sorrell
Chairperson of the Supervisory Board: Nicole Lau
Registered Seat: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928
  

Hello Simon,

>The general issue is that safe-iterations should not be nested.  They should
>only
>be used with the intent of deleting the current item.  A comment may be more
>effective at preventing mis-use.
>
>Both for_each_thread() and find_thread() use all_threads_safe().  They
>probably
>shouldn't.  So, instead of fixing this one case by not using for_each_thread(),
>we
>should probably change for_each_thread() to use all_threads() and find those
>few
>cases, if any, that do use for_each_thread() to delete threads and change
>them
>to explicitly use all_threads_safe().

It turns out that there are only few calls to for_each_thread() and it looks like
one such call in mi/mi-main.c would run into a similar issue that this patch
fixes for infcmd.c.  I have not found a call that deletes the thread in the callback.

I'm testing a patch [1] that changes for_each_thread().

Regards,
markus.

[1]:
diff --git a/gdb/gdbthread.h b/gdb/gdbthread.h
index b3052b28d1a..1bdbf621982 100644
--- a/gdb/gdbthread.h
+++ b/gdb/gdbthread.h
@@ -796,7 +796,14 @@ void thread_change_ptid (process_stratum_target *targ,
 using for_each_thread_callback_ftype
   = gdb::function_view<void (thread_info *)>;
 
-/* Call CALLBACK once for each known thread.  */
+/* Call CALLBACK once for each known thread.
+
+   CALLBACK must not delete the thread.  To delete threads, use:
+
+     for (thread_info &t : all_threads_safe ())
+       if (some_condition ())
+        delete &t;
+*/
 
 extern void for_each_thread (for_each_thread_callback_ftype callback);
 
diff --git a/gdb/thread.c b/gdb/thread.c
index 4f62ca280cd..110e58a70aa 100644
--- a/gdb/thread.c
+++ b/gdb/thread.c
@@ -605,7 +605,7 @@ find_thread_by_handle (gdb::array_view<const gdb_byte> handle,
 void
 for_each_thread (for_each_thread_callback_ftype callback)
 {
-  for (thread_info &tp : all_threads_safe ())
+  for (thread_info &tp : all_threads ())
     callback (&tp);
 }

Intel Deutschland GmbH
Registered Address: Dornacher Strasse 1, 85622 Feldkirchen, Germany
Tel: +49 89 991 430, www.intel.de
Managing Directors: Harry Demas, Jeffrey Schneiderman, Yin Chong Sorrell
Chairperson of the Supervisory Board: Nicole Lau
Registered Seat: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928
  

>>The general issue is that safe-iterations should not be nested.  They should
>>only
>>be used with the intent of deleting the current item.  A comment may be
>more
>>effective at preventing mis-use.
>>
>>Both for_each_thread() and find_thread() use all_threads_safe().  They
>>probably
>>shouldn't.  So, instead of fixing this one case by not using for_each_thread(),
>>we
>>should probably change for_each_thread() to use all_threads() and find
>those
>>few
>>cases, if any, that do use for_each_thread() to delete threads and change
>>them
>>to explicitly use all_threads_safe().
>
>It turns out that there are only few calls to for_each_thread() and it looks like
>one such call in mi/mi-main.c would run into a similar issue that this patch
>fixes for infcmd.c.  I have not found a call that deletes the thread in the
>callback.
>
>I'm testing a patch [1] that changes for_each_thread().

Sent as https://sourceware.org/pipermail/gdb-patches/2026-May/227320.html.

This replaces this patch.

Regards,
Markus.
Intel Deutschland GmbH
Registered Address: Dornacher Strasse 1, 85622 Feldkirchen, Germany
Tel: +49 89 991 430, www.intel.de
Managing Directors: Harry Demas, Jeffrey Schneiderman, Yin Chong Sorrell
Chairperson of the Supervisory Board: Nicole Lau
Registered Seat: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928