Patchwork [v3,09/17] Teach non-stop to do in-line step-overs (stop all, step, restart)

login
register
mail settings
Submitter Pedro Alves
Date April 17, 2015, 10:45 a.m.
Message ID <1429267521-21047-10-git-send-email-palves@redhat.com>
Download mbox | patch
Permalink /patch/6292/
State New
Headers show

Comments

Pedro Alves - April 17, 2015, 10:45 a.m.
That is, step past breakpoints by:

 - pausing all threads
 - removing breakpoint at PC
 - single-step
 - reinsert breakpoint
 - restart threads

similarly to all-stop (with displaced stepping disabled).  This allows
non-stop to work on targets/architectures without displaced stepping
support.  That is, it makes displaced stepping an optimization instead
of a requirement.  For example, in principle, all GNU/Linux ports
support non-stop mode at the target_ops level, but not all
corresponding gdbarch's implement displaced stepping.  This should
make non-stop work for all (albeit, not as efficiently).  And then
there are scenarios where even if the architecture supports displaced
stepping, we can't use it, because we e.g., don't find a usable
address to use as displaced step scratch pad.  It should also fix
stepping past watchpoints on targets that have non-continuable
watchpoints in non-stop mode (e.g., PPC, untested).  Running the
instruction out of line in the displaced stepping scratch pad doesn't
help that case, as the copied instruction reads/writes the same
watched memory...  We can fix that too by teaching GDB to only remove
the watchpoint from the thread that we want to move past the
watchpoint (currently, removing a watchpoint always removes it from
all threads), but again, that can be considered an optimization; not
all targets would support it.

For those familiar with the gdb and gdbserver Linux target_ops
backends, the implementation should look similar, except it is done on
the core side.  When we pause threads, we may find they stop with an
interesting event that should be handled later when the thread is
re-resumed, thus we store such events in the thread object, and mark
the event as pending.  We should only consume pending events if the
thread is indeed resumed, thus we add a new "resumed" flag to the
thread object.  At a later stage, we might add new target methods to
accelerate some of this, like "pause all threads", with corresponding
RSP packets, but we'd still need a fallback method for remote targets
that don't support such packets, so, again, that can be deferred as
optimization.

My _real_ motivation here is making it possible to reimplement
all-stop mode on top of the target always working on non-stop mode, so
that e.g., we can send RSP packets to a remote target even while the
target is running -- can't do that in the all-stop RSP variant, by
design).

Tested on x86_64 Fedora 20, with and without "set displaced off"
forced.  The latter forces the new code paths whenever GDB needs to
step past a breakpoint.

gdb/ChangeLog:
2015-04-17  Pedro Alves  <pedro@codesourcery.com>

	* breakpoint.c (breakpoints_should_be_inserted_now): If any thread
	has a pending status, return true.
	* gdbthread.h: Include target/waitstatus.h.
	(struct thread_suspend_state) <stop_reason, waitstatus_pending_p,
	stop_pc>: New fields.
	(struct thread_info) <resumed>: New field.
	(set_resumed): Declare.
	* infrun.c: Include "event-loop.h".
	(infrun_async_inferior_event_token, infrun_is_async): New globals.
	(infrun_async): New function.
	(clear_step_over_info): Add debug output.
	(displaced_step_in_progress_any_inferior): New function.
	(displaced_step_fixup): New returns int.
	(start_step_over): Handle in-line step-overs too.  Assert the
	thread is marked resumed.
	(resume_cleanups): Clear the thread's resumed flag.
	(resume): Set the thread's resumed flag.  Return early if the
	thread has a pending status.  Allow stepping a breakpoint with no
	signal.
	(proceed): Adjust to check 'resumed' instead of 'executing'.
	(clear_proceed_status_thread): If the thread has a pending status,
	and that status is a finished step, discard the pending status.
	(clear_proceed_status): Don't clear step_over_info here.
	(random_pending_event_thread, do_target_wait): New functions.
	(prepare_for_detach, wait_for_inferior, fetch_inferior_event): Use
	do_target_wait.
	(wait_one): New function.
	(THREAD_STOPPED_BY): New macro.
	(thread_stopped_by_watchpoint, thread_stopped_by_sw_breakpoint)
	(thread_stopped_by_hw_breakpoint): New functions.
	(switch_to_thread_cleanup, save_waitstatus, stop_all_threads): New
	functions.
	(handle_inferior_event): Also call set_resumed(false) on all
	threads implicitly stopped by the event.
	(restart_threads, resumed_thread_with_pending_status): New
	functions.
	(finish_step_over): If we were doing an in-line step-over before,
	and no longer are after trying to start a new step-over, restart
	all threads.  If we have multiple threads with pending events,
	save the current event and go through the event loop again.
	(handle_signal_stop): Return early if finish_step_over returns
	false.
	<random signal>: Clear step_over_info before delivering
	the signal.
	(keep_going_stepped_thread): Use internal_error instead of
	gdb_assert.  Mark the thread as resumed.
	(keep_going_pass): Assert the thread isn't already resumed.  If
	some other thread is doing an in-line step-over, defer the resume.
	If we just started a new in-line step-over, stop all threads.
	Don't clear step_over_info.
	(infrun_async_inferior_event_handler): New function.
	(_initialize_infrun): Create async event handler with
	infrun_async_inferior_event_handler as callback.
	(infrun_async): New declaration.
	* target.c (target_async): New function.
	* target.h (target_async): Declare macro and readd as function
	declaration.
	* target/waitstatus.h (enum target_stop_reason)
	<TARGET_STOPPED_BY_SINGLE_STEP>: New value.
	* thread.c (new_thread): Clear the new waitstatus field.
	(set_resumed): New function.

v3:

  - pending status handing moved from keep_going_stepped_thread to
    restart_threads.  Fixes an assert triggered while testing with sss
    and no displaced stepping.

  - proceed and restart_threads now check "resumed" instead of
    "executing", because the thread that was started might have a
    pending status, and thus ends resumed, but not executing.  Caught
    by an assertion in keep_going_stepped_thread.

  - do_target_wait was missing a call to mark the infrun event source
    in case we are returning a pending wait status and there are other
    pending events to process.  Caught by testing with gdbserver + no
    displaced stepping + sss.

  - had finish_step_over detect that there are multiple pending events
    that could be consumed after an in-line step-over finishes, and
    pick one at random.  Fixes non-stop-fair-events.exp timeout due to
    event starvation on targets without displaced stepping.

  - start_step_over now waits for all displaced steps in progress
    before starting an in-line step over (avoid starvation/bouncing
    effects caused by pausing threads doing the displaced step, thus
    cancelling the displaced step, only to restart it again later).

  - resume now only sets the resume flag on success, instead of
    undoing it on error.  avoids the need for resume_cleanups change
    in v2.

  - Adjust to use the new thread_is_in_step_over_chain.

v2:

  - `proceed' changes dropped.  no longer needed after fix to previous patch.
  - "infrun.c" -> "infrun" in debug output.
  - clear trap_expected when displaced_step_fixup returns -1, when
    stopping threads (as the step-over was cancelled).  Testing on
    PPC64 caught this was missing ("inconsistent state" internal error
    in start_step_over_inferior).
---
 gdb/breakpoint.c        |    9 +
 gdb/gdbthread.h         |   31 ++
 gdb/infrun.c            | 1058 ++++++++++++++++++++++++++++++++++++++++++++---
 gdb/infrun.h            |    3 +
 gdb/target.c            |    9 +
 gdb/target.h            |    3 +-
 gdb/target/waitstatus.h |    5 +-
 gdb/thread.c            |   23 ++
 8 files changed, 1085 insertions(+), 56 deletions(-)
Pedro Alves - April 17, 2015, 11:01 a.m.
On 04/17/2015 11:45 AM, Pedro Alves wrote:
> make non-stop work for all (albeit, not as efficiently).  And then
> there are scenarios where even if the architecture supports displaced
> stepping, we can't use it, because we e.g., don't find a usable
> address to use as displaced step scratch pad.  It should also fix
> stepping past watchpoints on targets that have non-continuable
> watchpoints in non-stop mode (e.g., PPC, untested).

Reading back, I notice that wrote that "should fix" and "untested"
back before v1, and forgot to update it since.  This definitely
fixes that scenario.

Since:
 https://sourceware.org/ml/gdb-patches/2015-04/msg00384.html
 PPC64: Fix step-over-trips-on-watchpoint.exp with displaced stepping on

GDB now does not start a displaced step if it need to step
over a watchpoint, instead if steps the instruction at the
original address.  What was missing was pausing all threads
in order to be able to remove the watchpoint without other
threads missing it -- that's what's fixed by this patch.

> Running the
> instruction out of line in the displaced stepping scratch pad doesn't
> help that case, as the copied instruction reads/writes the same
> watched memory...  We can fix that too by teaching GDB to only remove
> the watchpoint from the thread that we want to move past the
> watchpoint (currently, removing a watchpoint always removes it from
> all threads), but again, that can be considered an optimization; not
> all targets would support it.

I'll update the commit log before pushing.

Thanks,
Pedro Alves
Yao Qi - April 21, 2015, 3:01 p.m.
Pedro Alves <palves@redhat.com> writes:

> -	 that adjust_pc_after_break doesn't end up confused.  */
> -      gdb_assert (sig != GDB_SIGNAL_0);
> +	 that adjust_pc_after_break doesn't end up confused.
> +
> +         - In non-stop if we insert a breakpoint (e.g., a step-resume)
> +	 in one thread after another thread that was stepping had been
> +	 momentarily paused for a step-over.  When we re-resume the
> +	 stepping thread, it may be resumed from that address with a
> +	 breakpoint that hasn't trapped yet.  Seen with
> +	 gdb.threads/non-stop-fair-events.exp, on targets that don't
> +	 do displaced stepping.  */

Does this comment mean the sequence like this?

 thread B is paused,
 a breakpoint is inserted at address PC for thread C,
 thread B may be resumed from address PC

>  
> +/* Select a thread at random, out of those which are resumed and have
> +   had events.  */
> +
> +static struct thread_info *
> +random_pending_event_thread (ptid_t waiton_ptid)
> +{

If GDB core knows select events at random, then we don't have to do
something similar in linux-nat.c:select_event_lwp, right?
Pedro Alves - April 22, 2015, 8:03 p.m.
On 04/21/2015 04:01 PM, Yao Qi wrote:
> Pedro Alves <palves@redhat.com> writes:
> 
>> -	 that adjust_pc_after_break doesn't end up confused.  */
>> -      gdb_assert (sig != GDB_SIGNAL_0);
>> +	 that adjust_pc_after_break doesn't end up confused.
>> +
>> +         - In non-stop if we insert a breakpoint (e.g., a step-resume)
>> +	 in one thread after another thread that was stepping had been
>> +	 momentarily paused for a step-over.  When we re-resume the
>> +	 stepping thread, it may be resumed from that address with a
>> +	 breakpoint that hasn't trapped yet.  Seen with
>> +	 gdb.threads/non-stop-fair-events.exp, on targets that don't
>> +	 do displaced stepping.  */
> 
> Does this comment mean the sequence like this?
> 
>  thread B is paused,
>  a breakpoint is inserted at address PC for thread C,
>  thread B may be resumed from address PC

Yep.  In that case, we should _not_ step over the breakpoint,
as thread B has not reported that breakpoint event.  Detecting the
case and reporting the breakpoint immediately without actually
resuming isn't correct, as in that case we'd lose the signal.
It's just simpler to let the breakpoint trap.

> 
>>  
>> +/* Select a thread at random, out of those which are resumed and have
>> +   had events.  */
>> +
>> +static struct thread_info *
>> +random_pending_event_thread (ptid_t waiton_ptid)
>> +{
> 
> If GDB core knows select events at random, then we don't have to do
> something similar in linux-nat.c:select_event_lwp, right?
> 

I think so, yes.  Though I'd rather leave it be for a while,
so that the "maint set target-non-stop off" fallback option
is fully functional as backup plan.  In the long term, if we manage
to merge linux-nat.c and gdbserver/linux-low.c, the resulting
code will still need it as long as we'd like to support gdbserver
connected to a gdb that doesn't force non-stop mode though, and,
probably as long as gdbserver's own step-over-breakpoints/tracepoints
implementation is baked in the backend too.  :-/

Thanks,
Pedro Alves
Yao Qi - April 24, 2015, 9:06 a.m.
Pedro Alves <palves@redhat.com> writes:

> I think so, yes.  Though I'd rather leave it be for a while,
> so that the "maint set target-non-stop off" fallback option
> is fully functional as backup plan.  In the long term, if we manage
> to merge linux-nat.c and gdbserver/linux-low.c, the resulting
> code will still need it as long as we'd like to support gdbserver
> connected to a gdb that doesn't force non-stop mode though, and,
> probably as long as gdbserver's own step-over-breakpoints/tracepoints
> implementation is baked in the backend too.  :-/

Hmm, that is OK.
Doug Evans - April 27, 2015, 7:27 p.m.
Pedro Alves writes:
 > ...
 > gdb/ChangeLog:
 > 2015-04-17  Pedro Alves  <pedro@codesourcery.com>
 > 
 > 	* breakpoint.c (breakpoints_should_be_inserted_now): If any thread
 > 	has a pending status, return true.
 > 	* gdbthread.h: Include target/waitstatus.h.
 > 	(struct thread_suspend_state) <stop_reason, waitstatus_pending_p,
 > 	stop_pc>: New fields.
 > 	(struct thread_info) <resumed>: New field.
 > 	(set_resumed): Declare.
 > 	* infrun.c: Include "event-loop.h".
 > 	(infrun_async_inferior_event_token, infrun_is_async): New globals.
 > 	(infrun_async): New function.
 > 	(clear_step_over_info): Add debug output.
 > 	(displaced_step_in_progress_any_inferior): New function.
 > 	(displaced_step_fixup): New returns int.
 > 	(start_step_over): Handle in-line step-overs too.  Assert the
 > 	thread is marked resumed.
 > 	(resume_cleanups): Clear the thread's resumed flag.
 > 	(resume): Set the thread's resumed flag.  Return early if the
 > 	thread has a pending status.  Allow stepping a breakpoint with no
 > 	signal.
 > 	(proceed): Adjust to check 'resumed' instead of 'executing'.
 > 	(clear_proceed_status_thread): If the thread has a pending status,
 > 	and that status is a finished step, discard the pending status.
 > 	(clear_proceed_status): Don't clear step_over_info here.
 > 	(random_pending_event_thread, do_target_wait): New functions.
 > 	(prepare_for_detach, wait_for_inferior, fetch_inferior_event): Use
 > 	do_target_wait.
 > 	(wait_one): New function.
 > 	(THREAD_STOPPED_BY): New macro.
 > 	(thread_stopped_by_watchpoint, thread_stopped_by_sw_breakpoint)
 > 	(thread_stopped_by_hw_breakpoint): New functions.
 > 	(switch_to_thread_cleanup, save_waitstatus, stop_all_threads): New
 > 	functions.
 > 	(handle_inferior_event): Also call set_resumed(false) on all
 > 	threads implicitly stopped by the event.
 > 	(restart_threads, resumed_thread_with_pending_status): New
 > 	functions.
 > 	(finish_step_over): If we were doing an in-line step-over before,
 > 	and no longer are after trying to start a new step-over, restart
 > 	all threads.  If we have multiple threads with pending events,
 > 	save the current event and go through the event loop again.
 > 	(handle_signal_stop): Return early if finish_step_over returns
 > 	false.
 > 	<random signal>: Clear step_over_info before delivering
 > 	the signal.
 > 	(keep_going_stepped_thread): Use internal_error instead of
 > 	gdb_assert.  Mark the thread as resumed.
 > 	(keep_going_pass): Assert the thread isn't already resumed.  If
 > 	some other thread is doing an in-line step-over, defer the resume.
 > 	If we just started a new in-line step-over, stop all threads.
 > 	Don't clear step_over_info.
 > 	(infrun_async_inferior_event_handler): New function.
 > 	(_initialize_infrun): Create async event handler with
 > 	infrun_async_inferior_event_handler as callback.
 > 	(infrun_async): New declaration.
 > 	* target.c (target_async): New function.
 > 	* target.h (target_async): Declare macro and readd as function
 > 	declaration.
 > 	* target/waitstatus.h (enum target_stop_reason)
 > 	<TARGET_STOPPED_BY_SINGLE_STEP>: New value.
 > 	* thread.c (new_thread): Clear the new waitstatus field.
 > 	(set_resumed): New function.

Hi.
A few comments inline.

 > ...
 > diff --git a/gdb/breakpoint.c b/gdb/breakpoint.c
 > index 31869f1..e23d223 100644
 > --- a/gdb/breakpoint.c
 > +++ b/gdb/breakpoint.c
 > @@ -468,6 +468,8 @@ breakpoints_should_be_inserted_now (void)
 >      }
 >    else if (target_has_execution)
 >      {
 > +      struct thread_info *tp;
 > +
 >        if (always_inserted_mode)
 >  	{
 >  	  /* The user wants breakpoints inserted even if all threads
 > @@ -477,6 +479,13 @@ breakpoints_should_be_inserted_now (void)
 >  
 >        if (threads_are_executing ())
 >  	return 1;

Not a problem introduced by this patch, but as an fyi, the terminology
employed here is a bit confusing.
Why would we want to insert breakpoints into executing threads,
or when threads are executing? That's what a simple reading of this
code says is happening. This reading can't be correct of course. :-)

 > +
 > +      /* Don't remove breakpoints yet if, even though all threads are
 > +	 stopped, we still have events to process.  */
 > +      ALL_NON_EXITED_THREADS (tp)
 > +	if (tp->resumed
 > +	    && tp->suspend.waitstatus_pending_p)
 > +	  return 1;

Plus, this function is named "breakpoints_should_be_inserted_now"
but the comment is talking about whether breakpoints should be removed.

Can you elaborate on how to interpret the name of this function?
Guessing at how I'm supposed to interpret what this function is for,
is a better name
"breakpoints_should_have_been_inserted_by_now_or_should_remain_inserted"?
[Not that that's my recommendation :-). Just trying to understand how
to read this function.]

 >      }
 >    return 0;
 >  }
 > diff --git a/gdb/gdbthread.h b/gdb/gdbthread.h
 > index 7052ee1..22c8eb2 100644
 > --- a/gdb/gdbthread.h
 > +++ b/gdb/gdbthread.h
 > @@ -29,6 +29,7 @@ struct symtab;
 >  #include "inferior.h"
 >  #include "btrace.h"
 >  #include "common/vec.h"
 > +#include "target/waitstatus.h"
 >  
 >  /* Frontend view of the thread state.  Possible extensions: stepping,
 >     finishing, until(ling),...  */
 > @@ -159,6 +160,23 @@ struct thread_suspend_state
 >       should be suppressed, the core will take care of clearing this
 >       before the target is resumed.  */
 >    enum gdb_signal stop_signal;
 > +
 > +  /* The reason the thread last stopped, if we need to track it
 > +     (breakpoint, watchpoint, etc.)  */
 > +  enum target_stop_reason stop_reason;
 > +
 > +  /* The waitstatus for this thread's last event.  */
 > +  struct target_waitstatus waitstatus;
 > +  /* If true WAITSTATUS hasn't been handled yet.  */
 > +  int waitstatus_pending_p;
 > +
 > +  /* Record the pc of the thread the last time it stopped.  (This is
 > +     not the current thread's PC as that may have changed since the
 > +     last stop, e.g., "return" command, or "p $pc = 0xf000").  This
 > +     used in coordination with stop_reason and waitstatus_pending_p:

s/used/is used/ ?

 > +     if the thread's PC is changed since it last stopped, a pending
 > +     breakpoint waitstatus is discarded.  */
 > +  CORE_ADDR stop_pc;
 >  };
 >  
 >  typedef struct value *value_ptr;
 > @@ -183,6 +201,14 @@ struct thread_info
 >       thread is off and running.  */
 >    int executing;
 >  
 > +  /* Non-zero if this thread will be/has been resumed.  Note that a
 > +     thread can be marked both as not-executing and resumed at the
 > +     same time.  This happens if we try to resume a thread that has a
 > +     wait status pending.  We shouldn't let the thread run until that
 > +     wait status has been processed, but we should not process that
 > +     wait status if we didn't try to let the thread run.  */
 > +  int resumed;

I suspect this will be another source of confusion, but I don't have
a good suggestion at the moment for how to improve it.
The "will be" in the comment speaks in future tense, but the name
"resumed" is past tense. Maybe (though I'd have to spend more time
reading the code to be sure) it would make sense to have this be
multi-state: not-resumed, to-be-resumed, and resumed; thus splitting up
"will be" resumed from "has been" resumed.

 > +
 >    /* Frontend view of the thread state.  Note that the THREAD_RUNNING/
 >       THREAD_STOPPED states are different from EXECUTING.  When the
 >       thread is stopped internally while handling an internal event,
 > @@ -400,6 +426,11 @@ extern int thread_count (void);
 >  /* Switch from one thread to another.  */
 >  extern void switch_to_thread (ptid_t ptid);
 >  
 > +/* Marks or clears thread(s) PTID as resumed.  If PTID is
 > +   MINUS_ONE_PTID, applies to all threads.  If ptid_is_pid(PTID) is
 > +   true, applies to all threads of the process pointed at by PTID.  */
 > +extern void set_resumed (ptid_t ptid, int resumed);
 > +
 >  /* Marks thread PTID is running, or stopped. 
 >     If PTID is minus_one_ptid, marks all threads.  */
 >  extern void set_running (ptid_t ptid, int running);
 > diff --git a/gdb/infrun.c b/gdb/infrun.c
 > index 6415fa2..8859b9f 100644
 > --- a/gdb/infrun.c
 > +++ b/gdb/infrun.c
 > @@ -60,6 +60,36 @@
 >  #include "target-descriptions.h"
 >  #include "target-dcache.h"
 >  #include "terminal.h"
 > +#include "event-loop.h"
 > +
 > +/* Asynchronous signal handler registered as event loop source for
 > +   when we have pending events ready to be passed to the core.  */
 > +static struct async_event_handler *infrun_async_inferior_event_token;
 > +
 > +/* Stores whether infrun_async was previously enabled or disabled.
 > +   Starts off as -1, indicating "never enabled/disabled".  */
 > +static int infrun_is_async = -1;
 > +
 > +/* See infrun.h.  */
 > +
 > +void
 > +infrun_async (int enable)
 > +{
 > +  if (infrun_is_async != enable)
 > +    {
 > +      infrun_is_async = enable;
 > +
 > +      if (debug_infrun)
 > +	fprintf_unfiltered (gdb_stdlog,
 > +			    "infrun: infrun_async(%d)\n",
 > +			    enable);
 > +
 > +      if (enable)
 > +	mark_async_event_handler (infrun_async_inferior_event_token);
 > +      else
 > +	clear_async_event_handler (infrun_async_inferior_event_token);
 > +    }
 > +}
 >  
 >  /* Prototypes for local functions */
 >  
 > @@ -1305,6 +1335,9 @@ set_step_over_info (struct address_space *aspace, CORE_ADDR address,
 >  static void
 >  clear_step_over_info (void)
 >  {
 > +  if (debug_infrun)
 > +    fprintf_unfiltered (gdb_stdlog,
 > +			"infrun: clear_step_over_info\n");
 >    step_over_info.aspace = NULL;
 >    step_over_info.address = 0;
 >    step_over_info.nonsteppable_watchpoint_p = 0;
 > @@ -1477,6 +1510,23 @@ get_displaced_stepping_state (int pid)
 >    return NULL;
 >  }
 >  
 > +/* Returns true if any inferior has a thread doing a displaced
 > +   step.  */
 > +
 > +static int
 > +displaced_step_in_progress_any_inferior (void)
 > +{
 > +  struct displaced_step_inferior_state *state;
 > +
 > +  for (state = displaced_step_inferior_states;
 > +       state != NULL;
 > +       state = state->next)
 > +    if (!ptid_equal (state->step_ptid, null_ptid))
 > +      return 1;
 > +
 > +  return 0;
 > +}
 > +
 >  /* Return true if process PID has a thread doing a displaced step.  */
 >  
 >  static int
 > @@ -1790,21 +1840,28 @@ displaced_step_restore (struct displaced_step_inferior_state *displaced,
 >  				  displaced->step_copy));
 >  }
 >  
 > -static void
 > +/* If we displaced stepped an instruction successfully, adjust
 > +   registers and memory to yield the same effect the instruction would
 > +   have had if we had executed it at its original address, and return
 > +   1.  If the instruction didn't complete, relocate the PC and return
 > +   -1.  If the thread wasn't displaced stepping, return 0.  */
 > +
 > +static int
 >  displaced_step_fixup (ptid_t event_ptid, enum gdb_signal signal)
 >  {
 >    struct cleanup *old_cleanups;
 >    struct displaced_step_inferior_state *displaced
 >      = get_displaced_stepping_state (ptid_get_pid (event_ptid));
 > +  int ret;
 >  
 >    /* Was any thread of this process doing a displaced step?  */
 >    if (displaced == NULL)
 > -    return;
 > +    return 0;
 >  
 >    /* Was this event for the pid we displaced?  */
 >    if (ptid_equal (displaced->step_ptid, null_ptid)
 >        || ! ptid_equal (displaced->step_ptid, event_ptid))
 > -    return;
 > +    return 0;
 >  
 >    old_cleanups = make_cleanup (displaced_step_clear_cleanup, displaced);
 >  
 > @@ -1827,6 +1884,7 @@ displaced_step_fixup (ptid_t event_ptid, enum gdb_signal signal)
 >                                      displaced->step_original,
 >                                      displaced->step_copy,
 >                                      get_thread_regcache (displaced->step_ptid));
 > +      ret = 1;
 >      }
 >    else
 >      {
 > @@ -1837,11 +1895,14 @@ displaced_step_fixup (ptid_t event_ptid, enum gdb_signal signal)
 >  
 >        pc = displaced->step_original + (pc - displaced->step_copy);
 >        regcache_write_pc (regcache, pc);
 > +      ret = -1;
 >      }
 >  
 >    do_cleanups (old_cleanups);
 >  
 >    displaced->step_ptid = null_ptid;
 > +
 > +  return ret;
 >  }
 >  
 >  /* Data to be passed around while handling an event.  This data is
 > @@ -1890,10 +1951,19 @@ start_step_over (void)
 >  {
 >    struct thread_info *tp, *next;
 >  
 > +  /* Don't start a new step-over if we already have a step-over
 > +     operation ongoing.  */
 > +  if (step_over_info_valid_p ())
 > +    return 0;
 > +
 >    for (tp = step_over_queue_head; tp != NULL; tp = next)
 >      {
 >        struct execution_control_state ecss;
 >        struct execution_control_state *ecs = &ecss;
 > +      enum step_over_what step_what;
 > +      int must_be_in_line;
 > +      struct regcache *regcache = get_thread_regcache (tp->ptid);
 > +      struct gdbarch *gdbarch = get_regcache_arch (regcache);
 >  
 >        next = thread_step_over_chain_next (tp);
 >  
 > @@ -1902,6 +1972,17 @@ start_step_over (void)
 >        if (displaced_step_in_progress (ptid_get_pid (tp->ptid)))
 >  	continue;
 >  
 > +      step_what = thread_still_needs_step_over (tp);
 > +      must_be_in_line = ((step_what & STEP_OVER_WATCHPOINT)
 > +			 || ((step_what & STEP_OVER_BREAKPOINT)
 > +			     && !use_displaced_stepping (gdbarch)));
 > +
 > +      /* We currently stop all threads of all processes to step-over
 > +	 in-line.  If we need to start a new in-line step-over, let
 > +	 any pending displace steps finish first.  */
 > +      if (must_be_in_line && displaced_step_in_progress_any_inferior ())
 > +	return 0;
 > +
 >        thread_step_over_chain_remove (tp);
 >  
 >        if (step_over_queue_head == NULL)
 > @@ -1911,13 +1992,16 @@ start_step_over (void)
 >  				"infrun: step-over queue now empty\n");
 >  	}
 >  
 > -      if (tp->control.trap_expected || tp->executing)
 > +      if (tp->control.trap_expected
 > +	  || tp->resumed
 > +	  || tp->executing)
 >  	{
 >  	  internal_error (__FILE__, __LINE__,
 >  			  "[%s] has inconsistent state: "
 > -			  "trap_expected=%d, executing=%d\n",
 > +			  "trap_expected=%d, resumed=%d, executing=%d\n",
 >  			  target_pid_to_str (tp->ptid),
 >  			  tp->control.trap_expected,
 > +			  tp->resumed,
 >  			  tp->executing);
 >  	}
 >  
 > @@ -1932,7 +2016,7 @@ start_step_over (void)
 >  	 wouldn't be able to resume anything else until the target
 >  	 stops again.  In non-stop, the resume always resumes only TP,
 >  	 so it's OK to let the thread resume freely.  */
 > -      if (!non_stop && !thread_still_needs_step_over (tp))
 > +      if (!non_stop && !step_what)
 >  	continue;
 >  
 >        switch_to_thread (tp->ptid);
 > @@ -1942,6 +2026,15 @@ start_step_over (void)
 >        if (!ecs->wait_some_more)
 >  	error (_("Command aborted."));
 >  
 > +      gdb_assert (tp->resumed);
 > +
 > +      /* If we started a new in-line step-over, we're done.  */
 > +      if (step_over_info_valid_p ())
 > +	{
 > +	  gdb_assert (tp->control.trap_expected);
 > +	  return 1;
 > +	}
 > +
 >        if (!non_stop)
 >  	{
 >  	  /* On all-stop, shouldn't have resumed unless we needed a
 > @@ -2157,12 +2250,38 @@ resume (enum gdb_signal sig)
 >       single-step).  */
 >    int step;
 >  
 > -  tp->stepped_breakpoint = 0;
 > -
 >    gdb_assert (!thread_is_in_step_over_chain (tp));
 >  
 >    QUIT;
 >  
 > +  if (tp->suspend.waitstatus_pending_p)
 > +    {
 > +      if (debug_infrun)
 > +	{
 > +	  char *statstr;
 > +
 > +	  statstr = target_waitstatus_to_string (&tp->suspend.waitstatus);
 > +	  fprintf_unfiltered (gdb_stdlog,
 > +			      "infrun: resume: thread %s has pending wait status %s "
 > +			      "(currently_stepping=%d).\n",
 > +			      target_pid_to_str (tp->ptid),  statstr,
 > +			      currently_stepping (tp));
 > +	  xfree (statstr);

Not something that has to be done with this patch of course,
but it's nice that we don't have to track the memory of target_pid_to_str;
IWBN to be able to do the same for target_waitstatus_to_string.
[In C++ it could just return a string, and we *could* just wait for C++.
Just a thought.]

 > +	}
 > +
 > +      tp->resumed = 1;
 > +      /* Avoid confusing the next resume, if the next stop/resume
 > +	 happens to apply to another thread.  */
 > +      tp->suspend.stop_signal = GDB_SIGNAL_0;

This is a bit confusing. How will the value of stop_signal in one
thread affect stop/resume in another thread?
Plus, the reader is left worried that we just clobbered the signal
that we need to resume thread tp with. Can you elaborate on what's
happening here?

 > +      discard_cleanups (old_cleanups);
 > +
 > +      if (target_can_async_p ())
 > +	target_async (1);
 > +      return;
 > +    }
 > +
 > +  tp->stepped_breakpoint = 0;
 > +
 >    /* Depends on stepped_breakpoint.  */
 >    step = currently_stepping (tp);
 >  
 > @@ -2267,6 +2386,7 @@ resume (enum gdb_signal sig)
 >  	      resume_ptid = user_visible_resume_ptid (user_step);
 >  	      do_target_resume (resume_ptid, 0, GDB_SIGNAL_0);
 >  	      discard_cleanups (old_cleanups);
 > +	      tp->resumed = 1;
 >  	      return;
 >  	    }
 >  	}
 > @@ -2392,11 +2512,12 @@ resume (enum gdb_signal sig)
 >    if (execution_direction != EXEC_REVERSE
 >        && step && breakpoint_inserted_here_p (aspace, pc))
 >      {
 > -      /* The only case we currently need to step a breakpoint
 > -	 instruction is when we have a signal to deliver.  See
 > -	 handle_signal_stop where we handle random signals that could
 > -	 take out us out of the stepping range.  Normally, in that
 > -	 case we end up continuing (instead of stepping) over the
 > +      /* There are two cases where we currently need to step a
 > +	 breakpoint instruction when we have a signal to deliver:
 > +
 > +	 - See handle_signal_stop where we handle random signals that
 > +	 could take out us out of the stepping range.  Normally, in
 > +	 that case we end up continuing (instead of stepping) over the
 >  	 signal handler with a breakpoint at PC, but there are cases
 >  	 where we should _always_ single-step, even if we have a
 >  	 step-resume breakpoint, like when a software watchpoint is
 > @@ -2409,8 +2530,20 @@ resume (enum gdb_signal sig)
 >  	 recurses and executes PC again, it'll miss the breakpoint.
 >  	 So we leave the breakpoint inserted anyway, but we need to
 >  	 record that we tried to step a breakpoint instruction, so
 > -	 that adjust_pc_after_break doesn't end up confused.  */
 > -      gdb_assert (sig != GDB_SIGNAL_0);
 > +	 that adjust_pc_after_break doesn't end up confused.
 > +
 > +         - In non-stop if we insert a breakpoint (e.g., a step-resume)
 > +	 in one thread after another thread that was stepping had been
 > +	 momentarily paused for a step-over.  When we re-resume the
 > +	 stepping thread, it may be resumed from that address with a
 > +	 breakpoint that hasn't trapped yet.  Seen with
 > +	 gdb.threads/non-stop-fair-events.exp, on targets that don't
 > +	 do displaced stepping.  */
 > +
 > +      if (debug_infrun)
 > +	fprintf_unfiltered (gdb_stdlog,
 > +			    "infrun: resume: [%s] stepped breakpoint\n",
 > +			    target_pid_to_str (tp->ptid));
 >  
 >        tp->stepped_breakpoint = 1;
 >  
 > @@ -2448,6 +2581,7 @@ resume (enum gdb_signal sig)
 >      }
 >  
 >    do_target_resume (resume_ptid, step, sig);
 > +  tp->resumed = 1;
 >    discard_cleanups (old_cleanups);
 >  }
 >  
 > @@ -2464,6 +2598,37 @@ clear_proceed_status_thread (struct thread_info *tp)
 >  			"infrun: clear_proceed_status_thread (%s)\n",
 >  			target_pid_to_str (tp->ptid));
 >  
 > +  /* If we're starting a new sequence, then the previous finished
 > +     single-step is no longer relevant.  */
 > +  if (tp->suspend.waitstatus_pending_p)
 > +    {
 > +      if (tp->suspend.stop_reason == TARGET_STOPPED_BY_SINGLE_STEP)
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: clear_proceed_status: pending "
 > +				"event of %s was a finished step. "
 > +				"Discarding.\n",
 > +				target_pid_to_str (tp->ptid));
 > +
 > +	  tp->suspend.waitstatus_pending_p = 0;
 > +	  tp->suspend.stop_reason = TARGET_STOPPED_BY_NO_REASON;
 > +	}
 > +      else if (debug_infrun)
 > +	{
 > +	  char *statstr;
 > +
 > +	  statstr = target_waitstatus_to_string (&tp->suspend.waitstatus);
 > +	  fprintf_unfiltered (gdb_stdlog,
 > +			      "infrun: clear_proceed_status_thread: thread %s "
 > +			      "has pending wait status %s "
 > +			      "(currently_stepping=%d).\n",
 > +			      target_pid_to_str (tp->ptid), statstr,
 > +			      currently_stepping (tp));
 > +	  xfree (statstr);
 > +	}
 > +    }
 > +
 >    /* If this signal should not be seen by program, give it zero.
 >       Used for debugging signals.  */
 >    if (!signal_pass_state (tp->suspend.stop_signal))
 > @@ -2527,8 +2692,6 @@ clear_proceed_status (int step)
 >  
 >    stop_after_trap = 0;
 >  
 > -  clear_step_over_info ();
 > -
 >    observer_notify_about_to_proceed ();
 >  
 >    if (stop_registers)
 > @@ -2777,7 +2940,7 @@ proceed (CORE_ADDR addr, enum gdb_signal siggnal)
 >        /* A new displaced stepping sequence was started.  In all-stop,
 >  	 we can't talk to the target anymore until it next stops.  */
 >      }
 > -  else if (!tp->executing && !thread_is_in_step_over_chain (tp))
 > +  else if (!tp->resumed && !thread_is_in_step_over_chain (tp))
 >      {
 >        /* The thread wasn't started, and isn't queued, run it now.  */
 >        reset_ecs (ecs, tp);
 > @@ -3076,6 +3239,179 @@ print_target_wait_results (ptid_t waiton_ptid, ptid_t result_ptid,
 >    ui_file_delete (tmp_stream);
 >  }
 >  
 > +/* Select a thread at random, out of those which are resumed and have
 > +   had events.  */
 > +
 > +static struct thread_info *
 > +random_pending_event_thread (ptid_t waiton_ptid)
 > +{
 > +  struct thread_info *event_tp;
 > +  int num_events = 0;
 > +  int random_selector;
 > +
 > +  /* First see how many events we have.  Count only resumed threads
 > +     that have an event pending.  */
 > +  ALL_NON_EXITED_THREADS (event_tp)
 > +    if (ptid_match (event_tp->ptid, waiton_ptid)
 > +	&& event_tp->resumed
 > +	&& event_tp->suspend.waitstatus_pending_p)
 > +      num_events++;
 > +
 > +  if (num_events == 0)
 > +    return NULL;
 > +
 > +  /* Now randomly pick a thread out of those that have had events.  */
 > +  random_selector = (int)
 > +    ((num_events * (double) rand ()) / (RAND_MAX + 1.0));
 > +
 > +  if (debug_infrun && num_events > 1)
 > +    fprintf_unfiltered (gdb_stdlog,
 > +			"infrun: Found %d events, selecting #%d\n",
 > +			num_events, random_selector);
 > +
 > +  /* Select the Nth thread that has had an event.  */
 > +  ALL_NON_EXITED_THREADS (event_tp)
 > +    if (ptid_match (event_tp->ptid, waiton_ptid)
 > +	&& event_tp->resumed
 > +	&& event_tp->suspend.waitstatus_pending_p)
 > +      if (random_selector-- == 0)
 > +	break;
 > +
 > +  return event_tp;
 > +}
 > +
 > +/* Wrapper for target_wait that first checks whether threads have
 > +   pending status to report before actually asking the target for more
 > +   events.  */
 > +
 > +static ptid_t
 > +do_target_wait (ptid_t ptid, struct target_waitstatus *status, int options)
 > +{
 > +  ptid_t event_ptid;
 > +  struct thread_info *tp;
 > +
 > +  /* First check if there is a resumed thread with a wait status
 > +     pending.  */
 > +  if (ptid_equal (ptid, minus_one_ptid) || ptid_is_pid (ptid))
 > +    {
 > +      tp = random_pending_event_thread (ptid);
 > +    }
 > +  else
 > +    {
 > +      if (debug_infrun)
 > +	fprintf_unfiltered (gdb_stdlog,
 > +			    "infrun: Waiting for specific thread %s.\n",
 > +			    target_pid_to_str (ptid));
 > +
 > +      /* We have a specific thread to check.  */
 > +      tp = find_thread_ptid (ptid);
 > +      gdb_assert (tp != NULL);
 > +      if (!tp->suspend.waitstatus_pending_p)
 > +	tp = NULL;
 > +    }
 > +
 > +  if (tp != NULL
 > +      && (tp->suspend.stop_reason == TARGET_STOPPED_BY_SW_BREAKPOINT
 > +	  || tp->suspend.stop_reason == TARGET_STOPPED_BY_HW_BREAKPOINT))
 > +    {
 > +      struct regcache *regcache = get_thread_regcache (tp->ptid);
 > +      struct gdbarch *gdbarch = get_regcache_arch (regcache);
 > +      CORE_ADDR pc;
 > +      int discard = 0;
 > +
 > +      pc = regcache_read_pc (regcache);
 > +
 > +      if (pc != tp->suspend.stop_pc)
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: PC of %s changed.  was=%s, now=%s\n",
 > +				target_pid_to_str (tp->ptid),
 > +				paddress (target_gdbarch (), tp->prev_pc),
 > +				paddress (target_gdbarch (), pc));

s/target_gdbarch ()/gdbarch/ ?

 > +	  discard = 1;
 > +	}
 > +      else if (!breakpoint_inserted_here_p (get_regcache_aspace (regcache), pc))
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: previous breakpoint of %s, at %s gone\n",
 > +				target_pid_to_str (tp->ptid),
 > +				paddress (target_gdbarch (), pc));
 > +
 > +	  discard = 1;
 > +	}
 > +
 > +      if (discard)
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: pending event of %s cancelled.\n",
 > +				target_pid_to_str (tp->ptid));
 > +
 > +	  tp->suspend.waitstatus.kind = TARGET_WAITKIND_SPURIOUS;
 > +	  tp->suspend.stop_reason = TARGET_STOPPED_BY_NO_REASON;
 > +	}
 > +    }
 > +
 > +  if (tp != NULL)
 > +    {
 > +      if (debug_infrun)
 > +	{
 > +	  char *statstr;
 > +
 > +	  statstr = target_waitstatus_to_string (&tp->suspend.waitstatus);
 > +	  fprintf_unfiltered (gdb_stdlog,
 > +			      "infrun: Using pending wait status %s for %s.\n",
 > +			      statstr,
 > +			      target_pid_to_str (tp->ptid));
 > +	  xfree (statstr);
 > +	}
 > +
 > +      /* Now that we've selected our final event LWP, un-adjust its PC
 > +	 if it was a software breakpoint (and the target doesn't
 > +	 always adjust the PC itself).  */
 > +      if (tp->suspend.stop_reason == TARGET_STOPPED_BY_SW_BREAKPOINT
 > +	  && !target_supports_stopped_by_sw_breakpoint ())
 > +	{
 > +	  struct regcache *regcache;
 > +	  struct gdbarch *gdbarch;
 > +	  int decr_pc;
 > +
 > +	  regcache = get_thread_regcache (tp->ptid);
 > +	  gdbarch = get_regcache_arch (regcache);
 > +
 > +	  decr_pc = gdbarch_decr_pc_after_break (gdbarch);
 > +	  if (decr_pc != 0)
 > +	    {
 > +	      CORE_ADDR pc;
 > +
 > +	      pc = regcache_read_pc (regcache);
 > +	      regcache_write_pc (regcache, pc + decr_pc);
 > +	    }
 > +	}
 > +
 > +      tp->suspend.stop_reason = TARGET_STOPPED_BY_NO_REASON;
 > +      *status = tp->suspend.waitstatus;
 > +      tp->suspend.waitstatus_pending_p = 0;
 > +
 > +      /* Wake up the event loop again, until all pending events are
 > +	 processed.  */
 > +      if (target_is_async_p ())
 > +	mark_async_event_handler (infrun_async_inferior_event_token);
 > +      return tp->ptid;
 > +    }
 > +
 > +  /* But if we don't find one, we'll have to wait.  */
 > +
 > +  if (deprecated_target_wait_hook)
 > +    event_ptid = deprecated_target_wait_hook (ptid, status, options);
 > +  else
 > +    event_ptid = target_wait (ptid, status, options);
 > +
 > +  return event_ptid;
 > +}
 > +
 >  /* Prepare and stabilize the inferior for detaching it.  E.g.,
 >     detaching while a thread is displaced stepping is a recipe for
 >     crashing it, as nothing would readjust the PC out of the scratch
 > @@ -3119,10 +3455,7 @@ prepare_for_detach (void)
 >  	 don't get any event.  */
 >        target_dcache_invalidate ();
 >  
 > -      if (deprecated_target_wait_hook)
 > -	ecs->ptid = deprecated_target_wait_hook (pid_ptid, &ecs->ws, 0);
 > -      else
 > -	ecs->ptid = target_wait (pid_ptid, &ecs->ws, 0);
 > +      ecs->ptid = do_target_wait (pid_ptid, &ecs->ws, 0);
 >  
 >        if (debug_infrun)
 >  	print_target_wait_results (pid_ptid, ecs->ptid, &ecs->ws);
 > @@ -3194,10 +3527,7 @@ wait_for_inferior (void)
 >  	 don't get any event.  */
 >        target_dcache_invalidate ();
 >  
 > -      if (deprecated_target_wait_hook)
 > -	ecs->ptid = deprecated_target_wait_hook (waiton_ptid, &ecs->ws, 0);
 > -      else
 > -	ecs->ptid = target_wait (waiton_ptid, &ecs->ws, 0);
 > +      ecs->ptid = do_target_wait (waiton_ptid, &ecs->ws, 0);
 >  
 >        if (debug_infrun)
 >  	print_target_wait_results (waiton_ptid, ecs->ptid, &ecs->ws);
 > @@ -3294,11 +3624,7 @@ fetch_inferior_event (void *client_data)
 >    make_cleanup_restore_integer (&execution_direction);
 >    execution_direction = target_execution_direction ();
 >  
 > -  if (deprecated_target_wait_hook)
 > -    ecs->ptid =
 > -      deprecated_target_wait_hook (waiton_ptid, &ecs->ws, TARGET_WNOHANG);
 > -  else
 > -    ecs->ptid = target_wait (waiton_ptid, &ecs->ws, TARGET_WNOHANG);
 > +  ecs->ptid = do_target_wait (waiton_ptid, &ecs->ws, TARGET_WNOHANG);
 >  
 >    if (debug_infrun)
 >      print_target_wait_results (waiton_ptid, ecs->ptid, &ecs->ws);
 > @@ -3671,6 +3997,349 @@ get_inferior_stop_soon (ptid_t ptid)
 >    return inf->control.stop_soon;
 >  }
 >  
 > +/* Wait for one event.  Store the resulting waitstatus in WS, and
 > +   return the event ptid.  */
 > +
 > +static ptid_t
 > +wait_one (struct target_waitstatus *ws)
 > +{
 > +  ptid_t event_ptid;
 > +  ptid_t wait_ptid = minus_one_ptid;
 > +
 > +  overlay_cache_invalid = 1;
 > +
 > +  /* Flush target cache before starting to handle each event.
 > +     Target was running and cache could be stale.  This is just a
 > +     heuristic.  Running threads may modify target memory, but we
 > +     don't get any event.  */
 > +  target_dcache_invalidate ();
 > +
 > +  if (deprecated_target_wait_hook)
 > +    event_ptid = deprecated_target_wait_hook (wait_ptid, ws, 0);
 > +  else
 > +    event_ptid = target_wait (wait_ptid, ws, 0);
 > +
 > +  if (debug_infrun)
 > +    print_target_wait_results (wait_ptid, event_ptid, ws);
 > +
 > +  if (ws->kind == TARGET_WAITKIND_SYSCALL_ENTRY
 > +      || ws->kind == TARGET_WAITKIND_SYSCALL_RETURN)
 > +    ws->value.syscall_number = UNKNOWN_SYSCALL;

IWBN to have a comment explaining why we set UNKNOWN_SYSCALL here.

 > +
 > +  return event_ptid;
 > +}
 > +
 > +/* Generate a wrapper for target_stopped_by_REASON that works on PTID
 > +   instead of the current thread.  */
 > +#define THREAD_STOPPED_BY(REASON)		\
 > +static int					\
 > +thread_stopped_by_ ## REASON (ptid_t ptid)	\
 > +{						\
 > +  struct cleanup *old_chain;			\
 > +  int res;					\
 > +						\
 > +  old_chain = save_inferior_ptid ();		\
 > +  inferior_ptid = ptid;				\
 > +						\
 > +  res = target_stopped_by_ ## REASON ();	\
 > +						\
 > +  do_cleanups (old_chain);			\
 > +						\
 > +  return res;					\
 > +}
 > +
 > +/* Generate thread_stopped_by_watchpoint.  */
 > +THREAD_STOPPED_BY (watchpoint)
 > +/* Generate thread_stopped_by_sw_breakpoint.  */
 > +THREAD_STOPPED_BY (sw_breakpoint)
 > +/* Generate thread_stopped_by_hw_breakpoint.  */
 > +THREAD_STOPPED_BY (hw_breakpoint)

People always nitpick when I do stuff like this. :-)
Don't change this on my account though.

 > +
 > +/* Cleanups that switches to the PTID pointed at by PTID_P.  */
 > +
 > +static void
 > +switch_to_thread_cleanup (void *ptid_p)
 > +{
 > +  ptid_t ptid = *(ptid_t *) ptid_p;
 > +
 > +  switch_to_thread (ptid);
 > +}
 > +
 > +/* Save the thread's event and stop reason to process it later.  */
 > +
 > +static void
 > +save_waitstatus (struct thread_info *tp, struct target_waitstatus *ws)
 > +{
 > +  struct regcache *regcache;
 > +  struct address_space *aspace;
 > +
 > +  if (debug_infrun)
 > +    {
 > +      char *statstr;
 > +
 > +      statstr = target_waitstatus_to_string (ws);
 > +      fprintf_unfiltered (gdb_stdlog,
 > +			  "infrun: saving status %s for %d.%ld.%ld\n",
 > +			  statstr,
 > +			  ptid_get_pid (tp->ptid),
 > +			  ptid_get_lwp (tp->ptid),
 > +			  ptid_get_tid (tp->ptid));
 > +      xfree (statstr);
 > +    }
 > +
 > +  /* Record for later.  */
 > +  tp->suspend.waitstatus = *ws;
 > +  tp->suspend.waitstatus_pending_p = 1;
 > +
 > +  regcache = get_thread_regcache (tp->ptid);
 > +  aspace = get_regcache_aspace (regcache);
 > +
 > +  if (ws->kind == TARGET_WAITKIND_STOPPED
 > +      && ws->value.sig == GDB_SIGNAL_TRAP)
 > +    {
 > +      CORE_ADDR pc = regcache_read_pc (regcache);
 > +
 > +      adjust_pc_after_break (tp, &tp->suspend.waitstatus);
 > +
 > +      if (thread_stopped_by_watchpoint (tp->ptid))
 > +	{
 > +	  tp->suspend.stop_reason
 > +	    = TARGET_STOPPED_BY_WATCHPOINT;
 > +	}
 > +      else if (target_supports_stopped_by_sw_breakpoint ()
 > +	       && thread_stopped_by_sw_breakpoint (tp->ptid))
 > +	{
 > +	  tp->suspend.stop_reason
 > +	    = TARGET_STOPPED_BY_SW_BREAKPOINT;
 > +	}
 > +      else if (target_supports_stopped_by_hw_breakpoint ()
 > +	       && thread_stopped_by_hw_breakpoint (tp->ptid))
 > +	{
 > +	  tp->suspend.stop_reason
 > +	    = TARGET_STOPPED_BY_HW_BREAKPOINT;
 > +	}
 > +      else if (!target_supports_stopped_by_hw_breakpoint ()
 > +	       && hardware_breakpoint_inserted_here_p (aspace,
 > +						       pc))
 > +	{
 > +	  tp->suspend.stop_reason
 > +	    = TARGET_STOPPED_BY_HW_BREAKPOINT;
 > +	}
 > +      else if (!target_supports_stopped_by_sw_breakpoint ()
 > +	       && software_breakpoint_inserted_here_p (aspace,
 > +						       pc))
 > +	{
 > +	  tp->suspend.stop_reason
 > +	    = TARGET_STOPPED_BY_SW_BREAKPOINT;
 > +	}
 > +      else if (!thread_has_single_step_breakpoints_set (tp)
 > +	       && currently_stepping (tp))
 > +	{
 > +	  tp->suspend.stop_reason
 > +	    = TARGET_STOPPED_BY_SINGLE_STEP;
 > +	}
 > +    }
 > +}
 > +
 > +/* Stop all threads.  */
 > +
 > +static void
 > +stop_all_threads (void)
 > +{
 > +  /* We may need multiple passes to discover all threads.  */
 > +  int pass;
 > +  int iterations = 0;
 > +  ptid_t entry_ptid;
 > +  struct cleanup *old_chain;
 > +
 > +  gdb_assert (non_stop);
 > +
 > +  if (debug_infrun)
 > +    fprintf_unfiltered (gdb_stdlog, "infrun: stop_all_threads\n");
 > +
 > +  entry_ptid = inferior_ptid;
 > +  old_chain = make_cleanup (switch_to_thread_cleanup, &entry_ptid);
 > +
 > +  /* Stop threads in two passes since threads could be spawning as we
 > +     go through the first pass.  In the second pass, we will stop such
 > +     spawned threads.  */
 > +  for (pass = 0; pass < 2; pass++, iterations++)

Can you rephrase the "Stop threads in two passes ... In the second pass ..."
comment? What's happening here is that we keep iterating until two passes
find no new threads (IIUC).

 > +    {
 > +      if (debug_infrun)
 > +	fprintf_unfiltered (gdb_stdlog,
 > +			    "infrun: stop_all_threads, pass=%d, "
 > +			    "iterations=%d\n", pass, iterations);
 > +      while (1)
 > +	{
 > +	  ptid_t event_ptid;
 > +	  struct target_waitstatus ws;
 > +	  int need_wait = 0;
 > +	  struct thread_info *t;
 > +
 > +	  update_thread_list ();
 > +
 > +	  /* Go through all threads looking for threads that we need
 > +	     to tell the target to stop.  */
 > +	  ALL_NON_EXITED_THREADS (t)
 > +	    {
 > +	      if (t->executing)
 > +		{
 > +		  /* If already stopping, don't request a stop again.
 > +		     We just haven't seen the notification yet.  */
 > +		  if (!t->stop_requested)
 > +		    {
 > +		      if (debug_infrun)
 > +			fprintf_unfiltered (gdb_stdlog,
 > +					    "infrun:   %s executing, "
 > +					    "need stop\n",
 > +					    target_pid_to_str (t->ptid));
 > +		      target_stop (t->ptid);
 > +		      t->stop_requested = 1;
 > +		    }
 > +		  else
 > +		    {
 > +		      if (debug_infrun)
 > +			fprintf_unfiltered (gdb_stdlog,
 > +					    "infrun:   %s executing, "
 > +					    "already stopping\n",
 > +					    target_pid_to_str (t->ptid));
 > +		    }
 > +
 > +		  if (t->stop_requested)
 > +		    need_wait = 1;
 > +		}
 > +	      else
 > +		{
 > +		  if (debug_infrun)
 > +		    fprintf_unfiltered (gdb_stdlog,
 > +					"infrun:   %s not executing\n",
 > +					target_pid_to_str (t->ptid));
 > +
 > +		  /* The thread may be not executing, but still be
 > +		     resumed with a pending status to process.  */
 > +		  t->resumed = 0;
 > +		}
 > +	    }
 > +
 > +	  if (!need_wait)
 > +	    break;
 > +
 > +	  /* If we find new threads on the second iteration, restart
 > +	     over.  We want to see two iterations in a row with all
 > +	     threads stopped.  */
 > +	  if (pass > 0)
 > +	    pass = -1;
 > +
 > +	  event_ptid = wait_one (&ws);
 > +	  if (ws.kind == TARGET_WAITKIND_NO_RESUMED)
 > +	    /* All resumed threads exited.  */
 > +	    ;
 > +	  else if (ws.kind == TARGET_WAITKIND_EXITED
 > +		   || ws.kind == TARGET_WAITKIND_SIGNALLED)
 > +	    {
 > +	      if (debug_infrun)
 > +		{
 > +		  ptid_t ptid = pid_to_ptid (ws.value.integer);
 > +
 > +		  fprintf_unfiltered (gdb_stdlog,
 > +				      "infrun: %s exited while "
 > +				      "stopping threads\n",
 > +				      target_pid_to_str (ptid));
 > +		}
 > +	    }
 > +	  else
 > +	    {
 > +	      if (!in_thread_list (event_ptid))
 > +		t = add_thread (event_ptid);
 > +	      else
 > +		t = find_thread_ptid (event_ptid);
 > +
 > +	      t->stop_requested = 0;
 > +	      t->executing = 0;
 > +	      t->resumed = 0;
 > +	      t->control.may_range_step = 0;
 > +
 > +	      if (ws.kind == TARGET_WAITKIND_STOPPED
 > +		  && ws.value.sig == GDB_SIGNAL_0)
 > +		{
 > +		  /* We caught the event that we intended to catch, so
 > +		     there's no event pending.  */
 > +		  t->suspend.waitstatus.kind = TARGET_WAITKIND_IGNORE;
 > +		  t->suspend.waitstatus_pending_p = 0;
 > +
 > +		  if (displaced_step_fixup (t->ptid, GDB_SIGNAL_0) < 0)
 > +		    {
 > +		      /* Add it back to the step-over queue.  */
 > +		      if (debug_infrun)
 > +			{
 > +			  fprintf_unfiltered (gdb_stdlog,
 > +					      "infrun: displaced-step of %s "
 > +					      "canceled: adding back to the "
 > +					      "step-over queue\n",
 > +					      target_pid_to_str (t->ptid));
 > +			}
 > +		      t->control.trap_expected = 0;
 > +		      thread_step_over_chain_enqueue (t);
 > +		    }
 > +		}
 > +	      else
 > +		{
 > +		  enum gdb_signal sig;
 > +		  struct regcache *regcache;
 > +		  struct address_space *aspace;
 > +
 > +		  if (debug_infrun)
 > +		    {
 > +		      char *statstr;
 > +
 > +		      statstr = target_waitstatus_to_string (&ws);
 > +		      fprintf_unfiltered (gdb_stdlog,
 > +					  "infrun: target_wait %s, saving "
 > +					  "status for %d.%ld.%ld\n",
 > +					  statstr,
 > +					  ptid_get_pid (t->ptid),
 > +					  ptid_get_lwp (t->ptid),
 > +					  ptid_get_tid (t->ptid));
 > +		      xfree (statstr);
 > +		    }
 > +
 > +		  /* Record for later.  */
 > +		  save_waitstatus (t, &ws);
 > +
 > +		  sig = (ws.kind == TARGET_WAITKIND_STOPPED
 > +			 ? ws.value.sig : GDB_SIGNAL_0);
 > +
 > +		  if (displaced_step_fixup (t->ptid, sig) < 0)
 > +		    {
 > +		      /* Add it back to the step-over queue.  */
 > +		      t->control.trap_expected = 0;
 > +		      thread_step_over_chain_enqueue (t);
 > +		    }
 > +
 > +		  regcache = get_thread_regcache (t->ptid);
 > +		  t->suspend.stop_pc = regcache_read_pc (regcache);
 > +
 > +		  if (debug_infrun)
 > +		    {
 > +		      fprintf_unfiltered (gdb_stdlog,
 > +					  "infrun: saved stop_pc=%s for %s "
 > +					  "(currently_stepping=%d)\n",
 > +					  paddress (target_gdbarch (),
 > +						    t->suspend.stop_pc),
 > +					  target_pid_to_str (t->ptid),
 > +					  currently_stepping (t));
 > +		    }
 > +		}
 > +	    }
 > +	}
 > +    }
 > +
 > +  do_cleanups (old_chain);
 > +
 > +  if (debug_infrun)
 > +    fprintf_unfiltered (gdb_stdlog, "infrun: stop_all_threads done\n");
 > +}
 > +
 >  /* Given an execution control state that has been freshly filled in by
 >     an event from the inferior, figure out what it means and take
 >     appropriate action.
 > @@ -3793,21 +4462,28 @@ handle_inferior_event (struct execution_control_state *ecs)
 >       we're handling a process exit in non-stop mode, there's nothing
 >       to do, as threads of the dead process are gone, and threads of
 >       any other process were left running.  */
 > -  if (!non_stop)
 > -    set_executing (minus_one_ptid, 0);
 > -  else if (ecs->ws.kind == TARGET_WAITKIND_SIGNALLED
 > -	   && ecs->ws.kind == TARGET_WAITKIND_EXITED)
 > -    {
 > -      ptid_t pid_ptid;
 >  
 > -      /* Some targets still have execution when a process exits.
 > -	 E.g., for "checkpoint", when when a fork exits and is
 > -	 mourned, linux-fork.c switches to another fork.  */
 > -      pid_ptid = pid_to_ptid (ptid_get_pid (ecs->ptid));
 > -      set_executing (pid_ptid, 0);
 > -    }
 > -  else
 > -    set_executing (ecs->ptid, 0);
 > +  {
 > +    ptid_t mark_ptid;
 > +
 > +    if (!non_stop)
 > +      mark_ptid = minus_one_ptid;
 > +    else if (ecs->ws.kind == TARGET_WAITKIND_SIGNALLED
 > +	     && ecs->ws.kind == TARGET_WAITKIND_EXITED)
 > +      {
 > +	/* Some targets still have execution when a process exits.
 > +	   E.g., for "checkpoint", when when a fork exits and is
 > +	   mourned, linux-fork.c switches to another fork.  */
 > +	mark_ptid = pid_to_ptid (ptid_get_pid (ecs->ptid));
 > +      }
 > +    else
 > +      mark_ptid = ecs->ptid;
 > +
 > +    set_executing (mark_ptid, 0);
 > +
 > +    /* Likewise the resumed flag.  */
 > +    set_resumed (mark_ptid, 0);
 > +  }
 >  
 >    switch (ecs->ws.kind)
 >      {
 > @@ -4220,16 +4896,137 @@ Cannot fill $_exitsignal with the correct signal number.\n"));
 >      }
 >  }
 >  
 > -/* Called when we get an event that may finish an in-line or
 > -   out-of-line (displaced stepping) step-over started previously.  */
 > +/* Restart threads back to what they were trying to do back when we
 > +   paused them for an in-line step-over.  The EVENT_THREAD thread is
 > +   ignored.  */
 >  
 >  static void
 > +restart_threads (struct thread_info *event_thread)
 > +{
 > +  struct thread_info *tp;
 > +  struct thread_info *step_over = NULL;
 > +
 > +  /* In case the instruction just stepped spawned a new thread.  */
 > +  update_thread_list ();
 > +
 > +  ALL_NON_EXITED_THREADS (tp)
 > +    {
 > +      struct execution_control_state ecss;
 > +      struct execution_control_state *ecs = &ecss;
 > +
 > +      if (tp == event_thread)
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: restart threads: "
 > +				"[%s] is event thread\n",
 > +				target_pid_to_str (tp->ptid));
 > +	  continue;
 > +	}
 > +
 > +      if (!(tp->state == THREAD_RUNNING || tp->control.in_infcall))
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: restart threads: "
 > +				"[%s] not meant to be running\n",
 > +				target_pid_to_str (tp->ptid));
 > +	  continue;
 > +	}
 > +
 > +      if (tp->resumed)
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: restart threads: [%s] resumed\n",
 > +				target_pid_to_str (tp->ptid));
 > +	  gdb_assert (tp->executing || tp->suspend.waitstatus_pending_p);
 > +	  continue;
 > +	}
 > +
 > +      if (thread_is_in_step_over_chain (tp))
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: restart threads: "
 > +				"[%s] needs step-over\n",
 > +				target_pid_to_str (tp->ptid));
 > +	  gdb_assert (!tp->resumed);
 > +	  continue;
 > +	}
 > +
 > +
 > +      if (tp->suspend.waitstatus_pending_p)
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: restart threads: "
 > +				"[%s] has pending status\n",
 > +				target_pid_to_str (tp->ptid));
 > +	  tp->resumed = 1;
 > +	  continue;
 > +	}
 > +
 > +      /* If some thread needs to start a step-over at this point, it
 > +	 should still be in the step-over queue, and thus skipped
 > +	 above.  */
 > +      if (thread_still_needs_step_over (tp))
 > +	{
 > +	  internal_error (__FILE__, __LINE__,
 > +			  "thread [%s] needs a step-over, but not in "
 > +			  "step-over queue\n",
 > +			  target_pid_to_str (tp->ptid));
 > +	}
 > +
 > +      if (currently_stepping (tp))
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: restart threads: [%s] was stepping\n",
 > +				target_pid_to_str (tp->ptid));
 > +	  keep_going_stepped_thread (tp);
 > +	}
 > +      else
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: restart threads: [%s] continuing\n",
 > +				target_pid_to_str (tp->ptid));
 > +	  reset_ecs (ecs, tp);
 > +	  switch_to_thread (tp->ptid);
 > +	  keep_going_pass (ecs);
 > +	}
 > +    }
 > +}
 > +
 > +/* Callback for iterate_over_threads.  Find a resumed thread that has
 > +   a pending waitstatus.  */
 > +
 > +static int
 > +resumed_thread_with_pending_status (struct thread_info *tp,
 > +				    void *arg)
 > +{
 > +  return (tp->resumed
 > +	  && tp->suspend.waitstatus_pending_p);
 > +}
 > +
 > +/* Called when we get an event that may finish an in-line or
 > +   out-of-line (displaced stepping) step-over started previously.
 > +   Return true if the event is processed and we should go back to the
 > +   event loop; false if the caller should continue processing the
 > +   event.  */
 > +
 > +static int
 >  finish_step_over (struct execution_control_state *ecs)
 >  {
 > +  int had_step_over_info;
 > +
 >    displaced_step_fixup (ecs->ptid,
 >  			ecs->event_thread->suspend.stop_signal);
 >  
 > -  if (step_over_info_valid_p ())
 > +  had_step_over_info = step_over_info_valid_p ();
 > +
 > +  if (had_step_over_info)
 >      {
 >        /* If we're stepping over a breakpoint with all threads locked,
 >  	 then only the thread that was stepped should be reporting
 > @@ -4241,11 +5038,99 @@ finish_step_over (struct execution_control_state *ecs)
 >      }
 >  
 >    if (!non_stop)
 > -    return;
 > +    return 0;
 >  
 >    /* Start a new step-over in another thread if there's one that
 >       needs it.  */
 >    start_step_over ();
 > +
 > +  /* If we were stepping over a breakpoint before, and haven't started
 > +     a new in-line step-over sequence, then restart all other threads
 > +     (except the event thread).  We can't do this in all-stop, as then
 > +     e.g., we wouldn't be able to issue any other remote packet until
 > +     these other threads stop.  */
 > +  if (had_step_over_info && !step_over_info_valid_p ())
 > +    {
 > +      struct thread_info *pending;
 > +
 > +      /* If we only have threads with pending statuses, the restart
 > +	 below won't restart any thread and so nothing re-inserts the
 > +	 breakpoint we just stepped over.  But we need it inserted
 > +	 when we later process the pending events, otherwise if
 > +	 another thread has a pending event for this breakpoint too,
 > +	 we'd discard its event (because the breakpoint that
 > +	 originally caused the event was no longer inserted).  */
 > +      context_switch (ecs->ptid);
 > +      insert_breakpoints ();
 > +
 > +      restart_threads (ecs->event_thread);
 > +
 > +      /* If we have events pending, go through handle_inferior_event
 > +	 again, picking up a pending event at random.  This avoids
 > +	 thread starvation.  */
 > +
 > +      /* But not if we just stepped over a watchpoint in order to let
 > +	 the instruction execute so we can evaluate its expression.
 > +	 The set of watchpoints that triggered is recorded in the
 > +	 breakpoint objects themselves (see bp->watchpoint_triggered).
 > +	 If we processed another event first, that other event could
 > +	 clobber this info.  */
 > +      if (ecs->event_thread->stepping_over_watchpoint)
 > +	return 0;
 > +
 > +      pending = iterate_over_threads (resumed_thread_with_pending_status,
 > +				      NULL);
 > +      if (pending != NULL)
 > +	{
 > +	  struct thread_info *tp = ecs->event_thread;
 > +	  struct regcache *regcache;
 > +
 > +	  if (debug_infrun)
 > +	    {
 > +	      fprintf_unfiltered (gdb_stdlog,
 > +				  "infrun: found resumed threads with "
 > +				  "pending events, saving status\n");
 > +	    }
 > +
 > +	  gdb_assert (pending != tp);
 > +
 > +	  /* Record the event thread's event for later.  */
 > +	  save_waitstatus (tp, &ecs->ws);
 > +	  /* This was cleared early, by handle_inferior_event.  Set it
 > +	     so this pending event is considered by
 > +	     do_target_wait.  */
 > +	  tp->resumed = 1;
 > +
 > +	  gdb_assert (!tp->executing);
 > +
 > +	  regcache = get_thread_regcache (tp->ptid);
 > +	  tp->suspend.stop_pc = regcache_read_pc (regcache);
 > +
 > +	  if (debug_infrun)
 > +	    {
 > +	      fprintf_unfiltered (gdb_stdlog,
 > +				  "infrun: saved stop_pc=%s for %s "
 > +				  "(currently_stepping=%d)\n",
 > +				  paddress (target_gdbarch (),
 > +					    tp->suspend.stop_pc),
 > +				  target_pid_to_str (tp->ptid),
 > +				  currently_stepping (tp));
 > +	    }
 > +
 > +	  /* This in-line step-over finished; clear this so we won't
 > +	     start a new one.  This is what handle_signal_stop would
 > +	     do, if we returned false.  */
 > +	  tp->stepping_over_breakpoint = 0;
 > +
 > +	  /* Wake up the event loop again.  */
 > +	  mark_async_event_handler (infrun_async_inferior_event_token);
 > +
 > +	  prepare_to_wait (ecs);
 > +	  return 1;
 > +	}
 > +    }
 > +
 > +  return 0;
 >  }
 >  
 >  /* Come here when the program has stopped with a signal.  */
 > @@ -4264,7 +5149,8 @@ handle_signal_stop (struct execution_control_state *ecs)
 >    /* Do we need to clean up the state of a thread that has
 >       completed a displaced single-step?  (Doing so usually affects
 >       the PC, so do it here, before we set stop_pc.)  */
 > -  finish_step_over (ecs);
 > +  if (finish_step_over (ecs))
 > +    return;
 >  
 >    /* If we either finished a single-step or hit a breakpoint, but
 >       the user wanted this thread to be stopped, pretend we got a
 > @@ -4722,6 +5608,7 @@ handle_signal_stop (struct execution_control_state *ecs)
 >                                  "infrun: signal arrived while stepping over "
 >                                  "breakpoint\n");
 >  
 > +	  clear_step_over_info ();
 >  	  insert_hp_step_resume_breakpoint_at_frame (frame);
 >  	  ecs->event_thread->step_after_step_resume_breakpoint = 1;
 >  	  /* Reset trap_expected to ensure breakpoints are re-inserted.  */
 > @@ -4755,6 +5642,7 @@ handle_signal_stop (struct execution_control_state *ecs)
 >                                  "infrun: signal may take us out of "
 >                                  "single-step range\n");
 >  
 > +	  clear_step_over_info ();
 >  	  insert_hp_step_resume_breakpoint_at_frame (frame);
 >  	  ecs->event_thread->step_after_step_resume_breakpoint = 1;
 >  	  /* Reset trap_expected to ensure breakpoints are re-inserted.  */
 > @@ -5710,7 +6598,14 @@ switch_back_to_stepped_thread (struct execution_control_state *ecs)
 >  	     except the one that needs to move past the breakpoint.
 >  	     If a non-event thread has this set, the "incomplete
 >  	     step-over" check above should have caught it earlier.  */
 > -	  gdb_assert (!tp->control.trap_expected);
 > +	  if (tp->control.trap_expected)
 > +	    {
 > +	      internal_error (__FILE__, __LINE__,
 > +			      "[%s] has inconsistent state: "
 > +			      "trap_expected=%d\n",
 > +			      target_pid_to_str (tp->ptid),
 > +			      tp->control.trap_expected);
 > +	    }
 >  
 >  	  /* Did we find the stepping thread?  */
 >  	  if (tp->control.step_range_end)
 > @@ -5839,6 +6734,7 @@ keep_going_stepped_thread (struct thread_info *tp)
 >  				     get_frame_address_space (frame),
 >  				     stop_pc);
 >  
 > +      tp->resumed = 1;
 >        resume_ptid = user_visible_resume_ptid (tp->control.stepping_command);
 >        do_target_resume (resume_ptid, 0, GDB_SIGNAL_0);
 >      }
 > @@ -6270,6 +7166,7 @@ keep_going_pass (struct execution_control_state *ecs)
 >    struct cleanup *old_cleanups = make_cleanup (resume_cleanups, 0);
 >  
 >    gdb_assert (ptid_equal (ecs->event_thread->ptid, inferior_ptid));
 > +  gdb_assert (!ecs->event_thread->resumed);
 >  
 >    /* Save the pc before execution, to compare with pc after stop.  */
 >    ecs->event_thread->prev_pc
 > @@ -6292,6 +7189,34 @@ keep_going_pass (struct execution_control_state *ecs)
 >        discard_cleanups (old_cleanups);
 >        resume (ecs->event_thread->suspend.stop_signal);
 >      }
 > +  else if (step_over_info_valid_p ())
 > +    {
 > +      /* Another thread is stepping over a breakpoint in-line.  If
 > +	 this thread needs a step-over too, queue the request.  In
 > +	 either case, this resume must be deferred for later.  */
 > +      struct thread_info *tp = ecs->event_thread;
 > +
 > +      if (ecs->hit_singlestep_breakpoint
 > +	  || thread_still_needs_step_over (tp))
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: step-over already in progress: "
 > +				"step-over for %s deferred\n",
 > +				target_pid_to_str (tp->ptid));
 > +	  thread_step_over_chain_enqueue (tp);
 > +	}
 > +      else
 > +	{
 > +	  if (debug_infrun)
 > +	    fprintf_unfiltered (gdb_stdlog,
 > +				"infrun: step-over in progress: "
 > +				"resume of %s deferred\n",
 > +				target_pid_to_str (tp->ptid));
 > +	}
 > +
 > +      discard_cleanups (old_cleanups);
 > +    }
 >    else
 >      {
 >        struct regcache *regcache = get_current_regcache ();
 > @@ -6336,8 +7261,14 @@ keep_going_pass (struct execution_control_state *ecs)
 >  	}
 >        else if (remove_wps)
 >  	set_step_over_info (NULL, 0, remove_wps);
 > -      else
 > -	clear_step_over_info ();
 > +
 > +      /* If we now need to do an in-line step-over, we need to stop
 > +	 all other threads.  Note this must be done before
 > +	 insert_breakpoints below, because that removes the breakpoint
 > +	 we're about to step over, otherwise other threads could miss
 > +	 it.  */
 > +      if (step_over_info_valid_p () && non_stop)
 > +	stop_all_threads ();
 >  
 >        /* Stop stepping if inserting breakpoints fails.  */
 >        TRY
 > @@ -7705,6 +8636,23 @@ static const struct internalvar_funcs siginfo_funcs =
 >    NULL
 >  };
 >  
 > +/* Callback for infrun's target events source.  This is marked when a
 > +   thread has a pending status to process.  */
 > +
 > +static void
 > +infrun_async_inferior_event_handler (gdb_client_data data)
 > +{
 > +  /* If the target is closed while this event source is marked, we
 > +     will reach here without execution, or a target to call
 > +     target_wait on, which is an error.  Instead of tracking whether
 > +     the target has been popped already, or whether we do have threads
 > +     with pending statutes, simply ignore the event.  */
 > +  if (!target_is_async_p ())
 > +    return;
 > +
 > +  inferior_event_handler (INF_REG_EVENT, NULL);
 > +}
 > +
 >  void
 >  _initialize_infrun (void)
 >  {
 > @@ -7712,6 +8660,10 @@ _initialize_infrun (void)
 >    int numsigs;
 >    struct cmd_list_element *c;
 >  
 > +  /* Register extra event sources in the event loop.  */
 > +  infrun_async_inferior_event_token
 > +    = create_async_event_handler (infrun_async_inferior_event_handler, NULL);
 > +
 >    add_info ("signals", signals_info, _("\
 >  What debugger does when program gets various signals.\n\
 >  Specify a signal as argument to print info on that signal only."));
 > diff --git a/gdb/infrun.h b/gdb/infrun.h
 > index bfce810..2dc55a9 100644
 > --- a/gdb/infrun.h
 > +++ b/gdb/infrun.h
 > @@ -196,6 +196,9 @@ extern void signal_catch_update (const unsigned int *);
 >     systems.  Use of symbolic signal names is strongly encouraged.  */
 >  enum gdb_signal gdb_signal_from_command (int num);
 >  
 > +/* Enables/disables infrun's async event source in the event loop.  */
 > +extern void infrun_async (int enable);
 > +
 >  /* The global queue of threads that need to do a step-over operation
 >     to get past e.g., a breakpoint.  */
 >  extern struct thread_info *step_over_queue_head;
 > diff --git a/gdb/target.c b/gdb/target.c
 > index 306c21d..e992a35 100644
 > --- a/gdb/target.c
 > +++ b/gdb/target.c
 > @@ -3712,6 +3712,15 @@ maintenance_print_target_stack (char *cmd, int from_tty)
 >      }
 >  }
 >  
 > +/* See target.h.  */
 > +
 > +void
 > +target_async (int enable)
 > +{
 > +  infrun_async (enable);
 > +  current_target.to_async (&current_target, enable);
 > +}
 > +
 >  /* Controls if targets can report that they can/are async.  This is
 >     just for maintainers to use when debugging gdb.  */
 >  int target_async_permitted = 1;
 > diff --git a/gdb/target.h b/gdb/target.h
 > index 66bf91e..3f1a8b0 100644
 > --- a/gdb/target.h
 > +++ b/gdb/target.h
 > @@ -1704,8 +1704,7 @@ extern int target_async_permitted;
 >  #define target_is_async_p() (current_target.to_is_async_p (&current_target))
 >  
 >  /* Enables/disabled async target events.  */
 > -#define target_async(ENABLE) \
 > -     (current_target.to_async (&current_target, (ENABLE)))
 > +extern void target_async (int enable);
 >  
 >  #define target_execution_direction() \
 >    (current_target.to_execution_direction (&current_target))
 > diff --git a/gdb/target/waitstatus.h b/gdb/target/waitstatus.h
 > index d4ef3b8..ffaddc1 100644
 > --- a/gdb/target/waitstatus.h
 > +++ b/gdb/target/waitstatus.h
 > @@ -131,7 +131,10 @@ enum target_stop_reason
 >    TARGET_STOPPED_BY_HW_BREAKPOINT,
 >  
 >    /* Stopped by a watchpoint.  */
 > -  TARGET_STOPPED_BY_WATCHPOINT
 > +  TARGET_STOPPED_BY_WATCHPOINT,
 > +
 > +  /* Stopped by a single step finishing.  */
 > +  TARGET_STOPPED_BY_SINGLE_STEP
 >  };
 >  
 >  /* Prototypes */
 > diff --git a/gdb/thread.c b/gdb/thread.c
 > index 3e3f419..4dde722 100644
 > --- a/gdb/thread.c
 > +++ b/gdb/thread.c
 > @@ -232,6 +232,7 @@ new_thread (ptid_t ptid)
 >    /* Nothing to follow yet.  */
 >    tp->pending_follow.kind = TARGET_WAITKIND_SPURIOUS;
 >    tp->state = THREAD_STOPPED;
 > +  tp->suspend.waitstatus.kind = TARGET_WAITKIND_IGNORE;
 >  
 >    return tp;
 >  }
 > @@ -852,6 +853,28 @@ thread_change_ptid (ptid_t old_ptid, ptid_t new_ptid)
 >    observer_notify_thread_ptid_changed (old_ptid, new_ptid);
 >  }
 >  
 > +/* See gdbthread.h.  */
 > +
 > +void
 > +set_resumed (ptid_t ptid, int resumed)
 > +{
 > +  struct thread_info *tp;
 > +  int all = ptid_equal (ptid, minus_one_ptid);
 > +
 > +  if (all || ptid_is_pid (ptid))
 > +    {
 > +      for (tp = thread_list; tp; tp = tp->next)
 > +	if (all || ptid_get_pid (tp->ptid) == ptid_get_pid (ptid))
 > +	  tp->resumed = resumed;
 > +    }
 > +  else
 > +    {
 > +      tp = find_thread_ptid (ptid);
 > +      gdb_assert (tp != NULL);
 > +      tp->resumed = resumed;
 > +    }
 > +}
 > +
 >  /* Helper for set_running, that marks one thread either running or
 >     stopped.  */
 >  
 > -- 
 > 1.9.3
 >

Patch

diff --git a/gdb/breakpoint.c b/gdb/breakpoint.c
index 31869f1..e23d223 100644
--- a/gdb/breakpoint.c
+++ b/gdb/breakpoint.c
@@ -468,6 +468,8 @@  breakpoints_should_be_inserted_now (void)
     }
   else if (target_has_execution)
     {
+      struct thread_info *tp;
+
       if (always_inserted_mode)
 	{
 	  /* The user wants breakpoints inserted even if all threads
@@ -477,6 +479,13 @@  breakpoints_should_be_inserted_now (void)
 
       if (threads_are_executing ())
 	return 1;
+
+      /* Don't remove breakpoints yet if, even though all threads are
+	 stopped, we still have events to process.  */
+      ALL_NON_EXITED_THREADS (tp)
+	if (tp->resumed
+	    && tp->suspend.waitstatus_pending_p)
+	  return 1;
     }
   return 0;
 }
diff --git a/gdb/gdbthread.h b/gdb/gdbthread.h
index 7052ee1..22c8eb2 100644
--- a/gdb/gdbthread.h
+++ b/gdb/gdbthread.h
@@ -29,6 +29,7 @@  struct symtab;
 #include "inferior.h"
 #include "btrace.h"
 #include "common/vec.h"
+#include "target/waitstatus.h"
 
 /* Frontend view of the thread state.  Possible extensions: stepping,
    finishing, until(ling),...  */
@@ -159,6 +160,23 @@  struct thread_suspend_state
      should be suppressed, the core will take care of clearing this
      before the target is resumed.  */
   enum gdb_signal stop_signal;
+
+  /* The reason the thread last stopped, if we need to track it
+     (breakpoint, watchpoint, etc.)  */
+  enum target_stop_reason stop_reason;
+
+  /* The waitstatus for this thread's last event.  */
+  struct target_waitstatus waitstatus;
+  /* If true WAITSTATUS hasn't been handled yet.  */
+  int waitstatus_pending_p;
+
+  /* Record the pc of the thread the last time it stopped.  (This is
+     not the current thread's PC as that may have changed since the
+     last stop, e.g., "return" command, or "p $pc = 0xf000").  This
+     used in coordination with stop_reason and waitstatus_pending_p:
+     if the thread's PC is changed since it last stopped, a pending
+     breakpoint waitstatus is discarded.  */
+  CORE_ADDR stop_pc;
 };
 
 typedef struct value *value_ptr;
@@ -183,6 +201,14 @@  struct thread_info
      thread is off and running.  */
   int executing;
 
+  /* Non-zero if this thread will be/has been resumed.  Note that a
+     thread can be marked both as not-executing and resumed at the
+     same time.  This happens if we try to resume a thread that has a
+     wait status pending.  We shouldn't let the thread run until that
+     wait status has been processed, but we should not process that
+     wait status if we didn't try to let the thread run.  */
+  int resumed;
+
   /* Frontend view of the thread state.  Note that the THREAD_RUNNING/
      THREAD_STOPPED states are different from EXECUTING.  When the
      thread is stopped internally while handling an internal event,
@@ -400,6 +426,11 @@  extern int thread_count (void);
 /* Switch from one thread to another.  */
 extern void switch_to_thread (ptid_t ptid);
 
+/* Marks or clears thread(s) PTID as resumed.  If PTID is
+   MINUS_ONE_PTID, applies to all threads.  If ptid_is_pid(PTID) is
+   true, applies to all threads of the process pointed at by PTID.  */
+extern void set_resumed (ptid_t ptid, int resumed);
+
 /* Marks thread PTID is running, or stopped. 
    If PTID is minus_one_ptid, marks all threads.  */
 extern void set_running (ptid_t ptid, int running);
diff --git a/gdb/infrun.c b/gdb/infrun.c
index 6415fa2..8859b9f 100644
--- a/gdb/infrun.c
+++ b/gdb/infrun.c
@@ -60,6 +60,36 @@ 
 #include "target-descriptions.h"
 #include "target-dcache.h"
 #include "terminal.h"
+#include "event-loop.h"
+
+/* Asynchronous signal handler registered as event loop source for
+   when we have pending events ready to be passed to the core.  */
+static struct async_event_handler *infrun_async_inferior_event_token;
+
+/* Stores whether infrun_async was previously enabled or disabled.
+   Starts off as -1, indicating "never enabled/disabled".  */
+static int infrun_is_async = -1;
+
+/* See infrun.h.  */
+
+void
+infrun_async (int enable)
+{
+  if (infrun_is_async != enable)
+    {
+      infrun_is_async = enable;
+
+      if (debug_infrun)
+	fprintf_unfiltered (gdb_stdlog,
+			    "infrun: infrun_async(%d)\n",
+			    enable);
+
+      if (enable)
+	mark_async_event_handler (infrun_async_inferior_event_token);
+      else
+	clear_async_event_handler (infrun_async_inferior_event_token);
+    }
+}
 
 /* Prototypes for local functions */
 
@@ -1305,6 +1335,9 @@  set_step_over_info (struct address_space *aspace, CORE_ADDR address,
 static void
 clear_step_over_info (void)
 {
+  if (debug_infrun)
+    fprintf_unfiltered (gdb_stdlog,
+			"infrun: clear_step_over_info\n");
   step_over_info.aspace = NULL;
   step_over_info.address = 0;
   step_over_info.nonsteppable_watchpoint_p = 0;
@@ -1477,6 +1510,23 @@  get_displaced_stepping_state (int pid)
   return NULL;
 }
 
+/* Returns true if any inferior has a thread doing a displaced
+   step.  */
+
+static int
+displaced_step_in_progress_any_inferior (void)
+{
+  struct displaced_step_inferior_state *state;
+
+  for (state = displaced_step_inferior_states;
+       state != NULL;
+       state = state->next)
+    if (!ptid_equal (state->step_ptid, null_ptid))
+      return 1;
+
+  return 0;
+}
+
 /* Return true if process PID has a thread doing a displaced step.  */
 
 static int
@@ -1790,21 +1840,28 @@  displaced_step_restore (struct displaced_step_inferior_state *displaced,
 				  displaced->step_copy));
 }
 
-static void
+/* If we displaced stepped an instruction successfully, adjust
+   registers and memory to yield the same effect the instruction would
+   have had if we had executed it at its original address, and return
+   1.  If the instruction didn't complete, relocate the PC and return
+   -1.  If the thread wasn't displaced stepping, return 0.  */
+
+static int
 displaced_step_fixup (ptid_t event_ptid, enum gdb_signal signal)
 {
   struct cleanup *old_cleanups;
   struct displaced_step_inferior_state *displaced
     = get_displaced_stepping_state (ptid_get_pid (event_ptid));
+  int ret;
 
   /* Was any thread of this process doing a displaced step?  */
   if (displaced == NULL)
-    return;
+    return 0;
 
   /* Was this event for the pid we displaced?  */
   if (ptid_equal (displaced->step_ptid, null_ptid)
       || ! ptid_equal (displaced->step_ptid, event_ptid))
-    return;
+    return 0;
 
   old_cleanups = make_cleanup (displaced_step_clear_cleanup, displaced);
 
@@ -1827,6 +1884,7 @@  displaced_step_fixup (ptid_t event_ptid, enum gdb_signal signal)
                                     displaced->step_original,
                                     displaced->step_copy,
                                     get_thread_regcache (displaced->step_ptid));
+      ret = 1;
     }
   else
     {
@@ -1837,11 +1895,14 @@  displaced_step_fixup (ptid_t event_ptid, enum gdb_signal signal)
 
       pc = displaced->step_original + (pc - displaced->step_copy);
       regcache_write_pc (regcache, pc);
+      ret = -1;
     }
 
   do_cleanups (old_cleanups);
 
   displaced->step_ptid = null_ptid;
+
+  return ret;
 }
 
 /* Data to be passed around while handling an event.  This data is
@@ -1890,10 +1951,19 @@  start_step_over (void)
 {
   struct thread_info *tp, *next;
 
+  /* Don't start a new step-over if we already have a step-over
+     operation ongoing.  */
+  if (step_over_info_valid_p ())
+    return 0;
+
   for (tp = step_over_queue_head; tp != NULL; tp = next)
     {
       struct execution_control_state ecss;
       struct execution_control_state *ecs = &ecss;
+      enum step_over_what step_what;
+      int must_be_in_line;
+      struct regcache *regcache = get_thread_regcache (tp->ptid);
+      struct gdbarch *gdbarch = get_regcache_arch (regcache);
 
       next = thread_step_over_chain_next (tp);
 
@@ -1902,6 +1972,17 @@  start_step_over (void)
       if (displaced_step_in_progress (ptid_get_pid (tp->ptid)))
 	continue;
 
+      step_what = thread_still_needs_step_over (tp);
+      must_be_in_line = ((step_what & STEP_OVER_WATCHPOINT)
+			 || ((step_what & STEP_OVER_BREAKPOINT)
+			     && !use_displaced_stepping (gdbarch)));
+
+      /* We currently stop all threads of all processes to step-over
+	 in-line.  If we need to start a new in-line step-over, let
+	 any pending displace steps finish first.  */
+      if (must_be_in_line && displaced_step_in_progress_any_inferior ())
+	return 0;
+
       thread_step_over_chain_remove (tp);
 
       if (step_over_queue_head == NULL)
@@ -1911,13 +1992,16 @@  start_step_over (void)
 				"infrun: step-over queue now empty\n");
 	}
 
-      if (tp->control.trap_expected || tp->executing)
+      if (tp->control.trap_expected
+	  || tp->resumed
+	  || tp->executing)
 	{
 	  internal_error (__FILE__, __LINE__,
 			  "[%s] has inconsistent state: "
-			  "trap_expected=%d, executing=%d\n",
+			  "trap_expected=%d, resumed=%d, executing=%d\n",
 			  target_pid_to_str (tp->ptid),
 			  tp->control.trap_expected,
+			  tp->resumed,
 			  tp->executing);
 	}
 
@@ -1932,7 +2016,7 @@  start_step_over (void)
 	 wouldn't be able to resume anything else until the target
 	 stops again.  In non-stop, the resume always resumes only TP,
 	 so it's OK to let the thread resume freely.  */
-      if (!non_stop && !thread_still_needs_step_over (tp))
+      if (!non_stop && !step_what)
 	continue;
 
       switch_to_thread (tp->ptid);
@@ -1942,6 +2026,15 @@  start_step_over (void)
       if (!ecs->wait_some_more)
 	error (_("Command aborted."));
 
+      gdb_assert (tp->resumed);
+
+      /* If we started a new in-line step-over, we're done.  */
+      if (step_over_info_valid_p ())
+	{
+	  gdb_assert (tp->control.trap_expected);
+	  return 1;
+	}
+
       if (!non_stop)
 	{
 	  /* On all-stop, shouldn't have resumed unless we needed a
@@ -2157,12 +2250,38 @@  resume (enum gdb_signal sig)
      single-step).  */
   int step;
 
-  tp->stepped_breakpoint = 0;
-
   gdb_assert (!thread_is_in_step_over_chain (tp));
 
   QUIT;
 
+  if (tp->suspend.waitstatus_pending_p)
+    {
+      if (debug_infrun)
+	{
+	  char *statstr;
+
+	  statstr = target_waitstatus_to_string (&tp->suspend.waitstatus);
+	  fprintf_unfiltered (gdb_stdlog,
+			      "infrun: resume: thread %s has pending wait status %s "
+			      "(currently_stepping=%d).\n",
+			      target_pid_to_str (tp->ptid),  statstr,
+			      currently_stepping (tp));
+	  xfree (statstr);
+	}
+
+      tp->resumed = 1;
+      /* Avoid confusing the next resume, if the next stop/resume
+	 happens to apply to another thread.  */
+      tp->suspend.stop_signal = GDB_SIGNAL_0;
+      discard_cleanups (old_cleanups);
+
+      if (target_can_async_p ())
+	target_async (1);
+      return;
+    }
+
+  tp->stepped_breakpoint = 0;
+
   /* Depends on stepped_breakpoint.  */
   step = currently_stepping (tp);
 
@@ -2267,6 +2386,7 @@  resume (enum gdb_signal sig)
 	      resume_ptid = user_visible_resume_ptid (user_step);
 	      do_target_resume (resume_ptid, 0, GDB_SIGNAL_0);
 	      discard_cleanups (old_cleanups);
+	      tp->resumed = 1;
 	      return;
 	    }
 	}
@@ -2392,11 +2512,12 @@  resume (enum gdb_signal sig)
   if (execution_direction != EXEC_REVERSE
       && step && breakpoint_inserted_here_p (aspace, pc))
     {
-      /* The only case we currently need to step a breakpoint
-	 instruction is when we have a signal to deliver.  See
-	 handle_signal_stop where we handle random signals that could
-	 take out us out of the stepping range.  Normally, in that
-	 case we end up continuing (instead of stepping) over the
+      /* There are two cases where we currently need to step a
+	 breakpoint instruction when we have a signal to deliver:
+
+	 - See handle_signal_stop where we handle random signals that
+	 could take out us out of the stepping range.  Normally, in
+	 that case we end up continuing (instead of stepping) over the
 	 signal handler with a breakpoint at PC, but there are cases
 	 where we should _always_ single-step, even if we have a
 	 step-resume breakpoint, like when a software watchpoint is
@@ -2409,8 +2530,20 @@  resume (enum gdb_signal sig)
 	 recurses and executes PC again, it'll miss the breakpoint.
 	 So we leave the breakpoint inserted anyway, but we need to
 	 record that we tried to step a breakpoint instruction, so
-	 that adjust_pc_after_break doesn't end up confused.  */
-      gdb_assert (sig != GDB_SIGNAL_0);
+	 that adjust_pc_after_break doesn't end up confused.
+
+         - In non-stop if we insert a breakpoint (e.g., a step-resume)
+	 in one thread after another thread that was stepping had been
+	 momentarily paused for a step-over.  When we re-resume the
+	 stepping thread, it may be resumed from that address with a
+	 breakpoint that hasn't trapped yet.  Seen with
+	 gdb.threads/non-stop-fair-events.exp, on targets that don't
+	 do displaced stepping.  */
+
+      if (debug_infrun)
+	fprintf_unfiltered (gdb_stdlog,
+			    "infrun: resume: [%s] stepped breakpoint\n",
+			    target_pid_to_str (tp->ptid));
 
       tp->stepped_breakpoint = 1;
 
@@ -2448,6 +2581,7 @@  resume (enum gdb_signal sig)
     }
 
   do_target_resume (resume_ptid, step, sig);
+  tp->resumed = 1;
   discard_cleanups (old_cleanups);
 }
 
@@ -2464,6 +2598,37 @@  clear_proceed_status_thread (struct thread_info *tp)
 			"infrun: clear_proceed_status_thread (%s)\n",
 			target_pid_to_str (tp->ptid));
 
+  /* If we're starting a new sequence, then the previous finished
+     single-step is no longer relevant.  */
+  if (tp->suspend.waitstatus_pending_p)
+    {
+      if (tp->suspend.stop_reason == TARGET_STOPPED_BY_SINGLE_STEP)
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: clear_proceed_status: pending "
+				"event of %s was a finished step. "
+				"Discarding.\n",
+				target_pid_to_str (tp->ptid));
+
+	  tp->suspend.waitstatus_pending_p = 0;
+	  tp->suspend.stop_reason = TARGET_STOPPED_BY_NO_REASON;
+	}
+      else if (debug_infrun)
+	{
+	  char *statstr;
+
+	  statstr = target_waitstatus_to_string (&tp->suspend.waitstatus);
+	  fprintf_unfiltered (gdb_stdlog,
+			      "infrun: clear_proceed_status_thread: thread %s "
+			      "has pending wait status %s "
+			      "(currently_stepping=%d).\n",
+			      target_pid_to_str (tp->ptid), statstr,
+			      currently_stepping (tp));
+	  xfree (statstr);
+	}
+    }
+
   /* If this signal should not be seen by program, give it zero.
      Used for debugging signals.  */
   if (!signal_pass_state (tp->suspend.stop_signal))
@@ -2527,8 +2692,6 @@  clear_proceed_status (int step)
 
   stop_after_trap = 0;
 
-  clear_step_over_info ();
-
   observer_notify_about_to_proceed ();
 
   if (stop_registers)
@@ -2777,7 +2940,7 @@  proceed (CORE_ADDR addr, enum gdb_signal siggnal)
       /* A new displaced stepping sequence was started.  In all-stop,
 	 we can't talk to the target anymore until it next stops.  */
     }
-  else if (!tp->executing && !thread_is_in_step_over_chain (tp))
+  else if (!tp->resumed && !thread_is_in_step_over_chain (tp))
     {
       /* The thread wasn't started, and isn't queued, run it now.  */
       reset_ecs (ecs, tp);
@@ -3076,6 +3239,179 @@  print_target_wait_results (ptid_t waiton_ptid, ptid_t result_ptid,
   ui_file_delete (tmp_stream);
 }
 
+/* Select a thread at random, out of those which are resumed and have
+   had events.  */
+
+static struct thread_info *
+random_pending_event_thread (ptid_t waiton_ptid)
+{
+  struct thread_info *event_tp;
+  int num_events = 0;
+  int random_selector;
+
+  /* First see how many events we have.  Count only resumed threads
+     that have an event pending.  */
+  ALL_NON_EXITED_THREADS (event_tp)
+    if (ptid_match (event_tp->ptid, waiton_ptid)
+	&& event_tp->resumed
+	&& event_tp->suspend.waitstatus_pending_p)
+      num_events++;
+
+  if (num_events == 0)
+    return NULL;
+
+  /* Now randomly pick a thread out of those that have had events.  */
+  random_selector = (int)
+    ((num_events * (double) rand ()) / (RAND_MAX + 1.0));
+
+  if (debug_infrun && num_events > 1)
+    fprintf_unfiltered (gdb_stdlog,
+			"infrun: Found %d events, selecting #%d\n",
+			num_events, random_selector);
+
+  /* Select the Nth thread that has had an event.  */
+  ALL_NON_EXITED_THREADS (event_tp)
+    if (ptid_match (event_tp->ptid, waiton_ptid)
+	&& event_tp->resumed
+	&& event_tp->suspend.waitstatus_pending_p)
+      if (random_selector-- == 0)
+	break;
+
+  return event_tp;
+}
+
+/* Wrapper for target_wait that first checks whether threads have
+   pending status to report before actually asking the target for more
+   events.  */
+
+static ptid_t
+do_target_wait (ptid_t ptid, struct target_waitstatus *status, int options)
+{
+  ptid_t event_ptid;
+  struct thread_info *tp;
+
+  /* First check if there is a resumed thread with a wait status
+     pending.  */
+  if (ptid_equal (ptid, minus_one_ptid) || ptid_is_pid (ptid))
+    {
+      tp = random_pending_event_thread (ptid);
+    }
+  else
+    {
+      if (debug_infrun)
+	fprintf_unfiltered (gdb_stdlog,
+			    "infrun: Waiting for specific thread %s.\n",
+			    target_pid_to_str (ptid));
+
+      /* We have a specific thread to check.  */
+      tp = find_thread_ptid (ptid);
+      gdb_assert (tp != NULL);
+      if (!tp->suspend.waitstatus_pending_p)
+	tp = NULL;
+    }
+
+  if (tp != NULL
+      && (tp->suspend.stop_reason == TARGET_STOPPED_BY_SW_BREAKPOINT
+	  || tp->suspend.stop_reason == TARGET_STOPPED_BY_HW_BREAKPOINT))
+    {
+      struct regcache *regcache = get_thread_regcache (tp->ptid);
+      struct gdbarch *gdbarch = get_regcache_arch (regcache);
+      CORE_ADDR pc;
+      int discard = 0;
+
+      pc = regcache_read_pc (regcache);
+
+      if (pc != tp->suspend.stop_pc)
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: PC of %s changed.  was=%s, now=%s\n",
+				target_pid_to_str (tp->ptid),
+				paddress (target_gdbarch (), tp->prev_pc),
+				paddress (target_gdbarch (), pc));
+	  discard = 1;
+	}
+      else if (!breakpoint_inserted_here_p (get_regcache_aspace (regcache), pc))
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: previous breakpoint of %s, at %s gone\n",
+				target_pid_to_str (tp->ptid),
+				paddress (target_gdbarch (), pc));
+
+	  discard = 1;
+	}
+
+      if (discard)
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: pending event of %s cancelled.\n",
+				target_pid_to_str (tp->ptid));
+
+	  tp->suspend.waitstatus.kind = TARGET_WAITKIND_SPURIOUS;
+	  tp->suspend.stop_reason = TARGET_STOPPED_BY_NO_REASON;
+	}
+    }
+
+  if (tp != NULL)
+    {
+      if (debug_infrun)
+	{
+	  char *statstr;
+
+	  statstr = target_waitstatus_to_string (&tp->suspend.waitstatus);
+	  fprintf_unfiltered (gdb_stdlog,
+			      "infrun: Using pending wait status %s for %s.\n",
+			      statstr,
+			      target_pid_to_str (tp->ptid));
+	  xfree (statstr);
+	}
+
+      /* Now that we've selected our final event LWP, un-adjust its PC
+	 if it was a software breakpoint (and the target doesn't
+	 always adjust the PC itself).  */
+      if (tp->suspend.stop_reason == TARGET_STOPPED_BY_SW_BREAKPOINT
+	  && !target_supports_stopped_by_sw_breakpoint ())
+	{
+	  struct regcache *regcache;
+	  struct gdbarch *gdbarch;
+	  int decr_pc;
+
+	  regcache = get_thread_regcache (tp->ptid);
+	  gdbarch = get_regcache_arch (regcache);
+
+	  decr_pc = gdbarch_decr_pc_after_break (gdbarch);
+	  if (decr_pc != 0)
+	    {
+	      CORE_ADDR pc;
+
+	      pc = regcache_read_pc (regcache);
+	      regcache_write_pc (regcache, pc + decr_pc);
+	    }
+	}
+
+      tp->suspend.stop_reason = TARGET_STOPPED_BY_NO_REASON;
+      *status = tp->suspend.waitstatus;
+      tp->suspend.waitstatus_pending_p = 0;
+
+      /* Wake up the event loop again, until all pending events are
+	 processed.  */
+      if (target_is_async_p ())
+	mark_async_event_handler (infrun_async_inferior_event_token);
+      return tp->ptid;
+    }
+
+  /* But if we don't find one, we'll have to wait.  */
+
+  if (deprecated_target_wait_hook)
+    event_ptid = deprecated_target_wait_hook (ptid, status, options);
+  else
+    event_ptid = target_wait (ptid, status, options);
+
+  return event_ptid;
+}
+
 /* Prepare and stabilize the inferior for detaching it.  E.g.,
    detaching while a thread is displaced stepping is a recipe for
    crashing it, as nothing would readjust the PC out of the scratch
@@ -3119,10 +3455,7 @@  prepare_for_detach (void)
 	 don't get any event.  */
       target_dcache_invalidate ();
 
-      if (deprecated_target_wait_hook)
-	ecs->ptid = deprecated_target_wait_hook (pid_ptid, &ecs->ws, 0);
-      else
-	ecs->ptid = target_wait (pid_ptid, &ecs->ws, 0);
+      ecs->ptid = do_target_wait (pid_ptid, &ecs->ws, 0);
 
       if (debug_infrun)
 	print_target_wait_results (pid_ptid, ecs->ptid, &ecs->ws);
@@ -3194,10 +3527,7 @@  wait_for_inferior (void)
 	 don't get any event.  */
       target_dcache_invalidate ();
 
-      if (deprecated_target_wait_hook)
-	ecs->ptid = deprecated_target_wait_hook (waiton_ptid, &ecs->ws, 0);
-      else
-	ecs->ptid = target_wait (waiton_ptid, &ecs->ws, 0);
+      ecs->ptid = do_target_wait (waiton_ptid, &ecs->ws, 0);
 
       if (debug_infrun)
 	print_target_wait_results (waiton_ptid, ecs->ptid, &ecs->ws);
@@ -3294,11 +3624,7 @@  fetch_inferior_event (void *client_data)
   make_cleanup_restore_integer (&execution_direction);
   execution_direction = target_execution_direction ();
 
-  if (deprecated_target_wait_hook)
-    ecs->ptid =
-      deprecated_target_wait_hook (waiton_ptid, &ecs->ws, TARGET_WNOHANG);
-  else
-    ecs->ptid = target_wait (waiton_ptid, &ecs->ws, TARGET_WNOHANG);
+  ecs->ptid = do_target_wait (waiton_ptid, &ecs->ws, TARGET_WNOHANG);
 
   if (debug_infrun)
     print_target_wait_results (waiton_ptid, ecs->ptid, &ecs->ws);
@@ -3671,6 +3997,349 @@  get_inferior_stop_soon (ptid_t ptid)
   return inf->control.stop_soon;
 }
 
+/* Wait for one event.  Store the resulting waitstatus in WS, and
+   return the event ptid.  */
+
+static ptid_t
+wait_one (struct target_waitstatus *ws)
+{
+  ptid_t event_ptid;
+  ptid_t wait_ptid = minus_one_ptid;
+
+  overlay_cache_invalid = 1;
+
+  /* Flush target cache before starting to handle each event.
+     Target was running and cache could be stale.  This is just a
+     heuristic.  Running threads may modify target memory, but we
+     don't get any event.  */
+  target_dcache_invalidate ();
+
+  if (deprecated_target_wait_hook)
+    event_ptid = deprecated_target_wait_hook (wait_ptid, ws, 0);
+  else
+    event_ptid = target_wait (wait_ptid, ws, 0);
+
+  if (debug_infrun)
+    print_target_wait_results (wait_ptid, event_ptid, ws);
+
+  if (ws->kind == TARGET_WAITKIND_SYSCALL_ENTRY
+      || ws->kind == TARGET_WAITKIND_SYSCALL_RETURN)
+    ws->value.syscall_number = UNKNOWN_SYSCALL;
+
+  return event_ptid;
+}
+
+/* Generate a wrapper for target_stopped_by_REASON that works on PTID
+   instead of the current thread.  */
+#define THREAD_STOPPED_BY(REASON)		\
+static int					\
+thread_stopped_by_ ## REASON (ptid_t ptid)	\
+{						\
+  struct cleanup *old_chain;			\
+  int res;					\
+						\
+  old_chain = save_inferior_ptid ();		\
+  inferior_ptid = ptid;				\
+						\
+  res = target_stopped_by_ ## REASON ();	\
+						\
+  do_cleanups (old_chain);			\
+						\
+  return res;					\
+}
+
+/* Generate thread_stopped_by_watchpoint.  */
+THREAD_STOPPED_BY (watchpoint)
+/* Generate thread_stopped_by_sw_breakpoint.  */
+THREAD_STOPPED_BY (sw_breakpoint)
+/* Generate thread_stopped_by_hw_breakpoint.  */
+THREAD_STOPPED_BY (hw_breakpoint)
+
+/* Cleanups that switches to the PTID pointed at by PTID_P.  */
+
+static void
+switch_to_thread_cleanup (void *ptid_p)
+{
+  ptid_t ptid = *(ptid_t *) ptid_p;
+
+  switch_to_thread (ptid);
+}
+
+/* Save the thread's event and stop reason to process it later.  */
+
+static void
+save_waitstatus (struct thread_info *tp, struct target_waitstatus *ws)
+{
+  struct regcache *regcache;
+  struct address_space *aspace;
+
+  if (debug_infrun)
+    {
+      char *statstr;
+
+      statstr = target_waitstatus_to_string (ws);
+      fprintf_unfiltered (gdb_stdlog,
+			  "infrun: saving status %s for %d.%ld.%ld\n",
+			  statstr,
+			  ptid_get_pid (tp->ptid),
+			  ptid_get_lwp (tp->ptid),
+			  ptid_get_tid (tp->ptid));
+      xfree (statstr);
+    }
+
+  /* Record for later.  */
+  tp->suspend.waitstatus = *ws;
+  tp->suspend.waitstatus_pending_p = 1;
+
+  regcache = get_thread_regcache (tp->ptid);
+  aspace = get_regcache_aspace (regcache);
+
+  if (ws->kind == TARGET_WAITKIND_STOPPED
+      && ws->value.sig == GDB_SIGNAL_TRAP)
+    {
+      CORE_ADDR pc = regcache_read_pc (regcache);
+
+      adjust_pc_after_break (tp, &tp->suspend.waitstatus);
+
+      if (thread_stopped_by_watchpoint (tp->ptid))
+	{
+	  tp->suspend.stop_reason
+	    = TARGET_STOPPED_BY_WATCHPOINT;
+	}
+      else if (target_supports_stopped_by_sw_breakpoint ()
+	       && thread_stopped_by_sw_breakpoint (tp->ptid))
+	{
+	  tp->suspend.stop_reason
+	    = TARGET_STOPPED_BY_SW_BREAKPOINT;
+	}
+      else if (target_supports_stopped_by_hw_breakpoint ()
+	       && thread_stopped_by_hw_breakpoint (tp->ptid))
+	{
+	  tp->suspend.stop_reason
+	    = TARGET_STOPPED_BY_HW_BREAKPOINT;
+	}
+      else if (!target_supports_stopped_by_hw_breakpoint ()
+	       && hardware_breakpoint_inserted_here_p (aspace,
+						       pc))
+	{
+	  tp->suspend.stop_reason
+	    = TARGET_STOPPED_BY_HW_BREAKPOINT;
+	}
+      else if (!target_supports_stopped_by_sw_breakpoint ()
+	       && software_breakpoint_inserted_here_p (aspace,
+						       pc))
+	{
+	  tp->suspend.stop_reason
+	    = TARGET_STOPPED_BY_SW_BREAKPOINT;
+	}
+      else if (!thread_has_single_step_breakpoints_set (tp)
+	       && currently_stepping (tp))
+	{
+	  tp->suspend.stop_reason
+	    = TARGET_STOPPED_BY_SINGLE_STEP;
+	}
+    }
+}
+
+/* Stop all threads.  */
+
+static void
+stop_all_threads (void)
+{
+  /* We may need multiple passes to discover all threads.  */
+  int pass;
+  int iterations = 0;
+  ptid_t entry_ptid;
+  struct cleanup *old_chain;
+
+  gdb_assert (non_stop);
+
+  if (debug_infrun)
+    fprintf_unfiltered (gdb_stdlog, "infrun: stop_all_threads\n");
+
+  entry_ptid = inferior_ptid;
+  old_chain = make_cleanup (switch_to_thread_cleanup, &entry_ptid);
+
+  /* Stop threads in two passes since threads could be spawning as we
+     go through the first pass.  In the second pass, we will stop such
+     spawned threads.  */
+  for (pass = 0; pass < 2; pass++, iterations++)
+    {
+      if (debug_infrun)
+	fprintf_unfiltered (gdb_stdlog,
+			    "infrun: stop_all_threads, pass=%d, "
+			    "iterations=%d\n", pass, iterations);
+      while (1)
+	{
+	  ptid_t event_ptid;
+	  struct target_waitstatus ws;
+	  int need_wait = 0;
+	  struct thread_info *t;
+
+	  update_thread_list ();
+
+	  /* Go through all threads looking for threads that we need
+	     to tell the target to stop.  */
+	  ALL_NON_EXITED_THREADS (t)
+	    {
+	      if (t->executing)
+		{
+		  /* If already stopping, don't request a stop again.
+		     We just haven't seen the notification yet.  */
+		  if (!t->stop_requested)
+		    {
+		      if (debug_infrun)
+			fprintf_unfiltered (gdb_stdlog,
+					    "infrun:   %s executing, "
+					    "need stop\n",
+					    target_pid_to_str (t->ptid));
+		      target_stop (t->ptid);
+		      t->stop_requested = 1;
+		    }
+		  else
+		    {
+		      if (debug_infrun)
+			fprintf_unfiltered (gdb_stdlog,
+					    "infrun:   %s executing, "
+					    "already stopping\n",
+					    target_pid_to_str (t->ptid));
+		    }
+
+		  if (t->stop_requested)
+		    need_wait = 1;
+		}
+	      else
+		{
+		  if (debug_infrun)
+		    fprintf_unfiltered (gdb_stdlog,
+					"infrun:   %s not executing\n",
+					target_pid_to_str (t->ptid));
+
+		  /* The thread may be not executing, but still be
+		     resumed with a pending status to process.  */
+		  t->resumed = 0;
+		}
+	    }
+
+	  if (!need_wait)
+	    break;
+
+	  /* If we find new threads on the second iteration, restart
+	     over.  We want to see two iterations in a row with all
+	     threads stopped.  */
+	  if (pass > 0)
+	    pass = -1;
+
+	  event_ptid = wait_one (&ws);
+	  if (ws.kind == TARGET_WAITKIND_NO_RESUMED)
+	    /* All resumed threads exited.  */
+	    ;
+	  else if (ws.kind == TARGET_WAITKIND_EXITED
+		   || ws.kind == TARGET_WAITKIND_SIGNALLED)
+	    {
+	      if (debug_infrun)
+		{
+		  ptid_t ptid = pid_to_ptid (ws.value.integer);
+
+		  fprintf_unfiltered (gdb_stdlog,
+				      "infrun: %s exited while "
+				      "stopping threads\n",
+				      target_pid_to_str (ptid));
+		}
+	    }
+	  else
+	    {
+	      if (!in_thread_list (event_ptid))
+		t = add_thread (event_ptid);
+	      else
+		t = find_thread_ptid (event_ptid);
+
+	      t->stop_requested = 0;
+	      t->executing = 0;
+	      t->resumed = 0;
+	      t->control.may_range_step = 0;
+
+	      if (ws.kind == TARGET_WAITKIND_STOPPED
+		  && ws.value.sig == GDB_SIGNAL_0)
+		{
+		  /* We caught the event that we intended to catch, so
+		     there's no event pending.  */
+		  t->suspend.waitstatus.kind = TARGET_WAITKIND_IGNORE;
+		  t->suspend.waitstatus_pending_p = 0;
+
+		  if (displaced_step_fixup (t->ptid, GDB_SIGNAL_0) < 0)
+		    {
+		      /* Add it back to the step-over queue.  */
+		      if (debug_infrun)
+			{
+			  fprintf_unfiltered (gdb_stdlog,
+					      "infrun: displaced-step of %s "
+					      "canceled: adding back to the "
+					      "step-over queue\n",
+					      target_pid_to_str (t->ptid));
+			}
+		      t->control.trap_expected = 0;
+		      thread_step_over_chain_enqueue (t);
+		    }
+		}
+	      else
+		{
+		  enum gdb_signal sig;
+		  struct regcache *regcache;
+		  struct address_space *aspace;
+
+		  if (debug_infrun)
+		    {
+		      char *statstr;
+
+		      statstr = target_waitstatus_to_string (&ws);
+		      fprintf_unfiltered (gdb_stdlog,
+					  "infrun: target_wait %s, saving "
+					  "status for %d.%ld.%ld\n",
+					  statstr,
+					  ptid_get_pid (t->ptid),
+					  ptid_get_lwp (t->ptid),
+					  ptid_get_tid (t->ptid));
+		      xfree (statstr);
+		    }
+
+		  /* Record for later.  */
+		  save_waitstatus (t, &ws);
+
+		  sig = (ws.kind == TARGET_WAITKIND_STOPPED
+			 ? ws.value.sig : GDB_SIGNAL_0);
+
+		  if (displaced_step_fixup (t->ptid, sig) < 0)
+		    {
+		      /* Add it back to the step-over queue.  */
+		      t->control.trap_expected = 0;
+		      thread_step_over_chain_enqueue (t);
+		    }
+
+		  regcache = get_thread_regcache (t->ptid);
+		  t->suspend.stop_pc = regcache_read_pc (regcache);
+
+		  if (debug_infrun)
+		    {
+		      fprintf_unfiltered (gdb_stdlog,
+					  "infrun: saved stop_pc=%s for %s "
+					  "(currently_stepping=%d)\n",
+					  paddress (target_gdbarch (),
+						    t->suspend.stop_pc),
+					  target_pid_to_str (t->ptid),
+					  currently_stepping (t));
+		    }
+		}
+	    }
+	}
+    }
+
+  do_cleanups (old_chain);
+
+  if (debug_infrun)
+    fprintf_unfiltered (gdb_stdlog, "infrun: stop_all_threads done\n");
+}
+
 /* Given an execution control state that has been freshly filled in by
    an event from the inferior, figure out what it means and take
    appropriate action.
@@ -3793,21 +4462,28 @@  handle_inferior_event (struct execution_control_state *ecs)
      we're handling a process exit in non-stop mode, there's nothing
      to do, as threads of the dead process are gone, and threads of
      any other process were left running.  */
-  if (!non_stop)
-    set_executing (minus_one_ptid, 0);
-  else if (ecs->ws.kind == TARGET_WAITKIND_SIGNALLED
-	   && ecs->ws.kind == TARGET_WAITKIND_EXITED)
-    {
-      ptid_t pid_ptid;
 
-      /* Some targets still have execution when a process exits.
-	 E.g., for "checkpoint", when when a fork exits and is
-	 mourned, linux-fork.c switches to another fork.  */
-      pid_ptid = pid_to_ptid (ptid_get_pid (ecs->ptid));
-      set_executing (pid_ptid, 0);
-    }
-  else
-    set_executing (ecs->ptid, 0);
+  {
+    ptid_t mark_ptid;
+
+    if (!non_stop)
+      mark_ptid = minus_one_ptid;
+    else if (ecs->ws.kind == TARGET_WAITKIND_SIGNALLED
+	     && ecs->ws.kind == TARGET_WAITKIND_EXITED)
+      {
+	/* Some targets still have execution when a process exits.
+	   E.g., for "checkpoint", when when a fork exits and is
+	   mourned, linux-fork.c switches to another fork.  */
+	mark_ptid = pid_to_ptid (ptid_get_pid (ecs->ptid));
+      }
+    else
+      mark_ptid = ecs->ptid;
+
+    set_executing (mark_ptid, 0);
+
+    /* Likewise the resumed flag.  */
+    set_resumed (mark_ptid, 0);
+  }
 
   switch (ecs->ws.kind)
     {
@@ -4220,16 +4896,137 @@  Cannot fill $_exitsignal with the correct signal number.\n"));
     }
 }
 
-/* Called when we get an event that may finish an in-line or
-   out-of-line (displaced stepping) step-over started previously.  */
+/* Restart threads back to what they were trying to do back when we
+   paused them for an in-line step-over.  The EVENT_THREAD thread is
+   ignored.  */
 
 static void
+restart_threads (struct thread_info *event_thread)
+{
+  struct thread_info *tp;
+  struct thread_info *step_over = NULL;
+
+  /* In case the instruction just stepped spawned a new thread.  */
+  update_thread_list ();
+
+  ALL_NON_EXITED_THREADS (tp)
+    {
+      struct execution_control_state ecss;
+      struct execution_control_state *ecs = &ecss;
+
+      if (tp == event_thread)
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: restart threads: "
+				"[%s] is event thread\n",
+				target_pid_to_str (tp->ptid));
+	  continue;
+	}
+
+      if (!(tp->state == THREAD_RUNNING || tp->control.in_infcall))
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: restart threads: "
+				"[%s] not meant to be running\n",
+				target_pid_to_str (tp->ptid));
+	  continue;
+	}
+
+      if (tp->resumed)
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: restart threads: [%s] resumed\n",
+				target_pid_to_str (tp->ptid));
+	  gdb_assert (tp->executing || tp->suspend.waitstatus_pending_p);
+	  continue;
+	}
+
+      if (thread_is_in_step_over_chain (tp))
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: restart threads: "
+				"[%s] needs step-over\n",
+				target_pid_to_str (tp->ptid));
+	  gdb_assert (!tp->resumed);
+	  continue;
+	}
+
+
+      if (tp->suspend.waitstatus_pending_p)
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: restart threads: "
+				"[%s] has pending status\n",
+				target_pid_to_str (tp->ptid));
+	  tp->resumed = 1;
+	  continue;
+	}
+
+      /* If some thread needs to start a step-over at this point, it
+	 should still be in the step-over queue, and thus skipped
+	 above.  */
+      if (thread_still_needs_step_over (tp))
+	{
+	  internal_error (__FILE__, __LINE__,
+			  "thread [%s] needs a step-over, but not in "
+			  "step-over queue\n",
+			  target_pid_to_str (tp->ptid));
+	}
+
+      if (currently_stepping (tp))
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: restart threads: [%s] was stepping\n",
+				target_pid_to_str (tp->ptid));
+	  keep_going_stepped_thread (tp);
+	}
+      else
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: restart threads: [%s] continuing\n",
+				target_pid_to_str (tp->ptid));
+	  reset_ecs (ecs, tp);
+	  switch_to_thread (tp->ptid);
+	  keep_going_pass (ecs);
+	}
+    }
+}
+
+/* Callback for iterate_over_threads.  Find a resumed thread that has
+   a pending waitstatus.  */
+
+static int
+resumed_thread_with_pending_status (struct thread_info *tp,
+				    void *arg)
+{
+  return (tp->resumed
+	  && tp->suspend.waitstatus_pending_p);
+}
+
+/* Called when we get an event that may finish an in-line or
+   out-of-line (displaced stepping) step-over started previously.
+   Return true if the event is processed and we should go back to the
+   event loop; false if the caller should continue processing the
+   event.  */
+
+static int
 finish_step_over (struct execution_control_state *ecs)
 {
+  int had_step_over_info;
+
   displaced_step_fixup (ecs->ptid,
 			ecs->event_thread->suspend.stop_signal);
 
-  if (step_over_info_valid_p ())
+  had_step_over_info = step_over_info_valid_p ();
+
+  if (had_step_over_info)
     {
       /* If we're stepping over a breakpoint with all threads locked,
 	 then only the thread that was stepped should be reporting
@@ -4241,11 +5038,99 @@  finish_step_over (struct execution_control_state *ecs)
     }
 
   if (!non_stop)
-    return;
+    return 0;
 
   /* Start a new step-over in another thread if there's one that
      needs it.  */
   start_step_over ();
+
+  /* If we were stepping over a breakpoint before, and haven't started
+     a new in-line step-over sequence, then restart all other threads
+     (except the event thread).  We can't do this in all-stop, as then
+     e.g., we wouldn't be able to issue any other remote packet until
+     these other threads stop.  */
+  if (had_step_over_info && !step_over_info_valid_p ())
+    {
+      struct thread_info *pending;
+
+      /* If we only have threads with pending statuses, the restart
+	 below won't restart any thread and so nothing re-inserts the
+	 breakpoint we just stepped over.  But we need it inserted
+	 when we later process the pending events, otherwise if
+	 another thread has a pending event for this breakpoint too,
+	 we'd discard its event (because the breakpoint that
+	 originally caused the event was no longer inserted).  */
+      context_switch (ecs->ptid);
+      insert_breakpoints ();
+
+      restart_threads (ecs->event_thread);
+
+      /* If we have events pending, go through handle_inferior_event
+	 again, picking up a pending event at random.  This avoids
+	 thread starvation.  */
+
+      /* But not if we just stepped over a watchpoint in order to let
+	 the instruction execute so we can evaluate its expression.
+	 The set of watchpoints that triggered is recorded in the
+	 breakpoint objects themselves (see bp->watchpoint_triggered).
+	 If we processed another event first, that other event could
+	 clobber this info.  */
+      if (ecs->event_thread->stepping_over_watchpoint)
+	return 0;
+
+      pending = iterate_over_threads (resumed_thread_with_pending_status,
+				      NULL);
+      if (pending != NULL)
+	{
+	  struct thread_info *tp = ecs->event_thread;
+	  struct regcache *regcache;
+
+	  if (debug_infrun)
+	    {
+	      fprintf_unfiltered (gdb_stdlog,
+				  "infrun: found resumed threads with "
+				  "pending events, saving status\n");
+	    }
+
+	  gdb_assert (pending != tp);
+
+	  /* Record the event thread's event for later.  */
+	  save_waitstatus (tp, &ecs->ws);
+	  /* This was cleared early, by handle_inferior_event.  Set it
+	     so this pending event is considered by
+	     do_target_wait.  */
+	  tp->resumed = 1;
+
+	  gdb_assert (!tp->executing);
+
+	  regcache = get_thread_regcache (tp->ptid);
+	  tp->suspend.stop_pc = regcache_read_pc (regcache);
+
+	  if (debug_infrun)
+	    {
+	      fprintf_unfiltered (gdb_stdlog,
+				  "infrun: saved stop_pc=%s for %s "
+				  "(currently_stepping=%d)\n",
+				  paddress (target_gdbarch (),
+					    tp->suspend.stop_pc),
+				  target_pid_to_str (tp->ptid),
+				  currently_stepping (tp));
+	    }
+
+	  /* This in-line step-over finished; clear this so we won't
+	     start a new one.  This is what handle_signal_stop would
+	     do, if we returned false.  */
+	  tp->stepping_over_breakpoint = 0;
+
+	  /* Wake up the event loop again.  */
+	  mark_async_event_handler (infrun_async_inferior_event_token);
+
+	  prepare_to_wait (ecs);
+	  return 1;
+	}
+    }
+
+  return 0;
 }
 
 /* Come here when the program has stopped with a signal.  */
@@ -4264,7 +5149,8 @@  handle_signal_stop (struct execution_control_state *ecs)
   /* Do we need to clean up the state of a thread that has
      completed a displaced single-step?  (Doing so usually affects
      the PC, so do it here, before we set stop_pc.)  */
-  finish_step_over (ecs);
+  if (finish_step_over (ecs))
+    return;
 
   /* If we either finished a single-step or hit a breakpoint, but
      the user wanted this thread to be stopped, pretend we got a
@@ -4722,6 +5608,7 @@  handle_signal_stop (struct execution_control_state *ecs)
                                 "infrun: signal arrived while stepping over "
                                 "breakpoint\n");
 
+	  clear_step_over_info ();
 	  insert_hp_step_resume_breakpoint_at_frame (frame);
 	  ecs->event_thread->step_after_step_resume_breakpoint = 1;
 	  /* Reset trap_expected to ensure breakpoints are re-inserted.  */
@@ -4755,6 +5642,7 @@  handle_signal_stop (struct execution_control_state *ecs)
                                 "infrun: signal may take us out of "
                                 "single-step range\n");
 
+	  clear_step_over_info ();
 	  insert_hp_step_resume_breakpoint_at_frame (frame);
 	  ecs->event_thread->step_after_step_resume_breakpoint = 1;
 	  /* Reset trap_expected to ensure breakpoints are re-inserted.  */
@@ -5710,7 +6598,14 @@  switch_back_to_stepped_thread (struct execution_control_state *ecs)
 	     except the one that needs to move past the breakpoint.
 	     If a non-event thread has this set, the "incomplete
 	     step-over" check above should have caught it earlier.  */
-	  gdb_assert (!tp->control.trap_expected);
+	  if (tp->control.trap_expected)
+	    {
+	      internal_error (__FILE__, __LINE__,
+			      "[%s] has inconsistent state: "
+			      "trap_expected=%d\n",
+			      target_pid_to_str (tp->ptid),
+			      tp->control.trap_expected);
+	    }
 
 	  /* Did we find the stepping thread?  */
 	  if (tp->control.step_range_end)
@@ -5839,6 +6734,7 @@  keep_going_stepped_thread (struct thread_info *tp)
 				     get_frame_address_space (frame),
 				     stop_pc);
 
+      tp->resumed = 1;
       resume_ptid = user_visible_resume_ptid (tp->control.stepping_command);
       do_target_resume (resume_ptid, 0, GDB_SIGNAL_0);
     }
@@ -6270,6 +7166,7 @@  keep_going_pass (struct execution_control_state *ecs)
   struct cleanup *old_cleanups = make_cleanup (resume_cleanups, 0);
 
   gdb_assert (ptid_equal (ecs->event_thread->ptid, inferior_ptid));
+  gdb_assert (!ecs->event_thread->resumed);
 
   /* Save the pc before execution, to compare with pc after stop.  */
   ecs->event_thread->prev_pc
@@ -6292,6 +7189,34 @@  keep_going_pass (struct execution_control_state *ecs)
       discard_cleanups (old_cleanups);
       resume (ecs->event_thread->suspend.stop_signal);
     }
+  else if (step_over_info_valid_p ())
+    {
+      /* Another thread is stepping over a breakpoint in-line.  If
+	 this thread needs a step-over too, queue the request.  In
+	 either case, this resume must be deferred for later.  */
+      struct thread_info *tp = ecs->event_thread;
+
+      if (ecs->hit_singlestep_breakpoint
+	  || thread_still_needs_step_over (tp))
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: step-over already in progress: "
+				"step-over for %s deferred\n",
+				target_pid_to_str (tp->ptid));
+	  thread_step_over_chain_enqueue (tp);
+	}
+      else
+	{
+	  if (debug_infrun)
+	    fprintf_unfiltered (gdb_stdlog,
+				"infrun: step-over in progress: "
+				"resume of %s deferred\n",
+				target_pid_to_str (tp->ptid));
+	}
+
+      discard_cleanups (old_cleanups);
+    }
   else
     {
       struct regcache *regcache = get_current_regcache ();
@@ -6336,8 +7261,14 @@  keep_going_pass (struct execution_control_state *ecs)
 	}
       else if (remove_wps)
 	set_step_over_info (NULL, 0, remove_wps);
-      else
-	clear_step_over_info ();
+
+      /* If we now need to do an in-line step-over, we need to stop
+	 all other threads.  Note this must be done before
+	 insert_breakpoints below, because that removes the breakpoint
+	 we're about to step over, otherwise other threads could miss
+	 it.  */
+      if (step_over_info_valid_p () && non_stop)
+	stop_all_threads ();
 
       /* Stop stepping if inserting breakpoints fails.  */
       TRY
@@ -7705,6 +8636,23 @@  static const struct internalvar_funcs siginfo_funcs =
   NULL
 };
 
+/* Callback for infrun's target events source.  This is marked when a
+   thread has a pending status to process.  */
+
+static void
+infrun_async_inferior_event_handler (gdb_client_data data)
+{
+  /* If the target is closed while this event source is marked, we
+     will reach here without execution, or a target to call
+     target_wait on, which is an error.  Instead of tracking whether
+     the target has been popped already, or whether we do have threads
+     with pending statutes, simply ignore the event.  */
+  if (!target_is_async_p ())
+    return;
+
+  inferior_event_handler (INF_REG_EVENT, NULL);
+}
+
 void
 _initialize_infrun (void)
 {
@@ -7712,6 +8660,10 @@  _initialize_infrun (void)
   int numsigs;
   struct cmd_list_element *c;
 
+  /* Register extra event sources in the event loop.  */
+  infrun_async_inferior_event_token
+    = create_async_event_handler (infrun_async_inferior_event_handler, NULL);
+
   add_info ("signals", signals_info, _("\
 What debugger does when program gets various signals.\n\
 Specify a signal as argument to print info on that signal only."));
diff --git a/gdb/infrun.h b/gdb/infrun.h
index bfce810..2dc55a9 100644
--- a/gdb/infrun.h
+++ b/gdb/infrun.h
@@ -196,6 +196,9 @@  extern void signal_catch_update (const unsigned int *);
    systems.  Use of symbolic signal names is strongly encouraged.  */
 enum gdb_signal gdb_signal_from_command (int num);
 
+/* Enables/disables infrun's async event source in the event loop.  */
+extern void infrun_async (int enable);
+
 /* The global queue of threads that need to do a step-over operation
    to get past e.g., a breakpoint.  */
 extern struct thread_info *step_over_queue_head;
diff --git a/gdb/target.c b/gdb/target.c
index 306c21d..e992a35 100644
--- a/gdb/target.c
+++ b/gdb/target.c
@@ -3712,6 +3712,15 @@  maintenance_print_target_stack (char *cmd, int from_tty)
     }
 }
 
+/* See target.h.  */
+
+void
+target_async (int enable)
+{
+  infrun_async (enable);
+  current_target.to_async (&current_target, enable);
+}
+
 /* Controls if targets can report that they can/are async.  This is
    just for maintainers to use when debugging gdb.  */
 int target_async_permitted = 1;
diff --git a/gdb/target.h b/gdb/target.h
index 66bf91e..3f1a8b0 100644
--- a/gdb/target.h
+++ b/gdb/target.h
@@ -1704,8 +1704,7 @@  extern int target_async_permitted;
 #define target_is_async_p() (current_target.to_is_async_p (&current_target))
 
 /* Enables/disabled async target events.  */
-#define target_async(ENABLE) \
-     (current_target.to_async (&current_target, (ENABLE)))
+extern void target_async (int enable);
 
 #define target_execution_direction() \
   (current_target.to_execution_direction (&current_target))
diff --git a/gdb/target/waitstatus.h b/gdb/target/waitstatus.h
index d4ef3b8..ffaddc1 100644
--- a/gdb/target/waitstatus.h
+++ b/gdb/target/waitstatus.h
@@ -131,7 +131,10 @@  enum target_stop_reason
   TARGET_STOPPED_BY_HW_BREAKPOINT,
 
   /* Stopped by a watchpoint.  */
-  TARGET_STOPPED_BY_WATCHPOINT
+  TARGET_STOPPED_BY_WATCHPOINT,
+
+  /* Stopped by a single step finishing.  */
+  TARGET_STOPPED_BY_SINGLE_STEP
 };
 
 /* Prototypes */
diff --git a/gdb/thread.c b/gdb/thread.c
index 3e3f419..4dde722 100644
--- a/gdb/thread.c
+++ b/gdb/thread.c
@@ -232,6 +232,7 @@  new_thread (ptid_t ptid)
   /* Nothing to follow yet.  */
   tp->pending_follow.kind = TARGET_WAITKIND_SPURIOUS;
   tp->state = THREAD_STOPPED;
+  tp->suspend.waitstatus.kind = TARGET_WAITKIND_IGNORE;
 
   return tp;
 }
@@ -852,6 +853,28 @@  thread_change_ptid (ptid_t old_ptid, ptid_t new_ptid)
   observer_notify_thread_ptid_changed (old_ptid, new_ptid);
 }
 
+/* See gdbthread.h.  */
+
+void
+set_resumed (ptid_t ptid, int resumed)
+{
+  struct thread_info *tp;
+  int all = ptid_equal (ptid, minus_one_ptid);
+
+  if (all || ptid_is_pid (ptid))
+    {
+      for (tp = thread_list; tp; tp = tp->next)
+	if (all || ptid_get_pid (tp->ptid) == ptid_get_pid (ptid))
+	  tp->resumed = resumed;
+    }
+  else
+    {
+      tp = find_thread_ptid (ptid);
+      gdb_assert (tp != NULL);
+      tp->resumed = resumed;
+    }
+}
+
 /* Helper for set_running, that marks one thread either running or
    stopped.  */