[review] Handle pending stops from the Windows kernel

  Change URL: https://gnutoolchain-gerrit.osci.io/r/c/binutils-gdb/+/414
......................................................................

Handle pending stops from the Windows kernel

PR gdb/22992 concerns an assertion failure in gdb when debugging a
certain inferior:

    int finish_step_over(execution_control_state*): Assertion `ecs->event_thread->control.trap_expected' failed.

Initially the investigation centered on the discovery that gdb was not
suspending other threads when attempting to single-step.  This
oversight is corrected in this patch: now, when stepping a thread, gdb
will call SuspendThread on all other threads.

However, the bug persisted even after this change.  In particular,
WaitForDebugEvent could see a stop for a thread that was ostensibly
suspended.  Our theory of what is happening here is that there are
actually simultaneous breakpoint hits, and the Windows kernel queues
the events, causing the second stop to be reported on a suspended
thread.

In Windows 10 or later gdb could use the DBG_REPLY_LATER flag to
ContinueDebugEvent to request that such events be re-reported later.
However, relying on that did not seem advisable, so this patch instead
arranges to queue such "pending" stops, and then to report them later,
once the step has completed.

In the PR, Pedro pointed out that it's best in this scenario to
implement the stopped_by_sw_breakpoint method, so this patch does this
as well.

gdb/ChangeLog
2019-10-29  Tom Tromey  <tromey@adacore.com>

	PR gdb/22992
	* windows-nat.c (current_event): Update comment.
	(last_wait_event, desired_stop_thread_id): New globals.
	(struct pending_stop): New.
	(pending_stops, last_stop_was_breakpoint): New globals.
	(windows_nat_target) <stopped_by_sw_breakpoint>
	<supports_stopped_by_sw_breakpoint>: New methods.
	(suspend_thread): New function.
	(thread_rec): Use suspend_thread.
	(windows_continue): Handle pending stops.  Suspend other threads
	when stepping.  Use last_wait_event
	(wait_for_debug_event): New function.
	(get_windows_debug_event): Use wait_for_debug_event.  Handle
	pending stops.  Queue spurious stops.
	(windows_nat_target::wait): Un-bias PC when needed.  Set
	last_stop_was_breakpoint.
	(windows_nat_target::kill): Use wait_for_debug_event.

Change-Id: I5d0dd64c3d2d8220b534d3e02aeaa6f6815264ab
---
M gdb/ChangeLog
M gdb/windows-nat.c
2 files changed, 184 insertions(+), 12 deletions(-)
  

Luis Machado has posted comments on this change.

Change URL: https://gnutoolchain-gerrit.osci.io/r/c/binutils-gdb/+/414
......................................................................


Patch Set 1: Code-Review+1

Though i'm not too familiar with Windows handling, the change looks sane to me.
  

On 10/29/19 5:57 PM, Tom Tromey (Code Review) wrote:

> +/* A single pending stop.  See "pending_stops" for more
> +   information.  */
> +struct pending_stop
> +{
> +  /* The thread id.  */
> +  DWORD thread_id;
> +
> +  /* The target waitstatus we computed.  */
> +  target_waitstatus status;
> +
> +  /* The event.  A few fields of this can be referenced after a stop,
> +     and it seemed simplest to store the entire event.  */
> +  DEBUG_EVENT event;
> +};
> +
> +/* A vector of pending stops.  Sometimes, Windows will report a stop
> +   on a thread that has been ostensibly suspended.  We believe what
> +   happens here is that two threads hit a breakpoint simultaneously,
> +   and the Windows kernel queues the stop events.  

I'm very much suspect that this must be happening also with other events,
not just breakpoints.  I think the comment should be adjusted in
that direction.

> However, this can
> +   result in the strange effect of trying to single step thread A --
> +   leaving all other threads suspended -- and then seeing a stop in
> +   thread B.  To handle this scenario, we queue all such "pending"
> +   stops here, and then process them once the step has completed.  See
> +   PR gdb/22992.  */
> +static std::vector<pending_stop> pending_stops;
> +
> +/* This is true if the most recently reported stop was due to a
> +   breakpoint.  */
> +static bool last_stop_was_breakpoint = false;
> +
>  struct windows_nat_target final : public x86_nat_target<inf_child_target>
>  {
>    void close () override;
> @@ -307,6 +351,16 @@
>    void fetch_registers (struct regcache *, int) override;
>    void store_registers (struct regcache *, int) override;
>  
> +  bool stopped_by_sw_breakpoint () override
> +  {
> +    return last_stop_was_breakpoint;
> +  }
> +
> +  bool supports_stopped_by_sw_breakpoint () override
> +  {
> +    return true;
> +  }
> +
>    enum target_xfer_status xfer_partial (enum target_object object,
>  					const char *annex,
>  					gdb_byte *readbuf,
> @@ -1297,16 +1351,37 @@

I'm disappointed that gerrit doesn't use "diff -up".  :-/

>  {
>    BOOL res;
>  
> +  desired_stop_thread_id = id;
> +
> +  /* If there are pending stops, and we might plausibly hit one of
> +     them, we don't want to actually continue the inferior -- we just
> +     want to report the stop.  In this case, we just pretend to
> +     continue.  See the comment by the definition of "pending_stops"
> +     for details on why this is needed.  */
> +  for (auto &item : pending_stops)

const auto ?

> +    {
> +      if (desired_stop_thread_id == -1
> +	  || desired_stop_thread_id == item.thread_id)
> +	{
> +	  DEBUG_EVENTS (("windows_continue - pending stop anticipated, "
> +			 "desired=0x%x, item=0x%x\n",
> +			 desired_stop_thread_id, item.thread_id));
> +	  debug_registers_changed = 0;

This clearing of debug_registers_changed looks suspicious.
If the debug registers did change, won't we miss propagating them
to the threads, when they're latter actually resumed?

> +	  return TRUE;
> +	}
> +    }
> +
>    DEBUG_EVENTS (("ContinueDebugEvent (cpid=%d, ctid=0x%x, %s);\n",
> -		  (unsigned) current_event.dwProcessId,
> -		  (unsigned) current_event.dwThreadId,
> +		  (unsigned) last_wait_event.dwProcessId,
> +		  (unsigned) last_wait_event.dwThreadId,
>  		  continue_status == DBG_CONTINUE ?
>  		  "DBG_CONTINUE" : "DBG_EXCEPTION_NOT_HANDLED"));
>  
>    for (windows_thread_info *th : thread_list)
> -    if ((id == -1 || id == (int) th->tid)
> -	&& th->suspended)
> +    if (id == -1 || id == (int) th->tid)
>        {
> +	if (!th->suspended)
> +	  continue;
>  	if (debug_registers_changed)
>  	  {
>  	    th->context.ContextFlags |= CONTEXT_DEBUG_REGISTERS;
> @@ -1333,9 +1408,15 @@
>  	  }
>  	th->resume ();
>        }
> +    else
> +      {
> +	/* When single-stepping a specific thread, other threads must
> +	   be suspended.  */
> +	th->suspend ();
> +      }
>  
> -  res = ContinueDebugEvent (current_event.dwProcessId,
> -			    current_event.dwThreadId,
> +  res = ContinueDebugEvent (last_wait_event.dwProcessId,
> +			    last_wait_event.dwThreadId,
>  			    continue_status);
>  
>    if (!res)
> @@ -1487,6 +1568,17 @@
>    return TRUE;
>  }
>  


> +  else if (desired_stop_thread_id != -1 && desired_stop_thread_id != thread_id)
> +    {
> +      /* Pending stop.  See the comment by the definition of
> +	 "pending_stops" for details on why this is needed.  */
> +      DEBUG_EVENTS (("get_windows_debug_event - "
> +		     "unexpected stop in 0x%x (expecting 0x%x)\n",
> +		     thread_id, desired_stop_thread_id));
> +
> +      pending_stops.push_back ({thread_id, *ourstatus, current_event});

I think you should unwind the PC here, not only when returning the pending
event to GDB core.  Consider the case of two threads hitting a breakpoint
at the same time.  When that happens, and do you "info threads", you want to
see the PC of all threads pointing at a valid instruction.  If you don't
unwind the PC of pending breakpoints, then the threads with pending breakpoints
will have their PC offset by one.

> +      thread_id = 0;
> +      CHECK (windows_continue (continue_status, desired_stop_thread_id, 0));
>      }
>    else
>      {
> @@ -1733,7 +1864,28 @@
>        SetConsoleCtrlHandler (&ctrl_c_handler, FALSE);
>  
>        if (retval)
> -	return ptid_t (current_event.dwProcessId, retval, 0);
> +	{
> +	  ptid_t result = ptid_t (current_event.dwProcessId, retval, 0);
> +
> +	  last_stop_was_breakpoint = false;
> +	  if (current_event.dwDebugEventCode == EXCEPTION_DEBUG_EVENT
> +	      && (current_event.u.Exception.ExceptionRecord.ExceptionCode
> +		  == EXCEPTION_BREAKPOINT))
> +	    {
> +	      struct regcache *regcache = get_thread_regcache (result);
> +	      gdbarch *gdbarch = regcache->arch ();
> +
> +	      CORE_ADDR pc = (regcache_read_pc (regcache)
> +			      - gdbarch_decr_pc_after_break (gdbarch));
> +	      if (software_breakpoint_inserted_here_p (regcache->aspace (), pc))

Why is software_breakpoint_inserted_here_p needed?  

> +		{
> +		  last_stop_was_breakpoint = true;
> +		  regcache_write_pc (regcache, pc);
> +		}
> +	    }
> +
> +	  return result;
> +	}
>        else
>  	{
>  	  int detach = 0;
Thanks,
Pedro Alves
  

Joel Brobecker has posted comments on this change.

Change URL: https://gnutoolchain-gerrit.osci.io/r/c/binutils-gdb/+/414
......................................................................


Patch Set 1: Code-Review+1
  

>>>>> "Pedro" == Pedro Alves <palves@redhat.com> writes:

>> +  else if (desired_stop_thread_id != -1 && desired_stop_thread_id != thread_id)
>> +    {
>> +      /* Pending stop.  See the comment by the definition of
>> +	 "pending_stops" for details on why this is needed.  */
>> +      DEBUG_EVENTS (("get_windows_debug_event - "
>> +		     "unexpected stop in 0x%x (expecting 0x%x)\n",
>> +		     thread_id, desired_stop_thread_id));
>> +
>> +      pending_stops.push_back ({thread_id, *ourstatus, current_event});

Pedro> I think you should unwind the PC here, not only when returning the pending
Pedro> event to GDB core.  Consider the case of two threads hitting a breakpoint
Pedro> at the same time.  When that happens, and do you "info threads", you want to
Pedro> see the PC of all threads pointing at a valid instruction.  If you don't
Pedro> unwind the PC of pending breakpoints, then the threads with pending breakpoints
Pedro> will have their PC offset by one.

I think I tried this, but I can try again.

>> +	      if (software_breakpoint_inserted_here_p (regcache->aspace (), pc))

Pedro> Why is software_breakpoint_inserted_here_p needed?  

Offsetting the PC did not work without this.
I tried to document my findings here:

https://sourceware.org/ml/gdb-patches/2019-10/msg00338.html

IIRC what happened is that gdb would sometimes resume the inferior with
wrong PC, causing it to crash.  However, I don't really recall, since it
was a long time ago now.  I guess I'll re-do the experiments.

Tom
  

On 11/19/19 2:20 PM, Tom Tromey wrote:
>>>>>> "Pedro" == Pedro Alves <palves@redhat.com> writes:

> Pedro> I think you should unwind the PC here, not only when returning the pending
> Pedro> event to GDB core.  Consider the case of two threads hitting a breakpoint
> Pedro> at the same time.  When that happens, and do you "info threads", you want to
> Pedro> see the PC of all threads pointing at a valid instruction.  If you don't
> Pedro> unwind the PC of pending breakpoints, then the threads with pending breakpoints
> Pedro> will have their PC offset by one.
> 
> I think I tried this, but I can try again.

Thanks.

> 
>>> +	      if (software_breakpoint_inserted_here_p (regcache->aspace (), pc))
> 
> Pedro> Why is software_breakpoint_inserted_here_p needed?  
> 
> Offsetting the PC did not work without this.
> I tried to document my findings here:
> 
> https://sourceware.org/ml/gdb-patches/2019-10/msg00338.html
> 

Off hand that doesn't sound right.  Linux doesn't do that.
See linux-nat.c:save_stop_reason, in the USE_SIGTRAP_SIGINFO case
(the #else case is probably and hopefully dead by now).

With the software_breakpoint_inserted_here_p check in place, what I imagine
would happen is:

- thread A and B hit a breakpoint
- the event for thread B is left pending
- event for thread A is reported
- user/GDB removes the breakpoint before the event for thread B is processed
- user continues
- windows-nat.c prepares to return the pending event for B
- software_breakpoint_inserted_here_p returns false, so the PC is left unadjusted
- gdb core reports a spurious SIGTRAP, with the PC left unadjusted
- if the inferior is resumed, it starts execution with a bogus PC

Without the check, what should happen, and is the right behavior, is:

- thread A and B hit a breakpoint
- the event for thread B is left pending
- event for thread A is reported
- user/GDB removes the breakpoint before the event for thread B is processed
- user continues
- windows-nat.c prepares to return the pending event for B, adjusts the PC
- gdb core sees a TARGET_STOPPED_BY_SW_BREAKPOINT event, with the PC
  already adjusted.
- there's no breakpoint at that address, so gdb re-resumes the inferior
  transparently, here, in infrun.c:

	  /* A delayed software breakpoint event.  Ignore the trap.  */
	  if (debug_infrun)
	    fprintf_unfiltered (gdb_stdlog,
				"infrun: delayed software breakpoint "
				"trap, ignoring\n");


Basically, this mechanism replaces the old moribund locations heuristic.

> IIRC what happened is that gdb would sometimes resume the inferior with
> wrong PC, causing it to crash.  However, I don't really recall, since it
> was a long time ago now.  I guess I'll re-do the experiments.

Thanks.  I'm mystified.

Pedro Alves
  

>>>>> "Pedro" == Pedro Alves <palves@redhat.com> writes:

>>>> +	      if (software_breakpoint_inserted_here_p (regcache->aspace (), pc))

Pedro> Why is software_breakpoint_inserted_here_p needed?  

>> Offsetting the PC did not work without this.

The problem turned out to be that this was the line preventing a
spurious stop at the initial "breakpoint" that occurs when starting an
inferior on Windows.  Checking windows_initialization_done worked as
well.

I've fixed this and also changed the code to adjust the PC for suspended
threads.

Tom
  

Pedro Alves has posted comments on this change.

Change URL: https://gnutoolchain-gerrit.osci.io/r/c/binutils-gdb/+/414
......................................................................


Patch Set 2: Code-Review+2

(2 comments)

Minor nit below.  Otherwise this version LGTM.

| --- gdb/nat/windows-nat.h
| +++ gdb/nat/windows-nat.h
| @@ -64,15 +64,19 @@ #endif
|    CONTEXT context {};
|  
|    /* Whether debug registers changed since we last set CONTEXT back to
|       the thread.  */
|    bool debug_registers_changed = false;
|  
|    /* Nonzero if CONTEXT is invalidated and must be re-read from the
|       inferior thread.  */
|    bool reload_context = false;
|  
| +  /* True if this thread is currently stopped at a breakpoint.  This
| +     is used to offset the PC when needed.  */
| +  bool stopped_at_breakpoint = false;

PS2, Line 76:

I think it would be better if this said "stopped at a software
breakpoint", and the field was called stopped_at_sw_breakpoint (for
example).  This is because you won't do the offsetting with hardware
breakpoints.

| +
|    /* The name of the thread, allocated by xmalloc.  */
|    gdb::unique_xmalloc_ptr<char> name;
|  };
|  
|  #endif
| --- gdb/windows-nat.c
| +++ gdb/windows-nat.c
| @@ -556,16 +610,37 @@ windows_fetch_one_register (struct regcache *regcache,
|      {
|        /* GDB treats segment registers as 32bit registers, but they are
|  	 in fact only 16 bits long.  Make sure we do not read extra
|  	 bits from our source buffer.  */
|        long l = *((long *) context_offset) & 0xffff;
|        regcache->raw_supply (r, (char *) &l);
|      }
|    else
| -    regcache->raw_supply (r, context_offset);
| +    {
| +      if (th->stopped_at_breakpoint && r == gdbarch_pc_regnum (gdbarch))
| +	{
| +	  int size = register_size (gdbarch, r);
| +	  if (size == 4)
| +	    {
| +	      uint32_t value;
| +	      memcpy (&value, context_offset, size);
| +	      value -= gdbarch_decr_pc_after_break (gdbarch);
| +	      memcpy (context_offset, &value, size);
| +	    }
| +	  else
| +	    {
| +	      gdb_assert (size == 8);
| +	      uint64_t value;
| +	      memcpy (&value, context_offset, size);
| +	      value -= gdbarch_decr_pc_after_break (gdbarch);
| +	      memcpy (context_offset, &value, size);
| +	    }
| +	}

PS2, Line 637:

Clever to do this here.

| +      regcache->raw_supply (r, context_offset);
| +    }
|  }
|  
|  void
|  windows_nat_target::fetch_registers (struct regcache *regcache, int r)
|  {
|    DWORD tid = regcache->ptid ().lwp ();
|    windows_thread_info *th = thread_rec (tid, INVALIDATE_CONTEXT);