gdb: Print frame address in more cases

Message ID 20200304141258.17737-1-andrew.burgess@embecosm.com
State New, archived
Headers

Commit Message

Andrew Burgess March 4, 2020, 2:12 p.m. UTC
  Change GDB to display the frame address for all inline frames except
the inner most frame.  This issue was discussed briefly on the mailing
list here:

   https://sourceware.org/ml/gdb-patches/2019-12/msg01015.html

Consider this stack setup from the test gdb.dwarf2/dw2-inline-many-frames.exp:

   #11     #10    #9     #8     #7     #6     #5     #4     #3     #2     #1     #0
   main -> aaa -> bbb -> ccc -> ddd -> eee -> fff -> ggg -> hhh -> iii -> jjj -> kkk
   \_______________________/    \________/    \______________________/    \________/
      Inline sequence #1          Normal          Inline sequence #2        Normal

When we stop in GDB and backtrace the stack is displayed like
this (I've replaced the file and line information with "..."):

  (gdb) bt
  #0  kkk () at ...
  #1  0x0000000000400494 in jjj () at ...
  #2  0x00000000004004b1 in iii () at ...
  #3  hhh () at ...
  #4  ggg () at ...
  #5  fff () at ...
  #6  0x0000000000400536 in eee () at ...
  #7  0x0000000000400519 in ddd () at ...
  #8  0x00000000004004f3 in ccc () at ...
  #9  bbb () at ...
  #10 aaa () at ...
  #11 main () at ...
  (gdb)

Notice that many frames are missing an address, take the sequence for
frames #6 to #2.  Frame #6 is a non-inline frame and has an address.
Frame #6 calls to frame #5 which is also non-inline.  Frames #4, #3,
and #2 are all inline within frame #5, and of these only frame #2 has
an address displayed.

This lack of address can be confusing, there's no clear indication why
the addresses are missing, a user needs to understand that the missing
address implies an inline frame, and can then infer the address based
on the address of frame #2.

After this patch the backtrace is now show like this:

  #0  kkk () at ...
  #1  0x0000000000400494 in jjj () at ...
  #2  0x00000000004004b1 in iii () at ...
  #3  0x00000000004004b1 in hhh () at ...
  #4  0x00000000004004b1 in ggg () at ...
  #5  0x00000000004004b1 in fff () at ...
  #6  0x0000000000400536 in eee () at ...
  #7  0x0000000000400519 in ddd () at ...
  #8  0x00000000004004f3 in ccc () at ...
  #9  0x00000000004004f3 in bbb () at ...
  #10 0x00000000004004f3 in aaa () at ...
  #11 0x00000000004004f3 in main () at ...

Now all frames (except #0) have an address.  Frames that are inline,
and previously lacked an address will now have the same address as the
inner-most inline frame in the sequence, so for the inline sequence #5
to #2, all frames have the same address as frame #2.  For the sequence
frame #11 to #8, all have the address of frame #8.

The duplicated address might also be confusing, though (personally) I
think it is slightly more obvious from the duplicated addresses that
the frames are inlined, however, if people strongly disagree and
prefer the no-address layout we could make this feature switchable.

One further location where the no address can crop up, is when the
user switches frame, previously we would see:

  (gdb) frame 3
  #3  hhh () at ...
  115	  return iii () + 1;
  (gdb)

After the patch we see:

  (gdb) frame 3
  #3  0x00000000004004b1 in hhh () at ...
  115	  return iii () + 1;
  (gdb)

There were a small number of tests that needed to have their expected
results updated, and I added a small section to the documentation to
help users understand what duplicate addresses in the backtrace might
mean.

gdb/ChangeLog:

	* stack.c (frame_show_address): Show the frame address for all but
	the inner-most inlined frame.

gdb/testsuite/ChangeLog:

	* gdb.dwarf2/dw2-inline-many-frames.exp: Update expected results.
	* gdb.dwarf2/dw2-inline-param.exp: Likewise.
	* gdb.guile/scm-frame-inline.exp: Likewise.
	* gdb.opt/inline-cmds.exp: Likewise.
	* gdb.python/py-frame-inline.exp: Likewise.

gdb/doc/ChangeLog:

	* gdb.texinfo (Backtrace): Mention duplicate addresses in the
	backtrace.
---
 gdb/ChangeLog                                       |  5 +++++
 gdb/doc/ChangeLog                                   |  5 +++++
 gdb/doc/gdb.texinfo                                 | 18 ++++++++++++++++++
 gdb/stack.c                                         |  2 +-
 gdb/testsuite/ChangeLog                             |  8 ++++++++
 gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp | 12 ++++++------
 gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp       |  4 +++-
 gdb/testsuite/gdb.guile/scm-frame-inline.exp        |  2 +-
 gdb/testsuite/gdb.opt/inline-cmds.exp               | 14 +++++++-------
 gdb/testsuite/gdb.python/py-frame-inline.exp        |  6 +++---
 10 files changed, 57 insertions(+), 19 deletions(-)
  

Comments

Terekhov, Mikhail via Gdb-patches March 4, 2020, 7:53 p.m. UTC | #1
On Wed, Mar 4, 2020 at 8:13 AM Andrew Burgess
<andrew.burgess@embecosm.com> wrote:
>
> Change GDB to display the frame address for all inline frames except
> the inner most frame.  This issue was discussed briefly on the mailing
> list here:
>
>    https://sourceware.org/ml/gdb-patches/2019-12/msg01015.html

Thanks! I think this makes more sense.

Christian

> Consider this stack setup from the test gdb.dwarf2/dw2-inline-many-frames.exp:
>
>    #11     #10    #9     #8     #7     #6     #5     #4     #3     #2     #1     #0
>    main -> aaa -> bbb -> ccc -> ddd -> eee -> fff -> ggg -> hhh -> iii -> jjj -> kkk
>    \_______________________/    \________/    \______________________/    \________/
>       Inline sequence #1          Normal          Inline sequence #2        Normal
>
> When we stop in GDB and backtrace the stack is displayed like
> this (I've replaced the file and line information with "..."):
>
>   (gdb) bt
>   #0  kkk () at ...
>   #1  0x0000000000400494 in jjj () at ...
>   #2  0x00000000004004b1 in iii () at ...
>   #3  hhh () at ...
>   #4  ggg () at ...
>   #5  fff () at ...
>   #6  0x0000000000400536 in eee () at ...
>   #7  0x0000000000400519 in ddd () at ...
>   #8  0x00000000004004f3 in ccc () at ...
>   #9  bbb () at ...
>   #10 aaa () at ...
>   #11 main () at ...
>   (gdb)
>
> Notice that many frames are missing an address, take the sequence for
> frames #6 to #2.  Frame #6 is a non-inline frame and has an address.
> Frame #6 calls to frame #5 which is also non-inline.  Frames #4, #3,
> and #2 are all inline within frame #5, and of these only frame #2 has
> an address displayed.
>
> This lack of address can be confusing, there's no clear indication why
> the addresses are missing, a user needs to understand that the missing
> address implies an inline frame, and can then infer the address based
> on the address of frame #2.
>
> After this patch the backtrace is now show like this:
>
>   #0  kkk () at ...
>   #1  0x0000000000400494 in jjj () at ...
>   #2  0x00000000004004b1 in iii () at ...
>   #3  0x00000000004004b1 in hhh () at ...
>   #4  0x00000000004004b1 in ggg () at ...
>   #5  0x00000000004004b1 in fff () at ...
>   #6  0x0000000000400536 in eee () at ...
>   #7  0x0000000000400519 in ddd () at ...
>   #8  0x00000000004004f3 in ccc () at ...
>   #9  0x00000000004004f3 in bbb () at ...
>   #10 0x00000000004004f3 in aaa () at ...
>   #11 0x00000000004004f3 in main () at ...
>
> Now all frames (except #0) have an address.  Frames that are inline,
> and previously lacked an address will now have the same address as the
> inner-most inline frame in the sequence, so for the inline sequence #5
> to #2, all frames have the same address as frame #2.  For the sequence
> frame #11 to #8, all have the address of frame #8.
>
> The duplicated address might also be confusing, though (personally) I
> think it is slightly more obvious from the duplicated addresses that
> the frames are inlined, however, if people strongly disagree and
> prefer the no-address layout we could make this feature switchable.
>
> One further location where the no address can crop up, is when the
> user switches frame, previously we would see:
>
>   (gdb) frame 3
>   #3  hhh () at ...
>   115     return iii () + 1;
>   (gdb)
>
> After the patch we see:
>
>   (gdb) frame 3
>   #3  0x00000000004004b1 in hhh () at ...
>   115     return iii () + 1;
>   (gdb)
>
> There were a small number of tests that needed to have their expected
> results updated, and I added a small section to the documentation to
> help users understand what duplicate addresses in the backtrace might
> mean.
>
> gdb/ChangeLog:
>
>         * stack.c (frame_show_address): Show the frame address for all but
>         the inner-most inlined frame.
>
> gdb/testsuite/ChangeLog:
>
>         * gdb.dwarf2/dw2-inline-many-frames.exp: Update expected results.
>         * gdb.dwarf2/dw2-inline-param.exp: Likewise.
>         * gdb.guile/scm-frame-inline.exp: Likewise.
>         * gdb.opt/inline-cmds.exp: Likewise.
>         * gdb.python/py-frame-inline.exp: Likewise.
>
> gdb/doc/ChangeLog:
>
>         * gdb.texinfo (Backtrace): Mention duplicate addresses in the
>         backtrace.
> ---
>  gdb/ChangeLog                                       |  5 +++++
>  gdb/doc/ChangeLog                                   |  5 +++++
>  gdb/doc/gdb.texinfo                                 | 18 ++++++++++++++++++
>  gdb/stack.c                                         |  2 +-
>  gdb/testsuite/ChangeLog                             |  8 ++++++++
>  gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp | 12 ++++++------
>  gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp       |  4 +++-
>  gdb/testsuite/gdb.guile/scm-frame-inline.exp        |  2 +-
>  gdb/testsuite/gdb.opt/inline-cmds.exp               | 14 +++++++-------
>  gdb/testsuite/gdb.python/py-frame-inline.exp        |  6 +++---
>  10 files changed, 57 insertions(+), 19 deletions(-)
>
> diff --git a/gdb/doc/gdb.texinfo b/gdb/doc/gdb.texinfo
> index 32e419ead02..74178faee37 100644
> --- a/gdb/doc/gdb.texinfo
> +++ b/gdb/doc/gdb.texinfo
> @@ -8013,6 +8013,24 @@
>  either deduce that from other variables whose values depend on the one
>  you are interested in, or recompile without optimizations.
>
> +If your compiler has performed the function inlining optimisation
> +(@pxref{Inline Functions}) then this will be reflected in the program
> +counter addresses displayed in the backtrace.  In the following
> +modified example the function @code{expand_macro} has been inlined
> +within @code{expand_token}, as a result the program counter values for
> +frame 1 and frame 2 are the same.
> +
> +@smallexample
> +@group
> +#0  m4_traceon (obs=0x24eb0, argc=1, argv=0x2b8c8)
> +    at builtin.c:993
> +#1  0x6e38 in expand_macro (sym=<optimized out>) at macro.c:242
> +#2  0x6e38 in expand_token (obs=0x0, t=<optimized out>, td=0xf7fffb08)
> +    at macro.c:71
> +(More stack frames follow...)
> +@end group
> +@end smallexample
> +
>  @cindex backtrace beyond @code{main} function
>  @cindex program entry point
>  @cindex startup code, and backtrace
> diff --git a/gdb/stack.c b/gdb/stack.c
> index 266d771e35f..4570eed1a2f 100644
> --- a/gdb/stack.c
> +++ b/gdb/stack.c
> @@ -327,7 +327,7 @@ frame_show_address (struct frame_info *frame,
>         gdb_assert (inline_skipped_frames (inferior_thread ()) > 0);
>        else
>         gdb_assert (get_frame_type (get_next_frame (frame)) == INLINE_FRAME);
> -      return false;
> +      return frame_relative_level (frame) > 0;
>      }
>
>    return get_frame_pc (frame) != sal.pc;
> diff --git a/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp b/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp
> index 146af8c6ef7..f5890fb2aca 100644
> --- a/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp
> +++ b/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp
> @@ -350,15 +350,15 @@ gdb_test "bt" [multi_line \
>                    "#0  kkk \\(\\) at \[^\r\n\]+${srcfile}:${line_in_kkk}" \
>                    "#1  $hex in jjj \\(\\) at \[^\r\n\]+${srcfile}:${line_in_jjj}" \
>                    "#2  $hex in iii \\(\\) at \[^\r\n\]+${srcfile}:${line_in_iii}" \
> -                  "#3  hhh \\(\\) at \[^\r\n\]+${srcfile}:${line_in_hhh}" \
> -                  "#4  ggg \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ggg}" \
> -                  "#5  fff \\(\\) at \[^\r\n\]+${srcfile}:${line_in_fff}" \
> +                  "#3  $hex in hhh \\(\\) at \[^\r\n\]+${srcfile}:${line_in_hhh}" \
> +                  "#4  $hex in ggg \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ggg}" \
> +                  "#5  $hex in fff \\(\\) at \[^\r\n\]+${srcfile}:${line_in_fff}" \
>                    "#6  $hex in eee \\(\\) at \[^\r\n\]+${srcfile}:${line_in_eee}" \
>                    "#7  $hex in ddd \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ddd}" \
>                    "#8  $hex in ccc \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ccc}" \
> -                  "#9  bbb \\(\\) at \[^\r\n\]+${srcfile}:${line_in_bbb}" \
> -                  "#10 aaa \\(\\) at \[^\r\n\]+${srcfile}:${line_in_aaa}" \
> -                  "#11 main \\(\\) at \[^\r\n\]+${srcfile}:${line_in_main}" ]
> +                  "#9  $hex in bbb \\(\\) at \[^\r\n\]+${srcfile}:${line_in_bbb}" \
> +                  "#10 $hex in aaa \\(\\) at \[^\r\n\]+${srcfile}:${line_in_aaa}" \
> +                  "#11 $hex in main \\(\\) at \[^\r\n\]+${srcfile}:${line_in_main}" ]
>
>  # Now check we can use 'up' to inspect each frame correctly.
>  set patterns [list  \
> diff --git a/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp b/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp
> index 1c1e75619da..e27ef8f3b38 100644
> --- a/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp
> +++ b/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp
> @@ -55,4 +55,6 @@ if ![runto "*${break_at}"] {
>      return -1
>  }
>
> -gdb_test "bt" "#0  (0x\[0-9a-f\]+ in )?func \\(funcparam=<optimized out>\\)\r\n#1  main \\(mainparam=<optimized out>\\)\[^\r\n\]*"
> +gdb_test "bt" [multi_line \
> +                  "#0  ($hex in )?func \\(funcparam=<optimized out>\\)" \
> +                  "#1  $hex in main \\(mainparam=<optimized out>\\)\[^\r\n\]*" ]
> diff --git a/gdb/testsuite/gdb.guile/scm-frame-inline.exp b/gdb/testsuite/gdb.guile/scm-frame-inline.exp
> index 8a4d8f893de..70882f841d7 100644
> --- a/gdb/testsuite/gdb.guile/scm-frame-inline.exp
> +++ b/gdb/testsuite/gdb.guile/scm-frame-inline.exp
> @@ -37,7 +37,7 @@ gdb_continue_to_breakpoint "break-here"
>
>  gdb_test "info frame" "inlined into frame 1\r\n.*"
>
> -gdb_test "up" "#1  g .*"
> +gdb_test "up" "#1  $hex in g .*"
>
>  gdb_test "guile (print (frame-read-var (selected-frame) \"l\"))" \
>      "= 42"
> diff --git a/gdb/testsuite/gdb.opt/inline-cmds.exp b/gdb/testsuite/gdb.opt/inline-cmds.exp
> index aa8c8c6bfa0..de5eb1dfee5 100644
> --- a/gdb/testsuite/gdb.opt/inline-cmds.exp
> +++ b/gdb/testsuite/gdb.opt/inline-cmds.exp
> @@ -288,15 +288,15 @@ gdb_test_multiple "finish" "finish from marker" {
>  }
>  gdb_test "bt" "#0  main.*" "backtrace at main of outer_inline"
>  gdb_test "step" "outer_inline2 \\\(\\\) at .*" "enter outer_inline2"
> -gdb_test "bt" "#0  outer_inline2.*#1  main.*" "backtrace at outer_inline2"
> +gdb_test "bt" "#0  outer_inline2.*#1  $hex in main.*" "backtrace at outer_inline2 xx"
>  gdb_test "step" "outer_inline1 \\\(\\\) at .*" "enter outer_inline1 from outer_inline2"
>
>  set msg "backtrace at outer_inline1"
>  gdb_test_multiple "bt" $msg {
> -    -re "#0  outer_inline1.*#1  outer_inline2.*#2  main.*$gdb_prompt $" {
> +    -re "#0  outer_inline1.*#1  $hex in outer_inline2.*#2  $hex in main.*$gdb_prompt $" {
>         pass $msg
>      }
> -    -re "#0  $hex in outer_inline1.*#1  outer_inline2.*#2  main.*$gdb_prompt $" {
> +    -re "#0  $hex in outer_inline1.*#1  $hex in outer_inline2.*#2  $hex in main.*$gdb_prompt $" {
>         # Binutils PR gas/6717.  Gas moves .loc past .p2align and the
>         # leading nop of the inlined call appears to be on the same line
>         # as main's call to marker.
> @@ -306,17 +306,17 @@ gdb_test_multiple "bt" $msg {
>  }
>
>  gdb_test "step" "noinline \\\(\\\) at .*" "enter noinline from outer_inline1"
> -gdb_test "bt" "#0  noinline.*#1  .*outer_inline1.*#2  .*outer_inline2.*#3  main.*" "backtrace at noinline from outer_inline1"
> +gdb_test "bt" "#0  noinline.*#1  .*outer_inline1.*#2  .*outer_inline2.*#3  $hex in main.*" "backtrace at noinline from outer_inline1"
>  gdb_test "step" "inlined_fn \\\(\\\) at .*" "enter inlined_fn from noinline"
> -gdb_test "bt" "#0  inlined_fn.*#1  noinline.*#2  .*outer_inline1.*#3  .*outer_inline2.*#4  main.*" "backtrace at inlined_fn from noinline"
> +gdb_test "bt" "#0  inlined_fn.*#1  $hex in noinline.*#2  .*outer_inline1.*#3  .*outer_inline2.*#4  $hex in main.*" "backtrace at inlined_fn from noinline"
>  gdb_test "info frame" ".*inlined into frame.*" "inlined_fn from noinline inlined"
> -gdb_test "up" "#1  noinline.*" "up to noinline"
> +gdb_test "up" "#1  $hex in noinline.*" "up to noinline"
>  gdb_test "info frame" ".*\n called by frame.*" "noinline from outer_inline1 not inlined"
>  gdb_test "up" "#2  .*outer_inline1.*" "up to outer_inline1"
>  gdb_test "info frame" ".*inlined into frame.*" "outer_inline1 inlined"
>  gdb_test "up" "#3  .*outer_inline2.*" "up to outer_inline2"
>  gdb_test "info frame" ".*inlined into frame.*" "outer_inline2 inlined"
> -gdb_test "up" "#4  main.*" "up from outer_inline2"
> +gdb_test "up" "#4  $hex in main.*" "up from outer_inline2"
>  gdb_test "info frame" ".*\n caller of frame.*" "main not inlined"
>
>  gdb_exit
> diff --git a/gdb/testsuite/gdb.python/py-frame-inline.exp b/gdb/testsuite/gdb.python/py-frame-inline.exp
> index 71bffd375db..aa22001b0d8 100644
> --- a/gdb/testsuite/gdb.python/py-frame-inline.exp
> +++ b/gdb/testsuite/gdb.python/py-frame-inline.exp
> @@ -30,11 +30,11 @@ if ![runto main] then {
>  }
>
>  gdb_breakpoint [gdb_get_line_number "break-here"]
> -gdb_continue_to_breakpoint "Block break here."
> +gdb_continue_to_breakpoint "continue to 'break here', first time"
>
>  gdb_test "info frame" "inlined into frame 1\r\n.*"
>
> -gdb_test "up" "#1  g .*"
> +gdb_test "up" "#1  $hex in g .*"
>
>  gdb_test "python print (gdb.selected_frame().read_var('l'))" "\r\n42"
>
> @@ -48,7 +48,7 @@ gdb_test "python print (gdb.selected_frame().read_var('l'))" "\r\n42"
>  # the frame cache is flushed somehow after setting the limit, to force
>  # frame id recomputation.
>  gdb_test_no_output "set backtrace limit 1"
> -gdb_continue_to_breakpoint "Block break here."
> +gdb_continue_to_breakpoint "continue to 'break here', second time"
>
>  gdb_test "python print (gdb.newest_frame())" ".*"
>
> --
> 2.14.5
>
  

Patch

diff --git a/gdb/doc/gdb.texinfo b/gdb/doc/gdb.texinfo
index 32e419ead02..74178faee37 100644
--- a/gdb/doc/gdb.texinfo
+++ b/gdb/doc/gdb.texinfo
@@ -8013,6 +8013,24 @@ 
 either deduce that from other variables whose values depend on the one
 you are interested in, or recompile without optimizations.
 
+If your compiler has performed the function inlining optimisation
+(@pxref{Inline Functions}) then this will be reflected in the program
+counter addresses displayed in the backtrace.  In the following
+modified example the function @code{expand_macro} has been inlined
+within @code{expand_token}, as a result the program counter values for
+frame 1 and frame 2 are the same.
+
+@smallexample
+@group
+#0  m4_traceon (obs=0x24eb0, argc=1, argv=0x2b8c8)
+    at builtin.c:993
+#1  0x6e38 in expand_macro (sym=<optimized out>) at macro.c:242
+#2  0x6e38 in expand_token (obs=0x0, t=<optimized out>, td=0xf7fffb08)
+    at macro.c:71
+(More stack frames follow...)
+@end group
+@end smallexample
+
 @cindex backtrace beyond @code{main} function
 @cindex program entry point
 @cindex startup code, and backtrace
diff --git a/gdb/stack.c b/gdb/stack.c
index 266d771e35f..4570eed1a2f 100644
--- a/gdb/stack.c
+++ b/gdb/stack.c
@@ -327,7 +327,7 @@  frame_show_address (struct frame_info *frame,
 	gdb_assert (inline_skipped_frames (inferior_thread ()) > 0);
       else
 	gdb_assert (get_frame_type (get_next_frame (frame)) == INLINE_FRAME);
-      return false;
+      return frame_relative_level (frame) > 0;
     }
 
   return get_frame_pc (frame) != sal.pc;
diff --git a/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp b/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp
index 146af8c6ef7..f5890fb2aca 100644
--- a/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp
+++ b/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp
@@ -350,15 +350,15 @@  gdb_test "bt" [multi_line \
 		   "#0  kkk \\(\\) at \[^\r\n\]+${srcfile}:${line_in_kkk}" \
 		   "#1  $hex in jjj \\(\\) at \[^\r\n\]+${srcfile}:${line_in_jjj}" \
 		   "#2  $hex in iii \\(\\) at \[^\r\n\]+${srcfile}:${line_in_iii}" \
-		   "#3  hhh \\(\\) at \[^\r\n\]+${srcfile}:${line_in_hhh}" \
-		   "#4  ggg \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ggg}" \
-		   "#5  fff \\(\\) at \[^\r\n\]+${srcfile}:${line_in_fff}" \
+		   "#3  $hex in hhh \\(\\) at \[^\r\n\]+${srcfile}:${line_in_hhh}" \
+		   "#4  $hex in ggg \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ggg}" \
+		   "#5  $hex in fff \\(\\) at \[^\r\n\]+${srcfile}:${line_in_fff}" \
 		   "#6  $hex in eee \\(\\) at \[^\r\n\]+${srcfile}:${line_in_eee}" \
 		   "#7  $hex in ddd \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ddd}" \
 		   "#8  $hex in ccc \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ccc}" \
-		   "#9  bbb \\(\\) at \[^\r\n\]+${srcfile}:${line_in_bbb}" \
-		   "#10 aaa \\(\\) at \[^\r\n\]+${srcfile}:${line_in_aaa}" \
-		   "#11 main \\(\\) at \[^\r\n\]+${srcfile}:${line_in_main}" ]
+		   "#9  $hex in bbb \\(\\) at \[^\r\n\]+${srcfile}:${line_in_bbb}" \
+		   "#10 $hex in aaa \\(\\) at \[^\r\n\]+${srcfile}:${line_in_aaa}" \
+		   "#11 $hex in main \\(\\) at \[^\r\n\]+${srcfile}:${line_in_main}" ]
 
 # Now check we can use 'up' to inspect each frame correctly.
 set patterns [list  \
diff --git a/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp b/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp
index 1c1e75619da..e27ef8f3b38 100644
--- a/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp
+++ b/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp
@@ -55,4 +55,6 @@  if ![runto "*${break_at}"] {
     return -1
 }
 
-gdb_test "bt" "#0  (0x\[0-9a-f\]+ in )?func \\(funcparam=<optimized out>\\)\r\n#1  main \\(mainparam=<optimized out>\\)\[^\r\n\]*"
+gdb_test "bt" [multi_line \
+		   "#0  ($hex in )?func \\(funcparam=<optimized out>\\)" \
+		   "#1  $hex in main \\(mainparam=<optimized out>\\)\[^\r\n\]*" ]
diff --git a/gdb/testsuite/gdb.guile/scm-frame-inline.exp b/gdb/testsuite/gdb.guile/scm-frame-inline.exp
index 8a4d8f893de..70882f841d7 100644
--- a/gdb/testsuite/gdb.guile/scm-frame-inline.exp
+++ b/gdb/testsuite/gdb.guile/scm-frame-inline.exp
@@ -37,7 +37,7 @@  gdb_continue_to_breakpoint "break-here"
 
 gdb_test "info frame" "inlined into frame 1\r\n.*"
 
-gdb_test "up" "#1  g .*"
+gdb_test "up" "#1  $hex in g .*"
 
 gdb_test "guile (print (frame-read-var (selected-frame) \"l\"))" \
     "= 42"
diff --git a/gdb/testsuite/gdb.opt/inline-cmds.exp b/gdb/testsuite/gdb.opt/inline-cmds.exp
index aa8c8c6bfa0..de5eb1dfee5 100644
--- a/gdb/testsuite/gdb.opt/inline-cmds.exp
+++ b/gdb/testsuite/gdb.opt/inline-cmds.exp
@@ -288,15 +288,15 @@  gdb_test_multiple "finish" "finish from marker" {
 }
 gdb_test "bt" "#0  main.*" "backtrace at main of outer_inline"
 gdb_test "step" "outer_inline2 \\\(\\\) at .*" "enter outer_inline2"
-gdb_test "bt" "#0  outer_inline2.*#1  main.*" "backtrace at outer_inline2"
+gdb_test "bt" "#0  outer_inline2.*#1  $hex in main.*" "backtrace at outer_inline2 xx"
 gdb_test "step" "outer_inline1 \\\(\\\) at .*" "enter outer_inline1 from outer_inline2"
 
 set msg "backtrace at outer_inline1"
 gdb_test_multiple "bt" $msg {
-    -re "#0  outer_inline1.*#1  outer_inline2.*#2  main.*$gdb_prompt $" {
+    -re "#0  outer_inline1.*#1  $hex in outer_inline2.*#2  $hex in main.*$gdb_prompt $" {
 	pass $msg
     }
-    -re "#0  $hex in outer_inline1.*#1  outer_inline2.*#2  main.*$gdb_prompt $" {
+    -re "#0  $hex in outer_inline1.*#1  $hex in outer_inline2.*#2  $hex in main.*$gdb_prompt $" {
 	# Binutils PR gas/6717.  Gas moves .loc past .p2align and the
 	# leading nop of the inlined call appears to be on the same line
 	# as main's call to marker.
@@ -306,17 +306,17 @@  gdb_test_multiple "bt" $msg {
 }
 
 gdb_test "step" "noinline \\\(\\\) at .*" "enter noinline from outer_inline1"
-gdb_test "bt" "#0  noinline.*#1  .*outer_inline1.*#2  .*outer_inline2.*#3  main.*" "backtrace at noinline from outer_inline1"
+gdb_test "bt" "#0  noinline.*#1  .*outer_inline1.*#2  .*outer_inline2.*#3  $hex in main.*" "backtrace at noinline from outer_inline1"
 gdb_test "step" "inlined_fn \\\(\\\) at .*" "enter inlined_fn from noinline"
-gdb_test "bt" "#0  inlined_fn.*#1  noinline.*#2  .*outer_inline1.*#3  .*outer_inline2.*#4  main.*" "backtrace at inlined_fn from noinline"
+gdb_test "bt" "#0  inlined_fn.*#1  $hex in noinline.*#2  .*outer_inline1.*#3  .*outer_inline2.*#4  $hex in main.*" "backtrace at inlined_fn from noinline"
 gdb_test "info frame" ".*inlined into frame.*" "inlined_fn from noinline inlined"
-gdb_test "up" "#1  noinline.*" "up to noinline"
+gdb_test "up" "#1  $hex in noinline.*" "up to noinline"
 gdb_test "info frame" ".*\n called by frame.*" "noinline from outer_inline1 not inlined"
 gdb_test "up" "#2  .*outer_inline1.*" "up to outer_inline1"
 gdb_test "info frame" ".*inlined into frame.*" "outer_inline1 inlined"
 gdb_test "up" "#3  .*outer_inline2.*" "up to outer_inline2"
 gdb_test "info frame" ".*inlined into frame.*" "outer_inline2 inlined"
-gdb_test "up" "#4  main.*" "up from outer_inline2"
+gdb_test "up" "#4  $hex in main.*" "up from outer_inline2"
 gdb_test "info frame" ".*\n caller of frame.*" "main not inlined"
 
 gdb_exit
diff --git a/gdb/testsuite/gdb.python/py-frame-inline.exp b/gdb/testsuite/gdb.python/py-frame-inline.exp
index 71bffd375db..aa22001b0d8 100644
--- a/gdb/testsuite/gdb.python/py-frame-inline.exp
+++ b/gdb/testsuite/gdb.python/py-frame-inline.exp
@@ -30,11 +30,11 @@  if ![runto main] then {
 }
 
 gdb_breakpoint [gdb_get_line_number "break-here"]
-gdb_continue_to_breakpoint "Block break here."
+gdb_continue_to_breakpoint "continue to 'break here', first time"
 
 gdb_test "info frame" "inlined into frame 1\r\n.*"
 
-gdb_test "up" "#1  g .*"
+gdb_test "up" "#1  $hex in g .*"
 
 gdb_test "python print (gdb.selected_frame().read_var('l'))" "\r\n42"
 
@@ -48,7 +48,7 @@  gdb_test "python print (gdb.selected_frame().read_var('l'))" "\r\n42"
 # the frame cache is flushed somehow after setting the limit, to force
 # frame id recomputation.
 gdb_test_no_output "set backtrace limit 1"
-gdb_continue_to_breakpoint "Block break here."
+gdb_continue_to_breakpoint "continue to 'break here', second time"
 
 gdb_test "python print (gdb.newest_frame())" ".*"