Add helper script for glibc debugging
Commit Message
From: "Gabriel F. T. Gomes" <gabrielftg@linux.ibm.com>
At Cauldron, while Florian showed his development workflow to us, we had
a brief discussion about debugging glibc with GDB. After that, I spent
some time with Arjun and we shared our debugging scripts and tricks,
then we came to the conclusion that this could be helpful to other
people, maybe even more so to newcomers. So, I got my off-tree script
and converted it into this auto-generated version.
Still at Cauldron, Carlos mentioned that it wasn't trivial to let GDB
know about the symbols within glibc. This script doesn't solve this for
every glibc debugging needs (I suppose it doesn't help at all when
debugging the initial steps of program loading, for example), but it
makes it a little easier when debugging test cases, as it automatically
loads the symbols from them.
Maybe there are more trivial ways to do it, but that's what I have been
using for some time.
-- 8< --
This patch adds a new make rule that generates a helper script for
debugging glibc test cases. The new script, debugglibc.sh, is similar
to testrun.sh, in the sense that it allows the execution of the
specified test case, however, it opens the test case in GDB, setting the
library path the same way that testrun.sh does. The commands are based
on the instruction on the wiki page for glibc debugging [1].
By default, the script tells GDB to load the test case for symbol
information, so that, when a breakpoint is hit, the call stack is
displayed correctly (instead of printing lots of '??'s). For instance,
after running 'make' and 'make check', one could do the following:
./debugglibc.sh -t $PWD/stdlib/tst-strfrom -B strfromf -B strfromd
Reading symbols from /home/gabriel/build/x86_64/glibc/elf/ld.so...
Breakpoint 1 at 0x1098
add symbol table from file "/home/gabriel/build/x86_64/glibc/stdlib/tst-strfrom"
Breakpoint 1, 0x00007ffff7fd5098 in _dl_start_user () from /home/gabriel/build/x86_64/glibc/elf/ld.so
Breakpoint 2 at 0x7ffff7e4dba0: file strfrom-skeleton.c, line 41.
Breakpoint 3 at 0x7ffff7e4ddd0: file strfrom-skeleton.c, line 41.
Notice that the script will always start GDB with the program running
and halted at _dl_start_user. So, in order to reach the actual
breakpoint of interest, one should hit 'c', not 'r':
>>> c
Continuing.
Testing in locale: C
Breakpoint 2, strfromf (dest=dest@entry=0x7fffffffe02b "\377", size=5, format=0x405010 "%g", f=12345.3447) at strfrom-skeleton.c:41
41 sfile.f._sbf._f._lock = NULL;
An inspect the call stack with 'bt', as usual, and see symbols from both
the test case and from the libraries themselves:
>>> bt
#0 strfromf (dest=dest@entry=0x7fffffffe02b "\377", size=5, format=0x405010 "%g", f=12345.3447) at strfrom-skeleton.c:41
#1 0x000000000040246a in test_f () at tst-strfrom.c:67
#2 test_locale (locale=locale@entry=0x405046 "C") at tst-strfrom.c:77
#3 0x000000000040358b in do_test () at tst-strfrom.c:84
#4 legacy_test_function (argc=<optimized out>, argv=<optimized out>) at ../test-skeleton.c:56
#5 0x0000000000403b25 in support_test_main (argc=1, argc@entry=2, argv=0x7fffffffe258, argv@entry=0x7fffffffe250, config=config@entry=0x7fffffffe120) at support_test_main.c:350
#6 0x0000000000402315 in main (argc=argc@entry=2, argv=argv@entry=0x7fffffffe250) at ../support/test-driver.c:168
#7 0x00007ffff7e35deb in __libc_start_main (main=0x4022e0 <main>, argc=2, argv=0x7fffffffe250, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe248) at ../csu/libc-start.c:308
#8 0x000000000040234a in _start () at ../sysdeps/x86_64/start.S:120
Tested for x86_64.
[1] https://sourceware.org/glibc/wiki/Debugging/Loader_Debugging
---
Makefile | 159 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 158 insertions(+), 1 deletion(-)
Comments
On 9/24/19 3:24 PM, Gabriel F. T. Gomes wrote:
> From: "Gabriel F. T. Gomes" <gabrielftg@linux.ibm.com>
>
> At Cauldron, while Florian showed his development workflow to us, we had
> a brief discussion about debugging glibc with GDB. After that, I spent
> some time with Arjun and we shared our debugging scripts and tricks,
> then we came to the conclusion that this could be helpful to other
> people, maybe even more so to newcomers. So, I got my off-tree script
> and converted it into this auto-generated version.
>
> Still at Cauldron, Carlos mentioned that it wasn't trivial to let GDB
> know about the symbols within glibc. This script doesn't solve this for
> every glibc debugging needs (I suppose it doesn't help at all when
> debugging the initial steps of program loading, for example), but it
> makes it a little easier when debugging test cases, as it automatically
> loads the symbols from them.
>
> Maybe there are more trivial ways to do it, but that's what I have been
> using for some time.
I like this script!
I don't immediately see how this solves the required gdb setup though?
https://sourceware.org/glibc/wiki/Testing/Builds#Required_gdb_setup
Did I miss something?
It looks like the script is *almost* there in terms of functionality
to make it possible to debug single-threaded and multi-threaded programs.
> -- 8< --
> This patch adds a new make rule that generates a helper script for
> debugging glibc test cases. The new script, debugglibc.sh, is similar
> to testrun.sh, in the sense that it allows the execution of the
> specified test case, however, it opens the test case in GDB, setting the
> library path the same way that testrun.sh does. The commands are based
> on the instruction on the wiki page for glibc debugging [1].
>
> By default, the script tells GDB to load the test case for symbol
> information, so that, when a breakpoint is hit, the call stack is
> displayed correctly (instead of printing lots of '??'s). For instance,
> after running 'make' and 'make check', one could do the following:
>
> ./debugglibc.sh -t $PWD/stdlib/tst-strfrom -B strfromf -B strfromd
>
> Reading symbols from /home/gabriel/build/x86_64/glibc/elf/ld.so...
> Breakpoint 1 at 0x1098
> add symbol table from file "/home/gabriel/build/x86_64/glibc/stdlib/tst-strfrom"
>
> Breakpoint 1, 0x00007ffff7fd5098 in _dl_start_user () from /home/gabriel/build/x86_64/glibc/elf/ld.so
> Breakpoint 2 at 0x7ffff7e4dba0: file strfrom-skeleton.c, line 41.
> Breakpoint 3 at 0x7ffff7e4ddd0: file strfrom-skeleton.c, line 41.
>
> Notice that the script will always start GDB with the program running
> and halted at _dl_start_user. So, in order to reach the actual
> breakpoint of interest, one should hit 'c', not 'r':
>
> >>> c
> Continuing.
> Testing in locale: C
>
> Breakpoint 2, strfromf (dest=dest@entry=0x7fffffffe02b "\377", size=5, format=0x405010 "%g", f=12345.3447) at strfrom-skeleton.c:41
> 41 sfile.f._sbf._f._lock = NULL;
>
> An inspect the call stack with 'bt', as usual, and see symbols from both
> the test case and from the libraries themselves:
>
> >>> bt
> #0 strfromf (dest=dest@entry=0x7fffffffe02b "\377", size=5, format=0x405010 "%g", f=12345.3447) at strfrom-skeleton.c:41
> #1 0x000000000040246a in test_f () at tst-strfrom.c:67
> #2 test_locale (locale=locale@entry=0x405046 "C") at tst-strfrom.c:77
> #3 0x000000000040358b in do_test () at tst-strfrom.c:84
> #4 legacy_test_function (argc=<optimized out>, argv=<optimized out>) at ../test-skeleton.c:56
> #5 0x0000000000403b25 in support_test_main (argc=1, argc@entry=2, argv=0x7fffffffe258, argv@entry=0x7fffffffe250, config=config@entry=0x7fffffffe120) at support_test_main.c:350
> #6 0x0000000000402315 in main (argc=argc@entry=2, argv=argv@entry=0x7fffffffe250) at ../support/test-driver.c:168
> #7 0x00007ffff7e35deb in __libc_start_main (main=0x4022e0 <main>, argc=2, argv=0x7fffffffe250, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe248) at ../csu/libc-start.c:308
> #8 0x000000000040234a in _start () at ../sysdeps/x86_64/start.S:120
>
> Tested for x86_64.
>
> [1] https://sourceware.org/glibc/wiki/Debugging/Loader_Debugging
> ---
> Makefile | 159 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 158 insertions(+), 1 deletion(-)
>
> diff --git a/Makefile b/Makefile
> index 67ddd01bfe..d62f5a348c 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -187,7 +187,164 @@ $(common-objpfx)testrun.sh: $(common-objpfx)config.make \
> mv -f $@T $@
> postclean-generated += testrun.sh
>
> -others: $(common-objpfx)testrun.sh
> +define debugglibc
> +#!/bin/bash
> +
> +# Defaults
> +SOURCE_DIR="$$HOME/src/glibc"
> +BUILD_DIR="$$PWD"
> +DIRECT=true
> +SYMBOLSFILE=true
> +
> +unset TESTCASE
> +unset CMD_FILE
> +unset BREAKPOINTS
> +
> +usage()
> +{
> +cat << EOF
> +Usage: $$0 [OPTION]
> +
> + -h, --help
> + Prints this message and leaves.
> +
> + The following options require one argument:
> +
> + -b, --build, --build-dir
> + Absolute path to glibc build directory
> + Defaults to \$$PWD
> + -s, --source, --source-dir
> + Absolute path to glibc source files
> + Defaults to \$$HOME/src/glibc
> + -t, --testcase
> + Absolute path to program to be tested
> + Defaults to \$$PWD/stdlib/tst-strfrom
> + -B, --breakpoint
> + Breakpoints to set at the beginning of the execution
> + (each breakpoint demands its own -B option, e.g. -B foo -B bar)
> +
> + The following options do not take arguments:
> +
> + -I, --no-direct
> + Selects whether to pass the flag --direct to gdb.
> + Required for glibc test cases and not allowed for non-glibc tests.
> + Default behaviour is to pass the flag --direct to gdb.
> + -S, --no-symbols-file
> + Do not tell GDB to load debug symbols from the testcase.
> +EOF
> +}
> +
> +# Parse input options
> +while [[ $$# > 0 ]]
> +do
> + key="$$1"
> + case $$key in
> + -h|--help)
> + usage
> + exit 0
> + ;;
> + -b|--build|--build-dir)
> + BUILD_DIR=$$2
> + shift
> + ;;
> + -s|--source|--source-dir)
> + SOURCE_DIR=$$2
> + shift
> + ;;
> + -t|--testcase)
> + TESTCASE=$$2
> + shift
> + ;;
> + -B|--breakpoint)
> + BREAKPOINTS="$$BREAKPOINTS\n break $$2"
> + shift
> + ;;
> + -I|--no-direct)
> + DIRECT=false
> + ;;
> + -S|--no-symbols-file)
> + SYMBOLSFILE=false
> + ;;
> + *)
> + usage
> + exit 1
> + ;;
> + esac
> + shift
> +done
> +
> +# Update arguments
> +if [ ! -v TESTCASE ]
> +then
> + TESTCASE="$$BUILD_DIR/stdlib/tst-strfrom"
> +fi
> +if [ ! -v CMD_FILE ]
> +then
> + CMD_FILE="$$BUILD_DIR/debugglibc.gdb"
> +fi
> +
> +# Expand direct argument
> +if [ "$$DIRECT" == true ]
> +then
> + DIRECT="--direct"
> +else
> + DIRECT=""
> +fi
> +
> +# Expand symbols loading command
> +if [ "$$SYMBOLSFILE" == true ]
> +then
> + SYMBOLSFILE="add-symbol-file $${TESTCASE}"
> +else
> + SYMBOLSFILE=""
> +fi
> +
> +# GDB commands template
> +template ()
> +{
> +cat <<EOF
> +set environment C -E -x c-header
> +break _dl_start_user
> +__SYMBOLSFILE__
> +run --library-path $(rtld-prefix) \
> +__TESTCASE__ __DIRECT__
> +__BREAKPOINTS__
> +EOF
> +}
> +
> +# Generate the commands file for gdb initialization
> +template | sed \
> + -e "s|__SYMBOLSFILE__|$$SYMBOLSFILE|" \
> + -e "s|__TESTCASE__|$$TESTCASE|" \
> + -e "s|__DIRECT__|$$DIRECT|" \
> + -e "s|__BREAKPOINTS__|$$BREAKPOINTS|" \
> + > $$CMD_FILE
> +
> +echo
> +echo "Debugging glibc..."
> +echo "Build directory : $$BUILD_DIR"
> +echo "Source directory : $$SOURCE_DIR"
> +echo "GLIBC Testcase : $$TESTCASE"
> +echo "GDB Commands : $$CMD_FILE"
> +echo
> +
> +# We need to make sure that gdb is linked against the standalone glibc
> +# so that it picks up the correct nptl_db/libthread_db.so. So that means
> +# invoking gdb using the standalone glibc's linker.
> +$(test-via-rtld-prefix) \
> +/usr/bin/gdb -q -x $${CMD_FILE} -d $${SOURCE_DIR} $${BUILD_DIR}/elf/ld.so
> +endef
> +
> +# This is another handy script for debugging dynamically linked program
> +# against the current libc build for testing.
> +$(common-objpfx)debugglibc.sh: $(common-objpfx)config.make \
> + $(..)Makeconfig $(..)Makefile
> + $(file >$@T,$(debugglibc))
> + chmod a+x $@T
> + mv -f $@T $@
> +postclean-generated += debugglibc.sh debugglibc.gdb
> +
> +others: $(common-objpfx)testrun.sh $(common-objpfx)debugglibc.sh
>
> # Makerules creates a file `stubs' in each subdirectory, which
> # contains `#define __stub_FUNCTION' for each function defined in that
>
Hi, Carlos,
On Tue, 24 Sep 2019, Carlos O'Donell wrote:
>I don't immediately see how this solves the required gdb setup though?
>
>https://sourceware.org/glibc/wiki/Testing/Builds#Required_gdb_setup
>
>Did I miss something?
I think you found a problem with the conversion from my off-tree script
to this auto-generated version.
In the off-tree script, the argument to --library-path was hardcoded and
it included the path to nptl_db, as mentioned in the wiki page I used [1]
(look for occurrences of "${GLIBC}/nptl_db:").
In the auto-generated version, I used the makefile expansion of
"$(rtld-prefix)", which I though gave the same path, but I failed to
notice that nptl_db was missing (nptl is there, but not nptl_db).
It should be easy to complement the argument to --library-path. I'll
check if that actually solves the problem (as the wiki [1] seems to
suggest), then send another version of this patch.
Thanks for your careful review! :)
[1] https://sourceware.org/glibc/wiki/Debugging/Loader_Debugging
On Tue, Sep 24, 2019 at 04:24:35PM -0300, Gabriel F. T. Gomes wrote:
[...]
> +$(test-via-rtld-prefix) \
> +/usr/bin/gdb -q -x $${CMD_FILE} -d $${SOURCE_DIR} $${BUILD_DIR}/elf/ld.so
Please do not hardcode the full path to gdb.
On Tue, 24 Sep 2019, Gabriel F. T. Gomes wrote:
> +# Defaults
> +SOURCE_DIR="$$HOME/src/glibc"
I think this should default to the actual source directory used when
configuring glibc.
> + -t, --testcase
> + Absolute path to program to be tested
> + Defaults to \$$PWD/stdlib/tst-strfrom
I don't think there should be a default for this at all.
(I find use of --enable-hardcoded-path-in-tests convenient for debugging
to avoid the complications this patch deals with.)
On Sep 24 2019, "Gabriel F. T. Gomes" <gabriel@inconstante.net.br> wrote:
> Maybe there are more trivial ways to do it, but that's what I have been
> using for some time.
The easiest way is to configure with --enable-hardcoded-path-in-tests or
setting build-hardcoded-path-in-tests, though that only works for test
programs. If you need to debug one of the included programs there needs
to be a way to inject
`-Wl,-dynamic-linker=$(elf-objpfx)$(rtld-installed-name)
-Wl,-rpath=$(rpath-link)$(patsubst %,:%,$(sysdep-library-path))' into
the link command (setting config-LDFLAGS used to accomplish this, but
that variable no longer exists).
Andreas.
@@ -187,7 +187,164 @@ $(common-objpfx)testrun.sh: $(common-objpfx)config.make \
mv -f $@T $@
postclean-generated += testrun.sh
-others: $(common-objpfx)testrun.sh
+define debugglibc
+#!/bin/bash
+
+# Defaults
+SOURCE_DIR="$$HOME/src/glibc"
+BUILD_DIR="$$PWD"
+DIRECT=true
+SYMBOLSFILE=true
+
+unset TESTCASE
+unset CMD_FILE
+unset BREAKPOINTS
+
+usage()
+{
+cat << EOF
+Usage: $$0 [OPTION]
+
+ -h, --help
+ Prints this message and leaves.
+
+ The following options require one argument:
+
+ -b, --build, --build-dir
+ Absolute path to glibc build directory
+ Defaults to \$$PWD
+ -s, --source, --source-dir
+ Absolute path to glibc source files
+ Defaults to \$$HOME/src/glibc
+ -t, --testcase
+ Absolute path to program to be tested
+ Defaults to \$$PWD/stdlib/tst-strfrom
+ -B, --breakpoint
+ Breakpoints to set at the beginning of the execution
+ (each breakpoint demands its own -B option, e.g. -B foo -B bar)
+
+ The following options do not take arguments:
+
+ -I, --no-direct
+ Selects whether to pass the flag --direct to gdb.
+ Required for glibc test cases and not allowed for non-glibc tests.
+ Default behaviour is to pass the flag --direct to gdb.
+ -S, --no-symbols-file
+ Do not tell GDB to load debug symbols from the testcase.
+EOF
+}
+
+# Parse input options
+while [[ $$# > 0 ]]
+do
+ key="$$1"
+ case $$key in
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ -b|--build|--build-dir)
+ BUILD_DIR=$$2
+ shift
+ ;;
+ -s|--source|--source-dir)
+ SOURCE_DIR=$$2
+ shift
+ ;;
+ -t|--testcase)
+ TESTCASE=$$2
+ shift
+ ;;
+ -B|--breakpoint)
+ BREAKPOINTS="$$BREAKPOINTS\n break $$2"
+ shift
+ ;;
+ -I|--no-direct)
+ DIRECT=false
+ ;;
+ -S|--no-symbols-file)
+ SYMBOLSFILE=false
+ ;;
+ *)
+ usage
+ exit 1
+ ;;
+ esac
+ shift
+done
+
+# Update arguments
+if [ ! -v TESTCASE ]
+then
+ TESTCASE="$$BUILD_DIR/stdlib/tst-strfrom"
+fi
+if [ ! -v CMD_FILE ]
+then
+ CMD_FILE="$$BUILD_DIR/debugglibc.gdb"
+fi
+
+# Expand direct argument
+if [ "$$DIRECT" == true ]
+then
+ DIRECT="--direct"
+else
+ DIRECT=""
+fi
+
+# Expand symbols loading command
+if [ "$$SYMBOLSFILE" == true ]
+then
+ SYMBOLSFILE="add-symbol-file $${TESTCASE}"
+else
+ SYMBOLSFILE=""
+fi
+
+# GDB commands template
+template ()
+{
+cat <<EOF
+set environment C -E -x c-header
+break _dl_start_user
+__SYMBOLSFILE__
+run --library-path $(rtld-prefix) \
+__TESTCASE__ __DIRECT__
+__BREAKPOINTS__
+EOF
+}
+
+# Generate the commands file for gdb initialization
+template | sed \
+ -e "s|__SYMBOLSFILE__|$$SYMBOLSFILE|" \
+ -e "s|__TESTCASE__|$$TESTCASE|" \
+ -e "s|__DIRECT__|$$DIRECT|" \
+ -e "s|__BREAKPOINTS__|$$BREAKPOINTS|" \
+ > $$CMD_FILE
+
+echo
+echo "Debugging glibc..."
+echo "Build directory : $$BUILD_DIR"
+echo "Source directory : $$SOURCE_DIR"
+echo "GLIBC Testcase : $$TESTCASE"
+echo "GDB Commands : $$CMD_FILE"
+echo
+
+# We need to make sure that gdb is linked against the standalone glibc
+# so that it picks up the correct nptl_db/libthread_db.so. So that means
+# invoking gdb using the standalone glibc's linker.
+$(test-via-rtld-prefix) \
+/usr/bin/gdb -q -x $${CMD_FILE} -d $${SOURCE_DIR} $${BUILD_DIR}/elf/ld.so
+endef
+
+# This is another handy script for debugging dynamically linked program
+# against the current libc build for testing.
+$(common-objpfx)debugglibc.sh: $(common-objpfx)config.make \
+ $(..)Makeconfig $(..)Makefile
+ $(file >$@T,$(debugglibc))
+ chmod a+x $@T
+ mv -f $@T $@
+postclean-generated += debugglibc.sh debugglibc.gdb
+
+others: $(common-objpfx)testrun.sh $(common-objpfx)debugglibc.sh
# Makerules creates a file `stubs' in each subdirectory, which
# contains `#define __stub_FUNCTION' for each function defined in that