[00/14] elfutils: DWARF package (.dwp) file support

  From: Omar Sandoval <osandov@fb.com>

Hi,

This patch series adds support for DWARF package files to libdw and the
elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1]
and the format standardized in DWARF 5 (section 7.3.5 "DWARF Package
Files"). It supports both automatically opening the .dwp file for a
skeleton unit and examining the .dwp file alone, just like for .dwo
files.

Patch 1 is a trivial cleanup I found while developing this. Patches 2-6
are fixes for unrelated bugs in split DWARF and/or DWARF 5 that I
encountered while adding test cases for this series. I didn't add test
cases explicitly for those fixes because they are covered by the dwp
tests later in the series.

The actual dwp support is in patches 7-10 and 13, including test cases.
Patches 11 and 12 enable testing macro information in dwp files. Patch
14 adds support and tests for an LLVM extension to the dwp format.

More details in the individual commit messages.

Thanks!
Omar

P.S. drgn's dwp branch [2] demonstrates how the new
dwarf_cu_dwp_section_info function will be used. With this patch series,
drgn's test suite passes against a Linux kernel build using .dwp.

1: https://gcc.gnu.org/wiki/DebugFissionDWP
2: https://github.com/osandov/drgn/tree/dwp

Omar Sandoval (14):
  libdw: Make try_split_file static
  libdw: Handle split DWARF in dwarf_entrypc
  libdw: Handle DW_AT_ranges in split DWARF 5 skeleton in dwarf_ranges
  libdw: Handle other string forms in dwarf_macro_param2
  libdw: Fix dwarf_macro_getsrcfiles for DWARF 5
  libdw: Handle split DWARF in dwarf_macro_getsrcfiles
  libdw: Recognize .debug_[ct]u_index sections in dwarf_elf_begin
  libdw: Parse DWARF package file index sections
  libdw, libdwfl: Save original path of ELF file
  libdw: Try .dwp file in __libdw_find_split_unit()
  tests: Handle DW_MACRO_{define,undef}_{strx,sup} in dwarf-getmacros
  tests: Optionally dump all units in dwarf-getmacros
  libdw: Apply DWARF package file section offsets where appropriate
  libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file
    indexes

 libdw/ChangeLog                               |   72 +
 libdw/Makefile.am                             |    2 +-
 libdw/dwarf_begin_elf.c                       |   88 +-
 libdw/dwarf_cu_dwp_section_info.c             |  529 ++++++
 libdw/dwarf_end.c                             |   27 +-
 libdw/dwarf_entrypc.c                         |   12 +-
 libdw/dwarf_error.c                           |    1 +
 libdw/dwarf_getlocation.c                     |    6 +
 libdw/dwarf_getmacros.c                       |   54 +-
 libdw/dwarf_macro_getsrcfiles.c               |    8 +-
 libdw/dwarf_macro_param2.c                    |   21 +-
 libdw/dwarf_ranges.c                          |    4 +-
 libdw/libdw.h                                 |   23 +
 libdw/libdw.map                               |    5 +
 libdw/libdwP.h                                |  112 +-
 libdw/libdw_find_split_unit.c                 |   77 +-
 libdw/libdw_findcu.c                          |    8 +
 libdwfl/ChangeLog                             |    9 +
 libdwfl/dwfl_module.c                         |    2 +-
 libdwfl/dwfl_module_getdwarf.c                |   11 +-
 libdwfl/libdwflP.h                            |    2 +-
 libdwfl/offline.c                             |    4 +-
 tests/.gitignore                              |    1 +
 tests/ChangeLog                               |   36 +
 tests/Makefile.am                             |   15 +-
 tests/cu-dwp-section-info.c                   |   74 +
 tests/dwarf-getmacros.c                       |   55 +-
 tests/run-all-dwarf-ranges.sh                 |  114 ++
 tests/run-cu-dwp-section-info.sh              |  168 ++
 tests/run-dwarf-getmacros.sh                  | 1412 +++++++++++++++++
 tests/run-get-units-split.sh                  |   18 +
 tests/run-large-elf-file.sh                   |  174 ++
 tests/run-varlocs.sh                          |  126 +-
 tests/testfile-dwp-4-cu-index-overflow.bz2    |  Bin 0 -> 4490 bytes
 .../testfile-dwp-4-cu-index-overflow.dwp.bz2  |  Bin 0 -> 5584 bytes
 tests/testfile-dwp-4-strict.bz2               |  Bin 0 -> 4169 bytes
 tests/testfile-dwp-4-strict.dwp.bz2           |  Bin 0 -> 6871 bytes
 tests/testfile-dwp-4.bz2                      |  Bin 0 -> 4194 bytes
 tests/testfile-dwp-4.dwp.bz2                  |  Bin 0 -> 10098 bytes
 tests/testfile-dwp-5-cu-index-overflow.bz2    |  Bin 0 -> 4544 bytes
 .../testfile-dwp-5-cu-index-overflow.dwp.bz2  |  Bin 0 -> 5790 bytes
 tests/testfile-dwp-5.bz2                      |  Bin 0 -> 4223 bytes
 tests/testfile-dwp-5.dwp.bz2                  |  Bin 0 -> 10313 bytes
 tests/testfile-dwp-cu-index-overflow.source   |   86 +
 tests/testfile-dwp.source                     |  102 ++
 45 files changed, 3337 insertions(+), 121 deletions(-)
 create mode 100644 libdw/dwarf_cu_dwp_section_info.c
 create mode 100644 tests/cu-dwp-section-info.c
 create mode 100755 tests/run-cu-dwp-section-info.sh
 create mode 100755 tests/testfile-dwp-4-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-4-cu-index-overflow.dwp.bz2
 create mode 100755 tests/testfile-dwp-4-strict.bz2
 create mode 100644 tests/testfile-dwp-4-strict.dwp.bz2
 create mode 100755 tests/testfile-dwp-4.bz2
 create mode 100644 tests/testfile-dwp-4.dwp.bz2
 create mode 100755 tests/testfile-dwp-5-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-5-cu-index-overflow.dwp.bz2
 create mode 100755 tests/testfile-dwp-5.bz2
 create mode 100644 tests/testfile-dwp-5.dwp.bz2
 create mode 100644 tests/testfile-dwp-cu-index-overflow.source
 create mode 100644 tests/testfile-dwp.source
  

Hi -

> This patch series adds support for DWARF package files to libdw and the
> elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1]
> and the format standardized in DWARF 5 (section 7.3.5 "DWARF Package
> Files"). It supports both automatically opening the .dwp file for a
> skeleton unit and examining the .dwp file alone, just like for .dwo
> files. [...]

Does this work have any implications for debuginfod?

- FChE
  

On Wed, Sep 27, 2023 at 03:20:40PM -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> > This patch series adds support for DWARF package files to libdw and the
> > elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1]
> > and the format standardized in DWARF 5 (section 7.3.5 "DWARF Package
> > Files"). It supports both automatically opening the .dwp file for a
> > skeleton unit and examining the .dwp file alone, just like for .dwo
> > files. [...]
> 
> Does this work have any implications for debuginfod?

Until anyone using debuginfod wants to use DWARF package files, not
really. The problems that split DWARF solves aren't super relevant to
Linux distributions, so I don't expect that will happen any time soon.

If that changes, then debuginfod could add a /$build_id/debuginfo.dwp
endpoint. Note that dwp files don't contain a build ID note. They are
associated with the main debug file only by name. To quote the DWARF 5
standard: "a package file is typically placed in the same directory as
the application, and is given the same name with a '.dwp' extension."
  

Hi Omar,

On Wed, Sep 27, 2023 at 11:20:49AM -0700, Omar Sandoval wrote:
> This patch series adds support for DWARF package files to libdw and the
> elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1]
> and the format standardized in DWARF 5 (section 7.3.5 "DWARF Package
> Files"). It supports both automatically opening the .dwp file for a
> skeleton unit and examining the .dwp file alone, just like for .dwo
> files.
> 
> Patch 1 is a trivial cleanup I found while developing this. Patches 2-6
> are fixes for unrelated bugs in split DWARF and/or DWARF 5 that I
> encountered while adding test cases for this series. I didn't add test
> cases explicitly for those fixes because they are covered by the dwp
> tests later in the series.
> 
> The actual dwp support is in patches 7-10 and 13, including test cases.
> Patches 11 and 12 enable testing macro information in dwp files. Patch
> 14 adds support and tests for an LLVM extension to the dwp format.

Thanks, I went through patches 1 to 6 and applied them all.

Sorry for the buildbot failures, neither the centos7 x86_64 one, nor
the gentoo sparc one is obviously caused by your patches. And both
were spurious, later builds are green again.

I'll go through the rest later. But there are more unreviewed patches,
so it might take a bit.

Thanks,

Mark
  

Hi Omar,

On Wed, Sep 27, 2023 at 11:20:49AM -0700, Omar Sandoval wrote:
> This patch series adds support for DWARF package files to libdw and the
> elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1]
> and the format standardized in DWARF 5 (section 7.3.5 "DWARF Package
> Files"). It supports both automatically opening the .dwp file for a
> skeleton unit and examining the .dwp file alone, just like for .dwo
> files.

I had hoped to review and apply all this for the 0.190 release
tomorrow, but wasn't fast enough. In general it looks very good and
most has been applied as is. Except for the last patch I don't have
real concerns, just a few questions (see the specific reviews).

> Patch 1 is a trivial cleanup I found while developing this. Patches 2-6
> are fixes for unrelated bugs in split DWARF and/or DWARF 5 that I
> encountered while adding test cases for this series.

These have all been applied some time ago.

> The actual dwp support is in patches 7-10 and 13, including test cases.

7 and 9 have been applied already. 8 has some comments/nitpicks, I can
probably easily be convinced to accept it. 10 looks good, but depends
on 8, some questions, specificly about handling split type units. 13
looks good, but depends on 8 and 10.

> Patches 11 and 12 enable testing macro information in dwp files.

Both applied.

> Patch 14 adds support and tests for an LLVM extension to the dwp
> format.

I have to think a bit more on this.

Cheers,

Mark
  

On Fri, Nov 03, 2023 at 12:05:57AM +0100, Mark Wielaard wrote:
> Hi Omar,
> 
> On Wed, Sep 27, 2023 at 11:20:49AM -0700, Omar Sandoval wrote:
> > This patch series adds support for DWARF package files to libdw and the
> > elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1]
> > and the format standardized in DWARF 5 (section 7.3.5 "DWARF Package
> > Files"). It supports both automatically opening the .dwp file for a
> > skeleton unit and examining the .dwp file alone, just like for .dwo
> > files.
> 
> I had hoped to review and apply all this for the 0.190 release
> tomorrow, but wasn't fast enough.

No problem, I'll be happy to see it in 0.191 instead.

> In general it looks very good and
> most has been applied as is. Except for the last patch I don't have
> real concerns, just a few questions (see the specific reviews).
> 
> > Patch 1 is a trivial cleanup I found while developing this. Patches 2-6
> > are fixes for unrelated bugs in split DWARF and/or DWARF 5 that I
> > encountered while adding test cases for this series.
> 
> These have all been applied some time ago.
> 
> > The actual dwp support is in patches 7-10 and 13, including test cases.
> 
> 7 and 9 have been applied already. 8 has some comments/nitpicks, I can
> probably easily be convinced to accept it. 10 looks good, but depends
> on 8, some questions, specificly about handling split type units. 13
> looks good, but depends on 8 and 10.

I replied to patch 8 with how drgn wants to use the new interface. I
also replied with a couple of questions about how you'd like me to split
up patch 8 and handle errors in a couple of the existing codepaths.

> > Patches 11 and 12 enable testing macro information in dwp files.
> 
> Both applied.
> 
> > Patch 14 adds support and tests for an LLVM extension to the dwp
> > format.
> 
> I have to think a bit more on this.

Yeah, the whole situation is very unfortunate. I tried to keep the
workaround as unintrusive as possible, so it wouldn't be the end of the
world if we had to carry our own patch for it. Of course, since LLVM
supports it upstream, I'd definitely prefer for elfutils to support it
upstream, too.

I'll rebase my remaining patches and address the comments so far while I
wait for your responses.

Thanks!