[v2,0/4] elfutils: DWARF package (.dwp) file support

  From: Omar Sandoval <osandov@fb.com>

Hi,

This is version 2 of my patch series adding support for DWARF package
files to libdw and the elfutils tools. Version 1 is here [1].

Patches 1-3 add the main implementation and tests for dwp files.

Most of this support is internal to libdw, but patch 1 adds a new public
function, dwarf_cu_dwp_section_info. drgn's dwp branch [2] demonstrates
how that function will be used. Also see [3] for more context on why
drgn needs this.

Patch 4 adds support and tests for an LLVM extension to the dwp format.
The "extension" is ugly because of an oversight in the design of the
format that LLVM had to make the best of, but unfortunately it's
necessary for a lot of our use cases.

With this patch series, drgn's test suite passes against a Linux kernel
build using .dwp.

Changes from v1:

* Rebased on main and dropped patches that were already merged.
* Moved ChangeLog entries to commit messages.
* Updated version in libdw.map to 0.191.
* Moved DW_SECT_TYPES definition to dwarf.h.
* Added copyright years.
* Added error handling for dwarf_cu_dwp_section_info calls in
  str_offsets_base_off, __libdw_cu_ranges_base, and __libdw_cu_locs_base
* Changed memset initialization of index->sections to an explicit
  loop.
* Added comment explaining __libdw_link_skel_split change.

There were a couple of things that were mentioned in review that I
didn't change:

* I kept dwarf_cu_dwp_section_info in patch 1 instead of separating it
  into its own patch so that I could test the dwp index implementation
  in the same commit that I introduced it in.
* I didn't make try_dwp_file return an error since try_split_file that
  it's based on doesn't either.

Thanks!
Omar

1: https://sourceware.org/pipermail/elfutils-devel/2023q3/006410.html
2: https://github.com/osandov/drgn/tree/dwp
3: https://sourceware.org/pipermail/elfutils-devel/2023q4/006630.html

Omar Sandoval (4):
  libdw: Parse DWARF package file index sections
  libdw: Try .dwp file in __libdw_find_split_unit()
  libdw: Apply DWARF package file section offsets where appropriate
  libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file
    indexes

 libdw/Makefile.am                             |    2 +-
 libdw/dwarf.h                                 |    2 +-
 libdw/dwarf_begin_elf.c                       |    1 +
 libdw/dwarf_cu_dwp_section_info.c             |  531 +++++++
 libdw/dwarf_end.c                             |   24 +-
 libdw/dwarf_error.c                           |    1 +
 libdw/dwarf_getlocation.c                     |    6 +
 libdw/dwarf_getmacros.c                       |   26 +-
 libdw/libdw.h                                 |   23 +
 libdw/libdw.map                               |    5 +
 libdw/libdwP.h                                |  101 +-
 libdw/libdw_find_split_unit.c                 |   75 +-
 libdw/libdw_findcu.c                          |    8 +
 tests/.gitignore                              |    1 +
 tests/Makefile.am                             |   15 +-
 tests/cu-dwp-section-info.c                   |   73 +
 tests/run-all-dwarf-ranges.sh                 |  114 ++
 tests/run-cu-dwp-section-info.sh              |  168 ++
 tests/run-dwarf-getmacros.sh                  | 1412 +++++++++++++++++
 tests/run-get-units-split.sh                  |   18 +
 tests/run-large-elf-file.sh                   |  174 ++
 tests/run-varlocs.sh                          |  112 ++
 tests/testfile-dwp-4-cu-index-overflow.bz2    |  Bin 0 -> 4490 bytes
 .../testfile-dwp-4-cu-index-overflow.dwp.bz2  |  Bin 0 -> 5584 bytes
 tests/testfile-dwp-4-strict.bz2               |  Bin 0 -> 4169 bytes
 tests/testfile-dwp-4-strict.dwp.bz2           |  Bin 0 -> 6871 bytes
 tests/testfile-dwp-4.bz2                      |  Bin 0 -> 4194 bytes
 tests/testfile-dwp-4.dwp.bz2                  |  Bin 0 -> 10098 bytes
 tests/testfile-dwp-5-cu-index-overflow.bz2    |  Bin 0 -> 4544 bytes
 .../testfile-dwp-5-cu-index-overflow.dwp.bz2  |  Bin 0 -> 5790 bytes
 tests/testfile-dwp-5.bz2                      |  Bin 0 -> 4223 bytes
 tests/testfile-dwp-5.dwp.bz2                  |  Bin 0 -> 10313 bytes
 tests/testfile-dwp-cu-index-overflow.source   |   86 +
 tests/testfile-dwp.source                     |  102 ++
 34 files changed, 3051 insertions(+), 29 deletions(-)
 create mode 100644 libdw/dwarf_cu_dwp_section_info.c
 create mode 100644 tests/cu-dwp-section-info.c
 create mode 100755 tests/run-cu-dwp-section-info.sh
 create mode 100755 tests/testfile-dwp-4-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-4-cu-index-overflow.dwp.bz2
 create mode 100755 tests/testfile-dwp-4-strict.bz2
 create mode 100644 tests/testfile-dwp-4-strict.dwp.bz2
 create mode 100755 tests/testfile-dwp-4.bz2
 create mode 100644 tests/testfile-dwp-4.dwp.bz2
 create mode 100755 tests/testfile-dwp-5-cu-index-overflow.bz2
 create mode 100644 tests/testfile-dwp-5-cu-index-overflow.dwp.bz2
 create mode 100755 tests/testfile-dwp-5.bz2
 create mode 100644 tests/testfile-dwp-5.dwp.bz2
 create mode 100644 tests/testfile-dwp-cu-index-overflow.source
 create mode 100644 tests/testfile-dwp.source
  

On Wed, Dec 06, 2023 at 01:22:15AM -0800, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
> 
> Hi,
> 
> This is version 2 of my patch series adding support for DWARF package
> files to libdw and the elfutils tools. Version 1 is here [1].
> 
> Patches 1-3 add the main implementation and tests for dwp files.
> 
> Most of this support is internal to libdw, but patch 1 adds a new public
> function, dwarf_cu_dwp_section_info. drgn's dwp branch [2] demonstrates
> how that function will be used. Also see [3] for more context on why
> drgn needs this.
> 
> Patch 4 adds support and tests for an LLVM extension to the dwp format.
> The "extension" is ugly because of an oversight in the design of the
> format that LLVM had to make the best of, but unfortunately it's
> necessary for a lot of our use cases.
> 
> With this patch series, drgn's test suite passes against a Linux kernel
> build using .dwp.
> 
> Changes from v1:
> 
> * Rebased on main and dropped patches that were already merged.
> * Moved ChangeLog entries to commit messages.
> * Updated version in libdw.map to 0.191.
> * Moved DW_SECT_TYPES definition to dwarf.h.
> * Added copyright years.
> * Added error handling for dwarf_cu_dwp_section_info calls in
>   str_offsets_base_off, __libdw_cu_ranges_base, and __libdw_cu_locs_base
> * Changed memset initialization of index->sections to an explicit
>   loop.
> * Added comment explaining __libdw_link_skel_split change.
> 
> There were a couple of things that were mentioned in review that I
> didn't change:
> 
> * I kept dwarf_cu_dwp_section_info in patch 1 instead of separating it
>   into its own patch so that I could test the dwp index implementation
>   in the same commit that I introduced it in.
> * I didn't make try_dwp_file return an error since try_split_file that
>   it's based on doesn't either.
> 
> Thanks!
> Omar
> 
> 1: https://sourceware.org/pipermail/elfutils-devel/2023q3/006410.html
> 2: https://github.com/osandov/drgn/tree/dwp
> 3: https://sourceware.org/pipermail/elfutils-devel/2023q4/006630.html
> 
> Omar Sandoval (4):
>   libdw: Parse DWARF package file index sections
>   libdw: Try .dwp file in __libdw_find_split_unit()
>   libdw: Apply DWARF package file section offsets where appropriate
>   libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file
>     indexes

Ping, and happy new year :)