[v4,0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

  From: Omar Sandoval <osandov@fb.com>

This is v4 of my patch series optimizing debuginfod for kernel
debuginfo.  v1 is here [1], v2 is here [2], v3 is here [3].  The only
changes from v3 in this version are fixing a bogus maybe-uninitialized
error on the Debian build and adding the new test files to EXTRA_DIST so
that make distcheck passes.

Thanks,
Omar

1: https://sourceware.org/pipermail/elfutils-devel/2024q3/007191.html
2: https://sourceware.org/pipermail/elfutils-devel/2024q3/007208.html
3: https://sourceware.org/pipermail/elfutils-devel/2024q3/007243.html

Omar Sandoval (7):
  debuginfod: fix skipping <built-in> source file
  tests/run-debuginfod-fd-prefetch-caches.sh: disable fdcache limit
    check
  debuginfod: factor out common code for responding from an archive
  debugifod: add new table and views for seekable archives
  debuginfod: optimize extraction from seekable xz archives
  debuginfod: populate _r_seekable on scan
  debuginfod: populate _r_seekable on request

 configure.ac                                  |   5 +
 debuginfod/Makefile.am                        |   2 +-
 debuginfod/debuginfod.cxx                     | 923 +++++++++++++++---
 tests/Makefile.am                             |  13 +-
 ...pressme-seekable-xz-dbgsym_1.0-1_amd64.deb | Bin 0 -> 6288 bytes
 ...compressme-seekable-xz_1.0-1.debian.tar.xz | Bin 0 -> 1440 bytes
 .../compressme-seekable-xz_1.0-1.dsc          |  19 +
 .../compressme-seekable-xz_1.0-1_amd64.deb    | Bin 0 -> 6208 bytes
 .../compressme-seekable-xz_1.0.orig.tar.xz    | Bin 0 -> 7160 bytes
 .../compressme-seekable-xz-1.0-1.src.rpm      | Bin 0 -> 15880 bytes
 .../compressme-seekable-xz-1.0-1.x86_64.rpm   | Bin 0 -> 31873 bytes
 ...sme-seekable-xz-debuginfo-1.0-1.x86_64.rpm | Bin 0 -> 21917 bytes
 ...e-seekable-xz-debugsource-1.0-1.x86_64.rpm | Bin 0 -> 7961 bytes
 tests/run-debuginfod-archive-groom.sh         |   2 +-
 tests/run-debuginfod-extraction.sh            |   2 +-
 tests/run-debuginfod-fd-prefetch-caches.sh    |   4 +
 tests/run-debuginfod-seekable.sh              | 186 ++++
 17 files changed, 1011 insertions(+), 145 deletions(-)
 create mode 100644 tests/debuginfod-debs/seekable-xz/compressme-seekable-xz-dbgsym_1.0-1_amd64.deb
 create mode 100644 tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.debian.tar.xz
 create mode 100644 tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1.dsc
 create mode 100644 tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0-1_amd64.deb
 create mode 100644 tests/debuginfod-debs/seekable-xz/compressme-seekable-xz_1.0.orig.tar.xz
 create mode 100644 tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.src.rpm
 create mode 100644 tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-1.0-1.x86_64.rpm
 create mode 100644 tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debuginfo-1.0-1.x86_64.rpm
 create mode 100644 tests/debuginfod-rpms/seekable-xz/compressme-seekable-xz-debugsource-1.0-1.x86_64.rpm
 create mode 100755 tests/run-debuginfod-seekable.sh
  

Hi Omar,

On Fri, Jul 19, 2024 at 2:24 PM Omar Sandoval <osandov@osandov.com> wrote:
>
> From: Omar Sandoval <osandov@fb.com>
>
> This is v4 of my patch series optimizing debuginfod for kernel
> debuginfo.  v1 is here [1], v2 is here [2], v3 is here [3].  The only
> changes from v3 in this version are fixing a bogus maybe-uninitialized
> error on the Debian build and adding the new test files to EXTRA_DIST so
> that make distcheck passes.
>
> Thanks,
> Omar
>
> 1: https://sourceware.org/pipermail/elfutils-devel/2024q3/007191.html
> 2: https://sourceware.org/pipermail/elfutils-devel/2024q3/007208.html
> 3: https://sourceware.org/pipermail/elfutils-devel/2024q3/007243.html

Thanks for working on these patches.  I ran v4 on the sourceware buildbots.
The new testcase fails on debian-ppc64 [1], debian-i386 [2] and
fedora-s390x [3].  There was a 4th failing buildbot but AFAICT the failure
is unrelated to your patches.

Do you mind taking a look at this?  The patches otherwise LGTM and I'll
merge them once this is passing on the buildbots.

Aaron

[1] https://builder.sourceware.org/buildbot/#/builders/214/builds/190
[2] https://builder.sourceware.org/buildbot/#/builders/210/builds/194
[3] https://builder.sourceware.org/buildbot/#/builders/211/builds/185
  

On Tue, Jul 23, 2024 at 05:47:50PM -0400, Aaron Merey wrote:
> Hi Omar,
> 
> On Fri, Jul 19, 2024 at 2:24 PM Omar Sandoval <osandov@osandov.com> wrote:
> >
> > From: Omar Sandoval <osandov@fb.com>
> >
> > This is v4 of my patch series optimizing debuginfod for kernel
> > debuginfo.  v1 is here [1], v2 is here [2], v3 is here [3].  The only
> > changes from v3 in this version are fixing a bogus maybe-uninitialized
> > error on the Debian build and adding the new test files to EXTRA_DIST so
> > that make distcheck passes.
> >
> > Thanks,
> > Omar
> >
> > 1: https://sourceware.org/pipermail/elfutils-devel/2024q3/007191.html
> > 2: https://sourceware.org/pipermail/elfutils-devel/2024q3/007208.html
> > 3: https://sourceware.org/pipermail/elfutils-devel/2024q3/007243.html
> 
> Thanks for working on these patches.  I ran v4 on the sourceware buildbots.
> The new testcase fails on debian-ppc64 [1], debian-i386 [2] and
> fedora-s390x [3].  There was a 4th failing buildbot but AFAICT the failure
> is unrelated to your patches.
> 
> Do you mind taking a look at this?  The patches otherwise LGTM and I'll
> merge them once this is passing on the buildbots.

Thanks for the test run, and sorry for the churn.  It looks like these
are all failing with "error: no fdcache hits" because every addition to
the fdcache results in "fdcache emergency flush for filling tmpdir".  I
see that run-debuginfod-fd-prefetch-caches.sh has a workaround for this
which I'll copy.

Thanks,
Omar