[00/17] Support self comparison of vmlinux & modules using BTF/CTF

  From: Dodji Seketeli <dodji@redhat.com>

Hello,

In libabigail lingo, self comparing a binary means building an ABI
corpus A for the binary, serialize it into ABIXML, read the ABIXML
back into a corpus A' and compare A' against A.  As A and A' should be
equivalent, the comparison should yield the empty set.

That self comparison is done using:

    abidw --abidiff <binary>


But then in the case of a Linux kernel build tree (vmlinux + kernel
modules), self comparison is dong using:

    abidw --abidiff --linux-tree --vmlinux </path/to/vmlinux> </path/to/kernel/modules>

If we want to use either the BTF or the CTF front-end for self
comparison, the command to use is:

    abidw --abidiff --{btf,ctf} --linux-tree --vmlinux </path/to/vmlinux> </path/to/kernel/modules>

It turns out the later command doesn't work.  Neither with the BTF not
with the CTF front-end.

This patch series intends to make self comparison of the linux kernel
work for the BTF and CTF front-ends.

The first two patches are fixes for generic issues that are
independent from the front-ends.

The third patch fixes the BTF front-end to make it take kernel modules
into account for its ABI analysis.

The fourth patch is a pre-requisite for allowing the CTF front end to
add translation units to a given ABI corpus.

Patches 5 to 7 are fixes specific to the CTF front-end.

Patches 8 to 12 are fixes specific to the ABIXML front-end.

Now that all front-ends can build a proper corpus group representing
the ABI of the union of vmlinux and its modules (a corpus
group), patches 13 to 15 are fixes to the general IR as well as the
diff IR.

Patch 16 adds support for the --abidiff option of abidw for Linux
kernel binaries.  Yup, weirdly enough, that is otherwise still not
supported to this day.

The last patch adds testing for the self-comparison of the Linux
kernel, validating all the patches above.  This is done using an
external Git repository that holds the tests data that would be too
big to be hosted in the current repository.

This patch set has been tested using the Sourceware CI builders.  I am
thus applying it to the mainline.

Dodji Seketeli (17):
  ir: Fix a potential crash in canonicalize_types
  elf-based-reader: Clean up logic of elf_based_reader::read_and_add_corpus_to_group
  tools-utils,btf-reader: Take modules into account for corpus group
  corpus: Support adding translation units with empty path
  ctf-reader: Do not set data member offsets for unions
  ctf-reader: During re-initialization, only clear canonicalize-able types
  ctf-reader: Fix analyzing single kernel binaries
  reader: Fix corpus group reading
  reader: Simplify type canonicalization invocation
  reader: Simplify logic of get_or_read_and_add_translation_unit
  reader: Fix building of void and void pointer types
  reader: Fix building of variadic parameter type
  ir: Don't strip typedefs from parms and return type when comparing fns
  ir: Rename integral_type into real_type
  ir,comparison,default-reporter: Consider sub-ranges in array diffs
  abidw: Support the --abidiff option for Linux Kernel trees
  configure: Support the optional 'big-tests' sub-directory

 .gitmodules                                   |     3 +
 CONTRIBUTING                                  |   159 +-
 Makefile.am                                   |     9 +-
 big-tests                                     |     1 +
 configure.ac                                  |    32 +
 include/abg-comparison.h                      |    21 +-
 include/abg-corpus.h                          |     2 +
 include/abg-fwd.h                             |     6 +
 src/abg-btf-reader.cc                         |   275 +-
 src/abg-comp-filter.cc                        |    44 +-
 src/abg-comparison-priv.h                     |     7 +-
 src/abg-comparison.cc                         |   111 +-
 src/abg-corpus.cc                             |    24 +-
 src/abg-ctf-reader.cc                         |    85 +-
 src/abg-default-reporter.cc                   |    18 +
 src/abg-dwarf-reader.cc                       |    12 +-
 src/abg-elf-based-reader.cc                   |     2 +-
 src/abg-fe-iface.cc                           |     1 -
 src/abg-ir-priv.h                             |    59 +-
 src/abg-ir.cc                                 |   331 +-
 src/abg-reader.cc                             |   221 +-
 src/abg-reporter-priv.cc                      |    27 +-
 src/abg-tools-utils.cc                        |     6 +-
 tests/Makefile.am                             |     2 +-
 .../PR30329/PR30329-report-1.txt              |     4 +-
 .../libsdl/libsdl-1.2.60-1.2.64-report.txt    |     2 +-
 .../reported/PR31513-reported-report-1.txt    |     2 +-
 .../reported/PR31513-reported-report-2.txt    |     2 +-
 .../test1-ada-subrange-report-1.txt           |     4 +-
 .../test1-ada-subrange-report-2.txt           |     4 +-
 .../test2-ada-subrange-redundant-report-1.txt |     6 +-
 .../test2-ada-subrange-redundant-report-2.txt |     6 +-
 .../test-allow-type-array-v0--v1-report-1.txt |     2 +
 .../test-allow-type-array-v0--v3-report-1.txt |     2 +
 .../data/test-abidiff/test-PR27985-report.txt |     2 +
 .../data/test-annotate/test14-pr18893.so.abi  |   267 +-
 .../data/test-annotate/test15-pr18892.so.abi  |  4703 +++---
 .../data/test-annotate/test17-pr19027.so.abi  |  7401 ++++-----
 ...19-pr19023-libtcmalloc_and_profiler.so.abi |  5197 +++---
 tests/data/test-diff-dwarf/test10-report.txt  |     2 +
 tests/data/test-diff-dwarf/test11-report.txt  |     4 +
 tests/data/test-diff-dwarf/test2-report.txt   |    11 +
 ...t35-pr19173-libfoo-long-clang-report-0.txt |     2 +
 ...est35-pr19173-libfoo-long-gcc-report-0.txt |     2 +
 .../PR24430-fold-qualified-array-report-0.txt |     3 +
 .../test-PR29811-0-report-1.txt               |     4 +
 tests/data/test-diff-filter/test3-report.txt  |     2 +-
 .../test30-pr18904-rvalueref-report0.txt      |     2 +
 .../test30-pr18904-rvalueref-report1.txt      |     2 +
 .../test30-pr18904-rvalueref-report2.txt      |     2 +
 .../test35-pr18754-no-added-syms-report-0.txt |     2 +
 .../gmp-6.x.x86_64-report-0.txt               |     2 +-
 ...libICE-1.0.9-2.el7.x86_64.rpm-report-0.txt |     4 +-
 .../nss-3.23.0-1.0.fc23.x86_64-report-0.txt   |     2 +
 ...bb-4.3-3.20141204.fc23.x86_64-report-0.txt |     2 +
 ...bb-4.3-3.20141204.fc23.x86_64-report-1.txt |     2 +
 ...has-data-member-inserted-at-1-report-1.txt |     2 +
 ...-array-data-member-conversion-report-2.txt |     2 +
 .../test-diff-suppr/test45-abi-report-1.txt   |     4 +-
 .../data/test-read-dwarf/PR25007-sdhci.ko.abi |  4202 ++---
 .../test-read-dwarf/test-libandroid.so.abi    |    26 +-
 .../test-read-dwarf/test14-pr18893.so.abi     |   262 +-
 .../test-read-dwarf/test15-pr18892.so.abi     |  4497 ++---
 .../test-read-dwarf/test17-pr19027.so.abi     |  7133 ++++----
 ...19-pr19023-libtcmalloc_and_profiler.so.abi |  5173 +++---
 .../test22-pr19097-libstdc++.so.6.0.17.so.abi | 13659 ++++++++--------
 tests/data/test-read-write/test17.xml         |    44 +-
 tests/data/test-read-write/test19.xml         |    12 +-
 tests/data/test-read-write/test20.xml         |    68 +-
 tests/data/test-read-write/test21.xml         |    14 +-
 tests/data/test-read-write/test22.xml         |     1 +
 tests/data/test-read-write/test23.xml         |     1 +
 tests/data/test-read-write/test25.xml         |    46 +-
 tests/data/test-read-write/test26.xml         |    16 +-
 .../test28-without-std-fns-ref.xml            |   937 +-
 .../test28-without-std-vars-ref.xml           |   865 +-
 tools/abidw.cc                                |   208 +-
 77 files changed, 28732 insertions(+), 27552 deletions(-)
 create mode 100644 .gitmodules
 create mode 160000 big-tests