[00/27] Implement type hashing & fix self-comparing gcc-gnat in fc37

Message ID	20240829154711.1883174-1-dodji@redhat.com
Headers	DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 0EDBD385B50D From: dodji@redhat.com To: libabigail@sourceware.org Cc: Dodji Seketeli <dodji@redhat.com> Subject: [PATCH 00/27] Implement type hashing & fix self-comparing gcc-gnat in fc37 Date: Thu, 29 Aug 2024 17:46:29 +0200 Message-ID: <20240829154711.1883174-1-dodji@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: list Errors-To: libabigail-bounces~patchwork=sourceware.org@sourceware.org
Series	Implement type hashing & fix self-comparing gcc-gnat in fc37 \| [00/27] Implement type hashing & fix self-comparing gcc-gnat in fc37 [02/27] ir: Remove the now useless type propagation optimization [03/27] ir: decl-only classes don't equal fully defined classes under ODR [04/27] comp-filter: Consider lvalue-ness changes on references as harmful [05/27] comp-filter: Ignore ptr size when detecting void ptr to ptr change [06/27] Don't strip typedefs in fn names when pretty-printing and comparing [07/27] reader: Avoid duplicating member types [08/27] ir: Cache the result of scope_decl::get_sorted_member_types [10/27] reader: Avoid duplicating recursive types [11/27] dwarf-reader: Speed-up decl-only resolution [12/27] ir: Fix name setting of a ptr-to-mbr-type [13/27] dwarf-reader: Support LLVM's lingo of declaration-ness [14/27] reader: Improve logging in the ABIXML reader [15/27] ir: Improve type logging during type canonicalization [16/27] tools-utils: Improve logging while reading a Linux kernel [17/27] dwarf-reader: Fix building of void, void* and variadic parm types [18/27] {dwarf,btf,ctf}-reader: Set the origin of the corpus group [19/27] {btf,ctf,dwarf}-reader, ir: Fix self-comparison debugging for corpus groups [20/27] abilint: Support --verbose option [21/27] big-tests: Update git sub-module [22/27] dwarf-reader,tools-utils: Add statistics about built/suppressed functions [23/27] abidw: Add a --kmi-stablelist option alongside existing --kmi-whitelist [24/27] dwarf-reader,ir: Fix endless loop while analyzing DWARF from Modula-2 [26/27] dwarf-reader, reader.cc: Fix function virtuality setting

Message ID

20240829154711.1883174-1-dodji@redhat.com

Headers

DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 0EDBD385B50D
From: dodji@redhat.com
To: libabigail@sourceware.org
Cc: Dodji Seketeli <dodji@redhat.com>
Subject: [PATCH 00/27] Implement type hashing & fix self-comparing gcc-gnat in
 fc37
Date: Thu, 29 Aug 2024 17:46:29 +0200
Message-ID: <20240829154711.1883174-1-dodji@redhat.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: libabigail-bounces~patchwork=sourceware.org@sourceware.org

Series

Implement type hashing & fix self-comparing gcc-gnat in fc37 |

Message

Dodji Seketeli Aug. 29, 2024, 3:46 p.m. UTC

  From: Dodji Seketeli <dodji@redhat.com>

This set of patches implements type hashing at the libabigail IR level
before doing type canonicalization.  The goal of this scheme is to
hopefully speed-up the type canonicalization process and also to make
it less error prone than what it is today.  I believe that the main
source of error of the type canonicalization process is the use of the
"canonical type propagation" optimization.  There has been a number of
bugs throughout the years that are related to that optimization and I
believe experience has proven that it becomes increasingly difficult
to maintain.

Hashing types allows the type canonicalization algorithm to quickly
determine when two types are different, as they'd have different hash
values.  When the hash values are equal however, structural comparison
is still needed.  But that amount of structural comparison can be
drastically reduced.

Once type hashing is implemented and it becomes much easier (and
precise) to determine that two types are different during type
canonicalization, I encountered a number of fixes, while trying to
make to the gcc-gnat package to self-compare.  The work that was laid
down also helped tackle the self-comparison of the libLLVM.so binary
as well as many other binaries.

In the end, the 27 patches allow libabigail to be able to self-compare
a lot of binaries than previously.  Among the 42 packages that were
failing self-comparison in fc36 at
https://sourceware.org/bugzilla/show_bug.cgi?id=29413#c5, only 5 still
fail today.  These are binutils, infinipath-psm, monodoc, OpenMolcas
and qt6-qtbase.  I'll probably try to fix self-comparison for these
packages in subsequent fixes now that fundamental things are fixed by
this patch series.

Of course, speed of processing has improved a lot for some packages,
going from near one hour to a matter of minutes in some instances.

Dodji Seketeli (27):
  Implement type hashing
  ir: Remove the now useless type propagation optimization
  ir: decl-only classes don't equal fully defined classes under ODR
  comp-filter: Consider lvalue-ness changes on references as harmful
  comp-filter: Ignore ptr size when detecting void ptr to ptr change
  Don't strip typedefs in fn names when pretty-printing and comparing
  reader: Avoid duplicating member types
  ir: Cache the result of scope_decl::get_sorted_member_types
  dwarf-reader: Avoid duplicating anonymous member types
  reader: Avoid duplicating recursive types
  dwarf-reader: Speed-up decl-only resolution
  ir: Fix name setting of a ptr-to-mbr-type
  dwarf-reader: Support LLVM's lingo of declaration-ness
  reader: Improve logging in the ABIXML reader
  ir: Improve type logging during type canonicalization
  tools-utils: Improve logging while reading a Linux kernel
  dwarf-reader: Fix building of void, void* and variadic parm types
  {dwarf,btf,ctf}-reader: Set the origin of the corpus group
  {btf,ctf,dwarf}-reader, ir: Fix self-comparison debugging for corpus groups
  abilint: Support --verbose option
  big-tests: Update git sub-module
  dwarf-reader,tools-utils: Add statistics about built/suppressed functions
  abidw: Add a --kmi-stablelist option alongside existing --kmi-whitelist
  dwarf-reader,ir: Fix endless loop while analyzing DWARF from Modula-2
  abipkgdiff: Extract devel and main packages in the same directory
  dwarf-reader,reader.cc: Fix function virtuality setting
  dwarf-reader,ir,writer: Better support for static member variables

 big-tests                                     |     2 +-
 configure.ac                                  |    27 +-
 doc/manuals/abidw.rst                         |    16 +-
 doc/manuals/abilint.rst                       |     4 +
 include/abg-comp-filter.h                     |    12 +
 include/abg-comparison.h                      |    35 +-
 include/abg-fwd.h                             |    28 +-
 include/abg-hash.h                            |   249 +-
 include/abg-ir.h                              |   457 +-
 src/abg-btf-reader.cc                         |    46 +-
 src/abg-comp-filter.cc                        |   245 +-
 src/abg-comparison-priv.h                     |     2 +-
 src/abg-comparison.cc                         |    73 +-
 src/abg-ctf-reader.cc                         |    58 +-
 src/abg-dwarf-reader.cc                       |  2063 +-
 src/abg-hash.cc                               |  1478 +-
 src/abg-ir-priv.h                             |  1329 +-
 src/abg-ir.cc                                 |  1602 +-
 src/abg-leaf-reporter.cc                      |    24 +-
 src/abg-reader.cc                             |   401 +-
 src/abg-tools-utils.cc                        |    24 +-
 src/abg-writer.cc                             |   457 +-
 tests/data/Makefile.am                        |     4 +
 .../libsdl/libsdl-1.2.60-1.2.64-report.txt    |    12 +-
 .../test-ptr-to-mbr2-output-1.txt             |     1 -
 .../qualifier-typedef-array-report-1.txt      |    64 +-
 .../test-ld-2.28-210.so--ld-2.28-211.so.txt   |     2 +-
 .../test-abidiff/test-PR18791-report0.txt     |    21 +
 .../PR29443-missing-xx.o.annotated.abi        |     8 +-
 tests/data/test-annotate/libtest23.so.abi     |   338 +-
 .../test-annotate/libtest24-drop-fns-2.so.abi |   377 +-
 .../test-annotate/libtest24-drop-fns.so.abi   |   377 +-
 .../test-anonymous-members-0.o.abi            |    34 +-
 .../test-pointer-to-member-1.o.annotated.abi  |    34 +-
 tests/data/test-annotate/test0.abi            |    46 +-
 tests/data/test-annotate/test1.abi            |    44 +-
 .../data/test-annotate/test13-pr18894.so.abi  |  1264 +-
 .../data/test-annotate/test14-pr18893.so.abi  |  3435 +-
 .../data/test-annotate/test15-pr18892.so.abi  | 18646 +++-----
 .../data/test-annotate/test17-pr19027.so.abi  |  9370 ++--
 ...st18-pr19037-libvtkRenderingLIC-6.1.so.abi | 10482 +++--
 ...19-pr19023-libtcmalloc_and_profiler.so.abi |  9933 ++--
 tests/data/test-annotate/test2.so.abi         |    30 +-
 ...st20-pr19025-libvtkParallelCore-6.1.so.abi |  9108 ++--
 .../data/test-annotate/test21-pr19092.so.abi  |  3848 +-
 tests/data/test-annotate/test3.so.abi         |     4 +-
 tests/data/test-annotate/test4.so.abi         |    18 +-
 tests/data/test-annotate/test5.o.abi          |     4 +-
 tests/data/test-annotate/test6.so.abi         |    18 +-
 tests/data/test-annotate/test7.so.abi         |    52 +-
 .../test8-qualified-this-pointer.so.abi       |    16 +-
 .../PR25409-librte_bus_dpaa.so.20.0.abi       |  2796 +-
 .../test0-pr19026-libvtkIOSQL-6.1.so.1.abi    |  7781 ++--
 .../PR25058-liblttng-ctl-report-1.txt         |    15 +-
 .../test28-vtable-changes-report-0.txt        |     3 +
 tests/data/test-diff-dwarf/test4-report.txt   |     6 +-
 .../test42-PR21296-clanggcc-report0.txt       |     6 +-
 .../test-diff-dwarf/test46-rust-report-0.txt  |     2 -
 tests/data/test-diff-dwarf/test5-report.txt   |     4 +-
 .../test-PR26739-2-report-0.txt               |    16 +-
 .../test30-pr18904-rvalueref-report0.txt      |    56 +-
 .../test30-pr18904-rvalueref-report1.txt      |    58 +-
 .../test30-pr18904-rvalueref-report2.txt      |    58 +-
 .../test31-pr18535-libstdc++-report-0.txt     |     2 +-
 .../test31-pr18535-libstdc++-report-1.txt     |     2 +-
 .../test35-pr18754-no-added-syms-report-0.txt |    56 +-
 .../test35-pr18754-no-added-syms-report-1.txt |     2 +-
 .../data/test-diff-filter/test41-report-0.txt |    40 +-
 ...t43-decl-only-def-change-leaf-report-0.txt |     5 +
 ...test47-filter-void-ptr-change-report-0.txt |     3 +
 tests/data/test-diff-filter/test5-report.txt  |     9 +-
 tests/data/test-diff-filter/test7-report.txt  |     1 -
 .../gcc-debuginfo-14.2.1-1.fc40.x86_64.rpm    |   Bin 0 -> 122904364 bytes
 ...libICE-1.0.9-2.el7.x86_64.rpm-report-0.txt |    13 +-
 ...14.2.1-1.fc40.x86_64-self-check-report.txt |     5 +
 .../libgm2-14.2.1-1.fc40.x86_64.rpm           |   Bin 0 -> 223952 bytes
 .../libgm2-debuginfo-14.2.1-1.fc40.x86_64.rpm |   Bin 0 -> 441515 bytes
 ...l7.x86_64-0.12.8-1.el7.x86_64-report-2.txt |    26 +-
 ...bb-4.3-3.20141204.fc23.x86_64-report-0.txt |    43 +-
 ...bb-4.3-3.20141204.fc23.x86_64-report-1.txt |    29 +-
 ...0-from-fc20-to-fc23-dbus-glib-report-0.txt |    26 +
 ...dbus-glib-0.106-1.fc23.x86_64-report-0.txt |    11 +
 ....fc20--dbus-glib-0.106-1.fc23-report-0.txt |    26 +
 ...--dbus-glib-0.106-1.fc23.i686-report-0.txt |    15 +
 ...-glib-0.106-1.fc23.x86_64.rpm-report-0.txt |    11 +
 ...f-compare-from-fc23-dbus-glib-report-0.txt |     2 +
 ...-vte291-0.39.90-1.fc22.x86_64-report-0.txt |     1 +
 tests/data/test-read-btf/test0.o.abi          |    74 +-
 tests/data/test-read-btf/test1.o.abi          |    12 +-
 .../test-read-ctf/PR27700/test-PR27700.abi    |    10 +-
 tests/data/test-read-ctf/test-PR26568-1.o.abi |    18 +-
 tests/data/test-read-ctf/test-PR26568-2.o.abi |    16 +-
 tests/data/test-read-ctf/test-alias.o.abi     |     6 +-
 .../test-ambiguous-struct-A.o.hash.abi        |    20 +-
 .../test-ambiguous-struct-B.o.hash.abi        |    10 +-
 .../test-read-ctf/test-anonymous-fields.o.abi |    14 +-
 .../test-read-ctf/test-array-mdimension.abi   |    16 +-
 .../test-read-ctf/test-array-of-pointers.abi  |    24 +-
 tests/data/test-read-ctf/test-array-size.abi  |    18 +-
 .../data/test-read-ctf/test-bitfield-enum.abi |     8 +-
 tests/data/test-read-ctf/test-bitfield.abi    |    10 +-
 tests/data/test-read-ctf/test-callback.abi    |    10 +-
 tests/data/test-read-ctf/test-callback2.abi   |    12 +-
 .../test-conflicting-type-syms-a.o.hash.abi   |    10 +-
 .../test-conflicting-type-syms-b.o.hash.abi   |     8 +-
 tests/data/test-read-ctf/test-const-array.abi |    12 +-
 .../test-read-ctf/test-dynamic-array.o.abi    |    18 +-
 .../test-read-ctf/test-enum-many.o.hash.abi   |     8 +-
 .../test-read-ctf/test-enum-symbol.o.hash.abi |     6 +-
 tests/data/test-read-ctf/test-enum.o.abi      |     8 +-
 tests/data/test-read-ctf/test-fallback.abi    |     4 +-
 .../test-read-ctf/test-forward-type-decl.abi  |    14 +-
 .../test-functions-declaration.abi            |    14 +-
 .../data/test-read-ctf/test-linux-module.abi  |    38 +-
 tests/data/test-read-ctf/test-list-struct.abi |     8 +-
 tests/data/test-read-ctf/test0.abi            |    40 +-
 tests/data/test-read-ctf/test0.hash.abi       |    40 +-
 tests/data/test-read-ctf/test1.so.abi         |    18 +-
 tests/data/test-read-ctf/test1.so.hash.abi    |    18 +-
 tests/data/test-read-ctf/test2.so.abi         |    18 +-
 tests/data/test-read-ctf/test2.so.hash.abi    |    18 +-
 tests/data/test-read-ctf/test3.so.abi         |     4 +-
 tests/data/test-read-ctf/test3.so.hash.abi    |     4 +-
 tests/data/test-read-ctf/test4.so.abi         |    18 +-
 tests/data/test-read-ctf/test4.so.hash.abi    |    18 +-
 tests/data/test-read-ctf/test5.o.abi          |    34 +-
 tests/data/test-read-ctf/test7.o.abi          |    22 +-
 tests/data/test-read-ctf/test8.o.abi          |     4 +-
 tests/data/test-read-ctf/test9.o.abi          |    48 +-
 .../PR22015-libboost_iostreams.so.abi         |  1579 +-
 .../test-read-dwarf/PR22122-libftdc.so.abi    |  6146 +--
 .../PR24378-fn-is-not-scope.abi               |    26 +-
 .../data/test-read-dwarf/PR25007-sdhci.ko.abi |  2938 +-
 .../PR25042-libgdbm-clang-dwarf5.so.6.0.0.abi |   609 +-
 .../test-read-dwarf/PR26261/PR26261-exe.abi   |    44 +-
 .../test-read-dwarf/PR27700/test-PR27700.abi  |    10 +-
 .../PR28584/PR28584-smv.clang.o.abi           |     6 +-
 .../test-read-dwarf/PR29443-missing-xx.o.abi  |     8 +-
 .../PR29692-kdelibs3-libkjava.so.1.0.0.abi    |     2 +-
 tests/data/test-read-dwarf/libtest23.so.abi   |   316 +-
 .../libtest24-drop-fns-2.so.abi               |   353 +-
 .../test-read-dwarf/libtest24-drop-fns.so.abi |   353 +-
 .../data/test-read-dwarf/test-PR26568-1.o.abi |    18 +-
 .../data/test-read-dwarf/test-PR26568-2.o.abi |    16 +-
 tests/data/test-read-dwarf/test-fallback.abi  |     4 +-
 .../test-read-dwarf/test-libaaudio.so.abi     |   424 +-
 .../test-read-dwarf/test-libandroid.so.abi    | 21142 +++++----
 .../test-pointer-to-member-1.o.abi            |    34 +-
 .../test-suppressed-alias.o.abi               |     6 +-
 tests/data/test-read-dwarf/test0.abi          |    46 +-
 tests/data/test-read-dwarf/test0.hash.abi     |    48 +-
 tests/data/test-read-dwarf/test1.abi          |    44 +-
 tests/data/test-read-dwarf/test1.hash.abi     |    50 +-
 .../test-read-dwarf/test10-pr18818-gcc.so.abi |  4194 +-
 .../test-read-dwarf/test11-pr18828.so.abi     | 11118 ++---
 .../test-read-dwarf/test12-pr18844.so.abi     | 21428 ++++-----
 .../test-read-dwarf/test13-pr18894.so.abi     |  1260 +-
 .../test-read-dwarf/test14-pr18893.so.abi     |  3091 +-
 .../test-read-dwarf/test15-pr18892.so.abi     | 20201 ++++-----
 .../test-read-dwarf/test16-pr18904.so.abi     | 19855 ++++----
 .../test-read-dwarf/test17-pr19027.so.abi     |  8996 ++--
 ...st18-pr19037-libvtkRenderingLIC-6.1.so.abi |  9051 ++--
 ...19-pr19023-libtcmalloc_and_profiler.so.abi |  9371 ++--
 tests/data/test-read-dwarf/test2.so.abi       |    30 +-
 tests/data/test-read-dwarf/test2.so.hash.abi  |    30 +-
 ...st20-pr19025-libvtkParallelCore-6.1.so.abi |  7774 ++--
 .../test-read-dwarf/test21-pr19092.so.abi     |  3804 +-
 .../test22-pr19097-libstdc++.so.6.0.17.so.abi | 37350 ++++++++--------
 .../test-read-dwarf/test3-alias-1.so.hash.abi |     4 +-
 .../test-read-dwarf/test3-alias-2.so.hash.abi |     4 +-
 .../test-read-dwarf/test3-alias-3.so.hash.abi |     4 +-
 .../test-read-dwarf/test3-alias-4.so.hash.abi |     4 +-
 tests/data/test-read-dwarf/test3.so.abi       |     4 +-
 tests/data/test-read-dwarf/test3.so.hash.abi  |     4 +-
 tests/data/test-read-dwarf/test4.so.abi       |    18 +-
 tests/data/test-read-dwarf/test4.so.hash.abi  |    18 +-
 tests/data/test-read-dwarf/test5.o.abi        |     4 +-
 tests/data/test-read-dwarf/test5.o.hash.abi   |     4 +-
 tests/data/test-read-dwarf/test6.so.abi       |    18 +-
 tests/data/test-read-dwarf/test6.so.hash.abi  |    18 +-
 tests/data/test-read-dwarf/test7.so.abi       |    52 +-
 tests/data/test-read-dwarf/test7.so.hash.abi  |    52 +-
 .../test8-qualified-this-pointer.so.abi       |    16 +-
 .../test8-qualified-this-pointer.so.hash.abi  |    16 +-
 .../test9-pr18818-clang.so.abi                |  3051 +-
 tests/data/test-read-write/test-crc.xml       |     2 +-
 tests/data/test-read-write/test0.xml          |     2 +-
 tests/data/test-read-write/test1.xml          |     2 +-
 tests/data/test-read-write/test10.xml         |    20 +-
 tests/data/test-read-write/test11.xml         |     4 +-
 tests/data/test-read-write/test12.xml         |     6 +-
 tests/data/test-read-write/test13.xml         |     8 +-
 tests/data/test-read-write/test14.xml         |     8 +-
 tests/data/test-read-write/test15.xml         |     6 +-
 tests/data/test-read-write/test16.xml         |     4 +-
 tests/data/test-read-write/test17.xml         |    54 +-
 tests/data/test-read-write/test18.xml         |    16 +-
 tests/data/test-read-write/test19.xml         |    22 +-
 tests/data/test-read-write/test2.xml          |     8 +-
 tests/data/test-read-write/test20.xml         |    88 +-
 tests/data/test-read-write/test21.xml         |    26 +-
 tests/data/test-read-write/test22.xml         |    12 +-
 tests/data/test-read-write/test23.xml         |    12 +-
 tests/data/test-read-write/test24.xml         |    16 +-
 tests/data/test-read-write/test25.xml         |    62 +-
 tests/data/test-read-write/test26.xml         |    22 +-
 tests/data/test-read-write/test27.xml         |    16 +-
 .../test28-without-std-fns-ref.xml            |  1190 +-
 .../test28-without-std-vars-ref.xml           |  1116 +-
 tests/data/test-read-write/test3.xml          |     8 +-
 tests/data/test-read-write/test4.xml          |     4 +-
 tests/data/test-read-write/test5.xml          |     4 +-
 tests/data/test-read-write/test6.xml          |    10 +-
 tests/data/test-read-write/test7.xml          |     4 +-
 tests/data/test-read-write/test8.xml          |     2 +-
 tests/data/test-read-write/test9.xml          |     8 +-
 tests/test-diff-pkg.cc                        |    13 +
 tools/abidw.cc                                |     3 +-
 tools/abilint.cc                              |    30 +-
 tools/abipkgdiff.cc                           |    38 +-
 220 files changed, 140901 insertions(+), 145092 deletions(-)
 create mode 100644 tests/data/test-diff-pkg/gcc-debuginfo-14.2.1-1.fc40.x86_64.rpm
 create mode 100644 tests/data/test-diff-pkg/libgm2-14.2.1-1.fc40.x86_64-self-check-report.txt
 create mode 100644 tests/data/test-diff-pkg/libgm2-14.2.1-1.fc40.x86_64.rpm
 create mode 100644 tests/data/test-diff-pkg/libgm2-debuginfo-14.2.1-1.fc40.x86_64.rpm

Comments

Dodji Seketeli Aug. 29, 2024, 3:52 p.m. UTC | #1

Hello,

The order in which types of the same name & kind (aka homonym types)
are canonicalized does have an impact on the final canonicalization
result.  This variation is at the root of the self-comparison error
happening on the gcc-gnat package in f37.  The command to reproduce
the issue is:

	$ fedabipkgdiff  -a --self-compare --from fc37 gcc-gnat

Note that the binary on which self-comparison is failing is gnat1 for
the s390 arch:

    $ pwd
    ~/git/libabigail/hash-types/prtests/extracts/gcc-gnat-12.3.1-1.fc37.s390x
    $
    $ abidw --debug-abidiff -d usr/lib/debug usr/libexec/gcc/s390x-redhat-linux/12/gnat1

That test is still failing with this patch, but is going to pass with
subsequent patches in the series.

Having persistent hashes that can be serialized along with the types
into ABIXML does help to suppress that variation.

This patch implements that idea.

Learning from the teachings of the DWZ tool, the patch uses the xxhash
algorithm for primitive hashing and hash combination.  The patch
prevents cycling while hashing type graphs by maintaining a hashing
state in objects being hashed, just like what DWZ does.

The patch cleans out the relics of the previous (failed) attempt of
using hashing in the IR.  Now, a virtual
type_or_decl_base::hash_value() is declared and implemented by
decl_base, type_base and descendants of type_base.

Each IR node has a "private" hashing function that knows how to hash
the specific parts of the IR node and combine the hashes of its
sub-type IR children nodes to come up with a hashing value for the
whole IR node, while preventing cycling.  There are some subtleties
introduced during the hashing.  For instance, a typedef hash is the
same as the hash of its underlying type.  This helps the hash value of
a pointer or reference to a typedef of class to be the same as the
hash value of a pointer or reference to the class.  Another subtlety
is to never hash the declaration of a class, if a definition is
present; rather, always hash the definition.  In general, types that
are not canonicalized (for which is_non_canonicalized_type returns
true) are not hashed.

That private hashing function is thus used by the ::hash_value()
of the IR node to compute the hash value for the first time and cache
it for it subsequent invocations.

After building the IR, the patch first sorts all types, then hashes
them and then canonicalizes them.  During the canonicalization phase,
two types with different hashes are deemed different.  Two types with
the same hash are compared structurally.

During ABIXML de-serialization, libabigail reads the hash value back
from each IR node and uses that instead of re-calculating the hash.

Note that calculating the hash of a node read from ABIXML could lead
to a different result than calculating the hash for the same node from
DWARF, because the order in which hashes are calculated counts,
especially for recursive types.  That is one of the reasons why hashes
are not re-calculated when reading from abixml.

In all fairness, I think we would need to test this hypothesis again,
because we now sort types before hashing them, so in theory, the
hashing should yield the same result as when done from ELF.
Regardless, just reading the hash values from ABIXML is faster than
re-calculating it.  We might want to re-calculate the hashes for
debugging purposes, however.  This is left as an exercise for the
astute reader of this patch.

If the hash of an ABIXML node equals the hash of an ELF node, then
during type canonicalization, the two nodes are compared structurally.
If the hashes are different, then the nodes compare different during
canonicalization.  That speeds up canonicalization at ABIXML reading
time.  But this implies that each change in a hashing scheme should
result in a change in the major ABIXML version number, rendering newer
ABIXML files incompatible with previous ones.  It even makes them
incompatible with newer libabigail code altogether.  In that case, we
might want to teach abidiff about ABIXML versioning, even when
comparing an ABIXML input against an ELF one.

Note that the patch uses the hashing during type canonicalization in
general; that is, if the hashes are present and different, then the
types are different.  That helps to do away with the need for the
"canonical type propagation" optimization that I suspect is causing
issues on redundant types.

It turns out that not using that optimization doesn't incur any
noticeable speed penalty so in a subsequent patch, that canonical type
propagation optimization code is going to be entirely removed.

The patch uses an optional value for hashes, to make the difference
between no-hash (for recursive sub-types and types supposed to have no
hash/canonical types) and a hash value of zero which is a valid hash
value.

Note that the patch emits hashes for function-decl elements.  This is
the hash of the type of the function.  When parsing ABIXML, the reader
sets that function-decl hash to the function_type built from it.

Also, note that for types having the same representation (homonym
types) and yet are canonically different, this patch emits a distinct
canonical type index (aka CTI).  That CTI is appended to the hash
value, following a "#" sign.

Later, during type canonicalization, two homonym types T and T' (where
T originates from ELF and T' originates from ABIXML) having the same
hash and CTI will be structurally compared.  If T equals T', then T
and T' will be considered canonically equivalent.  In other words,
CTIs help to match homonym types originating from ABIXML against
homonym types originating from ELF.

Note that 2 tests are still failing. They are fall-outs that are
addressed in subsequent patches.  This patch XFAIL them until they get
fixed by subsequent patches of the series.

	* configure.ac: Bump ABIXML version to 4.0.  Detect the new xxhash
	dependency and require the 0.8.0 version at minimum.
	* include/abg-fwd.h (look_through_decl_only_type): Declare new
	functions.
	(is_scope_decl): Const-ify the parameter and
	the returned value.  Also, do not include "abg-hash.h" here.
	* include/abg-hash.h: Make this include cstdint (for uint64_t) and
	abg-ir.h, instead of stdint.h.
	(enum hashing_state): Define new enum in the abigail::hashing
	namespace.
	(combine_hashes, fnv_has, hash, get_hashing_state)
	(set_hashing_state, is_recursive_artefact): Declare new functions
	in the abigail::hashing namespace.
	(struct {decl_base, type_base, type_decl, qualified_type_def,
	pointer_type_def, reference_type_def, ptr_to_mbr_type,
	array_type_def, enum_type_decl, typedef_decl, function_type,
	method_type, member_base, class_or_union, class_decl::base_spec,
	class_decl, union_decl}::hash): Declare new hash functors in the
	abigail::ir namespace.
	* include/abg-ir.h: Remove the inclusion of abg-hash.h from this
	header.
	(typedef hash_t): Define new type.
	(peek_hash_value): Declare new function.
	(get_canonical_types): Return a pointer to const vector.
	({type_or_decl_base, type_base, qualified_type_def,
	pointer_type_def, reference_type_def, ptr_to_mbr_type,
	array_type_def::subrange_type, typedef_decl, function_type,
	method_type, class_or_union, class_decl, class_decl::base_spec,
	union_decl}::hash_value): Add new virtual member functions.
	(type_or_decl_base::set_hash_value): Add new member function.
	(type_or_decl_base::priv_): Make this data member public.
	(peek_hash_value, set_or_get_cached_hash_value): Declare these
	functions as friends of class type_or_decl_base.
	({decl_base, scope_decl, var_decl, function_decl,
	function_decl::parameter}::get_hash): Remove member functions.
	* src/abg-comparison-priv.h (types_or_decls_hash::operator()):
	Adjust to using the new hash_t type.
	* src/abg-ctf-reader.cc
	(reader::{additional_types_to_canonicalize, types}): Add new data
	members.
	(reader::add_type(): Add new member function.
	(reader::canonicalize_all_types): Use the new types vector data
	member to stash types to be canonicalized and pass them to
	ir::hash_and_canonicalize_types for canonicalization.
	(process_ctf_base_type)
	(build_ir_node_for_variadic_parameter_type)
	(build_ir_node_for_void_type, build_ir_node_for_void_pointer_type)
	(build_array_ctf_range, process_ctf_enum_type): Do not use
	canonicalize anymore.  Rather, rely on reader::add_type to
	schedule types for canonicalization, doing the generic sorting and
	hashing before doing the actually canonicalization.
	* src/abg-dwarf-reader.cc ({dwarf_offset_pair_hash,offset_hash,
	offset_pair_hash}::operator()): Use the new hashing function.
	(reader::read_debug_info_into_corpus): Improve logging.  Use
	ir::hash_and_canonicalize_types in lieu of ir::canonicalize_types.
	(reader::types_to_canonicalize): Add
	non-const overload.
	(reader::canonicalize_types_scheduled): Add better logs & misc
	obvious cleanup.
	(maybe_canonicalize_type): Force scheduling canonicalization of
	all types at the end of the DWARF processing.
	* src/abg-hash.cc (combine_hashes, hash, get_hashing_state)
	(set_hashing_state, is_recursive_artefact): Define new functions.
	(MAYBE_RETURN_EARLY_FROM_HASHING_TO_AVOID_CYCLES)
	(MAYBE_FLAG_TYPE_AS_RECURSIVE)
	(MAYBE_RETURN_EARLY_IF_HASH_EXISTS): Define new macros.
	(struct {decl_base, type_base, type_decl, qualified_type_def,
	pointer_type_def, reference_type_def, ptr_to_mbr_type,
	array_type_def, ptr_to_mbr_type, enum_type_decl, typedef_decl,
	function_decl, function_type, method_type, member_base,
	class_or_union, class_decl::base_spec, class_decl,
	union_decl}::hash::operator): Define new hash functors.
	({template_parameter, template_decl, non_type_tparameter,
	template_tparameter, type_composition, type_composition,
	function_tdecl}::{hash, dynamic_hash, shared_ptr_hash): Remove.
	* src/abg-ir-priv.h: Don't include abg-ir.h anymore, rather
	include abg-hash.h.
	(struct type_or_decl_base::priv): Move this here, from abg-ir.cc.
	(type_or_decl_base::priv::{hashing_state_, hash_value_,
	is_recursive_artefact_}): Define new data members.
	(type_or_decl_base::priv::{get_hashing_state, set_hashing_state,
	set_hash_value, force_set_hash_value, is_recursive_artefact}):
	Define new member functions.
	(struct sort_for_hash_functor): Define new functor.
	(do_hash_value, set_or_get_cached_hash_value)
	(hash_and_canonicalize_types, sort_and_canonicalize_types): Define
	new function templates.
	(type_is_suitable_for_hash_computing)
	(sort_types_for_hash_computing_and_c14n)
	(get_canonical_type_index, get_decl_name_for_comparison): Declare
	new functions.
	(type_base::priv::canonical_type_index): New data member.
	(type_base::priv::{priv, clear_propagated_canonical_type}):
	Initialize it.
	(uint64_t_pair_hash::operator()): Adjust.
	(environment::priv::number_of_canonical_types): New data member.
	(environment::priv::priv): Initialize it.
	(environment::priv::get_new_canonical_type_index): New member
	function.
	(environment::priv::propagate_ct): Propagate the CTI too.
	(environment::priv::{confirm_ct_propagation_for_types_dependant_on,
	confirm_ct_propagation}): Assert that canonical type has been
	propagated and thus we have a canonical type.
	(struct type_topo_comp::operator()): Beef up sorting of types.
	Take into account the absolute path of the TU, the hash value, the
	CTI)
	(struct sort_for_hash_functor): Define new functor.
	(sort_types_for_hash_computing_and_c14n): Declare new function
	template and new overload.
	(canonicalize_types): Take new do_log and show_stats parameters.
	Improve logging.  Do not sort types in here.
	(hash_and_canonicalize_types): Define new function template.  This
	one does the sorting before the hashing and the canonicalization.
	(sort_and_canonicalize_types): Likewise, but this one does no
	hashing.
	(cache_type_comparison_result): Cache the result of the comparison
	now, unconditionally.  As we don't do canonical type propagation
	anymore, we should not get canonical types and equality to
	disagree anymore.
	(* src/abg-ir.cc (try_canonical_compare): If hash values are
	present and different then the two types are different.
	(environment::get_canonical_types): Constify return value.
	(environment::get_canonical_type): Adjust.
	(struct type_or_decl_base::priv): Move this to abg-ir-priv.h.
	(type_or_decl_base::hashing_started): Remove.
	({decl_base, scope_decl, var_decl, function_decl,
	function_decl::parameter, class_decl::base_spec,
	non_type_tparameter, type_composition}::get_hash): Likewise.
	({template_parameter}::get_hashing_has_started)
	(template_parameter::set_hashing_has_started): Likewise.
	(type_or_decl_base::{hash_value, set_hash_value})
	({type_base, type_decl, qualified_type_def, pointer_type_def,
	reference_type_def, ptr_to_mbr_type,
	array_type_def::subrange_type, array_type_def, enum_type_decl,
	typedef_decl, function_type, method_type, class_or_union,
	class_decl::base_spec, class_decl, union_decl}::hash_value):
	Define new member functions.
	(type_or_decl_base::set_hash_value): Likewise.
	(get_decl_name_for_comparison): Make this non-static.
	(is_scope_decl): Constify.
	(type_is_suitable_for_hash_computing)
	(peek_hash_value, read_type_hash, read_hash_and_stash)
	(look_through_decl_only_type)
	(candidate_matches_a_canonical_type_hash)
	(sort_types_for_hash_computing_and_c14n): Define new functions.
	(lookup_pointer_type, lookup_reference_type)
	(pointer_type_def::get_qualified_name)
	(reference_type_def::get_qualified_name)
	(reference_type_def::get_pretty_representation): Adjust.
	(get_type_name):  Better handle naming of anonymous type decls,
	used for enums.
	(get_debug_representation): Adjust to emit hashes & CTI of types
	as well as member types while debugging.
	(look_through_decl_only_type): Rename look_through_decl_only into
	this.
	(lookup_pointer_type, lookup_reference_type)
	(reference_type_def::get_qualified_name): Adjust.
	(compare_canonical_type_against_candidate): Stop doing canonical
	type propagation during type canonicalization.
	(type_base::get_canonical_type_for): Use the new
	candidate_matches_a_canonical_type_hash.  Set the canonical type
	index for homophone canonical types.  In debug mode, check that if
	a type equals a canonical type, their hash value must match.
	(canonicalize): Take new do_log and show_stats parameter.  Improve
	logging.  Also, a type must have the same CTI as its canonical
	type.
	(hash_type_or_decl): Adjust to use the new type
	hash_t.
	(type_topo_comp::operator()): Add an overload for type_base_wptr.
	If the two pointers are equal, get out early.  Otherwise, if
	everything else is equal, sort using the absolute path of the
	containing translation unit.
	(maybe_propagate_canonical_type): Propagate canonical type only if
	their CTI match.
	* src/abg-reader.cc (maybe_canonicalize_type): Improve logging.
	Schedule all types for late canonicalization.
	(reader::perform_type_canonicalization): Improve logging.  Call
	hash_and_canonicalize_types to hash and canonicalize types.
	(read_type_hash_and_cti, read_hash_and_stash): Define new static
	functions.
	(build_function_decl, build_type_decl)
	(build_qualified_type_decl)
	(build_pointer_type_def, build_reference_type_def)
	(build_ptr_to_mbr_type, build_function_type, build_subrange_type)
	(build_array_type_def, build_enum_type_decl, build_typedef_decl)
	(build_class_decl, build_union_decl): Read and set the hash value
	from ABIXML.
	(build_function_decl, build_qualified_type_decl)
	(build_pointer_type_def, build_reference_type_def)
	(build_function_type, build_subrange_type, build_array_type_def)
	(build_enum_type_decl, build_typedef_decl, build_class_decl)
	(build_union_decl): Read the hash and CTI and set them to the
	type.
	(build_reference_type_def): Build the referenced type before-hand.
	(build_class_tdecl, build_type_tparameter, build_type_composition)
	(build_template_tparameter, build_type, handle_type_decl)
	(handle_qualified_type_decl, handle_pointer_type_def)
	(handle_reference_type_def, handle_function_type)
	(handle_array_type_def, handle_enum_type_decl)
	(handle_typedef_decl, handle_class_decl, handle_union_decl):
	Adjust to the use of the new maybe_canonicalize_type signature.
	* src/abg-btf-reader.cc (reader::canonicalize_types): Use the new
	hash_and_canonicalize_types defined above.  Log the time taken by
	type canonicalization.
	* src/abg-ctf-reader.cc (reader::{types,
	additional_types_to_canonicalize}): New data members
	(reader::add_types): New member functions.
	(reader::canonicalize_all_types): Use the new
	hash_and_canonicalize_types defined above.
	(process_ctf_base_type): Do not call canonicalize here.
	(build_ir_node_for_variadic_parameter_type)
	(build_ir_node_for_void_type)
	(build_ir_node_for_void_pointer_type, process_ctf_enum_type):
	Likewise, and call reader::add_type instead.
	* src/abg-dwarf-reader.cc ({dwarf_offset_pair_hash, offset_hash,
	offset_pair_hash}::operator()): Adjust to using the new hash_t
	type.
	(reader::canonicalize_types_scheduled): Use the new
	hash_and_canonicalize_types above.
	(maybe_canonicalize_type): Schedule all types for late
	canonicalization.
	* src/abg-writer.cc (reader::get_id_for_type): Constify the
	parameter.
	(write_type_hash_and_cti, write_common_type_info)
	(write_fn_parm_and_return_types): Define new static functions.
	(write_type_decl, write_qualified_type_def)
	(write_pointer_type_def, write_reference_type_def)
	(write_array_subrange_type, write_array_type_def)
	(write_enum_type_decl, write_typedef_decl, write_function_decl)
	(write_function_type, write_class_decl_opening_tag)
	(write_union_decl_opening_tag): Emit hash value and CTI to the
	ABIXML.
	* tests/Makefile.am: XFAIL the tests runtestdifffilter and
	runtestabidiffexit for now.
	* tests/data/test-abidiff-exit/PR30329/PR30329-report-1.txt:
	Adjust.
	* tests/data/test-annotate/PR29443-missing-xx.o.annotated.abi:
	Likewise.
	* tests/data/test-annotate/libtest23.so.abi: Likewise.
	* tests/data/test-annotate/libtest24-drop-fns-2.so.abi: Likewise.
	* tests/data/test-annotate/libtest24-drop-fns.so.abi: Likewise.
	* tests/data/test-annotate/test-anonymous-members-0.o.abi:
	Likewise.
	* tests/data/test-annotate/test-pointer-to-member-1.o.annotated.abi:
	Likewise.
	* tests/data/test-annotate/test0.abi: Likewise.
	* tests/data/test-annotate/test1.abi: Likewise.
	* tests/data/test-annotate/test13-pr18894.so.abi: Likewise.
	* tests/data/test-annotate/test14-pr18893.so.abi: Likewise.
	* tests/data/test-annotate/test15-pr18892.so.abi: Likewise.
	* tests/data/test-annotate/test17-pr19027.so.abi: Likewise.
	* tests/data/test-annotate/test18-pr19037-libvtkRenderingLIC-6.1.so.abi:
	Likewise.
	* tests/data/test-annotate/test19-pr19023-libtcmalloc_and_profiler.so.abi:
	Likewise.
	* tests/data/test-annotate/test2.so.abi: Likewise.
	* tests/data/test-annotate/test20-pr19025-libvtkParallelCore-6.1.so.abi:
	Likewise.
	* tests/data/test-annotate/test21-pr19092.so.abi: Likewise.
	* tests/data/test-annotate/test3.so.abi: Likewise.
	* tests/data/test-annotate/test4.so.abi: Likewise.
	* tests/data/test-annotate/test5.o.abi: Likewise.
	* tests/data/test-annotate/test6.so.abi: Likewise.
	* tests/data/test-annotate/test7.so.abi: Likewise.
	* tests/data/test-annotate/test8-qualified-this-pointer.so.abi:
	Likewise.
	* tests/data/test-diff-dwarf-abixml/PR25409-librte_bus_dpaa.so.20.0.abi:
	Likewise.
	* tests/data/test-diff-dwarf-abixml/test0-pr19026-libvtkIOSQL-6.1.so.1.abi:
	Likewise.
	* tests/data/test-diff-filter/test-PR26739-2-report-0.txt:
	Likewise.
	* tests/data/test-diff-filter/test3-report.txt: Likewise.
	* tests/data/test-diff-filter/test30-pr18904-rvalueref-report0.txt:
	Likewise.
	* tests/data/test-diff-filter/test30-pr18904-rvalueref-report1.txt:
	Likewise.
	* tests/data/test-diff-filter/test30-pr18904-rvalueref-report2.txt:
	Likewise.
	* tests/data/test-diff-filter/test31-pr18535-libstdc++-report-0.txt:
	Likewise.
	* tests/data/test-diff-filter/test31-pr18535-libstdc++-report-1.txt:
	Likewise.
	* tests/data/test-diff-filter/test35-pr18754-no-added-syms-report-0.txt:
	Likewise.
	* tests/data/test-diff-filter/test35-pr18754-no-added-syms-report-1.txt:
	Likewise.
	* tests/data/test-diff-filter/test43-decl-only-def-change-leaf-report-0.txt:
	Likewise.
	* tests/data/test-diff-pkg-ctf/gmp-6.x.x86_64-report-0.txt:
	Likewise.
	* tests/data/test-diff-pkg/tbb-4.1-9.20130314.fc22.x86_64--tbb-4.3-3.20141204.fc23.x86_64-report-0.txt:
	Likewise.
	* tests/data/test-diff-pkg/tbb-4.1-9.20130314.fc22.x86_64--tbb-4.3-3.20141204.fc23.x86_64-report-1.txt:
	Likewise.
	* tests/data/test-read-btf/test0.o.abi: Likewise.
	* tests/data/test-read-btf/test1.o.abi: Likewise.
	* tests/data/test-read-ctf/PR27700/test-PR27700.abi: Likewise.
	* tests/data/test-read-ctf/test-PR26568-1.o.abi: Likewise.
	* tests/data/test-read-ctf/test-PR26568-2.o.abi: Likewise.
	* tests/data/test-read-ctf/test-alias.o.abi: Likewise.
	* tests/data/test-read-ctf/test-ambiguous-struct-A.o.hash.abi:
	Likewise.
	* tests/data/test-read-ctf/test-ambiguous-struct-B.o.hash.abi:
	Likewise.
	* tests/data/test-read-ctf/test-anonymous-fields.o.abi: Likewise.
	* tests/data/test-read-ctf/test-array-mdimension.abi: Likewise.
	* tests/data/test-read-ctf/test-array-of-pointers.abi: Likewise.
	* tests/data/test-read-ctf/test-array-size.abi: Likewise.
	* tests/data/test-read-ctf/test-bitfield-enum.abi: Likewise.
	* tests/data/test-read-ctf/test-bitfield.abi: Likewise.
	* tests/data/test-read-ctf/test-callback.abi: Likewise.
	* tests/data/test-read-ctf/test-callback2.abi: Likewise.
	* tests/data/test-read-ctf/test-conflicting-type-syms-a.o.hash.abi:
	Likewise.
	* tests/data/test-read-ctf/test-conflicting-type-syms-b.o.hash.abi:
	Likewise.
	* tests/data/test-read-ctf/test-const-array.abi: Likewise.
	* tests/data/test-read-ctf/test-dynamic-array.o.abi: Likewise.
	* tests/data/test-read-ctf/test-enum-many.o.hash.abi: Likewise.
	* tests/data/test-read-ctf/test-enum-symbol.o.hash.abi: Likewise.
	* tests/data/test-read-ctf/test-enum.o.abi: Likewise.
	* tests/data/test-read-ctf/test-fallback.abi: Likewise.
	* tests/data/test-read-ctf/test-forward-type-decl.abi: Likewise.
	* tests/data/test-read-ctf/test-functions-declaration.abi:
	Likewise.
	* tests/data/test-read-ctf/test-linux-module.abi: Likewise.
	* tests/data/test-read-ctf/test-list-struct.abi: Likewise.
	* tests/data/test-read-ctf/test0.abi: Likewise.
	* tests/data/test-read-ctf/test0.hash.abi: Likewise.
	* tests/data/test-read-ctf/test1.so.abi: Likewise.
	* tests/data/test-read-ctf/test1.so.hash.abi: Likewise.
	* tests/data/test-read-ctf/test2.so.abi: Likewise.
	* tests/data/test-read-ctf/test2.so.hash.abi: Likewise.
	* tests/data/test-read-ctf/test3.so.abi: Likewise.
	* tests/data/test-read-ctf/test3.so.hash.abi: Likewise.
	* tests/data/test-read-ctf/test4.so.abi: Likewise.
	* tests/data/test-read-ctf/test4.so.hash.abi: Likewise.
	* tests/data/test-read-ctf/test5.o.abi: Likewise.
	* tests/data/test-read-ctf/test7.o.abi: Likewise.
	* tests/data/test-read-ctf/test8.o.abi: Likewise.
	* tests/data/test-read-ctf/test9.o.abi: Likewise.
	* tests/data/test-read-dwarf/PR22015-libboost_iostreams.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/PR22122-libftdc.so.abi: Likewise.
	* tests/data/test-read-dwarf/PR24378-fn-is-not-scope.abi:
	Likewise.
	* tests/data/test-read-dwarf/PR25007-sdhci.ko.abi: Likewise.
	* tests/data/test-read-dwarf/PR25042-libgdbm-clang-dwarf5.so.6.0.0.abi:
	Likewise.
	* tests/data/test-read-dwarf/PR26261/PR26261-exe.abi: Likewise.
	* tests/data/test-read-dwarf/PR27700/test-PR27700.abi: Likewise.
	* tests/data/test-read-dwarf/PR28584/PR28584-smv.clang.o.abi:
	Likewise.
	* tests/data/test-read-dwarf/PR29443-missing-xx.o.abi: Likewise.
	* tests/data/test-read-dwarf/PR29692-kdelibs3-libkjava.so.1.0.0.abi:
	Likewise.
	* tests/data/test-read-dwarf/libtest23.so.abi: Likewise.
	* tests/data/test-read-dwarf/libtest24-drop-fns-2.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/libtest24-drop-fns.so.abi: Likewise.
	* tests/data/test-read-dwarf/test-PR26568-1.o.abi: Likewise.
	* tests/data/test-read-dwarf/test-PR26568-2.o.abi: Likewise.
	* tests/data/test-read-dwarf/test-fallback.abi: Likewise.
	* tests/data/test-read-dwarf/test-libaaudio.so.abi: Likewise.
	* tests/data/test-read-dwarf/test-libandroid.so.abi: Likewise.
	* tests/data/test-read-dwarf/test-pointer-to-member-1.o.abi:
	Likewise.
	* tests/data/test-read-dwarf/test-suppressed-alias.o.abi:
	Likewise.
	* tests/data/test-read-dwarf/test0.abi: Likewise.
	* tests/data/test-read-dwarf/test0.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test1.abi: Likewise.
	* tests/data/test-read-dwarf/test1.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test10-pr18818-gcc.so.abi: Likewise.
	* tests/data/test-read-dwarf/test11-pr18828.so.abi: Likewise.
	* tests/data/test-read-dwarf/test12-pr18844.so.abi: Likewise.
	* tests/data/test-read-dwarf/test13-pr18894.so.abi: Likewise.
	* tests/data/test-read-dwarf/test14-pr18893.so.abi: Likewise.
	* tests/data/test-read-dwarf/test15-pr18892.so.abi: Likewise.
	* tests/data/test-read-dwarf/test16-pr18904.so.abi: Likewise.
	* tests/data/test-read-dwarf/test17-pr19027.so.abi: Likewise.
	* tests/data/test-read-dwarf/test18-pr19037-libvtkRenderingLIC-6.1.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test19-pr19023-libtcmalloc_and_profiler.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test2.so.abi: Likewise.
	* tests/data/test-read-dwarf/test2.so.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test20-pr19025-libvtkParallelCore-6.1.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test21-pr19092.so.abi: Likewise.
	* tests/data/test-read-dwarf/test22-pr19097-libstdc++.so.6.0.17.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test3-alias-1.so.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test3-alias-2.so.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test3-alias-3.so.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test3-alias-4.so.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test3.so.abi: Likewise.
	* tests/data/test-read-dwarf/test3.so.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test4.so.abi: Likewise.
	* tests/data/test-read-dwarf/test4.so.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test5.o.abi: Likewise.
	* tests/data/test-read-dwarf/test5.o.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test6.so.abi: Likewise.
	* tests/data/test-read-dwarf/test6.so.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test7.so.abi: Likewise.
	* tests/data/test-read-dwarf/test7.so.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test8-qualified-this-pointer.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test8-qualified-this-pointer.so.hash.abi:
	Likewise.
	* tests/data/test-read-dwarf/test9-pr18818-clang.so.abi: Likewise.
	* tests/data/test-read-write/test-crc.xml: Likewise.
	* tests/data/test-read-write/test0.xml: Likewise.
	* tests/data/test-read-write/test1.xml: Likewise.
	* tests/data/test-read-write/test10.xml: Likewise.
	* tests/data/test-read-write/test11.xml: Likewise.
	* tests/data/test-read-write/test12.xml: Likewise.
	* tests/data/test-read-write/test13.xml: Likewise.
	* tests/data/test-read-write/test14.xml: Likewise.
	* tests/data/test-read-write/test15.xml: Likewise.
	* tests/data/test-read-write/test16.xml: Likewise.
	* tests/data/test-read-write/test17.xml: Likewise.
	* tests/data/test-read-write/test18.xml: Likewise.
	* tests/data/test-read-write/test19.xml: Likewise.
	* tests/data/test-read-write/test2.xml: Likewise.
	* tests/data/test-read-write/test20.xml: Likewise.
	* tests/data/test-read-write/test21.xml: Likewise.
	* tests/data/test-read-write/test22.xml: Likewise.
	* tests/data/test-read-write/test23.xml: Likewise.
	* tests/data/test-read-write/test24.xml: Likewise.
	* tests/data/test-read-write/test25.xml: Likewise.
	* tests/data/test-read-write/test26.xml: Likewise.
	* tests/data/test-read-write/test27.xml: Likewise.
	* tests/data/test-read-write/test28-without-std-fns-ref.xml:
	Likewise.
	* tests/data/test-read-write/test28-without-std-vars-ref.xml:
	Likewise.
	* tests/data/test-read-write/test3.xml: Likewise.
	* tests/data/test-read-write/test4.xml: Likewise.
	* tests/data/test-read-write/test5.xml: Likewise.
	* tests/data/test-read-write/test6.xml: Likewise.
	* tests/data/test-read-write/test7.xml: Likewise.
	* tests/data/test-read-write/test8.xml: Likewise.
	* tests/data/test-read-write/test9.xml: Likewise.

Signed-off-by: Dodji Seketeli <dodji@redhat.com>
---
 configure.ac                                  |    12 +-
 include/abg-fwd.h                             |    13 +-
 include/abg-hash.h                            |   249 +-
 include/abg-ir.h                              |   414 +-
 src/abg-btf-reader.cc                         |    30 +-
 src/abg-comparison-priv.h                     |     2 +-
 src/abg-ctf-reader.cc                         |    42 +-
 src/abg-dwarf-reader.cc                       |   102 +-
 src/abg-hash.cc                               |  1476 +-
 src/abg-ir-priv.h                             |   612 +-
 src/abg-ir.cc                                 |   917 +-
 src/abg-reader.cc                             |   145 +-
 src/abg-writer.cc                             |   409 +-
 tests/Makefile.am                             |     8 +
 .../test-abidiff/test-PR18791-report0.txt     |    21 +
 .../PR29443-missing-xx.o.annotated.abi        |     8 +-
 tests/data/test-annotate/libtest23.so.abi     |   230 +-
 .../test-annotate/libtest24-drop-fns-2.so.abi |   350 +-
 .../test-annotate/libtest24-drop-fns.so.abi   |   350 +-
 .../test-anonymous-members-0.o.abi            |    34 +-
 .../test-pointer-to-member-1.o.annotated.abi  |    34 +-
 tests/data/test-annotate/test0.abi            |    46 +-
 tests/data/test-annotate/test1.abi            |    44 +-
 .../data/test-annotate/test13-pr18894.so.abi  |  1264 +-
 .../data/test-annotate/test14-pr18893.so.abi  |  2923 +-
 .../data/test-annotate/test15-pr18892.so.abi  | 18646 ++++----
 .../data/test-annotate/test17-pr19027.so.abi  |  6106 +--
 ...st18-pr19037-libvtkRenderingLIC-6.1.so.abi | 10454 ++---
 ...19-pr19023-libtcmalloc_and_profiler.so.abi |  9864 +++--
 tests/data/test-annotate/test2.so.abi         |    30 +-
 ...st20-pr19025-libvtkParallelCore-6.1.so.abi |  9082 ++--
 .../data/test-annotate/test21-pr19092.so.abi  |  3848 +-
 tests/data/test-annotate/test3.so.abi         |     4 +-
 tests/data/test-annotate/test4.so.abi         |    18 +-
 tests/data/test-annotate/test5.o.abi          |     4 +-
 tests/data/test-annotate/test6.so.abi         |    18 +-
 tests/data/test-annotate/test7.so.abi         |    52 +-
 .../test8-qualified-this-pointer.so.abi       |    16 +-
 .../PR25409-librte_bus_dpaa.so.20.0.abi       |  2796 +-
 .../test0-pr19026-libvtkIOSQL-6.1.so.1.abi    |  7765 ++--
 .../test42-PR21296-clanggcc-report0.txt       |     6 +-
 .../test-diff-dwarf/test46-rust-report-0.txt  |     2 -
 .../test-PR26739-2-report-0.txt               |     2 +-
 .../test30-pr18904-rvalueref-report0.txt      |     2 +-
 .../test30-pr18904-rvalueref-report1.txt      |     2 +-
 .../test30-pr18904-rvalueref-report2.txt      |     2 +-
 .../test31-pr18535-libstdc++-report-0.txt     |     2 +-
 .../test31-pr18535-libstdc++-report-1.txt     |     2 +-
 .../test35-pr18754-no-added-syms-report-0.txt |     2 +-
 .../test35-pr18754-no-added-syms-report-1.txt |     2 +-
 ...t43-decl-only-def-change-leaf-report-0.txt |     5 +
 ...bb-4.3-3.20141204.fc23.x86_64-report-0.txt |    20 +-
 ...bb-4.3-3.20141204.fc23.x86_64-report-1.txt |    20 +-
 tests/data/test-read-btf/test0.o.abi          |    74 +-
 tests/data/test-read-btf/test1.o.abi          |    12 +-
 .../test-read-ctf/PR27700/test-PR27700.abi    |    10 +-
 tests/data/test-read-ctf/test-PR26568-1.o.abi |    18 +-
 tests/data/test-read-ctf/test-PR26568-2.o.abi |    16 +-
 tests/data/test-read-ctf/test-alias.o.abi     |     6 +-
 .../test-ambiguous-struct-A.o.hash.abi        |    20 +-
 .../test-ambiguous-struct-B.o.hash.abi        |    10 +-
 .../test-read-ctf/test-anonymous-fields.o.abi |    14 +-
 .../test-read-ctf/test-array-mdimension.abi   |    16 +-
 .../test-read-ctf/test-array-of-pointers.abi  |    24 +-
 tests/data/test-read-ctf/test-array-size.abi  |    18 +-
 .../data/test-read-ctf/test-bitfield-enum.abi |     8 +-
 tests/data/test-read-ctf/test-bitfield.abi    |    10 +-
 tests/data/test-read-ctf/test-callback.abi    |    10 +-
 tests/data/test-read-ctf/test-callback2.abi   |    12 +-
 .../test-conflicting-type-syms-a.o.hash.abi   |    10 +-
 .../test-conflicting-type-syms-b.o.hash.abi   |     8 +-
 tests/data/test-read-ctf/test-const-array.abi |    12 +-
 .../test-read-ctf/test-dynamic-array.o.abi    |    18 +-
 .../test-read-ctf/test-enum-many.o.hash.abi   |     8 +-
 .../test-read-ctf/test-enum-symbol.o.hash.abi |     6 +-
 tests/data/test-read-ctf/test-enum.o.abi      |     8 +-
 tests/data/test-read-ctf/test-fallback.abi    |     4 +-
 .../test-read-ctf/test-forward-type-decl.abi  |    14 +-
 .../test-functions-declaration.abi            |    14 +-
 .../data/test-read-ctf/test-linux-module.abi  |    38 +-
 tests/data/test-read-ctf/test-list-struct.abi |     8 +-
 tests/data/test-read-ctf/test0.abi            |    40 +-
 tests/data/test-read-ctf/test0.hash.abi       |    40 +-
 tests/data/test-read-ctf/test1.so.abi         |    18 +-
 tests/data/test-read-ctf/test1.so.hash.abi    |    18 +-
 tests/data/test-read-ctf/test2.so.abi         |    18 +-
 tests/data/test-read-ctf/test2.so.hash.abi    |    18 +-
 tests/data/test-read-ctf/test3.so.abi         |     4 +-
 tests/data/test-read-ctf/test3.so.hash.abi    |     4 +-
 tests/data/test-read-ctf/test4.so.abi         |    18 +-
 tests/data/test-read-ctf/test4.so.hash.abi    |    18 +-
 tests/data/test-read-ctf/test5.o.abi          |    34 +-
 tests/data/test-read-ctf/test7.o.abi          |    22 +-
 tests/data/test-read-ctf/test8.o.abi          |     4 +-
 tests/data/test-read-ctf/test9.o.abi          |    48 +-
 .../PR22015-libboost_iostreams.so.abi         |  1537 +-
 .../test-read-dwarf/PR22122-libftdc.so.abi    |  5737 +--
 .../PR24378-fn-is-not-scope.abi               |    10 +-
 .../data/test-read-dwarf/PR25007-sdhci.ko.abi |  2938 +-
 .../PR25042-libgdbm-clang-dwarf5.so.6.0.0.abi |   609 +-
 .../test-read-dwarf/PR26261/PR26261-exe.abi   |    44 +-
 .../test-read-dwarf/PR27700/test-PR27700.abi  |    10 +-
 .../PR28584/PR28584-smv.clang.o.abi           |     6 +-
 .../test-read-dwarf/PR29443-missing-xx.o.abi  |     8 +-
 .../PR29692-kdelibs3-libkjava.so.1.0.0.abi    |     2 +-
 tests/data/test-read-dwarf/libtest23.so.abi   |   229 +-
 .../libtest24-drop-fns-2.so.abi               |   346 +-
 .../test-read-dwarf/libtest24-drop-fns.so.abi |   346 +-
 .../data/test-read-dwarf/test-PR26568-1.o.abi |    18 +-
 .../data/test-read-dwarf/test-PR26568-2.o.abi |    16 +-
 tests/data/test-read-dwarf/test-fallback.abi  |     4 +-
 .../test-read-dwarf/test-libaaudio.so.abi     |   424 +-
 .../test-read-dwarf/test-libandroid.so.abi    | 20488 +++++----
 .../test-pointer-to-member-1.o.abi            |    34 +-
 .../test-suppressed-alias.o.abi               |     6 +-
 tests/data/test-read-dwarf/test0.abi          |    46 +-
 tests/data/test-read-dwarf/test0.hash.abi     |    48 +-
 tests/data/test-read-dwarf/test1.abi          |    44 +-
 tests/data/test-read-dwarf/test1.hash.abi     |    50 +-
 .../test-read-dwarf/test10-pr18818-gcc.so.abi |  3702 +-
 .../test-read-dwarf/test11-pr18828.so.abi     |  9682 ++--
 .../test-read-dwarf/test12-pr18844.so.abi     | 21211 ++++-----
 .../test-read-dwarf/test13-pr18894.so.abi     |  1260 +-
 .../test-read-dwarf/test14-pr18893.so.abi     |  2769 +-
 .../test-read-dwarf/test15-pr18892.so.abi     | 20073 ++++-----
 .../test-read-dwarf/test16-pr18904.so.abi     | 17081 +++----
 .../test-read-dwarf/test17-pr19027.so.abi     |  6127 +--
 ...st18-pr19037-libvtkRenderingLIC-6.1.so.abi |  9029 ++--
 ...19-pr19023-libtcmalloc_and_profiler.so.abi |  9372 ++--
 tests/data/test-read-dwarf/test2.so.abi       |    30 +-
 tests/data/test-read-dwarf/test2.so.hash.abi  |    30 +-
 ...st20-pr19025-libvtkParallelCore-6.1.so.abi |  7707 ++--
 .../test-read-dwarf/test21-pr19092.so.abi     |  3804 +-
 .../test22-pr19097-libstdc++.so.6.0.17.so.abi | 36979 ++++++++--------
 .../test-read-dwarf/test3-alias-1.so.hash.abi |     4 +-
 .../test-read-dwarf/test3-alias-2.so.hash.abi |     4 +-
 .../test-read-dwarf/test3-alias-3.so.hash.abi |     4 +-
 .../test-read-dwarf/test3-alias-4.so.hash.abi |     4 +-
 tests/data/test-read-dwarf/test3.so.abi       |     4 +-
 tests/data/test-read-dwarf/test3.so.hash.abi  |     4 +-
 tests/data/test-read-dwarf/test4.so.abi       |    18 +-
 tests/data/test-read-dwarf/test4.so.hash.abi  |    18 +-
 tests/data/test-read-dwarf/test5.o.abi        |     4 +-
 tests/data/test-read-dwarf/test5.o.hash.abi   |     4 +-
 tests/data/test-read-dwarf/test6.so.abi       |    18 +-
 tests/data/test-read-dwarf/test6.so.hash.abi  |    18 +-
 tests/data/test-read-dwarf/test7.so.abi       |    52 +-
 tests/data/test-read-dwarf/test7.so.hash.abi  |    52 +-
 .../test8-qualified-this-pointer.so.abi       |    16 +-
 .../test8-qualified-this-pointer.so.hash.abi  |    16 +-
 .../test9-pr18818-clang.so.abi                |  3464 +-
 tests/data/test-read-write/test-crc.xml       |     2 +-
 tests/data/test-read-write/test0.xml          |     2 +-
 tests/data/test-read-write/test1.xml          |     2 +-
 tests/data/test-read-write/test10.xml         |    14 +-
 tests/data/test-read-write/test11.xml         |     4 +-
 tests/data/test-read-write/test12.xml         |     6 +-
 tests/data/test-read-write/test13.xml         |     8 +-
 tests/data/test-read-write/test14.xml         |     8 +-
 tests/data/test-read-write/test15.xml         |     6 +-
 tests/data/test-read-write/test16.xml         |     4 +-
 tests/data/test-read-write/test17.xml         |    54 +-
 tests/data/test-read-write/test18.xml         |    16 +-
 tests/data/test-read-write/test19.xml         |    22 +-
 tests/data/test-read-write/test2.xml          |     8 +-
 tests/data/test-read-write/test20.xml         |    88 +-
 tests/data/test-read-write/test21.xml         |    26 +-
 tests/data/test-read-write/test22.xml         |    12 +-
 tests/data/test-read-write/test23.xml         |    12 +-
 tests/data/test-read-write/test24.xml         |    16 +-
 tests/data/test-read-write/test25.xml         |    62 +-
 tests/data/test-read-write/test26.xml         |    22 +-
 tests/data/test-read-write/test27.xml         |    16 +-
 .../test28-without-std-fns-ref.xml            |  1190 +-
 .../test28-without-std-vars-ref.xml           |  1116 +-
 tests/data/test-read-write/test3.xml          |     8 +-
 tests/data/test-read-write/test4.xml          |     4 +-
 tests/data/test-read-write/test5.xml          |     4 +-
 tests/data/test-read-write/test6.xml          |    10 +-
 tests/data/test-read-write/test7.xml          |     4 +-
 tests/data/test-read-write/test8.xml          |     2 +-
 tests/data/test-read-write/test9.xml          |     8 +-
 182 files changed, 137455 insertions(+), 130098 deletions(-)

The patch is too big so I am adding attaching it gzip'ed.

Dodji Seketeli Aug. 29, 2024, 4:01 p.m. UTC | #2

Hello,

Sometimes, anonymous member types are duplicated in some classes read
by the DWARF reader.  This is because the DWARF reader doesn't lookup
anonymous member types in their class scope before adding them into
their scope.

This patch teaches the DWARF reader how to name an anonymous class,
union or enum by using its flat pretty representation, e.g:

    struct {int blah; char meh;}

That flat representation is used as the name of the anonymous type to
look it up in a given class scope before adding it to that scope.  If
the scope already contains the type, then the type is not added.

Now that class scopes have their proper anonymous member types, it
appeared that the filtering of change reports needed a number of
adaptations because we are now seeing things that we were not seeing
before.  For instance two new change categories
NON_COMPATIBLE_DISTINCT_CHANGE_CATEGORY and
NON_COMPATIBLE_NAME_CHANGE_CATEGORY are added.  The former is for a
change where a given type becomes another type of a different kind in
an non-compatible way.  For instance, a struct type becomes an enum
type.  The later category is for a change in a name of the type,
resulting in a non-compatible change.  Thus, new non-compatible
changes can now be categorized and reported as such.

	* include/abg-comp-filter.h (has_void_ptr_to_ptr_change)
	(has_harmless_enum_to_int_change)
	(has_benign_array_of_unknown_size_change): Declare new functions.
	* include/abg-comparison.h (enum diff_category): Add
	NON_COMPATIBLE_DISTINCT_CHANGE_CATEGORY,
	NON_COMPATIBLE_NAME_CHANGE_CATEGORY enumerators.  Update the other
	enumerator values.  Also, update the EVERYTHING_CATEGORY
	enumerator value.
	* include/abg-ir.h (class_decl::find_base_class): Document the
	name of the parameter.
	* src/abg-comp-filter.cc (type_diff_has_cv_qual_change_only)
	(is_non_compatible_distinct_change, is_void_ptr_to_ptr): Define
	new static functions.
	(has_subtype_changes): Remove static function.
	(class_diff_has_only_harmless_changes): Do not bother checking if
	the class_diff has subtype changes.  If there are, the category of
	those changes is going to be propagated upward to this current
	diff node.  Update the comments.
	(has_harmless_name_change): A diff node carrying compatible types
	or types that differ only from their CV qualifiers should qualify
	as having a harmless name change.
	(has_harmless_enum_to_int_change): Make this function be
	non-static.
	(type_diff_has_cv_qual_change_only): Introduce an overload that
	takes two ABI artifacts instead of one diff node.  Make the diff
	node overload use the new one.  Simplify logic by essentially
	peeling off qualified or typedefs first.  Then if what remain is
	pointer or array types, look at their underlying types.  If the
	remaining underlying types are equal then return true.
	(has_void_ptr_to_ptr_change): Use the new is_void_ptr_to_ptr
	function.  This allows the function to detect void* -> pointer and
	pointer -> void* changes.  And it simplifies the logic.
	* src/abg-comparison.cc (distinct_diff::compatible_child_diff):
	Build a compatible child diff node iff the two diff subject are
	actually compatible meaning, they are equal modulo a typedef.
	(get_default_harmful_categories_bitmap): Add the new
	abigail::comparison::{NON_COMPATIBLE_DISTINCT_CHANGE_CATEGORY,NON_COMPATIBLE_NAME_CHANGE_CATEGORY}
	enumerators to the bitmap of harmful categories.
	(operator<<(ostream& o, diff_category c)): Update this to support
	emitting the new
	abigail::comparison::{NON_COMPATIBLE_DISTINCT_CHANGE_CATEGORY,NON_COMPATIBLE_NAME_CHANGE_CATEGORY}
	enumerators.
	(class_or_union_diff::priv::count_filtered_{subtype_}changed_dm):
	Fix logic for counting local changes.
	diff::has_local_changes_to_be_reported is not reliable so do not
	rely on it.  We might want to remove that function in the end.
	(category_propagation_visitor::visit_end): If the current diff
	node has a harmless void pointer to pointer, enum to int, or a
	benign array of unknown size change, do not propagate the
	NON_COMPATIBLE_{NAME,DISTINCT}_CHANGE_CATEGORY that have
	necessarily bubbled up from some of their distant children nodes.
	* src/abg-dwarf-reader.cc
	(maybe_set_member_type_access_specifier)
	(get_next_member_sibling_die): Declare this pre-existing static
	function.
	(get_internal_anonymous_die_name)
	(lookup_class_typedef_or_enum_type_from_corpus)
	(lookup_class_typedef_or_enum_type_from_corpus): Remove.
	(die_return_and_parm_names_from_fn_type_die): Add is_method_type
	and return_type_name parameters. Fix the representation of member
	functions.
	(die_function_signature): Add qualified_name parameter. Adjust
	call to die_return_and_parm_names_from_fn_type_die.
	(die_type_name, die_enum_flat_representation)
	(die_class_flat_representation)
	(die_class_or_enum_flat_representation): Define new static
	functions.
	(reader::get_die_qualified_type_name, die_qualified_type_name):
	For anonymous class or enum types, use the new
	die_class_or_enum_flat_representation.  Adjust the call to
	die_return_and_parm_names_from_fn_type_die. For function types fix
	the name representation.
	(die_pretty_print_type): Adjust call to
	die_return_and_parm_names_from_fn_type_die.
	(die_pretty_print_decl): Take qualified_name and include_fns
	parameters.  Use the new die_type_name for variable type names.
	Adjust call to die_function_signature.
	(die_pretty_print): Adjust call die_pretty_print_decl.
	(get_member_child_die): Remove useless vertical space.
	(build_enum_underlying_type): Sort this enum underlying type (and
	potentially hash it) before canonicalizing it.
	(add_or_update_class_type): If an anonymous type is in the global
	scope, take that into account when building its internal anonymous
	die name.  Lookup anonymous member types before adding them to the
	class scope to avoid duplicating them in their scope.
	(add_or_update_union_type): Likewise, lookup anonymous member
	types before adding them to the union scope to avoid duplicating
	them in their scope.
	(build_subranges_from_array_type_die): Const-ify the input reader.
	Adjust call to build_ir_node_from_die.  Associate the subrange DIE
	to the IR node built.
	(build_ir_node_from_die): Get the current corpus from the reader
	because it might be set for scope passed to this function, or the
	scope might be nullptr.  Schedule base types for canonicalization
	like all the other types.  Also, lookup all class/union types
	before adding them to their class scope.
	* src/abg-leaf-reporter.cc (leaf_reporter::report): In the
	overload for class_or_union_diff, report changed data members and
	their sub-types.
	* tests/Makefile.am: Do not XFAIL the test runtestabidiffexit.  No
	more tests are XFAILED.  All tests should now pass.
	* tests/data/test-abidiff-exit/PR30503/libsdl/libsdl-1.2.60-1.2.64-report.txt:
	Adjust.
	* tests/data/test-abidiff-exit/PR31513/reported/PR31513-reported-report-1.txt:
	Likewise.
	* tests/data/test-abidiff-exit/qualifier-typedef-array-report-1.txt:
	Likewise.
	* tests/data/test-annotate/libtest24-drop-fns-2.so.abi: Likewise.
	* tests/data/test-annotate/libtest24-drop-fns.so.abi: Likewise.
	* tests/data/test-annotate/test0.abi: Likewise.
	* tests/data/test-annotate/test1.abi: Likewise.
	* tests/data/test-annotate/test15-pr18892.so.abi: Likewise.
	* tests/data/test-annotate/test18-pr19037-libvtkRenderingLIC-6.1.so.abi:
	Likewise.
	* tests/data/test-annotate/test19-pr19023-libtcmalloc_and_profiler.so.abi:
	Likewise.
	* tests/data/test-annotate/test20-pr19025-libvtkParallelCore-6.1.so.abi:
	Likewise.
	* tests/data/test-annotate/test21-pr19092.so.abi: Likewise.
	* tests/data/test-diff-dwarf/PR25058-liblttng-ctl-report-1.txt:
	Likewise.
	* tests/data/test-diff-dwarf/test4-report.txt: Likewise.
	* tests/data/test-diff-dwarf/test5-report.txt: Likewise.
	* tests/data/test-diff-filter/test-PR26739-2-report-0.txt:
	Likewise.
	* tests/data/test-diff-filter/test30-pr18904-rvalueref-report0.txt:
	Likewise.
	* tests/data/test-diff-filter/test30-pr18904-rvalueref-report1.txt:
	Likewise.
	* tests/data/test-diff-filter/test30-pr18904-rvalueref-report2.txt:
	Likewise.
	* tests/data/test-diff-filter/test31-pr18535-libstdc++-report-0.txt:
	Likewise.
	* tests/data/test-diff-filter/test31-pr18535-libstdc++-report-1.txt:
	Likewise.
	* tests/data/test-diff-filter/test35-pr18754-no-added-syms-report-0.txt:
	Likewise.
	* tests/data/test-diff-filter/test41-report-0.txt: Likewise.
	* tests/data/test-diff-filter/test5-report.txt: Likewise.
	* tests/data/test-diff-pkg/libICE-1.0.6-1.el6.x86_64.rpm--libICE-1.0.9-2.el7.x86_64.rpm-report-0.txt:
	Likewise.
	* tests/data/test-diff-pkg/spice-server-0.12.4-19.el7.x86_64-0.12.8-1.el7.x86_64-report-2.txt:
	Likewise.
	* tests/data/test-diff-pkg/tbb-4.1-9.20130314.fc22.x86_64--tbb-4.3-3.20141204.fc23.x86_64-report-0.txt:
	Likewise.
	* tests/data/test-diff-pkg/tbb-4.1-9.20130314.fc22.x86_64--tbb-4.3-3.20141204.fc23.x86_64-report-1.txt:
	Likewise.
	* tests/data/test-read-dwarf/PR22015-libboost_iostreams.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/PR22122-libftdc.so.abi: Likewise.
	* tests/data/test-read-dwarf/PR24378-fn-is-not-scope.abi:
	Likewise.
	* tests/data/test-read-dwarf/PR25007-sdhci.ko.abi: Likewise.
	* tests/data/test-read-dwarf/libtest24-drop-fns-2.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/libtest24-drop-fns.so.abi: Likewise.
	* tests/data/test-read-dwarf/test-libaaudio.so.abi: Likewise.
	* tests/data/test-read-dwarf/test-libandroid.so.abi: Likewise.
	* tests/data/test-read-dwarf/test0.abi: Likewise.
	* tests/data/test-read-dwarf/test0.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test1.abi: Likewise.
	* tests/data/test-read-dwarf/test1.hash.abi: Likewise.
	* tests/data/test-read-dwarf/test10-pr18818-gcc.so.abi: Likewise.
	* tests/data/test-read-dwarf/test11-pr18828.so.abi: Likewise.
	* tests/data/test-read-dwarf/test12-pr18844.so.abi: Likewise.
	* tests/data/test-read-dwarf/test15-pr18892.so.abi: Likewise.
	* tests/data/test-read-dwarf/test16-pr18904.so.abi: Likewise.
	* tests/data/test-read-dwarf/test18-pr19037-libvtkRenderingLIC-6.1.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test19-pr19023-libtcmalloc_and_profiler.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test20-pr19025-libvtkParallelCore-6.1.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test21-pr19092.so.abi: Likewise.
	* tests/data/test-read-dwarf/test22-pr19097-libstdc++.so.6.0.17.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test9-pr18818-clang.so.abi: Likewise.

Signed-off-by: Dodji Seketeli <dodji@redhat.com>
---
 include/abg-comp-filter.h                     |     9 +
 include/abg-comparison.h                      |    32 +-
 include/abg-ir.h                              |     2 +-
 src/abg-comp-filter.cc                        |   225 +-
 src/abg-comparison.cc                         |    64 +-
 src/abg-dwarf-reader.cc                       |  1024 +-
 src/abg-leaf-reporter.cc                      |    24 +-
 tests/Makefile.am                             |     2 -
 .../libsdl/libsdl-1.2.60-1.2.64-report.txt    |    12 +-
 .../reported/PR31513-reported-report-1.txt    |    20 +-
 .../qualifier-typedef-array-report-1.txt      |    64 +-
 .../test-annotate/libtest24-drop-fns-2.so.abi |     4 +-
 .../test-annotate/libtest24-drop-fns.so.abi   |     4 +-
 tests/data/test-annotate/test0.abi            |     4 +-
 tests/data/test-annotate/test1.abi            |     4 +-
 .../data/test-annotate/test15-pr18892.so.abi  |  8368 +++---
 ...st18-pr19037-libvtkRenderingLIC-6.1.so.abi |  2181 +-
 ...19-pr19023-libtcmalloc_and_profiler.so.abi |  5514 ++--
 ...st20-pr19025-libvtkParallelCore-6.1.so.abi |  6075 ++--
 .../data/test-annotate/test21-pr19092.so.abi  |     6 +-
 .../PR25058-liblttng-ctl-report-1.txt         |    15 +-
 tests/data/test-diff-dwarf/test4-report.txt   |     6 +-
 tests/data/test-diff-dwarf/test5-report.txt   |     4 +-
 .../test-PR26739-2-report-0.txt               |    16 +-
 .../test30-pr18904-rvalueref-report0.txt      |    46 +-
 .../test30-pr18904-rvalueref-report1.txt      |    46 +-
 .../test30-pr18904-rvalueref-report2.txt      |    46 +-
 .../test31-pr18535-libstdc++-report-0.txt     |     2 +-
 .../test31-pr18535-libstdc++-report-1.txt     |     2 +-
 .../test35-pr18754-no-added-syms-report-0.txt |    46 +-
 .../data/test-diff-filter/test41-report-0.txt |    32 +-
 tests/data/test-diff-filter/test5-report.txt  |     9 +-
 ...libICE-1.0.9-2.el7.x86_64.rpm-report-0.txt |    13 +-
 ...l7.x86_64-0.12.8-1.el7.x86_64-report-2.txt |    23 +-
 ...bb-4.3-3.20141204.fc23.x86_64-report-0.txt |    59 +-
 ...bb-4.3-3.20141204.fc23.x86_64-report-1.txt |    45 +-
 .../PR22015-libboost_iostreams.so.abi         |     6 +-
 .../test-read-dwarf/PR22122-libftdc.so.abi    |  3741 ++-
 .../PR24378-fn-is-not-scope.abi               |    22 +-
 .../data/test-read-dwarf/PR25007-sdhci.ko.abi |     6 +-
 .../libtest24-drop-fns-2.so.abi               |     4 +-
 .../test-read-dwarf/libtest24-drop-fns.so.abi |     4 +-
 .../test-read-dwarf/test-libaaudio.so.abi     |     6 +-
 .../test-read-dwarf/test-libandroid.so.abi    |   197 +-
 tests/data/test-read-dwarf/test0.abi          |     4 +-
 tests/data/test-read-dwarf/test0.hash.abi     |     4 +-
 tests/data/test-read-dwarf/test1.abi          |     4 +-
 tests/data/test-read-dwarf/test1.hash.abi     |     4 +-
 .../test-read-dwarf/test10-pr18818-gcc.so.abi |   524 +-
 .../test-read-dwarf/test11-pr18828.so.abi     |    10 +-
 .../test-read-dwarf/test12-pr18844.so.abi     | 18079 +++++------
 .../test-read-dwarf/test15-pr18892.so.abi     |  8306 +++---
 .../test-read-dwarf/test16-pr18904.so.abi     |  8470 +++---
 ...st18-pr19037-libvtkRenderingLIC-6.1.so.abi |  2143 +-
 ...19-pr19023-libtcmalloc_and_profiler.so.abi |  5485 ++--
 ...st20-pr19025-libvtkParallelCore-6.1.so.abi |  5964 ++--
 .../test-read-dwarf/test21-pr19092.so.abi     |     6 +-
 .../test22-pr19097-libstdc++.so.6.0.17.so.abi | 24829 ++++++++--------
 .../test9-pr18818-clang.so.abi                |   857 +-
 59 files changed, 51445 insertions(+), 51278 deletions(-)

The patch is too big so I am attaching it gzip'ed.

Dodji Seketeli Aug. 29, 2024, 4:13 p.m. UTC | #3

Hello,

abipkgdiff extracts the two devel packages in directories named
"devel_package{1,2}, whereas the main packages are extracted in
directories named package{1,2}.  This is wrong because a devel package
should be extracted into the same directory tree as the main package.
For instance, there can be a foo.so symbolic link (coming from the
devel package) that points to the foo-1.0.0.so (coming from the main
package) below usr/lib64/ in the extracted directory.  If the devel
and main packages are extracted in different directories then the
symbolic link is going to be broken and some binaries won't be
ABI-compared.

This patch fixes the issue by extracting devel packages into the same
directory as the main one.  Then, it makes sure the devel package and
the main one are extracted sequentially, not in parallel since they
are not being extracted into the same directory.

By fixing this, a number of previously hidden issues or even change
output are now exposed.  The patch updates these.

	* tests/data/test-diff-pkg/gcc-debuginfo-14.2.1-1.fc40.x86_64.rpm:
	Add missing debug info package.
	* tests/data/Makefile.am: Add it to source distribution.
	* tests/test-diff-pkg.cc (in_out_specs): Add the debuginfo package
	to the libgm2 test.
	* tools/abipkgdiff.cc (main): Extract the devel packages into the
	"package1" and "package2" directories.
	(extract_package_and_map_its_content): Extract the main and devel
	packages sequentially, not in parallel.
	(self_compare_prepared_userspace_package): Add more logs.
	* tests/data/test-fedabipkgdiff/test0-from-fc20-to-fc23-dbus-glib-report-0.txt:
	Adjust.
	* tests/data/test-fedabipkgdiff/test1-from-fc20-to-dbus-glib-0.106-1.fc23.x86_64-report-0.txt:
	Likewise
	* tests/data/test-fedabipkgdiff/test2-dbus-glib-0.100.2-2.fc20--dbus-glib-0.106-1.fc23-report-0.txt:
	Likewise
	* tests/data/test-fedabipkgdiff/test3-dbus-glib-0.100.2-2.fc20.i686--dbus-glib-0.106-1.fc23.i686-report-0.txt:
	Likewise
	* tests/data/test-fedabipkgdiff/test4-glib-0.100.2-2.fc20.x86_64.rpm-glib-0.106-1.fc23.x86_64.rpm-report-0.txt:
	Likewise
	* tests/data/test-fedabipkgdiff/test7-self-compare-from-fc23-dbus-glib-report-0.txt:
	Likewise
	* tests/data/test-fedabipkgdiff/vte291-0.39.1-1.fc22.x86_64--vte291-0.39.90-1.fc22.x86_64-report-0.txt:
	Likewise

Signed-off-by: Dodji Seketeli <dodji@redhat.com>
---
 tests/data/Makefile.am                        |   1 +
 .../gcc-debuginfo-14.2.1-1.fc40.x86_64.rpm    | Bin 0 -> 122904364 bytes
 ...0-from-fc20-to-fc23-dbus-glib-report-0.txt |  26 ++++++++++++
 ...dbus-glib-0.106-1.fc23.x86_64-report-0.txt |  11 +++++
 ....fc20--dbus-glib-0.106-1.fc23-report-0.txt |  26 ++++++++++++
 ...--dbus-glib-0.106-1.fc23.i686-report-0.txt |  15 +++++++
 ...-glib-0.106-1.fc23.x86_64.rpm-report-0.txt |  11 +++++
 ...f-compare-from-fc23-dbus-glib-report-0.txt |   2 +
 ...-vte291-0.39.90-1.fc22.x86_64-report-0.txt |   1 +
 tests/test-diff-pkg.cc                        |   3 +-
 tools/abipkgdiff.cc                           |  38 +++++++++++-------
 11 files changed, 119 insertions(+), 15 deletions(-)
 create mode 100644 tests/data/test-diff-pkg/gcc-debuginfo-14.2.1-1.fc40.x86_64.rpm

The patch is too big.  So you can read it online at: https://sourceware.org/git/?p=libabigail.git;a=commit;h=5be5809f7c95f13cf50be312979c04efcb829aca

Dodji Seketeli Aug. 29, 2024, 4:15 p.m. UTC | #4

Hello,

To support the duality concrete/abstract instance of static data
member variables in DWARF, the reader needed a little bit of an
overhaul.  That overhaul is necessary to self-compare libLLVM.so, from
llvm-libs, in fc37.  Note that that self-compare is done using this
command:

    $ fedabipkgdiff -a--self-compare --from fc37 llvm-libs

So now, with this patch, the IR of a static data member is constructed
only when the concrete instance DIE of the static data member is
encountered.  The abstract instance DIE is then read first, the IR is
built from it, and then the concrete instance DIE is read for
additional attributes that might be needed, like the ELF symbol
attribute.

The patch fixes a number of issues related to static data members
representation in the IR, namely, it puts them all in their own array
in the class_or_union, just like what is done for non-static data
members.  The ABIXML writer is updated to emit static data member from
their new storage.

	* include/abg-ir.h (class_or_union::get_static_data_members):
	Declare new method.
	* src/abg-dwarf-reader.cc (add_or_update_class_type): Do not
	create any IR for the /declaration/ of static data member here.
	Wait for its definition from a concrete instance DIE to create the
	IR for it.
	(build_ir_node_from_die): In the DW_TAG_variable case, if we are
	looking at the concrete instance of a static variable (one that
	has a DW_AT_abstract_origin pointing to a static data member) then
	build the IR from it and add it to its class scope.
	* src/abg-ir-priv.h (class_or_union::priv::static_data_members_):
	Define new data member.
	(class_or_union::priv::priv): Stick the static data members that
	are in class_or_union::priv::data_members_ into
	class_or_union::priv::static_data_members_.
	* src/abg-ir.cc (maybe_adjust_canonical_type): Make sure static
	data members of the canonical type has the same symbols set as the
	data members of the canonicalized type.
	(class_or_union::add_data_member): Update the new
	class_or_union::priv::static_data_members_ when a data member is
	added.
	(class_or_union::get_non_static_data_members): Fix comments.
	(class_or_union::get_static_data_members): Define new member
	functions.
	(set_member_is_static): Update the new
	class_or_union::priv::static_data_members_ when static-ness is
	changed.
	* src/abg-writer.cc (write_decl_in_scope): Support writing pure
	declarations (not types).
	(write_translation_unit): Don't forget to write remaining global
	variables.
	(write_class_decl): Make sure to write static data members first,
	before writing the non-static data members.
	* tests/data/test-abidiff-exit/test-ld-2.28-210.so--ld-2.28-211.so.txt:
	Adjust.
	* tests/data/test-annotate/libtest23.so.abi: Likewise.
	* tests/data/test-annotate/libtest24-drop-fns-2.so.abi: Likewise.
	* tests/data/test-annotate/libtest24-drop-fns.so.abi: Likewise.
	* tests/data/test-annotate/test14-pr18893.so.abi: Likewise.
	* tests/data/test-annotate/test15-pr18892.so.abi: Likewise.
	* tests/data/test-annotate/test17-pr19027.so.abi: Likewise.
	* tests/data/test-annotate/test18-pr19037-libvtkRenderingLIC-6.1.so.abi:
	Likewise.
	* tests/data/test-annotate/test19-pr19023-libtcmalloc_and_profiler.so.abi:
	Likewise.
	* tests/data/test-annotate/test20-pr19025-libvtkParallelCore-6.1.so.abi:
	Likewise.
	* tests/data/test-diff-dwarf-abixml/PR25409-librte_bus_dpaa.so.20.0.abi:
	Likewise.
	* tests/data/test-diff-dwarf-abixml/test0-pr19026-libvtkIOSQL-6.1.so.1.abi:
	Likewise.
	* tests/data/test-diff-filter/test30-pr18904-rvalueref-report1.txt:
	Likewise.
	* tests/data/test-diff-filter/test30-pr18904-rvalueref-report2.txt:
	Likewise.
	* tests/data/test-read-dwarf/PR22015-libboost_iostreams.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/PR22122-libftdc.so.abi: Likewise.
	* tests/data/test-read-dwarf/libtest23.so.abi: Likewise.
	* tests/data/test-read-dwarf/libtest24-drop-fns-2.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/libtest24-drop-fns.so.abi: Likewise.
	* tests/data/test-read-dwarf/test-libandroid.so.abi: Likewise.
	* tests/data/test-read-dwarf/test10-pr18818-gcc.so.abi: Likewise.
	* tests/data/test-read-dwarf/test11-pr18828.so.abi: Likewise.
	* tests/data/test-read-dwarf/test12-pr18844.so.abi: Likewise.
	* tests/data/test-read-dwarf/test14-pr18893.so.abi: Likewise.
	* tests/data/test-read-dwarf/test15-pr18892.so.abi: Likewise.
	* tests/data/test-read-dwarf/test16-pr18904.so.abi: Likewise.
	* tests/data/test-read-dwarf/test17-pr19027.so.abi: Likewise.
	* tests/data/test-read-dwarf/test18-pr19037-libvtkRenderingLIC-6.1.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test19-pr19023-libtcmalloc_and_profiler.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test20-pr19025-libvtkParallelCore-6.1.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test22-pr19097-libstdc++.so.6.0.17.so.abi:
	Likewise.
	* tests/data/test-read-dwarf/test9-pr18818-clang.so.abi: Likewise.
	* tests/data/test-read-write/test10.xml: Likewise.

Signed-off-by: Dodji Seketeli <dodji@redhat.com>
---
 include/abg-ir.h                              |     3 +
 src/abg-dwarf-reader.cc                       |   107 +-
 src/abg-ir-priv.h                             |    11 +-
 src/abg-ir.cc                                 |   136 +-
 src/abg-writer.cc                             |    48 +-
 .../test-ld-2.28-210.so--ld-2.28-211.so.txt   |     2 +-
 tests/data/test-annotate/libtest23.so.abi     |   316 +-
 .../test-annotate/libtest24-drop-fns-2.so.abi |   309 +-
 .../test-annotate/libtest24-drop-fns.so.abi   |   309 +-
 .../data/test-annotate/test14-pr18893.so.abi  |  2977 +-
 .../data/test-annotate/test15-pr18892.so.abi  | 15324 ++++------
 .../data/test-annotate/test17-pr19027.so.abi  |  8826 +++---
 ...st18-pr19037-libvtkRenderingLIC-6.1.so.abi |  2777 +-
 ...19-pr19023-libtcmalloc_and_profiler.so.abi |  4431 ++-
 ...st20-pr19025-libvtkParallelCore-6.1.so.abi |  4525 ++-
 .../PR25409-librte_bus_dpaa.so.20.0.abi       |    10 +-
 .../test0-pr19026-libvtkIOSQL-6.1.so.1.abi    |  5990 ++--
 .../test30-pr18904-rvalueref-report1.txt      |     2 +-
 .../test30-pr18904-rvalueref-report2.txt      |     2 +-
 .../PR22015-libboost_iostreams.so.abi         |   748 +-
 .../test-read-dwarf/PR22122-libftdc.so.abi    |   620 +-
 tests/data/test-read-dwarf/libtest23.so.abi   |   295 +-
 .../libtest24-drop-fns-2.so.abi               |   289 +-
 .../test-read-dwarf/libtest24-drop-fns.so.abi |   289 +-
 .../test-read-dwarf/test-libandroid.so.abi    |   150 +-
 .../test-read-dwarf/test10-pr18818-gcc.so.abi |  3268 +--
 .../test-read-dwarf/test11-pr18828.so.abi     |  9940 ++++---
 .../test-read-dwarf/test12-pr18844.so.abi     | 15676 +++++-----
 .../test-read-dwarf/test14-pr18893.so.abi     |   340 +-
 .../test-read-dwarf/test15-pr18892.so.abi     | 12988 ++++-----
 .../test-read-dwarf/test16-pr18904.so.abi     | 17232 ++++++-----
 .../test-read-dwarf/test17-pr19027.so.abi     |  8441 +++---
 ...st18-pr19037-libvtkRenderingLIC-6.1.so.abi |  2731 +-
 ...19-pr19023-libtcmalloc_and_profiler.so.abi |  4300 ++-
 ...st20-pr19025-libvtkParallelCore-6.1.so.abi |  4503 ++-
 .../test22-pr19097-libstdc++.so.6.0.17.so.abi | 23768 ++++++++--------
 .../test9-pr18818-clang.so.abi                |  2278 +-
 tests/data/test-read-write/test10.xml         |     6 +-
 38 files changed, 72311 insertions(+), 81656 deletions(-)

The patch is too big so I am attaching it gzip'ed.