abidiff: improve treatment of array types in leaf changes mode

  If an array's element type doesn't change name but has some other
(local) change, then the change should not also be considered local to
the array type.

	* src/abg-ir.cc (types_have_similar_structure): When
	examining array types, always treat element type changes as
	indirect changes.
	* tests/data/Makefile.am: Add new test case files.
	* tests/data/test-abidiff-exit/test-non-leaf-array-report.txt:
	New test case showing correct --leaf-changes-only reporting.
	* tests/data/test-abidiff-exit/test-non-leaf-array-v0.c:
	Likewise.
	* tests/data/test-abidiff-exit/test-non-leaf-array-v0.o:
	Likewise.
	* tests/data/test-abidiff-exit/test-non-leaf-array-v1.c:
	Likewise.
	* tests/data/test-abidiff-exit/test-non-leaf-array-v1.o:
	Likewise.
	* tests/test-abidiff-exit.cc: Run new test case.

Signed-off-by: Giuliano Procida <gprocida@google.com>
---
 src/abg-ir.cc                                    |   2 +-
 tests/data/Makefile.am                           |   5 +++++
 .../test-non-leaf-array-report.txt               |  11 +++++++++++
 .../test-abidiff-exit/test-non-leaf-array-v0.c   |  12 ++++++++++++
 .../test-abidiff-exit/test-non-leaf-array-v0.o   | Bin 0 -> 3096 bytes
 .../test-abidiff-exit/test-non-leaf-array-v1.c   |  12 ++++++++++++
 .../test-abidiff-exit/test-non-leaf-array-v1.o   | Bin 0 -> 3072 bytes
 tests/test-abidiff-exit.cc                       |  11 +++++++++++
 8 files changed, 52 insertions(+), 1 deletion(-)
 create mode 100644 tests/data/test-abidiff-exit/test-non-leaf-array-report.txt
 create mode 100644 tests/data/test-abidiff-exit/test-non-leaf-array-v0.c
 create mode 100644 tests/data/test-abidiff-exit/test-non-leaf-array-v0.o
 create mode 100644 tests/data/test-abidiff-exit/test-non-leaf-array-v1.c
 create mode 100644 tests/data/test-abidiff-exit/test-non-leaf-array-v1.o
  

Hello Giuliano,

Giuliano Procida <gprocida@google.com> a écrit:

[...]

> diff --git a/src/abg-ir.cc b/src/abg-ir.cc
> index c6f7c13e..b0db9c39 100644
> --- a/src/abg-ir.cc
> +++ b/src/abg-ir.cc
> @@ -23606,7 +23606,7 @@ types_have_similar_structure(const type_base* first,
>  	  || ty1->get_dimension_count() != ty2->get_dimension_count()
>  	  || !types_have_similar_structure(ty1->get_element_type(),
>  					   ty2->get_element_type(),
> -					   indirect_type))
> +					   true))
>  	return false;
>  
>        return true;

I think this change is correct, thanks for spotting that.

However ...

[...]

> If an array's element type doesn't change name but has some other
> (local) change, then the change should not also be considered local to
> the array type.

I find this comment confusing, even if I think I see what you mean.

A change being local or not, is a concept that is not at the same
logical level as the concept of "type similarity" defined in the comment
of the function types_have_similar_structure.  I would say that the type
similarity concept is at a lower logical level.  So seeing this comment
that applies to a higher level concept for a change made to something

But I agree that the comments of types_have_similar_structure are
confusing as well.

So I am proposing two patches following this message, for this issue.
One patch amends the the comments for types_have_similar_structure, and
the second one is your patch, with amended comments.

And we can discuss from there as you like.

[...]

Cheers,
  

Hi.

On Fri, 4 Dec 2020 at 11:02, Dodji Seketeli <dodji@seketeli.org> wrote:

> Hello Giuliano,
>
> Giuliano Procida <gprocida@google.com> a écrit:
>
> [...]
>
> > diff --git a/src/abg-ir.cc b/src/abg-ir.cc
> > index c6f7c13e..b0db9c39 100644
> > --- a/src/abg-ir.cc
> > +++ b/src/abg-ir.cc
> > @@ -23606,7 +23606,7 @@ types_have_similar_structure(const type_base*
> first,
> >         || ty1->get_dimension_count() != ty2->get_dimension_count()
> >         || !types_have_similar_structure(ty1->get_element_type(),
> >                                          ty2->get_element_type(),
> > -                                        indirect_type))
> > +                                        true))
> >       return false;
> >
> >        return true;
>
> I think this change is correct, thanks for spotting that.
>
> However ...
>
> [...]
>
> > If an array's element type doesn't change name but has some other
> > (local) change, then the change should not also be considered local to
> > the array type.
>
> I find this comment confusing, even if I think I see what you mean.
>
> A change being local or not, is a concept that is not at the same
> logical level as the concept of "type similarity" defined in the comment
> of the function types_have_similar_structure.  I would say that the type
> similarity concept is at a lower logical level.  So seeing this comment
> that applies to a higher level concept for a change made to something
>
> But I agree that the comments of types_have_similar_structure are
> confusing as well.
>
> So I am proposing two patches following this message, for this issue.
> One patch amends the the comments for types_have_similar_structure, and
> the second one is your patch, with amended comments.
>
> And we can discuss from there as you like.
>
>
I'm happy with your patches and have replied to them. However, I'll be
honest, I've found this aspect (local changes and types with
similar structure) of abidiff really hard to reason about. I'm likely
looking at things the wrong way, but here's roughly how I see things.

We want to identify "leaf" changes. I think of them more as "root" (cause)
changes, but that's just inverting the edges in some graph. These should be
(user-defined) types that have not changed name but have changed in some
way "directly" (indirectly would be if there were a more remote cause of
the same sort). There are also direct changes to symbols' types.
User-defined types with local changes have paths to the symbols whose types
they affect - they are the impacted interfaces.

Now with libabigail, there is the notion of local vs non-local change which
might not correspond exactly with the notion sketched above, but it's
certainly related. And then there are at least two more
pieces: types_have_similar_structure which is used in decisions about
locality and lastly the logic used to filter changes for consideration in
the leaf report - they have to be local plus a lot of other conditions.

The decision-making that goes into leaf reporting seems to be spread across
several bits of code and I speculate that it was an inconsistency between
two of these that resulted in diff nodes being selected for inclusion but
actually have no local diff to report (by the represent helper function).
If I knew what types_have_similar_structure had to be consistent with I
could check all the cases or, more interestingly, see if the decisions
being made in two different places could be made in just one place.

Regards,
Giuliano.


> [...]
>
> Cheers,
>
> --
>                 Dodji
>
  

Hello Giuliano,

Giuliano Procida <gprocida@google.com> a écrit:

[...]

> I've found this aspect (local changes and types with similar
> structure) of abidiff really hard to reason about. I'm likely looking
> at things the wrong way, but here's roughly how I see things.

No, I don't think you are seeing things the "wrong way".  I think there
are several ways of seeing things.

There are several goals libabigail had to fulfill.  To accommodate them
all we adopted some point of views that might make things look less
obvious that they could have been should we have fewer goals to fulfill.

So, here are some of the concepts we use in Libabigail to analyze
changes.  I'll only talk about those related to your discussion here.

An ABI artifact is either a type or a declaration.  An ABI artifact is a
composite entity, meaning that it has sub-parts that are ABI artifacts
as well.  So, a declaration has a type.  A type might have have
sub-types.

In that context, we define the concept of a local change.

A local change to an ABI artifact is a change that does not involve any
of the sub-parts of the artifact.  For instance, a local change for a
typedef would be a change to its name.  But a change to the underlying
type of the typedef is not considered local, as the underlying type is
the sub-type of the typedef.

Said otherwise, a local change to an ABI artifact is a change to the
artifact itself, not a change to any of its sub-parts.

A leaf change is then a local change "that makes sense" from the
end-user point of view, in the context of the leaf reporter.

More precisely, a leaf change:

   * must impact an exported interface of the binary we are
     analyzing.  In other words, it must be a change to a sub-part of
     an ABI artifact (a declaration) that is exported by the binary we
     are looking at.

   * doesn't contain a name change

   * is about a function, variable, class, union or enum type.

There are other restrictions, but there are technical details that are
not useful to mention at this point, think.

The reason why we introduced the concept of the "local change" is that
it's useful to fulfill goals in libabigail, independently from the leaf
reporter requirements.  It's useful to be able to perform things like
redundancy detection and proper categorization of diff nodes.  But I
think this is a topic for a separate discussion.

> We want to identify "leaf" changes. I think of them more as "root" (cause)
> changes, but that's just inverting the edges in some graph. These should be
> (user-defined) types that have not changed name but have changed in some
> way "directly" (indirectly would be if there were a more remote cause of
> the same sort).

I find your way of seeing things fairly accurate.  It's doesn't seem to
go against the view that I exposed above.


> There are also direct changes to symbols' types.

I tend to think more in terms of 'exported interfaces'.  I.e, a binary
exports functions or global variables.  Those constitute the interfaces
of the application binary.  Those interfaces have several properties.
Their type is one of their properties.  In the particular case of ELF
binaries, those exported interfaces must have ELF symbols, which another
property of those exported interfaces.

So, yes, there are also "local changes to the binary interfaces".  This
is a particular case of the definition I laid down earlier.

So I agree with you.

> User-defined types with local changes have paths to the symbols whose types
> they affect - they are the impacted interfaces.

Yes.

> Now with libabigail, there is the notion of local vs non-local change which
> might not correspond exactly with the notion sketched above, but it's
> certainly related.

I hope it's clearer now.

> And then there are at least two more
> pieces: types_have_similar_structure which is used in decisions about
> locality.

Right.  The concept of "type structure similarity" is a detail of the
concept of "local change".

> and lastly the logic used to filter changes for consideration in
> the leaf report - they have to be local plus a lot of other conditions.

Right.

> The decision-making that goes into leaf reporting seems to be spread across
> several bits of code

The pass that walks through diff nodes to detect which ones are leaf
nodes is the 'struct leaf_diff_node_marker_visitor' defined in
abg-comparison.cc.

Now, there are a *lot* of concepts involved here, I agree.  But
ultimately, I think it's related to the complexity of what we are trying
to model here.

> and I speculate that it was an inconsistency between two of these that
> resulted in diff nodes being selected for inclusion but actually have
> no local diff to report (by the represent helper function).

Stricto sensu, I think the problem was us not respecting the definition
of the "type structure similarity" relation.  As that relation is at the
*bottom* of a stack of concepts, it's not surprising that a bug in there
has impacts possibly further down the road.

At the end of the day, this is a complex network of concepts we are
dealing with.

> If I knew what types_have_similar_structure had to be consistent with I
> could check all the cases or, more interestingly, see if the decisions
> being made in two different places could be made in just one place.

Well.  In this particular case, I'd argue that
types_have_similar_structure had to be consistent with itself to begin
with :-)

So I wouldn't beat myself over this too much, seriously.  To me, what is
paramount is that we keep the whole thing as "debuggable" as possible.
Because *that* is how we progressively get better.  At least in my
experience.

Thank you for initiating this discussion.  I hope I haven't made things
even more confusing than they are at the moment.

Cheers,