[v5,1/2] check-uapi: Introduce check-uapi.sh

  While the kernel community has been good at maintaining backwards
compatibility with kernel UAPIs, it would be helpful to have a tool
to check if a commit introduces changes that break backwards
compatibility.

To that end, introduce check-uapi.sh: a simple shell script that
checks for changes to UAPI headers using libabigail.

libabigail is "a framework which aims at helping developers and
software distributors to spot some ABI-related issues like interface
incompatibility in ELF shared libraries by performing a static
analysis of the ELF binaries at hand."

The script uses one of libabigail's tools, "abidiff", to compile the
changed header before and after the commit to detect any changes.

abidiff "compares the ABI of two shared libraries in ELF format. It
emits a meaningful report describing the differences between the two
ABIs."

The script also includes the ability to check the compatibility of
all UAPI headers across commits. This allows developers to inspect
the stability of the UAPIs over time.

Signed-off-by: John Moon <quic_johmoo@quicinc.com>
---
    - Improved git tree restore operation by going back to a user's
      branch if they were on one. This is better than returning to a
      detached commit.
    - Fixed bitwise AND check syntax which failed with earlier versions
      of Bash.
    - Removed suppression of "make headers_install" stderr for better
      user feedback.
    - Made handling of -q flag more elegant.

 scripts/check-uapi.sh | 489 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 489 insertions(+)
 create mode 100755 scripts/check-uapi.sh

--
2.17.1
  

On Sat, Apr 8, 2023 at 5:35 AM John Moon <quic_johmoo@quicinc.com> wrote:
>
> While the kernel community has been good at maintaining backwards
> compatibility with kernel UAPIs, it would be helpful to have a tool
> to check if a commit introduces changes that break backwards
> compatibility.
>
> To that end, introduce check-uapi.sh: a simple shell script that
> checks for changes to UAPI headers using libabigail.
>
> libabigail is "a framework which aims at helping developers and
> software distributors to spot some ABI-related issues like interface
> incompatibility in ELF shared libraries by performing a static
> analysis of the ELF binaries at hand."
>
> The script uses one of libabigail's tools, "abidiff", to compile the
> changed header before and after the commit to detect any changes.
>
> abidiff "compares the ABI of two shared libraries in ELF format. It
> emits a meaningful report describing the differences between the two
> ABIs."
>
> The script also includes the ability to check the compatibility of
> all UAPI headers across commits. This allows developers to inspect
> the stability of the UAPIs over time.
>
> Signed-off-by: John Moon <quic_johmoo@quicinc.com>



BTW, is there anybody (except the submitters) who loves this tool?
(or anybody who has ever evaluated this?)

I am the only person who pointed out something.



I put some more comments from users' point of view.





[1] The number of lines of the error log.


According to this tool, it looks like we broke a lot of UAPI
headers in the previous MW (between v6.2 and v6.3-rc1).

Not only in the previous MW, we always broke so many UAPI
headers, this script says.


The following command outputs 2380 lines.


$ ./scripts/check-uapi.sh -q -b v6.3-rc1 -p v6.2 2>&1  | wc
   2380    9438  104953


I do not know how many are real breakages, and
how many are false positives.

Anyway, I will not check the 2380 lines of the error log.


And, please note that the help message explains '-q' is the *quiet* mode.
It is the opposite. This is super noisy.




[2] Be careful!


While testing the patch submissions of this,
I messed up my repository multiple times.

The script takes some time because it builds many objects
internally.

However, once this script starts running, you must not hit Ctrl-C.
If you do it, your repository will be sprinkled with a ton
of untracked files.

Apply this patch, and run "./scripts/check-uapi.sh -p v6.0"
and hit Ctrl-C.

Repeat it a couple of times, and "git status" will show you
something horrible.


You will never know when git is checking out a commit
because this script hides it by 'git checkout --quiet'.


So, this tool should show a caveat at least.


'git checkout' should not be hidden, or
maybe a caveat message should be shown.


CAVEAT
This tool runs 'git checkout' a couple of times internally.
If you interrupt it, your worktree might be messed up.












--
Best Regards
Masahiro Yamada
  

On Mon, Apr 10, 2023 at 07:03:05PM +0900, Masahiro Yamada wrote:
> On Sat, Apr 8, 2023 at 5:35 AM John Moon <quic_johmoo@quicinc.com> wrote:
> >
> > While the kernel community has been good at maintaining backwards
> > compatibility with kernel UAPIs, it would be helpful to have a tool
> > to check if a commit introduces changes that break backwards
> > compatibility.
> >
> > To that end, introduce check-uapi.sh: a simple shell script that
> > checks for changes to UAPI headers using libabigail.
> >
> > libabigail is "a framework which aims at helping developers and
> > software distributors to spot some ABI-related issues like interface
> > incompatibility in ELF shared libraries by performing a static
> > analysis of the ELF binaries at hand."
> >
> > The script uses one of libabigail's tools, "abidiff", to compile the
> > changed header before and after the commit to detect any changes.
> >
> > abidiff "compares the ABI of two shared libraries in ELF format. It
> > emits a meaningful report describing the differences between the two
> > ABIs."
> >
> > The script also includes the ability to check the compatibility of
> > all UAPI headers across commits. This allows developers to inspect
> > the stability of the UAPIs over time.
> >
> > Signed-off-by: John Moon <quic_johmoo@quicinc.com>
> 
> 
> 
> BTW, is there anybody (except the submitters) who loves this tool?
> (or anybody who has ever evaluated this?)

I evaluated the first one, and yes, I do want this, but I haven't tested
it out yet, sorry.

I get patches for header files all the time and hand-verifying that they
don't break the abi is a pain at times

> According to this tool, it looks like we broke a lot of UAPI
> headers in the previous MW (between v6.2 and v6.3-rc1).

That's not ok, and needs to be fixed, otherwise this is useless as no
one can rely on it at all.

> The script takes some time because it builds many objects
> internally.
> 
> However, once this script starts running, you must not hit Ctrl-C.
> If you do it, your repository will be sprinkled with a ton
> of untracked files.

That needs to be unwound and fixed.

> CAVEAT
> This tool runs 'git checkout' a couple of times internally.
> If you interrupt it, your worktree might be messed up.

ctrl-c can be properly caught and the git state needs to be restored for
this to be able to be accepted.

thanks,

greg k-h
  

On 4/10/2023 11:45 AM, Greg Kroah-Hartman wrote:
> On Mon, Apr 10, 2023 at 07:03:05PM +0900, Masahiro Yamada wrote:
>> On Sat, Apr 8, 2023 at 5:35 AM John Moon <quic_johmoo@quicinc.com> wrote:
>>>
>>> While the kernel community has been good at maintaining backwards
>>> compatibility with kernel UAPIs, it would be helpful to have a tool
>>> to check if a commit introduces changes that break backwards
>>> compatibility.
>>>
>>> To that end, introduce check-uapi.sh: a simple shell script that
>>> checks for changes to UAPI headers using libabigail.
>>>
>>> libabigail is "a framework which aims at helping developers and
>>> software distributors to spot some ABI-related issues like interface
>>> incompatibility in ELF shared libraries by performing a static
>>> analysis of the ELF binaries at hand."
>>>
>>> The script uses one of libabigail's tools, "abidiff", to compile the
>>> changed header before and after the commit to detect any changes.
>>>
>>> abidiff "compares the ABI of two shared libraries in ELF format. It
>>> emits a meaningful report describing the differences between the two
>>> ABIs."
>>>
>>> The script also includes the ability to check the compatibility of
>>> all UAPI headers across commits. This allows developers to inspect
>>> the stability of the UAPIs over time.
>>>
>>> Signed-off-by: John Moon <quic_johmoo@quicinc.com>
>>
>>
>>
>> BTW, is there anybody (except the submitters) who loves this tool?
>> (or anybody who has ever evaluated this?)
> 
> I evaluated the first one, and yes, I do want this, but I haven't tested
> it out yet, sorry.
> 
> I get patches for header files all the time and hand-verifying that they
> don't break the abi is a pain at times
>

Agreed, this is the way we're using the tool internally. It's great as a 
quick spot-check on a change.

>> According to this tool, it looks like we broke a lot of UAPI
>> headers in the previous MW (between v6.2 and v6.3-rc1).
> 
> That's not ok, and needs to be fixed, otherwise this is useless as no
> one can rely on it at all.
> 

Right, there are several classes of false positives that we've 
documented and when examining thousands of commits at time, it'll flag 
many things.

For some comparison, if you run checkpatch on the same changeset 
(v6.2..v6.3-rc1), you get 995 errors and 7,313 warnings. Still, 
checkpatch is helpful for spot-checks.

"./scripts/check-uapi.sh -b v6.3-rc1 -p v6.2" flags 36 out of the 911 
files checked. Of those 36, 19 fell into the currently documented false 
positive categories:

Enum expansion: 17
Expanding into padded/reserved fields: 2

Beyond those, the tool appears to be flagging legitimate breakages.

Some fit into the definition of "intentional breakages" where support is 
being dropped or something is being refactored:

  File removals:
    - include/uapi/drm/i810_drm.h
    - include/uapi/drm/mga_drm.h
    - include/uapi/drm/r128_drm.h
    - include/uapi/drm/savage_drm.h
    - include/uapi/drm/sis_drm.h
    - include/uapi/drm/via_drm.h
    - include/uapi/linux/meye.h

  File moves:
    - include/uapi/misc/habanalabs.h

  Removal of struct:
    - include/uapi/linux/uuid.h (5e6a51787fef)
      - include/uapi/linux/mei.h (failed due to uuid.h)
      - include/uapi/linux/ublk_cmd.h (failed due to uuid.h)

Others do not seem to be intentional:

  Addition/use of flex arrays:
    - include/uapi/linux/rseq.h (f7b01bb0b57f)
    - include/uapi/scsi/scsi_bsg_mpi3mr.h (c6f2e6b6eaaf)

  Type change:
    - include/uapi/scsi/scsi_bsg_ufs.h (3f5145a615238)

  Additions into existing struct:
    - include/uapi/drm/amdgpu_drm.h (b299221faf9b)
    - include/uapi/linux/perf_event.h (09519ec3b19e)
    - include/uapi/linux/virtio_blk.h (95bfec41bd3d)

Is there something I'm missing that makes these changes false positives? 
If so, I'd be happy to add on to the documentation and work towards a 
way to filter them out.

In the mean time, we will start a thread on the libabigail mailing list 
to see if there's a way to add flags such as --ignore-enum-expansion, 
--ignore-expansion-into-reserved-fields, etc. Enum expansion seems to be 
making up the largest portion of false positives, so would be the best 
thing to filter out.

>> The script takes some time because it builds many objects
>> internally.
>>
>> However, once this script starts running, you must not hit Ctrl-C.
>> If you do it, your repository will be sprinkled with a ton
>> of untracked files.
> 
> That needs to be unwound and fixed.
> 
>> CAVEAT
>> This tool runs 'git checkout' a couple of times internally.
>> If you interrupt it, your worktree might be messed up.
> 
> ctrl-c can be properly caught and the git state needs to be restored for
> this to be able to be accepted.
>

Yes, this can be taken care of.

> thanks,
> 
> greg k-h
  

On Mon, Apr 10, 2023 at 04:32:49PM -0700, John Moon wrote:
> > > According to this tool, it looks like we broke a lot of UAPI
> > > headers in the previous MW (between v6.2 and v6.3-rc1).
> > 
> > That's not ok, and needs to be fixed, otherwise this is useless as no
> > one can rely on it at all.
> > 
> 
> Right, there are several classes of false positives that we've documented
> and when examining thousands of commits at time, it'll flag many things.
> 
> For some comparison, if you run checkpatch on the same changeset
> (v6.2..v6.3-rc1), you get 995 errors and 7,313 warnings. Still, checkpatch
> is helpful for spot-checks.

checkpatch.pl does not matter, it is a "hint", and many patches
explicitly ignore it (think about patches in the staging tree, you could
fix up one checkpatch issue for a line, but ignore another one as you
are not supposed to mix them up.)

Also for some subsystems, checkpatch does not matter because their
codebase is old and follows different rules.  And in some places,
checkpatch is just wrong, because it's a perl script and can not really
parse code.

So NEVER use that as a comparison to the user/kernel abi please.  It's a
false comparison.

> "./scripts/check-uapi.sh -b v6.3-rc1 -p v6.2" flags 36 out of the 911 files
> checked. Of those 36, 19 fell into the currently documented false positive
> categories:
> 
> Enum expansion: 17
> Expanding into padded/reserved fields: 2
> 
> Beyond those, the tool appears to be flagging legitimate breakages.
> 
> Some fit into the definition of "intentional breakages" where support is
> being dropped or something is being refactored:
> 
>  File removals:
>    - include/uapi/drm/i810_drm.h
>    - include/uapi/drm/mga_drm.h
>    - include/uapi/drm/r128_drm.h
>    - include/uapi/drm/savage_drm.h
>    - include/uapi/drm/sis_drm.h
>    - include/uapi/drm/via_drm.h
>    - include/uapi/linux/meye.h
> 
>  File moves:
>    - include/uapi/misc/habanalabs.h
> 
>  Removal of struct:
>    - include/uapi/linux/uuid.h (5e6a51787fef)
>      - include/uapi/linux/mei.h (failed due to uuid.h)
>      - include/uapi/linux/ublk_cmd.h (failed due to uuid.h)
> 
> Others do not seem to be intentional:
> 
>  Addition/use of flex arrays:
>    - include/uapi/linux/rseq.h (f7b01bb0b57f)
>    - include/uapi/scsi/scsi_bsg_mpi3mr.h (c6f2e6b6eaaf)

That is not a breakage, that's a tool problem.

>  Type change:
>    - include/uapi/scsi/scsi_bsg_ufs.h (3f5145a615238)

Again, not a real breakage, size is still the same.

>  Additions into existing struct:
>    - include/uapi/drm/amdgpu_drm.h (b299221faf9b)
>    - include/uapi/linux/perf_event.h (09519ec3b19e)
>    - include/uapi/linux/virtio_blk.h (95bfec41bd3d)

Adding data to the end of a structure is a well-known way to extend the
api, in SOME instances if it is used properly.

So again, not a break.

> Is there something I'm missing that makes these changes false positives? If
> so, I'd be happy to add on to the documentation and work towards a way to
> filter them out.
> 
> In the mean time, we will start a thread on the libabigail mailing list to
> see if there's a way to add flags such as --ignore-enum-expansion,
> --ignore-expansion-into-reserved-fields, etc. Enum expansion seems to be
> making up the largest portion of false positives, so would be the best thing
> to filter out.

Increasing enums is in no way an abi break unless the size of the
structure changes.

Using reserved fields too is not a breakage.

So yes, it looks like the tooling needs some work in order for us to be
able to use this properly, digging through false positives like this is
going to make it not used at all.

thanks,

greg k-h
  

On 4/10/2023 11:34 PM, Greg Kroah-Hartman wrote:
> On Mon, Apr 10, 2023 at 04:32:49PM -0700, John Moon wrote:
>>>> According to this tool, it looks like we broke a lot of UAPI
>>>> headers in the previous MW (between v6.2 and v6.3-rc1).
>>>
>>> That's not ok, and needs to be fixed, otherwise this is useless as no
>>> one can rely on it at all.
>>>
>>
>> Right, there are several classes of false positives that we've documented
>> and when examining thousands of commits at time, it'll flag many things.
>>
>> For some comparison, if you run checkpatch on the same changeset
>> (v6.2..v6.3-rc1), you get 995 errors and 7,313 warnings. Still, checkpatch
>> is helpful for spot-checks.
> 
> checkpatch.pl does not matter, it is a "hint", and many patches
> explicitly ignore it (think about patches in the staging tree, you could
> fix up one checkpatch issue for a line, but ignore another one as you
> are not supposed to mix them up.)
> 
> Also for some subsystems, checkpatch does not matter because their
> codebase is old and follows different rules.  And in some places,
> checkpatch is just wrong, because it's a perl script and can not really
> parse code.
> 
> So NEVER use that as a comparison to the user/kernel abi please.  It's a
> false comparison.
>

Fair enough. I was just trying to frame this tool as a "hint" as well. :)

>>
>> Others do not seem to be intentional:
>>
>>   Addition/use of flex arrays:
>>     - include/uapi/linux/rseq.h (f7b01bb0b57f)
>>     - include/uapi/scsi/scsi_bsg_mpi3mr.h (c6f2e6b6eaaf)
> 
> That is not a breakage, that's a tool problem.
> 
>>   Type change:
>>     - include/uapi/scsi/scsi_bsg_ufs.h (3f5145a615238)
> 
> Again, not a real breakage, size is still the same.
>

Would you find the tool more useful if it simply filtered out all 
instances where the size of the type did not change? This would filter 
out the following which the tool currently flags:

- enum expansions
- reserved field expansions
- expansions of a struct with a flex array at the end
- type changes
- re-ordering of existing members
- ...others?

These changes aren't _always_ safe, but if you assume the kernel 
developer is doing something reasonable, then maybe it's okay. Maybe we 
could hide these checks behind something like a "--pedantic" flag?

This logic is actually trivial to add. Filtering out issues where the 
type size stays the same brings us down to 14 failures in that same MW 
changeset. 8 of them are file removals and the 6 remaining flags are 
additions to existing structs or intentional removals of APIs.

LMK what you think and I can work this into a v6 patch.

>>   Additions into existing struct:
>>     - include/uapi/drm/amdgpu_drm.h (b299221faf9b)
>>     - include/uapi/linux/perf_event.h (09519ec3b19e)
>>     - include/uapi/linux/virtio_blk.h (95bfec41bd3d)
> 
> Adding data to the end of a structure is a well-known way to extend the
> api, in SOME instances if it is used properly.
> 
> So again, not a break.
>

I don't know of a way the tool could be smart enough to decide if the 
additions are done properly in the proper instances. Seems appropriate 
that the tool would flag these changes for closer human review.

Thanks,
John
  

On Tue, Apr 11, 2023 at 11:36:48AM -0700, John Moon wrote:
> > > 
> > > Others do not seem to be intentional:
> > > 
> > >   Addition/use of flex arrays:
> > >     - include/uapi/linux/rseq.h (f7b01bb0b57f)
> > >     - include/uapi/scsi/scsi_bsg_mpi3mr.h (c6f2e6b6eaaf)
> > 
> > That is not a breakage, that's a tool problem.
> > 
> > >   Type change:
> > >     - include/uapi/scsi/scsi_bsg_ufs.h (3f5145a615238)
> > 
> > Again, not a real breakage, size is still the same.
> > 
> 
> Would you find the tool more useful if it simply filtered out all instances
> where the size of the type did not change? This would filter out the
> following which the tool currently flags:
> 
> - enum expansions
> - reserved field expansions
> - expansions of a struct with a flex array at the end
> - type changes
> - re-ordering of existing members
> - ...others?

Obviously not, as some of those are real breakages, and some are not at
all.

Please understand what is an abi breakage.  Adding new enums is not.
Using a reserved field is not.  Reording existing members IS.

> These changes aren't _always_ safe, but if you assume the kernel developer
> is doing something reasonable, then maybe it's okay. Maybe we could hide
> these checks behind something like a "--pedantic" flag?

Again, no, that list above has totally different things in it, some are
completly safe, others totally break the abi.  Do NOT lump them all
together as that is wrong.

thanks,

greg k-h
  

On 4/11/2023 11:14 PM, Greg Kroah-Hartman wrote:
>> Would you find the tool more useful if it simply filtered out all instances
>> where the size of the type did not change? This would filter out the
>> following which the tool currently flags:
>>
>> - enum expansions
>> - reserved field expansions
>> - expansions of a struct with a flex array at the end
>> - type changes
>> - re-ordering of existing members
>> - ...others?
> 
> Obviously not, as some of those are real breakages, and some are not at
> all.
> 
> Please understand what is an abi breakage.  Adding new enums is not.
> Using a reserved field is not.  Reording existing members IS.
> 

Yes, understood that method would miss certain classes of breakages. I 
was suggesting it as a way to improve the signal-to-noise ratio of the 
tool since we don't currently have an algorithm for determining 
breakages with 100% accuracy.

We'll work internally and with the libabigail team to improve the story 
here and get back to you. Thanks for the discussion!

- John
  

On Wed, Apr 12, 2023 at 09:37:16AM -0700, John Moon wrote:
> On 4/11/2023 11:14 PM, Greg Kroah-Hartman wrote:
> > > Would you find the tool more useful if it simply filtered out all instances
> > > where the size of the type did not change? This would filter out the
> > > following which the tool currently flags:
> > > 
> > > - enum expansions
> > > - reserved field expansions
> > > - expansions of a struct with a flex array at the end
> > > - type changes
> > > - re-ordering of existing members
> > > - ...others?
> > 
> > Obviously not, as some of those are real breakages, and some are not at
> > all.
> > 
> > Please understand what is an abi breakage.  Adding new enums is not.
> > Using a reserved field is not.  Reording existing members IS.
> > 
> 
> Yes, understood that method would miss certain classes of breakages. I was
> suggesting it as a way to improve the signal-to-noise ratio of the tool
> since we don't currently have an algorithm for determining breakages with
> 100% accuracy.

Why not?  You know the different types of things here based on the
differences between the dwarf data, and they fall into different
categories, and those different categories mean different things.

If you have questions as to which type of change is allowed and which is
not, just ask us, the rules are not complex, nor impossible to describe,
otherwise we wouldn't have a stable api at all, right?

thanks,

greg k-h
  

Hi,

On Wed, 2023-04-12 at 09:37 -0700, John Moon via Libabigail wrote:
> On 4/11/2023 11:14 PM, Greg Kroah-Hartman wrote:
> > > Would you find the tool more useful if it simply filtered out all instances
> > > where the size of the type did not change? This would filter out the
> > > following which the tool currently flags:
> > > 
> > > - enum expansions
> > > - reserved field expansions
> > > - expansions of a struct with a flex array at the end
> > > - type changes
> > > - re-ordering of existing members
> > > - ...others?
> > 
> > Obviously not, as some of those are real breakages, and some are not at
> > all.
> > 
> > Please understand what is an abi breakage.  Adding new enums is not.
> > Using a reserved field is not.  Reording existing members IS.
> > 
> 
> Yes, understood that method would miss certain classes of breakages. I 
> was suggesting it as a way to improve the signal-to-noise ratio of the 
> tool since we don't currently have an algorithm for determining 
> breakages with 100% accuracy.

Note that you can check the exit code of libabigail's abidiff to see
whether something is an incompatible abi change or not, see:
https://sourceware.org/libabigail/manual/abidiff.html#return-values

You can also of course use suppressions to instruct abidiff to avoid
reporting changes involving certain ABI artifacts:
https://sourceware.org/libabigail/manual/libabigail-concepts.html#suppr-spec-label

Cheers,

Mark
  

On Mon, Apr 10, 2023 at 07:03:05PM +0900 Masahiro Yamada wrote:
[...]
> [2] Be careful!
> 
> 
> While testing the patch submissions of this,
> I messed up my repository multiple times.
> 
> The script takes some time because it builds many objects
> internally.
> 
> However, once this script starts running, you must not hit Ctrl-C.
> If you do it, your repository will be sprinkled with a ton
> of untracked files.
> 
> Apply this patch, and run "./scripts/check-uapi.sh -p v6.0"
> and hit Ctrl-C.
> 
> Repeat it a couple of times, and "git status" will show you
> something horrible.
> 
> 
> You will never know when git is checking out a commit
> because this script hides it by 'git checkout --quiet'.
> 
> 
> So, this tool should show a caveat at least.
> 
> 
> 'git checkout' should not be hidden, or
> maybe a caveat message should be shown.
> 
> 
> CAVEAT
> This tool runs 'git checkout' a couple of times internally.
> If you interrupt it, your worktree might be messed up.

John, did you consider using git export for obtaining a source tree that
can be used for headers_install and the following comparison, instead of
git stash+checkout?  AFACS the script does not depend on any other git
functionality than providing a specific version of the source tree.  I am
pretty sure that leaving the original working copy in its original state
is supporting the script's popularity.

Kind regards,
Nicolas
  

On 4/12/2023 9:43 AM, Greg Kroah-Hartman wrote:
> On Wed, Apr 12, 2023 at 09:37:16AM -0700, John Moon wrote:
>> On 4/11/2023 11:14 PM, Greg Kroah-Hartman wrote:
>>>> Would you find the tool more useful if it simply filtered out all instances
>>>> where the size of the type did not change? This would filter out the
>>>> following which the tool currently flags:
>>>>
>>>> - enum expansions
>>>> - reserved field expansions
>>>> - expansions of a struct with a flex array at the end
>>>> - type changes
>>>> - re-ordering of existing members
>>>> - ...others?
>>>
>>> Obviously not, as some of those are real breakages, and some are not at
>>> all.
>>>
>>> Please understand what is an abi breakage.  Adding new enums is not.
>>> Using a reserved field is not.  Reording existing members IS.
>>>
>>
>> Yes, understood that method would miss certain classes of breakages. I was
>> suggesting it as a way to improve the signal-to-noise ratio of the tool
>> since we don't currently have an algorithm for determining breakages with
>> 100% accuracy.
> 
> Why not?  You know the different types of things here based on the
> differences between the dwarf data, and they fall into different
> categories, and those different categories mean different things.
> 
> If you have questions as to which type of change is allowed and which is
> not, just ask us, the rules are not complex, nor impossible to describe,
> otherwise we wouldn't have a stable api at all, right?
> 

Right, it's currently a limitation of parsing the abidiff output.

Even in trivial situations like an enum expansion, the tool knows that a 
variant was added and another variant had its offset changed. There's 
not a good way to say for sure that the variant whose offset changed is 
a "*_MAX" variant. So if we simply ignored enum expansion, we'd miss 
breakages like this:

enum foo {
	FLAG_A,
+       FLAG_B,
	FLAG_C,
	FLAG_MAX
}

Maybe we can ignore an enum expansion if only the last variant's offset 
changed, but then we'd miss cases where enums don't have a MAX variant. 
Maybe we could limit the check to last variant's offset whose name 
contains string "MAX", but what if someone calls it "LAST" instead? It 
gets fragile.

Or situations like expanding into reserved fields. How can we detect the 
difference between this:

struct foo {
	__u32 x;
	__u32 y;
+       __u32 z;
+       __u8  padding[12];
-	__u8  padding[16];
}

And this:

struct foo {
	__u32 x;
	__u32 y;
+       __u32 z;
+       char  codename[4]; /* Takes "NAME" */
-	char  codename[8]; /* Takes "CODENAME" */
}

Maybe we grep for "pad" or "reserved", but again... fragile.

Another idea is to add some sort of in-line comment to give the checker 
a hint that the field is intentionally unstable. It could be implicit 
for "*_MAX" enum variants or "*padding" at the end of structs, but if 
you wanted to have something like "end[]" (like in the rseq change), you 
could add /* ABI-unstable */ next to it and the script would ignore it.

Beyond those issues, we have non-trivial situations like when it's safe 
to add members to a struct. We know the kernel will zero-extend 
mismatches between kernel and userspace, but how do we know the driver 
properly handles the case of an old userspace passing an old struct?

So far, we've erred on the side of flagging it if it _could_ be a break 
and relied on the reviewer to make the final determination.
  

Hi all.

On Thu, 13 Apr 2023 at 15:37, Mark Wielaard <mark@klomp.org> wrote:
>
> Hi,
>
> On Wed, 2023-04-12 at 09:37 -0700, John Moon via Libabigail wrote:
> > On 4/11/2023 11:14 PM, Greg Kroah-Hartman wrote:
> > > > Would you find the tool more useful if it simply filtered out all instances
> > > > where the size of the type did not change? This would filter out the
> > > > following which the tool currently flags:
> > > >
> > > > - enum expansions
> > > > - reserved field expansions
> > > > - expansions of a struct with a flex array at the end
> > > > - type changes
> > > > - re-ordering of existing members
> > > > - ...others?
> > >
> > > Obviously not, as some of those are real breakages, and some are not at
> > > all.
> > >
> > > Please understand what is an abi breakage.  Adding new enums is not.
> > > Using a reserved field is not.  Reording existing members IS.
> > >
> >
> > Yes, understood that method would miss certain classes of breakages. I
> > was suggesting it as a way to improve the signal-to-noise ratio of the
> > tool since we don't currently have an algorithm for determining
> > breakages with 100% accuracy.
>
> Note that you can check the exit code of libabigail's abidiff to see
> whether something is an incompatible abi change or not, see:
> https://sourceware.org/libabigail/manual/abidiff.html#return-values
>
> You can also of course use suppressions to instruct abidiff to avoid
> reporting changes involving certain ABI artifacts:
> https://sourceware.org/libabigail/manual/libabigail-concepts.html#suppr-spec-label

libabigail's abidiff already hides certain differences by default.
You can turn this behaviour off with --harmless.

Note that abidiff without --harmless treats certain ABI differences
asymmetrically,
hiding them one way around but not the other.

The ABI diff tool I designed for Android always treats differences symmetrically
and will only suppress certain kinds of diff if specially requested
(which we don't
do any more in production). [Technically, we also ignore qualifier changes on
function parameter and return types, but we achieve that by stripping them out
unconditionally.]

Once we get around to UAPI monitoring, we'll do the same there. We can always
review the SNR later.

Regards,
Giuliano.

> Cheers,
>
> Mark
>
> --
> To unsubscribe from this group and stop receiving emails from it, send an email to kernel-team+unsubscribe@android.com.
>
  

On 4/13/2023 7:37 AM, Mark Wielaard wrote:
> Hi,
> 
> On Wed, 2023-04-12 at 09:37 -0700, John Moon via Libabigail wrote:
>> On 4/11/2023 11:14 PM, Greg Kroah-Hartman wrote:
>>>> Would you find the tool more useful if it simply filtered out all instances
>>>> where the size of the type did not change? This would filter out the
>>>> following which the tool currently flags:
>>>>
>>>> - enum expansions
>>>> - reserved field expansions
>>>> - expansions of a struct with a flex array at the end
>>>> - type changes
>>>> - re-ordering of existing members
>>>> - ...others?
>>>
>>> Obviously not, as some of those are real breakages, and some are not at
>>> all.
>>>
>>> Please understand what is an abi breakage.  Adding new enums is not.
>>> Using a reserved field is not.  Reording existing members IS.
>>>
>>
>> Yes, understood that method would miss certain classes of breakages. I
>> was suggesting it as a way to improve the signal-to-noise ratio of the
>> tool since we don't currently have an algorithm for determining
>> breakages with 100% accuracy.
> 
> Note that you can check the exit code of libabigail's abidiff to see
> whether something is an incompatible abi change or not, see:
> https://sourceware.org/libabigail/manual/abidiff.html#return-values
> 
> You can also of course use suppressions to instruct abidiff to avoid
> reporting changes involving certain ABI artifacts:
> https://sourceware.org/libabigail/manual/libabigail-concepts.html#suppr-spec-label
> 
> Cheers,
> 
> Mark

Checking the ABIDIFF_ABI_INCOMPATIBLE_CHANGE flag in the return code is 
a good idea, but checking it doesn't change what the tool is currently 
outputting (i.e. the flag is set for all the changes currently 
reported). I think this is because of some filtering we're doing based 
on grepping stdout, but checking the return code would be more stable.

The suppressions may work for some cases, but I fear they would be too 
eager in other cases. Looking at the docs, I'm not sure how we could 
express something like:

"suppress changed enumerators if they end in 'MAX' or 'LAST' and appear 
at the end of the enumeration"

or

"suppress data member insertions into a struct if the last member in the 
struct has its size reduced by sizeof(new_member) and is named 'pad' or 
'reserved'"

They're complicated cases to detect in a general way.

Thanks,
John
  

On 4/13/2023 10:03 AM, Nicolas Schier wrote:
> On Mon, Apr 10, 2023 at 07:03:05PM +0900 Masahiro Yamada wrote:
>> CAVEAT
>> This tool runs 'git checkout' a couple of times internally.
>> If you interrupt it, your worktree might be messed up.
> 
> John, did you consider using git export for obtaining a source tree that
> can be used for headers_install and the following comparison, instead of
> git stash+checkout?  AFACS the script does not depend on any other git
> functionality than providing a specific version of the source tree.  I am
> pretty sure that leaving the original working copy in its original state
> is supporting the script's popularity.
> 
> Kind regards,
> Nicolas
> 

I hadn't considered that. Great suggestion! I'll use this method in the 
next rev. Thanks!
  

On Thu, Apr 13, 2023 at 10:07:23AM -0700, John Moon wrote:
> On 4/12/2023 9:43 AM, Greg Kroah-Hartman wrote:
> > On Wed, Apr 12, 2023 at 09:37:16AM -0700, John Moon wrote:
> > > On 4/11/2023 11:14 PM, Greg Kroah-Hartman wrote:
> > > > > Would you find the tool more useful if it simply filtered out all instances
> > > > > where the size of the type did not change? This would filter out the
> > > > > following which the tool currently flags:
> > > > > 
> > > > > - enum expansions
> > > > > - reserved field expansions
> > > > > - expansions of a struct with a flex array at the end
> > > > > - type changes
> > > > > - re-ordering of existing members
> > > > > - ...others?
> > > > 
> > > > Obviously not, as some of those are real breakages, and some are not at
> > > > all.
> > > > 
> > > > Please understand what is an abi breakage.  Adding new enums is not.
> > > > Using a reserved field is not.  Reording existing members IS.
> > > > 
> > > 
> > > Yes, understood that method would miss certain classes of breakages. I was
> > > suggesting it as a way to improve the signal-to-noise ratio of the tool
> > > since we don't currently have an algorithm for determining breakages with
> > > 100% accuracy.
> > 
> > Why not?  You know the different types of things here based on the
> > differences between the dwarf data, and they fall into different
> > categories, and those different categories mean different things.
> > 
> > If you have questions as to which type of change is allowed and which is
> > not, just ask us, the rules are not complex, nor impossible to describe,
> > otherwise we wouldn't have a stable api at all, right?
> > 
> 
> Right, it's currently a limitation of parsing the abidiff output.
> 
> Even in trivial situations like an enum expansion, the tool knows that a
> variant was added and another variant had its offset changed. There's not a
> good way to say for sure that the variant whose offset changed is a "*_MAX"
> variant. So if we simply ignored enum expansion, we'd miss breakages like
> this:
> 
> enum foo {
> 	FLAG_A,
> +       FLAG_B,
> 	FLAG_C,
> 	FLAG_MAX
> }
> 
> Maybe we can ignore an enum expansion if only the last variant's offset
> changed, but then we'd miss cases where enums don't have a MAX variant.
> Maybe we could limit the check to last variant's offset whose name contains
> string "MAX", but what if someone calls it "LAST" instead? It gets fragile.

That's what the regexes are for, you can make them on a per-file basis,
right?

> Or situations like expanding into reserved fields. How can we detect the
> difference between this:
> 
> struct foo {
> 	__u32 x;
> 	__u32 y;
> +       __u32 z;
> +       __u8  padding[12];
> -	__u8  padding[16];
> }
> 
> And this:
> 
> struct foo {
> 	__u32 x;
> 	__u32 y;
> +       __u32 z;
> +       char  codename[4]; /* Takes "NAME" */
> -	char  codename[8]; /* Takes "CODENAME" */
> }
> 
> Maybe we grep for "pad" or "reserved", but again... fragile.

Again, regexes.

But if this is too fragile, then yes, it's going to be useless as those
are obviously allowed changes and you are giving us a tool that would
say they are forbidden.

> Another idea is to add some sort of in-line comment to give the checker a
> hint that the field is intentionally unstable. It could be implicit for
> "*_MAX" enum variants or "*padding" at the end of structs, but if you wanted
> to have something like "end[]" (like in the rseq change), you could add /*
> ABI-unstable */ next to it and the script would ignore it.

The abi isn't "unstable", it's "extensible".  Those are two very
different things.

> Beyond those issues, we have non-trivial situations like when it's safe to
> add members to a struct. We know the kernel will zero-extend mismatches
> between kernel and userspace, but how do we know the driver properly handles
> the case of an old userspace passing an old struct?

Then flag it to make sure as the "driver" is the "kernel".

> So far, we've erred on the side of flagging it if it _could_ be a break and
> relied on the reviewer to make the final determination.

But don't give us loads of "this could be broken" if it really isn't
please.

You have decades of code history to run the tool on to get these things
worked out.  Please do so before expecting us to use it and complain
about things it flags that are not actual breakages.

In it's current form, would you use this tool if you were the maintainer
of a subsystem?

thanks,

greg k-h