[0/9] Add debug_annotate attributes

  Hello,

This patch series adds support for:

- Two new C-language-level attributes that allow to associate (to "annotate" or
  to "tag") particular declarations and types with arbitrary strings. As
  explained below, this is intended to be used to, for example, characterize
  certain pointer types.

- The conveyance of that information in the DWARF output in the form of a new
  DIE: DW_TAG_GNU_annotation.

- The conveyance of that information in the BTF output in the form of two new
  kinds of BTF objects: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.

All of these facilities are being added to the eBPF ecosystem, and support for
them exists in some form in LLVM.

Purpose
=======

1)  Addition of C-family language constructs (attributes) to specify free-text
    tags on certain language elements, such as struct fields.

    The purpose of these annotations is to provide additional information about
    types, variables, and function parameters of interest to the kernel. A
    driving use case is to tag pointer types within the linux kernel and eBPF
    programs with additional semantic information, such as '__user' or '__rcu'.

    For example, consider the linux kernel function do_execve with the
    following declaration:

      static int do_execve(struct filename *filename,
         const char __user *const __user *__argv,
         const char __user *const __user *__envp);

    Here, __user could be defined with these annotations to record semantic
    information about the pointer parameters (e.g., they are user-provided) in
    DWARF and BTF information. Other kernel facilites such as the eBPF verifier
    can read the tags and make use of the information.

2)  Conveying the tags in the generated DWARF debug info.

    The main motivation for emitting the tags in DWARF is that the Linux kernel
    generates its BTF information via pahole, using DWARF as a source:

        +--------+  BTF                  BTF   +----------+
        | pahole |-------> vmlinux.btf ------->| verifier |
        +--------+                             +----------+
            ^                                        ^
            |                                        |
      DWARF |                                    BTF |
            |                                        |
         vmlinux                              +-------------+
         module1.ko                           | BPF program |
         module2.ko                           +-------------+
           ...

    This is because:

    a)  Unlike GCC, LLVM will only generate BTF for BPF programs.

    b)  GCC can generate BTF for whatever target with -gbtf, but there is no
        support for linking/deduplicating BTF in the linker.

    In the scenario above, the verifier needs access to the pointer tags of
    both the kernel types/declarations (conveyed in the DWARF and translated
    to BTF by pahole) and those of the BPF program (available directly in BTF).

    Another motivation for having the tag information in DWARF, unrelated to
    BPF and BTF, is that the drgn project (another DWARF consumer) also wants
    to benefit from these tags in order to differentiate between different
    kinds of pointers in the kernel.

3)  Conveying the tags in the generated BTF debug info.

    This is easy: the main purpose of having this info in BTF is for the
    compiled eBPF programs. The kernel verifier can then access the tags
    of pointers used by the eBPF programs.


For more information about these tags and the motivation behind them, please
refer to the following linux kernel discussions:

  https://lore.kernel.org/bpf/20210914223004.244411-1-yhs@fb.com/
  https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
  https://lore.kernel.org/bpf/20211112012604.1504583-1-yhs@fb.com/


Implementation Overview
=======================

To enable these annotations, two new C language attributes are added:
__attribute__((debug_annotate_decl("foo"))) and
__attribute__((debug_annotate_type("bar"))). Both attributes accept a single
arbitrary string constant argument, which will be recorded in the generated
DWARF and/or BTF debug information. They have no effect on code generation.

Note that we are not using the same attribute names as LLVM (btf_decl_tag and
btf_type_tag, respectively). While these attributes are functionally very
similar, they have grown beyond purely BTF-specific uses, so inclusion of "btf"
in the attribute name seems misleading.

DWARF support is enabled via a new DW_TAG_GNU_annotation. When generating DWARF,
declarations and types will be checked for the corresponding attributes. If
present, a DW_TAG_GNU_annotation DIE will be created as a child of the DIE for
the annotated type or declaration, one for each tag. These DIEs link the
arbitrary tag value to the item they annotate.

For example, the following variable declaration:

  #define __typetag1 __attribute__((debug_annotate_type ("typetag1")))

  #define __decltag1 __attribute__((debug_annotate_decl ("decltag1")))
  #define __decltag2 __attribute__((debug_annotate_decl ("decltag2")))

  int * __typetag1 x __decltag1 __decltag2;

Produces the following DWARF information:

 <1><1e>: Abbrev Number: 3 (DW_TAG_variable)
    <1f>   DW_AT_name        : x
    <21>   DW_AT_decl_file   : 1
    <22>   DW_AT_decl_line   : 7
    <23>   DW_AT_decl_column : 18
    <24>   DW_AT_type        : <0x49>
    <28>   DW_AT_external    : 1
    <28>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
    <32>   DW_AT_sibling     : <0x49>
 <2><36>: Abbrev Number: 1 (User TAG value: 0x6000)
    <37>   DW_AT_name        : (indirect string, offset: 0xd6): debug_annotate_decl
    <3b>   DW_AT_const_value : (indirect string, offset: 0xcd): decltag2
 <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000)
    <40>   DW_AT_name        : (indirect string, offset: 0xd6): debug_annotate_decl
    <44>   DW_AT_const_value : (indirect string, offset: 0x0): decltag1
 <2><48>: Abbrev Number: 0
 <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type)
    <4a>   DW_AT_byte_size   : 8
    <4b>   DW_AT_type        : <0x5d>
    <4f>   DW_AT_sibling     : <0x5d>
 <2><53>: Abbrev Number: 1 (User TAG value: 0x6000)
    <54>   DW_AT_name        : (indirect string, offset: 0x9): debug_annotate_type
    <58>   DW_AT_const_value : (indirect string, offset: 0x1d): typetag1
 <2><5c>: Abbrev Number: 0
 <1><5d>: Abbrev Number: 5 (DW_TAG_base_type)
    <5e>   DW_AT_byte_size   : 4
    <5f>   DW_AT_encoding    : 5	(signed)
    <60>   DW_AT_name        : int
 <1><64>: Abbrev Number: 0

In the case of BTF, the annotations are recorded in two type kinds recently
added to the BTF specification: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
The above example declaration prodcues the following BTF information:

[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[2] PTR '(anon)' type_id=3
[3] TYPE_TAG 'typetag1' type_id=1
[4] DECL_TAG 'decltag1' type_id=6 component_idx=-1
[5] DECL_TAG 'decltag2' type_id=6 component_idx=-1
[6] VAR 'x' type_id=2, linkage=global
[7] DATASEC '.bss' size=0 vlen=1
	type_id=6 offset=0 size=8 (VAR 'x')


David Faust (9):
  dwarf: add dw_get_die_parent function
  include: Add new definitions
  c-family: Add debug_annotate attribute handlers
  dwarf: generate annotation DIEs
  ctfc: pass through debug annotations to BTF
  dwarf2ctf: convert annotation DIEs to CTF types
  btf: output decl_tag and type_tag records
  doc: document new attributes
  testsuite: add debug annotation tests

 gcc/btfout.cc                                 |  28 +++++
 gcc/c-family/c-attribs.cc                     |  43 +++++++
 gcc/ctf-int.h                                 |  29 +++++
 gcc/ctfc.cc                                   |  11 +-
 gcc/ctfc.h                                    |  17 ++-
 gcc/doc/extend.texi                           | 106 ++++++++++++++++
 gcc/dwarf2ctf.cc                              | 114 +++++++++++++++++-
 gcc/dwarf2out.cc                              | 102 ++++++++++++++++
 gcc/dwarf2out.h                               |   1 +
 .../gcc.dg/debug/btf/btf-decltag-func.c       |  18 +++
 .../gcc.dg/debug/btf/btf-decltag-sou.c        |  34 ++++++
 .../gcc.dg/debug/btf/btf-decltag-typedef.c    |  15 +++
 .../gcc.dg/debug/btf/btf-typetag-1.c          |  20 +++
 .../gcc.dg/debug/dwarf2/annotation-1.c        |  20 +++
 .../gcc.dg/debug/dwarf2/annotation-2.c        |  17 +++
 .../gcc.dg/debug/dwarf2/annotation-3.c        |  20 +++
 .../gcc.dg/debug/dwarf2/annotation-4.c        |  34 ++++++
 include/btf.h                                 |  17 ++-
 include/dwarf2.def                            |   4 +
 19 files changed, 639 insertions(+), 11 deletions(-)
 create mode 100644 gcc/ctf-int.h
 create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-decltag-func.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-decltag-sou.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-decltag-typedef.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-typetag-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2/annotation-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2/annotation-2.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2/annotation-3.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2/annotation-4.c
  

> On 6/17/22 10:18 AM, Jose E. Marchesi wrote:
>> Hi Yonghong.
>> 
>>> On 6/15/22 1:57 PM, David Faust wrote:
>>>>
>>>> On 6/14/22 22:53, Yonghong Song wrote:
>>>>>
>>>>>
>>>>> On 6/7/22 2:43 PM, David Faust wrote:
>>>>>> Hello,
>>>>>>
>>>>>> This patch series adds support for:
>>>>>>
>>>>>> - Two new C-language-level attributes that allow to associate (to "annotate" or
>>>>>>      to "tag") particular declarations and types with arbitrary strings. As
>>>>>>      explained below, this is intended to be used to, for example, characterize
>>>>>>      certain pointer types.
>>>>>>
>>>>>> - The conveyance of that information in the DWARF output in the form of a new
>>>>>>      DIE: DW_TAG_GNU_annotation.
>>>>>>
>>>>>> - The conveyance of that information in the BTF output in the form of two new
>>>>>>      kinds of BTF objects: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>
>>>>>> All of these facilities are being added to the eBPF ecosystem, and support for
>>>>>> them exists in some form in LLVM.
>>>>>>
>>>>>> Purpose
>>>>>> =======
>>>>>>
>>>>>> 1)  Addition of C-family language constructs (attributes) to specify free-text
>>>>>>        tags on certain language elements, such as struct fields.
>>>>>>
>>>>>>        The purpose of these annotations is to provide additional information about
>>>>>>        types, variables, and function parameters of interest to the kernel. A
>>>>>>        driving use case is to tag pointer types within the linux kernel and eBPF
>>>>>>        programs with additional semantic information, such as '__user' or '__rcu'.
>>>>>>
>>>>>>        For example, consider the linux kernel function do_execve with the
>>>>>>        following declaration:
>>>>>>
>>>>>>          static int do_execve(struct filename *filename,
>>>>>>             const char __user *const __user *__argv,
>>>>>>             const char __user *const __user *__envp);
>>>>>>
>>>>>>        Here, __user could be defined with these annotations to record semantic
>>>>>>        information about the pointer parameters (e.g., they are user-provided) in
>>>>>>        DWARF and BTF information. Other kernel facilites such as the eBPF verifier
>>>>>>        can read the tags and make use of the information.
>>>>>>
>>>>>> 2)  Conveying the tags in the generated DWARF debug info.
>>>>>>
>>>>>>        The main motivation for emitting the tags in DWARF is that the Linux kernel
>>>>>>        generates its BTF information via pahole, using DWARF as a source:
>>>>>>
>>>>>>            +--------+  BTF                  BTF   +----------+
>>>>>>            | pahole |-------> vmlinux.btf ------->| verifier |
>>>>>>            +--------+                             +----------+
>>>>>>                ^                                        ^
>>>>>>                |                                        |
>>>>>>          DWARF |                                    BTF |
>>>>>>                |                                        |
>>>>>>             vmlinux                              +-------------+
>>>>>>             module1.ko                           | BPF program |
>>>>>>             module2.ko                           +-------------+
>>>>>>               ...
>>>>>>
>>>>>>        This is because:
>>>>>>
>>>>>>        a)  Unlike GCC, LLVM will only generate BTF for BPF programs.
>>>>>>
>>>>>>        b)  GCC can generate BTF for whatever target with -gbtf, but there is no
>>>>>>            support for linking/deduplicating BTF in the linker.
>>>>>>
>>>>>>        In the scenario above, the verifier needs access to the pointer tags of
>>>>>>        both the kernel types/declarations (conveyed in the DWARF and translated
>>>>>>        to BTF by pahole) and those of the BPF program (available directly in BTF).
>>>>>>
>>>>>>        Another motivation for having the tag information in DWARF, unrelated to
>>>>>>        BPF and BTF, is that the drgn project (another DWARF consumer) also wants
>>>>>>        to benefit from these tags in order to differentiate between different
>>>>>>        kinds of pointers in the kernel.
>>>>>>
>>>>>> 3)  Conveying the tags in the generated BTF debug info.
>>>>>>
>>>>>>        This is easy: the main purpose of having this info in BTF is for the
>>>>>>        compiled eBPF programs. The kernel verifier can then access the tags
>>>>>>        of pointers used by the eBPF programs.
>>>>>>
>>>>>>
>>>>>> For more information about these tags and the motivation behind them, please
>>>>>> refer to the following linux kernel discussions:
>>>>>>
>>>>>>      https://lore.kernel.org/bpf/20210914223004.244411-1-yhs@fb.com/
>>>>>>      https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
>>>>>>      https://lore.kernel.org/bpf/20211112012604.1504583-1-yhs@fb.com/
>>>>>>
>>>>>>
>>>>>> Implementation Overview
>>>>>> =======================
>>>>>>
>>>>>> To enable these annotations, two new C language attributes are added:
>>>>>> __attribute__((debug_annotate_decl("foo"))) and
>>>>>> __attribute__((debug_annotate_type("bar"))). Both attributes accept a single
>>>>>> arbitrary string constant argument, which will be recorded in the generated
>>>>>> DWARF and/or BTF debug information. They have no effect on code generation.
>>>>>>
>>>>>> Note that we are not using the same attribute names as LLVM (btf_decl_tag and
>>>>>> btf_type_tag, respectively). While these attributes are functionally very
>>>>>> similar, they have grown beyond purely BTF-specific uses, so inclusion of "btf"
>>>>>> in the attribute name seems misleading.
>>>>>>
>>>>>> DWARF support is enabled via a new DW_TAG_GNU_annotation. When generating DWARF,
>>>>>> declarations and types will be checked for the corresponding attributes. If
>>>>>> present, a DW_TAG_GNU_annotation DIE will be created as a child of the DIE for
>>>>>> the annotated type or declaration, one for each tag. These DIEs link the
>>>>>> arbitrary tag value to the item they annotate.
>>>>>>
>>>>>> For example, the following variable declaration:
>>>>>>
>>>>>>      #define __typetag1 __attribute__((debug_annotate_type ("typetag1")))
>>>>>>
>>>>>>      #define __decltag1 __attribute__((debug_annotate_decl ("decltag1")))
>>>>>>      #define __decltag2 __attribute__((debug_annotate_decl ("decltag2")))
>>>>>>
>>>>>>      int * __typetag1 x __decltag1 __decltag2;
>>>>>
>>>>> Based on the above example
>>>>>            static int do_execve(struct filename *filename,
>>>>>              const char __user *const __user *__argv,
>>>>>              const char __user *const __user *__envp);
>>>>>
>>>>> Should the above example should be the below?
>>>>>        int __typetag1 * x __decltag1 __decltag2
>>>>>
>>>> This example is not related to the one above. It is just meant to
>>>> show the behavior of both attributes. My apologies for not making
>>>> that clear.
>>>
>>> Okay, it should be fine if the dwarf debug_info is shown.
>>>
>>>>
>>>>>>
>>>>>> Produces the following DWARF information:
>>>>>>
>>>>>>     <1><1e>: Abbrev Number: 3 (DW_TAG_variable)
>>>>>>        <1f>   DW_AT_name        : x
>>>>>>        <21>   DW_AT_decl_file   : 1
>>>>>>        <22>   DW_AT_decl_line   : 7
>>>>>>        <23>   DW_AT_decl_column : 18
>>>>>>        <24>   DW_AT_type        : <0x49>
>>>>>>        <28>   DW_AT_external    : 1
>>>>>>        <28>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
>>>>>>        <32>   DW_AT_sibling     : <0x49>
>>>>>>     <2><36>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>        <37>   DW_AT_name        : (indirect string, offset: 0xd6): debug_annotate_decl
>>>>>>        <3b>   DW_AT_const_value : (indirect string, offset: 0xcd): decltag2
>>>>>>     <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>        <40>   DW_AT_name        : (indirect string, offset: 0xd6): debug_annotate_decl
>>>>>>        <44>   DW_AT_const_value : (indirect string, offset: 0x0): decltag1
>>>>>>     <2><48>: Abbrev Number: 0
>>>>>>     <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type)
>>>>>>        <4a>   DW_AT_byte_size   : 8
>>>>>>        <4b>   DW_AT_type        : <0x5d>
>>>>>>        <4f>   DW_AT_sibling     : <0x5d>
>>>>>>     <2><53>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>        <54>   DW_AT_name        : (indirect string, offset: 0x9): debug_annotate_type
>>>>>>        <58>   DW_AT_const_value : (indirect string, offset: 0x1d): typetag1
>>>>>>     <2><5c>: Abbrev Number: 0
>>>>>>     <1><5d>: Abbrev Number: 5 (DW_TAG_base_type)
>>>>>>        <5e>   DW_AT_byte_size   : 4
>>>>>>        <5f>   DW_AT_encoding    : 5	(signed)
>>>>>>        <60>   DW_AT_name        : int
>>>>>>     <1><64>: Abbrev Number: 0
>>>
>>> This shows the info in .debug_abbrev. What I mean is to
>>> show the related info in .debug_info section which seems more useful to
>>> understand the relationships between different tags. Maybe this is due
>>> to that I am not fully understanding what <1>/<2> means in <1><49> and
>>> <2><53> etc.
>> I think that dump actually shows .debug_info, with the abbrevs
>> expanded...
>> Anyway, it seems to us that the root of this problem is the fact the
>> kernel sparse annotations, such as address_space(__user), are:
>> 1) To be processed by an external kernel-specific tool (
>>     https://sparse.docs.kernel.org/en/latest/annotations.html) and not a
>>     C compiler, and therefore,
>> 2) Not quite the same than compiler attributes (despite the way they
>>     look.)  In particular, they seem to assume an ordering different than
>>     of GNU attributes: in some cases given the same written order, they
>>     refer to different things!.  Which is quite unfortunate :(
>
> Yes, currently __user/__kernel macros (implemented with address_space
> attribute) are processed by macros.
>
>> Now, if I understood properly, you plan to change the definition of
>> __user and __kernel in the kernel sources in order to generate the tag
>> compiler attributes, correct?
>
> Right. The original __user definition likes:
>   # define __user         __attribute__((noderef, address_space(__user)))
>
> The new attribute looks like
>   # define BTF_TYPE_TAG(value) __attribute__((btf_type_tag(#value)))
>   #  define __user        BTF_TYPE_TAG(user)

Ok I see.  So the kernel will stop using sparse attributes to implement
__user and __kernel and start using compiler attributes for tags
instead.

>> Is that the reason why LLVM implements what we assume to be the
>> sparse
>> ordering, and not the correct GNU attributes ordering, for the tag
>> attributes?
>
> Note that __user attributes apply to pointee's and not pointers.
> Just like
>    const int *p;
> the 'const' is not applied to pointer 'p', but the pointee of 'p'.
>
> What current llvm dwarf generation with
>    pointer
>      <--- btf_type_tag
> is just ONE implementation. As I said earlier, I am okay to
> have dwarf implementation like
>    p->btf_type_tag->const->int.
> If you can propose an implementation like this in dwarf. I can propose
> to change implementation in llvm.

I think we are miscommunicating.

Looks like there is a divergence on what attributes apply to what
language entities between the sparse compiler and GCC/LLVM.  How to
represent that in DWARF is a different matter.

For this example:

  int __typetag1 * __typetag2 __typetag3 * g;

a) GCC associates __typetag1 with the pointer-to-pointer-to-int.
b) LLVM associates __typetag1 to pointer-to-int.

Where:

a) Is the expected behavior of a compiler attributes, as documented in
   the GCC manual.

b) Is presumably what the sparse compiler expects, but _not_ the
   ordering expected for a compiler GNU attribute.

So, if the kernel source __user and __kernel annotations (which
currently expand to sparse attributes) follow the sparse ordering, and
you want to implement __user and __kernel in terms of compiler
attributes instead (the annotation attributes) then you will have to:

1) Fix LLVM to implement the usual ordering for these attributes and
2) fix the kernel sources to use that ordering

[Incidentally, the same applies to another "ex-sparse" attribute you
 have in the kernel and also implemented in LLVM with a weird ordering:
 the address_space attribute.]

For 2), it may be possible to write a coccinnelle script to generate the
patch...

Does this make sense?

>> If that is so, we have quite a problem here: I don't think we can
>> change
>> the way GCC handles GNU-like attributes just because the kernel sources
>> want to hook on these __user/__kernel sparse annotations to generate the
>> compiler tags, even if we could mayhaps get GCC to handle
>> debug_annotate_type and debug_annotate_decl differently.  Some would say
>> doing so would perpetuate the mistake instead of fixing it...
>> Is my understanding correct?
>
> Let us just say that the btf_type_tag attribute applies to pointees.
> Does this help?
>
>> 
>>>>>
>>>>> Maybe you can also show what dwarf debug_info looks like
>>>> I am not sure what you mean. This is the .debug_info section as output
>>>> by readelf -w. I did trim some information not relevant to the discussion
>>>> such as the DW_TAG_compile_unit DIE, for brevity.
>>>>
>>>>>
>>>>>>
>>>>>> In the case of BTF, the annotations are recorded in two type kinds recently
>>>>>> added to the BTF specification: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>> The above example declaration prodcues the following BTF information:
>>>>>>
>>>>>> [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
>>>>>> [2] PTR '(anon)' type_id=3
>>>>>> [3] TYPE_TAG 'typetag1' type_id=1
>>>>>> [4] DECL_TAG 'decltag1' type_id=6 component_idx=-1
>>>>>> [5] DECL_TAG 'decltag2' type_id=6 component_idx=-1
>>>>>> [6] VAR 'x' type_id=2, linkage=global
>>>>>> [7] DATASEC '.bss' size=0 vlen=1
>>>>>> 	type_id=6 offset=0 size=8 (VAR 'x')
>>>>>>
>>>>>>
>>>>> [...]
  

On 6/21/22 9:12 AM, Jose E. Marchesi wrote:
> 
>> On 6/17/22 10:18 AM, Jose E. Marchesi wrote:
>>> Hi Yonghong.
>>>
>>>> On 6/15/22 1:57 PM, David Faust wrote:
>>>>>
>>>>> On 6/14/22 22:53, Yonghong Song wrote:
>>>>>>
>>>>>>
>>>>>> On 6/7/22 2:43 PM, David Faust wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> This patch series adds support for:
>>>>>>>
>>>>>>> - Two new C-language-level attributes that allow to associate (to "annotate" or
>>>>>>>       to "tag") particular declarations and types with arbitrary strings. As
>>>>>>>       explained below, this is intended to be used to, for example, characterize
>>>>>>>       certain pointer types.
>>>>>>>
>>>>>>> - The conveyance of that information in the DWARF output in the form of a new
>>>>>>>       DIE: DW_TAG_GNU_annotation.
>>>>>>>
>>>>>>> - The conveyance of that information in the BTF output in the form of two new
>>>>>>>       kinds of BTF objects: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>
>>>>>>> All of these facilities are being added to the eBPF ecosystem, and support for
>>>>>>> them exists in some form in LLVM.
>>>>>>>
>>>>>>> Purpose
>>>>>>> =======
>>>>>>>
>>>>>>> 1)  Addition of C-family language constructs (attributes) to specify free-text
>>>>>>>         tags on certain language elements, such as struct fields.
>>>>>>>
>>>>>>>         The purpose of these annotations is to provide additional information about
>>>>>>>         types, variables, and function parameters of interest to the kernel. A
>>>>>>>         driving use case is to tag pointer types within the linux kernel and eBPF
>>>>>>>         programs with additional semantic information, such as '__user' or '__rcu'.
>>>>>>>
>>>>>>>         For example, consider the linux kernel function do_execve with the
>>>>>>>         following declaration:
>>>>>>>
>>>>>>>           static int do_execve(struct filename *filename,
>>>>>>>              const char __user *const __user *__argv,
>>>>>>>              const char __user *const __user *__envp);
>>>>>>>
>>>>>>>         Here, __user could be defined with these annotations to record semantic
>>>>>>>         information about the pointer parameters (e.g., they are user-provided) in
>>>>>>>         DWARF and BTF information. Other kernel facilites such as the eBPF verifier
>>>>>>>         can read the tags and make use of the information.
>>>>>>>
>>>>>>> 2)  Conveying the tags in the generated DWARF debug info.
>>>>>>>
>>>>>>>         The main motivation for emitting the tags in DWARF is that the Linux kernel
>>>>>>>         generates its BTF information via pahole, using DWARF as a source:
>>>>>>>
>>>>>>>             +--------+  BTF                  BTF   +----------+
>>>>>>>             | pahole |-------> vmlinux.btf ------->| verifier |
>>>>>>>             +--------+                             +----------+
>>>>>>>                 ^                                        ^
>>>>>>>                 |                                        |
>>>>>>>           DWARF |                                    BTF |
>>>>>>>                 |                                        |
>>>>>>>              vmlinux                              +-------------+
>>>>>>>              module1.ko                           | BPF program |
>>>>>>>              module2.ko                           +-------------+
>>>>>>>                ...
>>>>>>>
>>>>>>>         This is because:
>>>>>>>
>>>>>>>         a)  Unlike GCC, LLVM will only generate BTF for BPF programs.
>>>>>>>
>>>>>>>         b)  GCC can generate BTF for whatever target with -gbtf, but there is no
>>>>>>>             support for linking/deduplicating BTF in the linker.
>>>>>>>
>>>>>>>         In the scenario above, the verifier needs access to the pointer tags of
>>>>>>>         both the kernel types/declarations (conveyed in the DWARF and translated
>>>>>>>         to BTF by pahole) and those of the BPF program (available directly in BTF).
>>>>>>>
>>>>>>>         Another motivation for having the tag information in DWARF, unrelated to
>>>>>>>         BPF and BTF, is that the drgn project (another DWARF consumer) also wants
>>>>>>>         to benefit from these tags in order to differentiate between different
>>>>>>>         kinds of pointers in the kernel.
>>>>>>>
>>>>>>> 3)  Conveying the tags in the generated BTF debug info.
>>>>>>>
>>>>>>>         This is easy: the main purpose of having this info in BTF is for the
>>>>>>>         compiled eBPF programs. The kernel verifier can then access the tags
>>>>>>>         of pointers used by the eBPF programs.
>>>>>>>
>>>>>>>
>>>>>>> For more information about these tags and the motivation behind them, please
>>>>>>> refer to the following linux kernel discussions:
>>>>>>>
>>>>>>>       https://lore.kernel.org/bpf/20210914223004.244411-1-yhs@fb.com/
>>>>>>>       https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
>>>>>>>       https://lore.kernel.org/bpf/20211112012604.1504583-1-yhs@fb.com/
>>>>>>>
>>>>>>>
>>>>>>> Implementation Overview
>>>>>>> =======================
>>>>>>>
>>>>>>> To enable these annotations, two new C language attributes are added:
>>>>>>> __attribute__((debug_annotate_decl("foo"))) and
>>>>>>> __attribute__((debug_annotate_type("bar"))). Both attributes accept a single
>>>>>>> arbitrary string constant argument, which will be recorded in the generated
>>>>>>> DWARF and/or BTF debug information. They have no effect on code generation.
>>>>>>>
>>>>>>> Note that we are not using the same attribute names as LLVM (btf_decl_tag and
>>>>>>> btf_type_tag, respectively). While these attributes are functionally very
>>>>>>> similar, they have grown beyond purely BTF-specific uses, so inclusion of "btf"
>>>>>>> in the attribute name seems misleading.
>>>>>>>
>>>>>>> DWARF support is enabled via a new DW_TAG_GNU_annotation. When generating DWARF,
>>>>>>> declarations and types will be checked for the corresponding attributes. If
>>>>>>> present, a DW_TAG_GNU_annotation DIE will be created as a child of the DIE for
>>>>>>> the annotated type or declaration, one for each tag. These DIEs link the
>>>>>>> arbitrary tag value to the item they annotate.
>>>>>>>
>>>>>>> For example, the following variable declaration:
>>>>>>>
>>>>>>>       #define __typetag1 __attribute__((debug_annotate_type ("typetag1")))
>>>>>>>
>>>>>>>       #define __decltag1 __attribute__((debug_annotate_decl ("decltag1")))
>>>>>>>       #define __decltag2 __attribute__((debug_annotate_decl ("decltag2")))
>>>>>>>
>>>>>>>       int * __typetag1 x __decltag1 __decltag2;
>>>>>>
>>>>>> Based on the above example
>>>>>>             static int do_execve(struct filename *filename,
>>>>>>               const char __user *const __user *__argv,
>>>>>>               const char __user *const __user *__envp);
>>>>>>
>>>>>> Should the above example should be the below?
>>>>>>         int __typetag1 * x __decltag1 __decltag2
>>>>>>
>>>>> This example is not related to the one above. It is just meant to
>>>>> show the behavior of both attributes. My apologies for not making
>>>>> that clear.
>>>>
>>>> Okay, it should be fine if the dwarf debug_info is shown.
>>>>
>>>>>
>>>>>>>
>>>>>>> Produces the following DWARF information:
>>>>>>>
>>>>>>>      <1><1e>: Abbrev Number: 3 (DW_TAG_variable)
>>>>>>>         <1f>   DW_AT_name        : x
>>>>>>>         <21>   DW_AT_decl_file   : 1
>>>>>>>         <22>   DW_AT_decl_line   : 7
>>>>>>>         <23>   DW_AT_decl_column : 18
>>>>>>>         <24>   DW_AT_type        : <0x49>
>>>>>>>         <28>   DW_AT_external    : 1
>>>>>>>         <28>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
>>>>>>>         <32>   DW_AT_sibling     : <0x49>
>>>>>>>      <2><36>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>         <37>   DW_AT_name        : (indirect string, offset: 0xd6): debug_annotate_decl
>>>>>>>         <3b>   DW_AT_const_value : (indirect string, offset: 0xcd): decltag2
>>>>>>>      <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>         <40>   DW_AT_name        : (indirect string, offset: 0xd6): debug_annotate_decl
>>>>>>>         <44>   DW_AT_const_value : (indirect string, offset: 0x0): decltag1
>>>>>>>      <2><48>: Abbrev Number: 0
>>>>>>>      <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type)
>>>>>>>         <4a>   DW_AT_byte_size   : 8
>>>>>>>         <4b>   DW_AT_type        : <0x5d>
>>>>>>>         <4f>   DW_AT_sibling     : <0x5d>
>>>>>>>      <2><53>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>         <54>   DW_AT_name        : (indirect string, offset: 0x9): debug_annotate_type
>>>>>>>         <58>   DW_AT_const_value : (indirect string, offset: 0x1d): typetag1
>>>>>>>      <2><5c>: Abbrev Number: 0
>>>>>>>      <1><5d>: Abbrev Number: 5 (DW_TAG_base_type)
>>>>>>>         <5e>   DW_AT_byte_size   : 4
>>>>>>>         <5f>   DW_AT_encoding    : 5	(signed)
>>>>>>>         <60>   DW_AT_name        : int
>>>>>>>      <1><64>: Abbrev Number: 0
>>>>
>>>> This shows the info in .debug_abbrev. What I mean is to
>>>> show the related info in .debug_info section which seems more useful to
>>>> understand the relationships between different tags. Maybe this is due
>>>> to that I am not fully understanding what <1>/<2> means in <1><49> and
>>>> <2><53> etc.
>>> I think that dump actually shows .debug_info, with the abbrevs
>>> expanded...
>>> Anyway, it seems to us that the root of this problem is the fact the
>>> kernel sparse annotations, such as address_space(__user), are:
>>> 1) To be processed by an external kernel-specific tool (
>>>      https://sparse.docs.kernel.org/en/latest/annotations.html) and not a
>>>      C compiler, and therefore,
>>> 2) Not quite the same than compiler attributes (despite the way they
>>>      look.)  In particular, they seem to assume an ordering different than
>>>      of GNU attributes: in some cases given the same written order, they
>>>      refer to different things!.  Which is quite unfortunate :(
>>
>> Yes, currently __user/__kernel macros (implemented with address_space
>> attribute) are processed by macros.
>>
>>> Now, if I understood properly, you plan to change the definition of
>>> __user and __kernel in the kernel sources in order to generate the tag
>>> compiler attributes, correct?
>>
>> Right. The original __user definition likes:
>>    # define __user         __attribute__((noderef, address_space(__user)))
>>
>> The new attribute looks like
>>    # define BTF_TYPE_TAG(value) __attribute__((btf_type_tag(#value)))
>>    #  define __user        BTF_TYPE_TAG(user)
> 
> Ok I see.  So the kernel will stop using sparse attributes to implement
> __user and __kernel and start using compiler attributes for tags
> instead.
> 
>>> Is that the reason why LLVM implements what we assume to be the
>>> sparse
>>> ordering, and not the correct GNU attributes ordering, for the tag
>>> attributes?
>>
>> Note that __user attributes apply to pointee's and not pointers.
>> Just like
>>     const int *p;
>> the 'const' is not applied to pointer 'p', but the pointee of 'p'.
>>
>> What current llvm dwarf generation with
>>     pointer
>>       <--- btf_type_tag
>> is just ONE implementation. As I said earlier, I am okay to
>> have dwarf implementation like
>>     p->btf_type_tag->const->int.
>> If you can propose an implementation like this in dwarf. I can propose
>> to change implementation in llvm.
> 
> I think we are miscommunicating.
> 
> Looks like there is a divergence on what attributes apply to what
> language entities between the sparse compiler and GCC/LLVM.  How to
> represent that in DWARF is a different matter.
> 
> For this example:
> 
>    int __typetag1 * __typetag2 __typetag3 * g;
> 
> a) GCC associates __typetag1 with the pointer-to-pointer-to-int.
> b) LLVM associates __typetag1 to pointer-to-int.
> 
> Where:
> 
> a) Is the expected behavior of a compiler attributes, as documented in
>     the GCC manual.
> 
> b) Is presumably what the sparse compiler expects, but _not_ the
>     ordering expected for a compiler GNU attribute.
> 
> So, if the kernel source __user and __kernel annotations (which
> currently expand to sparse attributes) follow the sparse ordering, and
> you want to implement __user and __kernel in terms of compiler
> attributes instead (the annotation attributes) then you will have to:
> 
> 1) Fix LLVM to implement the usual ordering for these attributes and
> 2) fix the kernel sources to use that ordering
> 
> [Incidentally, the same applies to another "ex-sparse" attribute you
>   have in the kernel and also implemented in LLVM with a weird ordering:
>   the address_space attribute.]
> 
> For 2), it may be possible to write a coccinnelle script to generate the
> patch...

I don't think (2) (to change kernel source for different attr ordering)
will work. So the only thing we can do is in compiler/pahole except 
macro replacement in kernel.

> 
> Does this make sense?
> 
>>> If that is so, we have quite a problem here: I don't think we can
>>> change
>>> the way GCC handles GNU-like attributes just because the kernel sources
>>> want to hook on these __user/__kernel sparse annotations to generate the
>>> compiler tags, even if we could mayhaps get GCC to handle
>>> debug_annotate_type and debug_annotate_decl differently.  Some would say
>>> doing so would perpetuate the mistake instead of fixing it...
>>> Is my understanding correct?
>>
>> Let us just say that the btf_type_tag attribute applies to pointees.
>> Does this help?
>>
>>>
>>>>>>
>>>>>> Maybe you can also show what dwarf debug_info looks like
>>>>> I am not sure what you mean. This is the .debug_info section as output
>>>>> by readelf -w. I did trim some information not relevant to the discussion
>>>>> such as the DW_TAG_compile_unit DIE, for brevity.
>>>>>
>>>>>>
>>>>>>>
>>>>>>> In the case of BTF, the annotations are recorded in two type kinds recently
>>>>>>> added to the BTF specification: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>> The above example declaration prodcues the following BTF information:
>>>>>>>
>>>>>>> [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
>>>>>>> [2] PTR '(anon)' type_id=3
>>>>>>> [3] TYPE_TAG 'typetag1' type_id=1
>>>>>>> [4] DECL_TAG 'decltag1' type_id=6 component_idx=-1
>>>>>>> [5] DECL_TAG 'decltag2' type_id=6 component_idx=-1
>>>>>>> [6] VAR 'x' type_id=2, linkage=global
>>>>>>> [7] DATASEC '.bss' size=0 vlen=1
>>>>>>> 	type_id=6 offset=0 size=8 (VAR 'x')
>>>>>>>
>>>>>>>
>>>>>> [...]
  

Hi Yonghong.

> On 6/21/22 9:12 AM, Jose E. Marchesi wrote:
>> 
>>> On 6/17/22 10:18 AM, Jose E. Marchesi wrote:
>>>> Hi Yonghong.
>>>>
>>>>> On 6/15/22 1:57 PM, David Faust wrote:
>>>>>>
>>>>>> On 6/14/22 22:53, Yonghong Song wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 6/7/22 2:43 PM, David Faust wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> This patch series adds support for:
>>>>>>>>
>>>>>>>> - Two new C-language-level attributes that allow to associate (to "annotate" or
>>>>>>>>       to "tag") particular declarations and types with arbitrary strings. As
>>>>>>>>       explained below, this is intended to be used to, for example, characterize
>>>>>>>>       certain pointer types.
>>>>>>>>
>>>>>>>> - The conveyance of that information in the DWARF output in the form of a new
>>>>>>>>       DIE: DW_TAG_GNU_annotation.
>>>>>>>>
>>>>>>>> - The conveyance of that information in the BTF output in the form of two new
>>>>>>>>       kinds of BTF objects: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>
>>>>>>>> All of these facilities are being added to the eBPF ecosystem, and support for
>>>>>>>> them exists in some form in LLVM.
>>>>>>>>
>>>>>>>> Purpose
>>>>>>>> =======
>>>>>>>>
>>>>>>>> 1)  Addition of C-family language constructs (attributes) to specify free-text
>>>>>>>>         tags on certain language elements, such as struct fields.
>>>>>>>>
>>>>>>>>         The purpose of these annotations is to provide additional information about
>>>>>>>>         types, variables, and function parameters of interest to the kernel. A
>>>>>>>>         driving use case is to tag pointer types within the linux kernel and eBPF
>>>>>>>>         programs with additional semantic information, such as '__user' or '__rcu'.
>>>>>>>>
>>>>>>>>         For example, consider the linux kernel function do_execve with the
>>>>>>>>         following declaration:
>>>>>>>>
>>>>>>>>           static int do_execve(struct filename *filename,
>>>>>>>>              const char __user *const __user *__argv,
>>>>>>>>              const char __user *const __user *__envp);
>>>>>>>>
>>>>>>>>         Here, __user could be defined with these annotations to record semantic
>>>>>>>>         information about the pointer parameters (e.g., they are user-provided) in
>>>>>>>>         DWARF and BTF information. Other kernel facilites such as the eBPF verifier
>>>>>>>>         can read the tags and make use of the information.
>>>>>>>>
>>>>>>>> 2)  Conveying the tags in the generated DWARF debug info.
>>>>>>>>
>>>>>>>>         The main motivation for emitting the tags in DWARF is that the Linux kernel
>>>>>>>>         generates its BTF information via pahole, using DWARF as a source:
>>>>>>>>
>>>>>>>>             +--------+  BTF                  BTF   +----------+
>>>>>>>>             | pahole |-------> vmlinux.btf ------->| verifier |
>>>>>>>>             +--------+                             +----------+
>>>>>>>>                 ^                                        ^
>>>>>>>>                 |                                        |
>>>>>>>>           DWARF |                                    BTF |
>>>>>>>>                 |                                        |
>>>>>>>>              vmlinux                              +-------------+
>>>>>>>>              module1.ko                           | BPF program |
>>>>>>>>              module2.ko                           +-------------+
>>>>>>>>                ...
>>>>>>>>
>>>>>>>>         This is because:
>>>>>>>>
>>>>>>>>         a)  Unlike GCC, LLVM will only generate BTF for BPF programs.
>>>>>>>>
>>>>>>>>         b)  GCC can generate BTF for whatever target with -gbtf, but there is no
>>>>>>>>             support for linking/deduplicating BTF in the linker.
>>>>>>>>
>>>>>>>>         In the scenario above, the verifier needs access to the pointer tags of
>>>>>>>>         both the kernel types/declarations (conveyed in the DWARF and translated
>>>>>>>>         to BTF by pahole) and those of the BPF program (available directly in BTF).
>>>>>>>>
>>>>>>>>         Another motivation for having the tag information in DWARF, unrelated to
>>>>>>>>         BPF and BTF, is that the drgn project (another DWARF consumer) also wants
>>>>>>>>         to benefit from these tags in order to differentiate between different
>>>>>>>>         kinds of pointers in the kernel.
>>>>>>>>
>>>>>>>> 3)  Conveying the tags in the generated BTF debug info.
>>>>>>>>
>>>>>>>>         This is easy: the main purpose of having this info in BTF is for the
>>>>>>>>         compiled eBPF programs. The kernel verifier can then access the tags
>>>>>>>>         of pointers used by the eBPF programs.
>>>>>>>>
>>>>>>>>
>>>>>>>> For more information about these tags and the motivation behind them, please
>>>>>>>> refer to the following linux kernel discussions:
>>>>>>>>
>>>>>>>>       https://lore.kernel.org/bpf/20210914223004.244411-1-yhs@fb.com/
>>>>>>>>       https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
>>>>>>>>       https://lore.kernel.org/bpf/20211112012604.1504583-1-yhs@fb.com/
>>>>>>>>
>>>>>>>>
>>>>>>>> Implementation Overview
>>>>>>>> =======================
>>>>>>>>
>>>>>>>> To enable these annotations, two new C language attributes are added:
>>>>>>>> __attribute__((debug_annotate_decl("foo"))) and
>>>>>>>> __attribute__((debug_annotate_type("bar"))). Both attributes accept a single
>>>>>>>> arbitrary string constant argument, which will be recorded in the generated
>>>>>>>> DWARF and/or BTF debug information. They have no effect on code generation.
>>>>>>>>
>>>>>>>> Note that we are not using the same attribute names as LLVM (btf_decl_tag and
>>>>>>>> btf_type_tag, respectively). While these attributes are functionally very
>>>>>>>> similar, they have grown beyond purely BTF-specific uses, so inclusion of "btf"
>>>>>>>> in the attribute name seems misleading.
>>>>>>>>
>>>>>>>> DWARF support is enabled via a new DW_TAG_GNU_annotation. When generating DWARF,
>>>>>>>> declarations and types will be checked for the corresponding attributes. If
>>>>>>>> present, a DW_TAG_GNU_annotation DIE will be created as a child of the DIE for
>>>>>>>> the annotated type or declaration, one for each tag. These DIEs link the
>>>>>>>> arbitrary tag value to the item they annotate.
>>>>>>>>
>>>>>>>> For example, the following variable declaration:
>>>>>>>>
>>>>>>>>       #define __typetag1 __attribute__((debug_annotate_type ("typetag1")))
>>>>>>>>
>>>>>>>>       #define __decltag1 __attribute__((debug_annotate_decl ("decltag1")))
>>>>>>>>       #define __decltag2 __attribute__((debug_annotate_decl ("decltag2")))
>>>>>>>>
>>>>>>>>       int * __typetag1 x __decltag1 __decltag2;
>>>>>>>
>>>>>>> Based on the above example
>>>>>>>             static int do_execve(struct filename *filename,
>>>>>>>               const char __user *const __user *__argv,
>>>>>>>               const char __user *const __user *__envp);
>>>>>>>
>>>>>>> Should the above example should be the below?
>>>>>>>         int __typetag1 * x __decltag1 __decltag2
>>>>>>>
>>>>>> This example is not related to the one above. It is just meant to
>>>>>> show the behavior of both attributes. My apologies for not making
>>>>>> that clear.
>>>>>
>>>>> Okay, it should be fine if the dwarf debug_info is shown.
>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> Produces the following DWARF information:
>>>>>>>>
>>>>>>>>      <1><1e>: Abbrev Number: 3 (DW_TAG_variable)
>>>>>>>>         <1f>   DW_AT_name        : x
>>>>>>>>         <21>   DW_AT_decl_file   : 1
>>>>>>>>         <22>   DW_AT_decl_line   : 7
>>>>>>>>         <23>   DW_AT_decl_column : 18
>>>>>>>>         <24>   DW_AT_type        : <0x49>
>>>>>>>>         <28>   DW_AT_external    : 1
>>>>>>>>         <28>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
>>>>>>>>         <32>   DW_AT_sibling     : <0x49>
>>>>>>>>      <2><36>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>         <37>   DW_AT_name        : (indirect string, offset: 0xd6): debug_annotate_decl
>>>>>>>>         <3b>   DW_AT_const_value : (indirect string, offset: 0xcd): decltag2
>>>>>>>>      <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>         <40>   DW_AT_name        : (indirect string, offset: 0xd6): debug_annotate_decl
>>>>>>>>         <44>   DW_AT_const_value : (indirect string, offset: 0x0): decltag1
>>>>>>>>      <2><48>: Abbrev Number: 0
>>>>>>>>      <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type)
>>>>>>>>         <4a>   DW_AT_byte_size   : 8
>>>>>>>>         <4b>   DW_AT_type        : <0x5d>
>>>>>>>>         <4f>   DW_AT_sibling     : <0x5d>
>>>>>>>>      <2><53>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>         <54>   DW_AT_name        : (indirect string, offset: 0x9): debug_annotate_type
>>>>>>>>         <58>   DW_AT_const_value : (indirect string, offset: 0x1d): typetag1
>>>>>>>>      <2><5c>: Abbrev Number: 0
>>>>>>>>      <1><5d>: Abbrev Number: 5 (DW_TAG_base_type)
>>>>>>>>         <5e>   DW_AT_byte_size   : 4
>>>>>>>>         <5f>   DW_AT_encoding    : 5	(signed)
>>>>>>>>         <60>   DW_AT_name        : int
>>>>>>>>      <1><64>: Abbrev Number: 0
>>>>>
>>>>> This shows the info in .debug_abbrev. What I mean is to
>>>>> show the related info in .debug_info section which seems more useful to
>>>>> understand the relationships between different tags. Maybe this is due
>>>>> to that I am not fully understanding what <1>/<2> means in <1><49> and
>>>>> <2><53> etc.
>>>> I think that dump actually shows .debug_info, with the abbrevs
>>>> expanded...
>>>> Anyway, it seems to us that the root of this problem is the fact the
>>>> kernel sparse annotations, such as address_space(__user), are:
>>>> 1) To be processed by an external kernel-specific tool (
>>>>      https://sparse.docs.kernel.org/en/latest/annotations.html) and not a
>>>>      C compiler, and therefore,
>>>> 2) Not quite the same than compiler attributes (despite the way they
>>>>      look.)  In particular, they seem to assume an ordering different than
>>>>      of GNU attributes: in some cases given the same written order, they
>>>>      refer to different things!.  Which is quite unfortunate :(
>>>
>>> Yes, currently __user/__kernel macros (implemented with address_space
>>> attribute) are processed by macros.
>>>
>>>> Now, if I understood properly, you plan to change the definition of
>>>> __user and __kernel in the kernel sources in order to generate the tag
>>>> compiler attributes, correct?
>>>
>>> Right. The original __user definition likes:
>>>    # define __user         __attribute__((noderef, address_space(__user)))
>>>
>>> The new attribute looks like
>>>    # define BTF_TYPE_TAG(value) __attribute__((btf_type_tag(#value)))
>>>    #  define __user        BTF_TYPE_TAG(user)
>> Ok I see.  So the kernel will stop using sparse attributes to
>> implement
>> __user and __kernel and start using compiler attributes for tags
>> instead.
>> 
>>>> Is that the reason why LLVM implements what we assume to be the
>>>> sparse
>>>> ordering, and not the correct GNU attributes ordering, for the tag
>>>> attributes?
>>>
>>> Note that __user attributes apply to pointee's and not pointers.
>>> Just like
>>>     const int *p;
>>> the 'const' is not applied to pointer 'p', but the pointee of 'p'.
>>>
>>> What current llvm dwarf generation with
>>>     pointer
>>>       <--- btf_type_tag
>>> is just ONE implementation. As I said earlier, I am okay to
>>> have dwarf implementation like
>>>     p->btf_type_tag->const->int.
>>> If you can propose an implementation like this in dwarf. I can propose
>>> to change implementation in llvm.
>> I think we are miscommunicating.
>> Looks like there is a divergence on what attributes apply to what
>> language entities between the sparse compiler and GCC/LLVM.  How to
>> represent that in DWARF is a different matter.
>> For this example:
>>    int __typetag1 * __typetag2 __typetag3 * g;
>> a) GCC associates __typetag1 with the pointer-to-pointer-to-int.
>> b) LLVM associates __typetag1 to pointer-to-int.
>> Where:
>> a) Is the expected behavior of a compiler attributes, as documented
>> in
>>     the GCC manual.
>> b) Is presumably what the sparse compiler expects, but _not_ the
>>     ordering expected for a compiler GNU attribute.
>> So, if the kernel source __user and __kernel annotations (which
>> currently expand to sparse attributes) follow the sparse ordering, and
>> you want to implement __user and __kernel in terms of compiler
>> attributes instead (the annotation attributes) then you will have to:
>> 1) Fix LLVM to implement the usual ordering for these attributes and
>> 2) fix the kernel sources to use that ordering
>> [Incidentally, the same applies to another "ex-sparse" attribute you
>>   have in the kernel and also implemented in LLVM with a weird ordering:
>>   the address_space attribute.]
>> For 2), it may be possible to write a coccinnelle script to generate
>> the
>> patch...
>
> I don't think (2) (to change kernel source for different attr ordering)
> will work. So the only thing we can do is in compiler/pahole except
> macro replacement in kernel.

I looked at sparse and its parser.  Wanted to be sure the ordering it
uses to interpret sparse annotations (such as address_space, alignment,
etc) is definitely _not_ the same ordering used by __attribute__ in C
compilers.

It is very different indeed and the same can be said about how sparse
interprets other modifiers like `const': in sparse both `int const *foo'
and `int *const foo' parse to a constant pointer to int, for example.

I am not to judge how sparse handles its annotations.  It may be very
well and pertinent for its particular purpose.

But I am not sure if it is reasonable to expect C compilers to implement
certain type __attributes__ to parse differently, just because it
happens these attributes are reused from sparse annotations in a
particular program (in this case the kernel.)  The debug_annotate_decl
and debug_annotate_type attributes are not even intended to be
kernel-specific.

So, if changing the kernel sources is not an option (why btw, other than
being a PITA?) at this point I really don't know what else to suggest :/

Any suggestion from the front-end people?

>> Does this make sense?
>> 
>>>> If that is so, we have quite a problem here: I don't think we can
>>>> change
>>>> the way GCC handles GNU-like attributes just because the kernel sources
>>>> want to hook on these __user/__kernel sparse annotations to generate the
>>>> compiler tags, even if we could mayhaps get GCC to handle
>>>> debug_annotate_type and debug_annotate_decl differently.  Some would say
>>>> doing so would perpetuate the mistake instead of fixing it...
>>>> Is my understanding correct?
>>>
>>> Let us just say that the btf_type_tag attribute applies to pointees.
>>> Does this help?
>>>
>>>>
>>>>>>>
>>>>>>> Maybe you can also show what dwarf debug_info looks like
>>>>>> I am not sure what you mean. This is the .debug_info section as output
>>>>>> by readelf -w. I did trim some information not relevant to the discussion
>>>>>> such as the DW_TAG_compile_unit DIE, for brevity.
>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> In the case of BTF, the annotations are recorded in two type kinds recently
>>>>>>>> added to the BTF specification: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>> The above example declaration prodcues the following BTF information:
>>>>>>>>
>>>>>>>> [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
>>>>>>>> [2] PTR '(anon)' type_id=3
>>>>>>>> [3] TYPE_TAG 'typetag1' type_id=1
>>>>>>>> [4] DECL_TAG 'decltag1' type_id=6 component_idx=-1
>>>>>>>> [5] DECL_TAG 'decltag2' type_id=6 component_idx=-1
>>>>>>>> [6] VAR 'x' type_id=2, linkage=global
>>>>>>>> [7] DATASEC '.bss' size=0 vlen=1
>>>>>>>> 	type_id=6 offset=0 size=8 (VAR 'x')
>>>>>>>>
>>>>>>>>
>>>>>>> [...]
  

On 7/7/22 1:24 PM, Jose E. Marchesi wrote:
> 
> Hi Yonghong.
> 
>> On 6/21/22 9:12 AM, Jose E. Marchesi wrote:
>>>
>>>> On 6/17/22 10:18 AM, Jose E. Marchesi wrote:
>>>>> Hi Yonghong.
>>>>>
>>>>>> On 6/15/22 1:57 PM, David Faust wrote:
>>>>>>>
>>>>>>> On 6/14/22 22:53, Yonghong Song wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 6/7/22 2:43 PM, David Faust wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> This patch series adds support for:
>>>>>>>>>
>>>>>>>>> - Two new C-language-level attributes that allow to associate (to "annotate" or
>>>>>>>>>        to "tag") particular declarations and types with arbitrary strings. As
>>>>>>>>>        explained below, this is intended to be used to, for example, characterize
>>>>>>>>>        certain pointer types.
>>>>>>>>>
>>>>>>>>> - The conveyance of that information in the DWARF output in the form of a new
>>>>>>>>>        DIE: DW_TAG_GNU_annotation.
>>>>>>>>>
>>>>>>>>> - The conveyance of that information in the BTF output in the form of two new
>>>>>>>>>        kinds of BTF objects: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>>
>>>>>>>>> All of these facilities are being added to the eBPF ecosystem, and support for
>>>>>>>>> them exists in some form in LLVM.
>>>>>>>>>
>>>>>>>>> Purpose
>>>>>>>>> =======
>>>>>>>>>
>>>>>>>>> 1)  Addition of C-family language constructs (attributes) to specify free-text
>>>>>>>>>          tags on certain language elements, such as struct fields.
>>>>>>>>>
>>>>>>>>>          The purpose of these annotations is to provide additional information about
>>>>>>>>>          types, variables, and function parameters of interest to the kernel. A
>>>>>>>>>          driving use case is to tag pointer types within the linux kernel and eBPF
>>>>>>>>>          programs with additional semantic information, such as '__user' or '__rcu'.
>>>>>>>>>
>>>>>>>>>          For example, consider the linux kernel function do_execve with the
>>>>>>>>>          following declaration:
>>>>>>>>>
>>>>>>>>>            static int do_execve(struct filename *filename,
>>>>>>>>>               const char __user *const __user *__argv,
>>>>>>>>>               const char __user *const __user *__envp);
>>>>>>>>>
>>>>>>>>>          Here, __user could be defined with these annotations to record semantic
>>>>>>>>>          information about the pointer parameters (e.g., they are user-provided) in
>>>>>>>>>          DWARF and BTF information. Other kernel facilites such as the eBPF verifier
>>>>>>>>>          can read the tags and make use of the information.
>>>>>>>>>
>>>>>>>>> 2)  Conveying the tags in the generated DWARF debug info.
>>>>>>>>>
>>>>>>>>>          The main motivation for emitting the tags in DWARF is that the Linux kernel
>>>>>>>>>          generates its BTF information via pahole, using DWARF as a source:
>>>>>>>>>
>>>>>>>>>              +--------+  BTF                  BTF   +----------+
>>>>>>>>>              | pahole |-------> vmlinux.btf ------->| verifier |
>>>>>>>>>              +--------+                             +----------+
>>>>>>>>>                  ^                                        ^
>>>>>>>>>                  |                                        |
>>>>>>>>>            DWARF |                                    BTF |
>>>>>>>>>                  |                                        |
>>>>>>>>>               vmlinux                              +-------------+
>>>>>>>>>               module1.ko                           | BPF program |
>>>>>>>>>               module2.ko                           +-------------+
>>>>>>>>>                 ...
>>>>>>>>>
>>>>>>>>>          This is because:
>>>>>>>>>
>>>>>>>>>          a)  Unlike GCC, LLVM will only generate BTF for BPF programs.
>>>>>>>>>
>>>>>>>>>          b)  GCC can generate BTF for whatever target with -gbtf, but there is no
>>>>>>>>>              support for linking/deduplicating BTF in the linker.
>>>>>>>>>
>>>>>>>>>          In the scenario above, the verifier needs access to the pointer tags of
>>>>>>>>>          both the kernel types/declarations (conveyed in the DWARF and translated
>>>>>>>>>          to BTF by pahole) and those of the BPF program (available directly in BTF).
>>>>>>>>>
>>>>>>>>>          Another motivation for having the tag information in DWARF, unrelated to
>>>>>>>>>          BPF and BTF, is that the drgn project (another DWARF consumer) also wants
>>>>>>>>>          to benefit from these tags in order to differentiate between different
>>>>>>>>>          kinds of pointers in the kernel.
>>>>>>>>>
>>>>>>>>> 3)  Conveying the tags in the generated BTF debug info.
>>>>>>>>>
>>>>>>>>>          This is easy: the main purpose of having this info in BTF is for the
>>>>>>>>>          compiled eBPF programs. The kernel verifier can then access the tags
>>>>>>>>>          of pointers used by the eBPF programs.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> For more information about these tags and the motivation behind them, please
>>>>>>>>> refer to the following linux kernel discussions:
>>>>>>>>>
>>>>>>>>>        https://lore.kernel.org/bpf/20210914223004.244411-1-yhs@fb.com/
>>>>>>>>>        https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
>>>>>>>>>        https://lore.kernel.org/bpf/20211112012604.1504583-1-yhs@fb.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Implementation Overview
>>>>>>>>> =======================
>>>>>>>>>
>>>>>>>>> To enable these annotations, two new C language attributes are added:
>>>>>>>>> __attribute__((debug_annotate_decl("foo"))) and
>>>>>>>>> __attribute__((debug_annotate_type("bar"))). Both attributes accept a single
>>>>>>>>> arbitrary string constant argument, which will be recorded in the generated
>>>>>>>>> DWARF and/or BTF debug information. They have no effect on code generation.
>>>>>>>>>
>>>>>>>>> Note that we are not using the same attribute names as LLVM (btf_decl_tag and
>>>>>>>>> btf_type_tag, respectively). While these attributes are functionally very
>>>>>>>>> similar, they have grown beyond purely BTF-specific uses, so inclusion of "btf"
>>>>>>>>> in the attribute name seems misleading.
>>>>>>>>>
>>>>>>>>> DWARF support is enabled via a new DW_TAG_GNU_annotation. When generating DWARF,
>>>>>>>>> declarations and types will be checked for the corresponding attributes. If
>>>>>>>>> present, a DW_TAG_GNU_annotation DIE will be created as a child of the DIE for
>>>>>>>>> the annotated type or declaration, one for each tag. These DIEs link the
>>>>>>>>> arbitrary tag value to the item they annotate.
>>>>>>>>>
>>>>>>>>> For example, the following variable declaration:
>>>>>>>>>
>>>>>>>>>        #define __typetag1 __attribute__((debug_annotate_type ("typetag1")))
>>>>>>>>>
>>>>>>>>>        #define __decltag1 __attribute__((debug_annotate_decl ("decltag1")))
>>>>>>>>>        #define __decltag2 __attribute__((debug_annotate_decl ("decltag2")))
>>>>>>>>>
>>>>>>>>>        int * __typetag1 x __decltag1 __decltag2;
>>>>>>>>
>>>>>>>> Based on the above example
>>>>>>>>              static int do_execve(struct filename *filename,
>>>>>>>>                const char __user *const __user *__argv,
>>>>>>>>                const char __user *const __user *__envp);
>>>>>>>>
>>>>>>>> Should the above example should be the below?
>>>>>>>>          int __typetag1 * x __decltag1 __decltag2
>>>>>>>>
>>>>>>> This example is not related to the one above. It is just meant to
>>>>>>> show the behavior of both attributes. My apologies for not making
>>>>>>> that clear.
>>>>>>
>>>>>> Okay, it should be fine if the dwarf debug_info is shown.
>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> Produces the following DWARF information:
>>>>>>>>>
>>>>>>>>>       <1><1e>: Abbrev Number: 3 (DW_TAG_variable)
>>>>>>>>>          <1f>   DW_AT_name        : x
>>>>>>>>>          <21>   DW_AT_decl_file   : 1
>>>>>>>>>          <22>   DW_AT_decl_line   : 7
>>>>>>>>>          <23>   DW_AT_decl_column : 18
>>>>>>>>>          <24>   DW_AT_type        : <0x49>
>>>>>>>>>          <28>   DW_AT_external    : 1
>>>>>>>>>          <28>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
>>>>>>>>>          <32>   DW_AT_sibling     : <0x49>
>>>>>>>>>       <2><36>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>          <37>   DW_AT_name        : (indirect string, offset: 0xd6): debug_annotate_decl
>>>>>>>>>          <3b>   DW_AT_const_value : (indirect string, offset: 0xcd): decltag2
>>>>>>>>>       <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>          <40>   DW_AT_name        : (indirect string, offset: 0xd6): debug_annotate_decl
>>>>>>>>>          <44>   DW_AT_const_value : (indirect string, offset: 0x0): decltag1
>>>>>>>>>       <2><48>: Abbrev Number: 0
>>>>>>>>>       <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type)
>>>>>>>>>          <4a>   DW_AT_byte_size   : 8
>>>>>>>>>          <4b>   DW_AT_type        : <0x5d>
>>>>>>>>>          <4f>   DW_AT_sibling     : <0x5d>
>>>>>>>>>       <2><53>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>          <54>   DW_AT_name        : (indirect string, offset: 0x9): debug_annotate_type
>>>>>>>>>          <58>   DW_AT_const_value : (indirect string, offset: 0x1d): typetag1
>>>>>>>>>       <2><5c>: Abbrev Number: 0
>>>>>>>>>       <1><5d>: Abbrev Number: 5 (DW_TAG_base_type)
>>>>>>>>>          <5e>   DW_AT_byte_size   : 4
>>>>>>>>>          <5f>   DW_AT_encoding    : 5	(signed)
>>>>>>>>>          <60>   DW_AT_name        : int
>>>>>>>>>       <1><64>: Abbrev Number: 0
>>>>>>
>>>>>> This shows the info in .debug_abbrev. What I mean is to
>>>>>> show the related info in .debug_info section which seems more useful to
>>>>>> understand the relationships between different tags. Maybe this is due
>>>>>> to that I am not fully understanding what <1>/<2> means in <1><49> and
>>>>>> <2><53> etc.
>>>>> I think that dump actually shows .debug_info, with the abbrevs
>>>>> expanded...
>>>>> Anyway, it seems to us that the root of this problem is the fact the
>>>>> kernel sparse annotations, such as address_space(__user), are:
>>>>> 1) To be processed by an external kernel-specific tool (
>>>>>       https://sparse.docs.kernel.org/en/latest/annotations.html) and not a
>>>>>       C compiler, and therefore,
>>>>> 2) Not quite the same than compiler attributes (despite the way they
>>>>>       look.)  In particular, they seem to assume an ordering different than
>>>>>       of GNU attributes: in some cases given the same written order, they
>>>>>       refer to different things!.  Which is quite unfortunate :(
>>>>
>>>> Yes, currently __user/__kernel macros (implemented with address_space
>>>> attribute) are processed by macros.
>>>>
>>>>> Now, if I understood properly, you plan to change the definition of
>>>>> __user and __kernel in the kernel sources in order to generate the tag
>>>>> compiler attributes, correct?
>>>>
>>>> Right. The original __user definition likes:
>>>>     # define __user         __attribute__((noderef, address_space(__user)))
>>>>
>>>> The new attribute looks like
>>>>     # define BTF_TYPE_TAG(value) __attribute__((btf_type_tag(#value)))
>>>>     #  define __user        BTF_TYPE_TAG(user)
>>> Ok I see.  So the kernel will stop using sparse attributes to
>>> implement
>>> __user and __kernel and start using compiler attributes for tags
>>> instead.
>>>
>>>>> Is that the reason why LLVM implements what we assume to be the
>>>>> sparse
>>>>> ordering, and not the correct GNU attributes ordering, for the tag
>>>>> attributes?
>>>>
>>>> Note that __user attributes apply to pointee's and not pointers.
>>>> Just like
>>>>      const int *p;
>>>> the 'const' is not applied to pointer 'p', but the pointee of 'p'.
>>>>
>>>> What current llvm dwarf generation with
>>>>      pointer
>>>>        <--- btf_type_tag
>>>> is just ONE implementation. As I said earlier, I am okay to
>>>> have dwarf implementation like
>>>>      p->btf_type_tag->const->int.
>>>> If you can propose an implementation like this in dwarf. I can propose
>>>> to change implementation in llvm.
>>> I think we are miscommunicating.
>>> Looks like there is a divergence on what attributes apply to what
>>> language entities between the sparse compiler and GCC/LLVM.  How to
>>> represent that in DWARF is a different matter.
>>> For this example:
>>>     int __typetag1 * __typetag2 __typetag3 * g;
>>> a) GCC associates __typetag1 with the pointer-to-pointer-to-int.
>>> b) LLVM associates __typetag1 to pointer-to-int.
>>> Where:
>>> a) Is the expected behavior of a compiler attributes, as documented
>>> in
>>>      the GCC manual.
>>> b) Is presumably what the sparse compiler expects, but _not_ the
>>>      ordering expected for a compiler GNU attribute.
>>> So, if the kernel source __user and __kernel annotations (which
>>> currently expand to sparse attributes) follow the sparse ordering, and
>>> you want to implement __user and __kernel in terms of compiler
>>> attributes instead (the annotation attributes) then you will have to:
>>> 1) Fix LLVM to implement the usual ordering for these attributes and
>>> 2) fix the kernel sources to use that ordering
>>> [Incidentally, the same applies to another "ex-sparse" attribute you
>>>    have in the kernel and also implemented in LLVM with a weird ordering:
>>>    the address_space attribute.]
>>> For 2), it may be possible to write a coccinnelle script to generate
>>> the
>>> patch...
>>
>> I don't think (2) (to change kernel source for different attr ordering)
>> will work. So the only thing we can do is in compiler/pahole except
>> macro replacement in kernel.
> 
> I looked at sparse and its parser.  Wanted to be sure the ordering it
> uses to interpret sparse annotations (such as address_space, alignment,
> etc) is definitely _not_ the same ordering used by __attribute__ in C
> compilers.
> 
> It is very different indeed and the same can be said about how sparse
> interprets other modifiers like `const': in sparse both `int const *foo'
> and `int *const foo' parse to a constant pointer to int, for example.
> 
> I am not to judge how sparse handles its annotations.  It may be very
> well and pertinent for its particular purpose.
> 
> But I am not sure if it is reasonable to expect C compilers to implement
> certain type __attributes__ to parse differently, just because it
> happens these attributes are reused from sparse annotations in a
> particular program (in this case the kernel.)  The debug_annotate_decl
> and debug_annotate_type attributes are not even intended to be
> kernel-specific.
> 
> So, if changing the kernel sources is not an option (why btw, other than
> being a PITA?) at this point I really don't know what else to suggest :/
> 
> Any suggestion from the front-end people?

Just want to understand the overall picture. So gcc can still emit
BTF properly with btf_type_tag right? The issue we are talking about
here is about the dwarf, right? If this is the case, we might have
a partial solution here.
   - gcc emits BTF for vmlinux
   - gcc emits dwarf for vmlinux ignoring btf_type_tag
   - in pahole, vmlinux BTF is amended with some additional misc things.
Although there are some use cases to have btf_type_tag in dwarf, but
that can be workarouned with BTF + dwarf both of which are generated
by the compiler. Not elegent, but probably works.

> 
>>> Does this make sense?
>>>
>>>>> If that is so, we have quite a problem here: I don't think we can
>>>>> change
>>>>> the way GCC handles GNU-like attributes just because the kernel sources
>>>>> want to hook on these __user/__kernel sparse annotations to generate the
>>>>> compiler tags, even if we could mayhaps get GCC to handle
>>>>> debug_annotate_type and debug_annotate_decl differently.  Some would say
>>>>> doing so would perpetuate the mistake instead of fixing it...
>>>>> Is my understanding correct?
>>>>
>>>> Let us just say that the btf_type_tag attribute applies to pointees.
>>>> Does this help?
>>>>
>>>>>
>>>>>>>>
>>>>>>>> Maybe you can also show what dwarf debug_info looks like
>>>>>>> I am not sure what you mean. This is the .debug_info section as output
>>>>>>> by readelf -w. I did trim some information not relevant to the discussion
>>>>>>> such as the DW_TAG_compile_unit DIE, for brevity.
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> In the case of BTF, the annotations are recorded in two type kinds recently
>>>>>>>>> added to the BTF specification: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>> The above example declaration prodcues the following BTF information:
>>>>>>>>>
>>>>>>>>> [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
>>>>>>>>> [2] PTR '(anon)' type_id=3
>>>>>>>>> [3] TYPE_TAG 'typetag1' type_id=1
>>>>>>>>> [4] DECL_TAG 'decltag1' type_id=6 component_idx=-1
>>>>>>>>> [5] DECL_TAG 'decltag2' type_id=6 component_idx=-1
>>>>>>>>> [6] VAR 'x' type_id=2, linkage=global
>>>>>>>>> [7] DATASEC '.bss' size=0 vlen=1
>>>>>>>>> 	type_id=6 offset=0 size=8 (VAR 'x')
>>>>>>>>>
>>>>>>>>>
>>>>>>>> [...]
  

Hi Yonghong.

> On 7/7/22 1:24 PM, Jose E. Marchesi wrote:
>> Hi Yonghong.
>> 
>>> On 6/21/22 9:12 AM, Jose E. Marchesi wrote:
>>>>
>>>>> On 6/17/22 10:18 AM, Jose E. Marchesi wrote:
>>>>>> Hi Yonghong.
>>>>>>
>>>>>>> On 6/15/22 1:57 PM, David Faust wrote:
>>>>>>>>
>>>>>>>> On 6/14/22 22:53, Yonghong Song wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 6/7/22 2:43 PM, David Faust wrote:
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> This patch series adds support for:
>>>>>>>>>>
>>>>>>>>>> - Two new C-language-level attributes that allow to associate (to "annotate" or
>>>>>>>>>>        to "tag") particular declarations and types with arbitrary strings. As
>>>>>>>>>>        explained below, this is intended to be used to, for example, characterize
>>>>>>>>>>        certain pointer types.
>>>>>>>>>>
>>>>>>>>>> - The conveyance of that information in the DWARF output in the form of a new
>>>>>>>>>>        DIE: DW_TAG_GNU_annotation.
>>>>>>>>>>
>>>>>>>>>> - The conveyance of that information in the BTF output in the form of two new
>>>>>>>>>>        kinds of BTF objects: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>>>
>>>>>>>>>> All of these facilities are being added to the eBPF ecosystem, and support for
>>>>>>>>>> them exists in some form in LLVM.
>>>>>>>>>>
>>>>>>>>>> Purpose
>>>>>>>>>> =======
>>>>>>>>>>
>>>>>>>>>> 1)  Addition of C-family language constructs (attributes) to specify free-text
>>>>>>>>>>          tags on certain language elements, such as struct fields.
>>>>>>>>>>
>>>>>>>>>>          The purpose of these annotations is to provide additional information about
>>>>>>>>>>          types, variables, and function parameters of interest to the kernel. A
>>>>>>>>>>          driving use case is to tag pointer types within the linux kernel and eBPF
>>>>>>>>>>          programs with additional semantic information, such as '__user' or '__rcu'.
>>>>>>>>>>
>>>>>>>>>>          For example, consider the linux kernel function do_execve with the
>>>>>>>>>>          following declaration:
>>>>>>>>>>
>>>>>>>>>>            static int do_execve(struct filename *filename,
>>>>>>>>>>               const char __user *const __user *__argv,
>>>>>>>>>>               const char __user *const __user *__envp);
>>>>>>>>>>
>>>>>>>>>>          Here, __user could be defined with these annotations to record semantic
>>>>>>>>>>          information about the pointer parameters (e.g., they are user-provided) in
>>>>>>>>>>          DWARF and BTF information. Other kernel facilites such as the eBPF verifier
>>>>>>>>>>          can read the tags and make use of the information.
>>>>>>>>>>
>>>>>>>>>> 2)  Conveying the tags in the generated DWARF debug info.
>>>>>>>>>>
>>>>>>>>>>          The main motivation for emitting the tags in DWARF is that the Linux kernel
>>>>>>>>>>          generates its BTF information via pahole, using DWARF as a source:
>>>>>>>>>>
>>>>>>>>>>              +--------+  BTF                  BTF   +----------+
>>>>>>>>>>              | pahole |-------> vmlinux.btf ------->| verifier |
>>>>>>>>>>              +--------+                             +----------+
>>>>>>>>>>                  ^                                        ^
>>>>>>>>>>                  |                                        |
>>>>>>>>>>            DWARF |                                    BTF |
>>>>>>>>>>                  |                                        |
>>>>>>>>>>               vmlinux                              +-------------+
>>>>>>>>>>               module1.ko                           | BPF program |
>>>>>>>>>>               module2.ko                           +-------------+
>>>>>>>>>>                 ...
>>>>>>>>>>
>>>>>>>>>>          This is because:
>>>>>>>>>>
>>>>>>>>>>          a)  Unlike GCC, LLVM will only generate BTF for BPF programs.
>>>>>>>>>>
>>>>>>>>>>          b)  GCC can generate BTF for whatever target with -gbtf, but there is no
>>>>>>>>>>              support for linking/deduplicating BTF in the linker.
>>>>>>>>>>
>>>>>>>>>>          In the scenario above, the verifier needs access to the pointer tags of
>>>>>>>>>>          both the kernel types/declarations (conveyed in the DWARF and translated
>>>>>>>>>>          to BTF by pahole) and those of the BPF program (available directly in BTF).
>>>>>>>>>>
>>>>>>>>>>          Another motivation for having the tag information in DWARF, unrelated to
>>>>>>>>>>          BPF and BTF, is that the drgn project (another DWARF consumer) also wants
>>>>>>>>>>          to benefit from these tags in order to differentiate between different
>>>>>>>>>>          kinds of pointers in the kernel.
>>>>>>>>>>
>>>>>>>>>> 3)  Conveying the tags in the generated BTF debug info.
>>>>>>>>>>
>>>>>>>>>>          This is easy: the main purpose of having this info in BTF is for the
>>>>>>>>>>          compiled eBPF programs. The kernel verifier can then access the tags
>>>>>>>>>>          of pointers used by the eBPF programs.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> For more information about these tags and the motivation behind them, please
>>>>>>>>>> refer to the following linux kernel discussions:
>>>>>>>>>>
>>>>>>>>>>        https://lore.kernel.org/bpf/20210914223004.244411-1-yhs@fb.com/
>>>>>>>>>>        https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
>>>>>>>>>>        https://lore.kernel.org/bpf/20211112012604.1504583-1-yhs@fb.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Implementation Overview
>>>>>>>>>> =======================
>>>>>>>>>>
>>>>>>>>>> To enable these annotations, two new C language attributes are added:
>>>>>>>>>> __attribute__((debug_annotate_decl("foo"))) and
>>>>>>>>>> __attribute__((debug_annotate_type("bar"))). Both attributes accept a single
>>>>>>>>>> arbitrary string constant argument, which will be recorded in the generated
>>>>>>>>>> DWARF and/or BTF debug information. They have no effect on code generation.
>>>>>>>>>>
>>>>>>>>>> Note that we are not using the same attribute names as LLVM (btf_decl_tag and
>>>>>>>>>> btf_type_tag, respectively). While these attributes are functionally very
>>>>>>>>>> similar, they have grown beyond purely BTF-specific uses, so inclusion of "btf"
>>>>>>>>>> in the attribute name seems misleading.
>>>>>>>>>>
>>>>>>>>>> DWARF support is enabled via a new DW_TAG_GNU_annotation. When generating DWARF,
>>>>>>>>>> declarations and types will be checked for the corresponding attributes. If
>>>>>>>>>> present, a DW_TAG_GNU_annotation DIE will be created as a child of the DIE for
>>>>>>>>>> the annotated type or declaration, one for each tag. These DIEs link the
>>>>>>>>>> arbitrary tag value to the item they annotate.
>>>>>>>>>>
>>>>>>>>>> For example, the following variable declaration:
>>>>>>>>>>
>>>>>>>>>>        #define __typetag1 __attribute__((debug_annotate_type ("typetag1")))
>>>>>>>>>>
>>>>>>>>>>        #define __decltag1 __attribute__((debug_annotate_decl ("decltag1")))
>>>>>>>>>>        #define __decltag2 __attribute__((debug_annotate_decl ("decltag2")))
>>>>>>>>>>
>>>>>>>>>>        int * __typetag1 x __decltag1 __decltag2;
>>>>>>>>>
>>>>>>>>> Based on the above example
>>>>>>>>>              static int do_execve(struct filename *filename,
>>>>>>>>>                const char __user *const __user *__argv,
>>>>>>>>>                const char __user *const __user *__envp);
>>>>>>>>>
>>>>>>>>> Should the above example should be the below?
>>>>>>>>>          int __typetag1 * x __decltag1 __decltag2
>>>>>>>>>
>>>>>>>> This example is not related to the one above. It is just meant to
>>>>>>>> show the behavior of both attributes. My apologies for not making
>>>>>>>> that clear.
>>>>>>>
>>>>>>> Okay, it should be fine if the dwarf debug_info is shown.
>>>>>>>
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Produces the following DWARF information:
>>>>>>>>>>
>>>>>>>>>>       <1><1e>: Abbrev Number: 3 (DW_TAG_variable)
>>>>>>>>>>          <1f>   DW_AT_name        : x
>>>>>>>>>>          <21>   DW_AT_decl_file   : 1
>>>>>>>>>>          <22>   DW_AT_decl_line   : 7
>>>>>>>>>>          <23>   DW_AT_decl_column : 18
>>>>>>>>>>          <24>   DW_AT_type        : <0x49>
>>>>>>>>>>          <28>   DW_AT_external    : 1
>>>>>>>>>>          <28>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
>>>>>>>>>>          <32>   DW_AT_sibling     : <0x49>
>>>>>>>>>>       <2><36>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>> <37> DW_AT_name : (indirect string, offset: 0xd6):
> debug_annotate_decl
>>>>>>>>>>          <3b>   DW_AT_const_value : (indirect string, offset: 0xcd): decltag2
>>>>>>>>>>       <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>> <40> DW_AT_name : (indirect string, offset: 0xd6):
> debug_annotate_decl
>>>>>>>>>>          <44>   DW_AT_const_value : (indirect string, offset: 0x0): decltag1
>>>>>>>>>>       <2><48>: Abbrev Number: 0
>>>>>>>>>>       <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type)
>>>>>>>>>>          <4a>   DW_AT_byte_size   : 8
>>>>>>>>>>          <4b>   DW_AT_type        : <0x5d>
>>>>>>>>>>          <4f>   DW_AT_sibling     : <0x5d>
>>>>>>>>>>       <2><53>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>> <54> DW_AT_name : (indirect string, offset: 0x9):
> debug_annotate_type
>>>>>>>>>>          <58>   DW_AT_const_value : (indirect string, offset: 0x1d): typetag1
>>>>>>>>>>       <2><5c>: Abbrev Number: 0
>>>>>>>>>>       <1><5d>: Abbrev Number: 5 (DW_TAG_base_type)
>>>>>>>>>>          <5e>   DW_AT_byte_size   : 4
>>>>>>>>>>          <5f>   DW_AT_encoding    : 5	(signed)
>>>>>>>>>>          <60>   DW_AT_name        : int
>>>>>>>>>>       <1><64>: Abbrev Number: 0
>>>>>>>
>>>>>>> This shows the info in .debug_abbrev. What I mean is to
>>>>>>> show the related info in .debug_info section which seems more useful to
>>>>>>> understand the relationships between different tags. Maybe this is due
>>>>>>> to that I am not fully understanding what <1>/<2> means in <1><49> and
>>>>>>> <2><53> etc.
>>>>>> I think that dump actually shows .debug_info, with the abbrevs
>>>>>> expanded...
>>>>>> Anyway, it seems to us that the root of this problem is the fact the
>>>>>> kernel sparse annotations, such as address_space(__user), are:
>>>>>> 1) To be processed by an external kernel-specific tool (
>>>>>>       https://sparse.docs.kernel.org/en/latest/annotations.html) and not a
>>>>>>       C compiler, and therefore,
>>>>>> 2) Not quite the same than compiler attributes (despite the way they
>>>>>>       look.)  In particular, they seem to assume an ordering different than
>>>>>>       of GNU attributes: in some cases given the same written order, they
>>>>>>       refer to different things!.  Which is quite unfortunate :(
>>>>>
>>>>> Yes, currently __user/__kernel macros (implemented with address_space
>>>>> attribute) are processed by macros.
>>>>>
>>>>>> Now, if I understood properly, you plan to change the definition of
>>>>>> __user and __kernel in the kernel sources in order to generate the tag
>>>>>> compiler attributes, correct?
>>>>>
>>>>> Right. The original __user definition likes:
>>>>>     # define __user         __attribute__((noderef, address_space(__user)))
>>>>>
>>>>> The new attribute looks like
>>>>>     # define BTF_TYPE_TAG(value) __attribute__((btf_type_tag(#value)))
>>>>>     #  define __user        BTF_TYPE_TAG(user)
>>>> Ok I see.  So the kernel will stop using sparse attributes to
>>>> implement
>>>> __user and __kernel and start using compiler attributes for tags
>>>> instead.
>>>>
>>>>>> Is that the reason why LLVM implements what we assume to be the
>>>>>> sparse
>>>>>> ordering, and not the correct GNU attributes ordering, for the tag
>>>>>> attributes?
>>>>>
>>>>> Note that __user attributes apply to pointee's and not pointers.
>>>>> Just like
>>>>>      const int *p;
>>>>> the 'const' is not applied to pointer 'p', but the pointee of 'p'.
>>>>>
>>>>> What current llvm dwarf generation with
>>>>>      pointer
>>>>>        <--- btf_type_tag
>>>>> is just ONE implementation. As I said earlier, I am okay to
>>>>> have dwarf implementation like
>>>>>      p->btf_type_tag->const->int.
>>>>> If you can propose an implementation like this in dwarf. I can propose
>>>>> to change implementation in llvm.
>>>> I think we are miscommunicating.
>>>> Looks like there is a divergence on what attributes apply to what
>>>> language entities between the sparse compiler and GCC/LLVM.  How to
>>>> represent that in DWARF is a different matter.
>>>> For this example:
>>>>     int __typetag1 * __typetag2 __typetag3 * g;
>>>> a) GCC associates __typetag1 with the pointer-to-pointer-to-int.
>>>> b) LLVM associates __typetag1 to pointer-to-int.
>>>> Where:
>>>> a) Is the expected behavior of a compiler attributes, as documented
>>>> in
>>>>      the GCC manual.
>>>> b) Is presumably what the sparse compiler expects, but _not_ the
>>>>      ordering expected for a compiler GNU attribute.
>>>> So, if the kernel source __user and __kernel annotations (which
>>>> currently expand to sparse attributes) follow the sparse ordering, and
>>>> you want to implement __user and __kernel in terms of compiler
>>>> attributes instead (the annotation attributes) then you will have to:
>>>> 1) Fix LLVM to implement the usual ordering for these attributes and
>>>> 2) fix the kernel sources to use that ordering
>>>> [Incidentally, the same applies to another "ex-sparse" attribute you
>>>>    have in the kernel and also implemented in LLVM with a weird ordering:
>>>>    the address_space attribute.]
>>>> For 2), it may be possible to write a coccinnelle script to generate
>>>> the
>>>> patch...
>>>
>>> I don't think (2) (to change kernel source for different attr ordering)
>>> will work. So the only thing we can do is in compiler/pahole except
>>> macro replacement in kernel.
>> I looked at sparse and its parser.  Wanted to be sure the ordering
>> it
>> uses to interpret sparse annotations (such as address_space, alignment,
>> etc) is definitely _not_ the same ordering used by __attribute__ in C
>> compilers.
>> It is very different indeed and the same can be said about how
>> sparse
>> interprets other modifiers like `const': in sparse both `int const *foo'
>> and `int *const foo' parse to a constant pointer to int, for example.
>> I am not to judge how sparse handles its annotations.  It may be
>> very
>> well and pertinent for its particular purpose.
>> But I am not sure if it is reasonable to expect C compilers to
>> implement
>> certain type __attributes__ to parse differently, just because it
>> happens these attributes are reused from sparse annotations in a
>> particular program (in this case the kernel.)  The debug_annotate_decl
>> and debug_annotate_type attributes are not even intended to be
>> kernel-specific.
>> So, if changing the kernel sources is not an option (why btw, other
>> than
>> being a PITA?) at this point I really don't know what else to suggest :/
>> Any suggestion from the front-end people?
>
> Just want to understand the overall picture. So gcc can still emit
> BTF properly with btf_type_tag right? The issue we are talking about
> here is about the dwarf, right?

If by "properly" you mean how sparse handles its annotations, then not
really.

The issue we are talking about is rather a language-level one: to what
entity/type the compiler attribute applies.

So, for:

  int __attribute__((debug_annotate_decl("user"))) *foo;

GCC will apply the attribute to the int type, following the rules for
type attributes (sparse would apply the annotation to the *int type
instead).  The emitted debug info (be it DWARF or BTF) will reflect
that, no more no less :/

> If this is the case, we might have
> a partial solution here.
>   - gcc emits BTF for vmlinux

Note that for emitting BTF for vmlinux we would need support in the
linker to merge and deduplicate BTF, which at the moment we don't have.

>   - gcc emits dwarf for vmlinux ignoring btf_type_tag
>   - in pahole, vmlinux BTF is amended with some additional misc things.
> Although there are some use cases to have btf_type_tag in dwarf, but
> that can be workarouned with BTF + dwarf both of which are generated
> by the compiler. Not elegent, but probably works.
>> 
>>>> Does this make sense?
>>>>
>>>>>> If that is so, we have quite a problem here: I don't think we can
>>>>>> change
>>>>>> the way GCC handles GNU-like attributes just because the kernel sources
>>>>>> want to hook on these __user/__kernel sparse annotations to generate the
>>>>>> compiler tags, even if we could mayhaps get GCC to handle
>>>>>> debug_annotate_type and debug_annotate_decl differently.  Some would say
>>>>>> doing so would perpetuate the mistake instead of fixing it...
>>>>>> Is my understanding correct?
>>>>>
>>>>> Let us just say that the btf_type_tag attribute applies to pointees.
>>>>> Does this help?
>>>>>
>>>>>>
>>>>>>>>>
>>>>>>>>> Maybe you can also show what dwarf debug_info looks like
>>>>>>>> I am not sure what you mean. This is the .debug_info section as output
>>>>>>>> by readelf -w. I did trim some information not relevant to the discussion
>>>>>>>> such as the DW_TAG_compile_unit DIE, for brevity.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> In the case of BTF, the annotations are recorded in two type kinds recently
>>>>>>>>>> added to the BTF specification: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>>> The above example declaration prodcues the following BTF information:
>>>>>>>>>>
>>>>>>>>>> [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
>>>>>>>>>> [2] PTR '(anon)' type_id=3
>>>>>>>>>> [3] TYPE_TAG 'typetag1' type_id=1
>>>>>>>>>> [4] DECL_TAG 'decltag1' type_id=6 component_idx=-1
>>>>>>>>>> [5] DECL_TAG 'decltag2' type_id=6 component_idx=-1
>>>>>>>>>> [6] VAR 'x' type_id=2, linkage=global
>>>>>>>>>> [7] DATASEC '.bss' size=0 vlen=1
>>>>>>>>>> 	type_id=6 offset=0 size=8 (VAR 'x')
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> [...]
  

On 7/14/22 8:09 AM, Jose E. Marchesi wrote:
> 
> Hi Yonghong.
> 
>> On 7/7/22 1:24 PM, Jose E. Marchesi wrote:
>>> Hi Yonghong.
>>>
>>>> On 6/21/22 9:12 AM, Jose E. Marchesi wrote:
>>>>>
>>>>>> On 6/17/22 10:18 AM, Jose E. Marchesi wrote:
>>>>>>> Hi Yonghong.
>>>>>>>
>>>>>>>> On 6/15/22 1:57 PM, David Faust wrote:
>>>>>>>>>
>>>>>>>>> On 6/14/22 22:53, Yonghong Song wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 6/7/22 2:43 PM, David Faust wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> This patch series adds support for:
>>>>>>>>>>>
>>>>>>>>>>> - Two new C-language-level attributes that allow to associate (to "annotate" or
>>>>>>>>>>>         to "tag") particular declarations and types with arbitrary strings. As
>>>>>>>>>>>         explained below, this is intended to be used to, for example, characterize
>>>>>>>>>>>         certain pointer types.
>>>>>>>>>>>
>>>>>>>>>>> - The conveyance of that information in the DWARF output in the form of a new
>>>>>>>>>>>         DIE: DW_TAG_GNU_annotation.
>>>>>>>>>>>
>>>>>>>>>>> - The conveyance of that information in the BTF output in the form of two new
>>>>>>>>>>>         kinds of BTF objects: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>>>>
>>>>>>>>>>> All of these facilities are being added to the eBPF ecosystem, and support for
>>>>>>>>>>> them exists in some form in LLVM.
>>>>>>>>>>>
>>>>>>>>>>> Purpose
>>>>>>>>>>> =======
>>>>>>>>>>>
>>>>>>>>>>> 1)  Addition of C-family language constructs (attributes) to specify free-text
>>>>>>>>>>>           tags on certain language elements, such as struct fields.
>>>>>>>>>>>
>>>>>>>>>>>           The purpose of these annotations is to provide additional information about
>>>>>>>>>>>           types, variables, and function parameters of interest to the kernel. A
>>>>>>>>>>>           driving use case is to tag pointer types within the linux kernel and eBPF
>>>>>>>>>>>           programs with additional semantic information, such as '__user' or '__rcu'.
>>>>>>>>>>>
>>>>>>>>>>>           For example, consider the linux kernel function do_execve with the
>>>>>>>>>>>           following declaration:
>>>>>>>>>>>
>>>>>>>>>>>             static int do_execve(struct filename *filename,
>>>>>>>>>>>                const char __user *const __user *__argv,
>>>>>>>>>>>                const char __user *const __user *__envp);
>>>>>>>>>>>
>>>>>>>>>>>           Here, __user could be defined with these annotations to record semantic
>>>>>>>>>>>           information about the pointer parameters (e.g., they are user-provided) in
>>>>>>>>>>>           DWARF and BTF information. Other kernel facilites such as the eBPF verifier
>>>>>>>>>>>           can read the tags and make use of the information.
>>>>>>>>>>>
>>>>>>>>>>> 2)  Conveying the tags in the generated DWARF debug info.
>>>>>>>>>>>
>>>>>>>>>>>           The main motivation for emitting the tags in DWARF is that the Linux kernel
>>>>>>>>>>>           generates its BTF information via pahole, using DWARF as a source:
>>>>>>>>>>>
>>>>>>>>>>>               +--------+  BTF                  BTF   +----------+
>>>>>>>>>>>               | pahole |-------> vmlinux.btf ------->| verifier |
>>>>>>>>>>>               +--------+                             +----------+
>>>>>>>>>>>                   ^                                        ^
>>>>>>>>>>>                   |                                        |
>>>>>>>>>>>             DWARF |                                    BTF |
>>>>>>>>>>>                   |                                        |
>>>>>>>>>>>                vmlinux                              +-------------+
>>>>>>>>>>>                module1.ko                           | BPF program |
>>>>>>>>>>>                module2.ko                           +-------------+
>>>>>>>>>>>                  ...
>>>>>>>>>>>
>>>>>>>>>>>           This is because:
>>>>>>>>>>>
>>>>>>>>>>>           a)  Unlike GCC, LLVM will only generate BTF for BPF programs.
>>>>>>>>>>>
>>>>>>>>>>>           b)  GCC can generate BTF for whatever target with -gbtf, but there is no
>>>>>>>>>>>               support for linking/deduplicating BTF in the linker.
>>>>>>>>>>>
>>>>>>>>>>>           In the scenario above, the verifier needs access to the pointer tags of
>>>>>>>>>>>           both the kernel types/declarations (conveyed in the DWARF and translated
>>>>>>>>>>>           to BTF by pahole) and those of the BPF program (available directly in BTF).
>>>>>>>>>>>
>>>>>>>>>>>           Another motivation for having the tag information in DWARF, unrelated to
>>>>>>>>>>>           BPF and BTF, is that the drgn project (another DWARF consumer) also wants
>>>>>>>>>>>           to benefit from these tags in order to differentiate between different
>>>>>>>>>>>           kinds of pointers in the kernel.
>>>>>>>>>>>
>>>>>>>>>>> 3)  Conveying the tags in the generated BTF debug info.
>>>>>>>>>>>
>>>>>>>>>>>           This is easy: the main purpose of having this info in BTF is for the
>>>>>>>>>>>           compiled eBPF programs. The kernel verifier can then access the tags
>>>>>>>>>>>           of pointers used by the eBPF programs.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> For more information about these tags and the motivation behind them, please
>>>>>>>>>>> refer to the following linux kernel discussions:
>>>>>>>>>>>
>>>>>>>>>>>         https://lore.kernel.org/bpf/20210914223004.244411-1-yhs@fb.com/
>>>>>>>>>>>         https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
>>>>>>>>>>>         https://lore.kernel.org/bpf/20211112012604.1504583-1-yhs@fb.com/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Implementation Overview
>>>>>>>>>>> =======================
>>>>>>>>>>>
>>>>>>>>>>> To enable these annotations, two new C language attributes are added:
>>>>>>>>>>> __attribute__((debug_annotate_decl("foo"))) and
>>>>>>>>>>> __attribute__((debug_annotate_type("bar"))). Both attributes accept a single
>>>>>>>>>>> arbitrary string constant argument, which will be recorded in the generated
>>>>>>>>>>> DWARF and/or BTF debug information. They have no effect on code generation.
>>>>>>>>>>>
>>>>>>>>>>> Note that we are not using the same attribute names as LLVM (btf_decl_tag and
>>>>>>>>>>> btf_type_tag, respectively). While these attributes are functionally very
>>>>>>>>>>> similar, they have grown beyond purely BTF-specific uses, so inclusion of "btf"
>>>>>>>>>>> in the attribute name seems misleading.
>>>>>>>>>>>
>>>>>>>>>>> DWARF support is enabled via a new DW_TAG_GNU_annotation. When generating DWARF,
>>>>>>>>>>> declarations and types will be checked for the corresponding attributes. If
>>>>>>>>>>> present, a DW_TAG_GNU_annotation DIE will be created as a child of the DIE for
>>>>>>>>>>> the annotated type or declaration, one for each tag. These DIEs link the
>>>>>>>>>>> arbitrary tag value to the item they annotate.
>>>>>>>>>>>
>>>>>>>>>>> For example, the following variable declaration:
>>>>>>>>>>>
>>>>>>>>>>>         #define __typetag1 __attribute__((debug_annotate_type ("typetag1")))
>>>>>>>>>>>
>>>>>>>>>>>         #define __decltag1 __attribute__((debug_annotate_decl ("decltag1")))
>>>>>>>>>>>         #define __decltag2 __attribute__((debug_annotate_decl ("decltag2")))
>>>>>>>>>>>
>>>>>>>>>>>         int * __typetag1 x __decltag1 __decltag2;
>>>>>>>>>>
>>>>>>>>>> Based on the above example
>>>>>>>>>>               static int do_execve(struct filename *filename,
>>>>>>>>>>                 const char __user *const __user *__argv,
>>>>>>>>>>                 const char __user *const __user *__envp);
>>>>>>>>>>
>>>>>>>>>> Should the above example should be the below?
>>>>>>>>>>           int __typetag1 * x __decltag1 __decltag2
>>>>>>>>>>
>>>>>>>>> This example is not related to the one above. It is just meant to
>>>>>>>>> show the behavior of both attributes. My apologies for not making
>>>>>>>>> that clear.
>>>>>>>>
>>>>>>>> Okay, it should be fine if the dwarf debug_info is shown.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Produces the following DWARF information:
>>>>>>>>>>>
>>>>>>>>>>>        <1><1e>: Abbrev Number: 3 (DW_TAG_variable)
>>>>>>>>>>>           <1f>   DW_AT_name        : x
>>>>>>>>>>>           <21>   DW_AT_decl_file   : 1
>>>>>>>>>>>           <22>   DW_AT_decl_line   : 7
>>>>>>>>>>>           <23>   DW_AT_decl_column : 18
>>>>>>>>>>>           <24>   DW_AT_type        : <0x49>
>>>>>>>>>>>           <28>   DW_AT_external    : 1
>>>>>>>>>>>           <28>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
>>>>>>>>>>>           <32>   DW_AT_sibling     : <0x49>
>>>>>>>>>>>        <2><36>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>>> <37> DW_AT_name : (indirect string, offset: 0xd6):
>> debug_annotate_decl
>>>>>>>>>>>           <3b>   DW_AT_const_value : (indirect string, offset: 0xcd): decltag2
>>>>>>>>>>>        <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>>> <40> DW_AT_name : (indirect string, offset: 0xd6):
>> debug_annotate_decl
>>>>>>>>>>>           <44>   DW_AT_const_value : (indirect string, offset: 0x0): decltag1
>>>>>>>>>>>        <2><48>: Abbrev Number: 0
>>>>>>>>>>>        <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type)
>>>>>>>>>>>           <4a>   DW_AT_byte_size   : 8
>>>>>>>>>>>           <4b>   DW_AT_type        : <0x5d>
>>>>>>>>>>>           <4f>   DW_AT_sibling     : <0x5d>
>>>>>>>>>>>        <2><53>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>>> <54> DW_AT_name : (indirect string, offset: 0x9):
>> debug_annotate_type
>>>>>>>>>>>           <58>   DW_AT_const_value : (indirect string, offset: 0x1d): typetag1
>>>>>>>>>>>        <2><5c>: Abbrev Number: 0
>>>>>>>>>>>        <1><5d>: Abbrev Number: 5 (DW_TAG_base_type)
>>>>>>>>>>>           <5e>   DW_AT_byte_size   : 4
>>>>>>>>>>>           <5f>   DW_AT_encoding    : 5	(signed)
>>>>>>>>>>>           <60>   DW_AT_name        : int
>>>>>>>>>>>        <1><64>: Abbrev Number: 0
>>>>>>>>
>>>>>>>> This shows the info in .debug_abbrev. What I mean is to
>>>>>>>> show the related info in .debug_info section which seems more useful to
>>>>>>>> understand the relationships between different tags. Maybe this is due
>>>>>>>> to that I am not fully understanding what <1>/<2> means in <1><49> and
>>>>>>>> <2><53> etc.
>>>>>>> I think that dump actually shows .debug_info, with the abbrevs
>>>>>>> expanded...
>>>>>>> Anyway, it seems to us that the root of this problem is the fact the
>>>>>>> kernel sparse annotations, such as address_space(__user), are:
>>>>>>> 1) To be processed by an external kernel-specific tool (
>>>>>>>        https://sparse.docs.kernel.org/en/latest/annotations.html) and not a
>>>>>>>        C compiler, and therefore,
>>>>>>> 2) Not quite the same than compiler attributes (despite the way they
>>>>>>>        look.)  In particular, they seem to assume an ordering different than
>>>>>>>        of GNU attributes: in some cases given the same written order, they
>>>>>>>        refer to different things!.  Which is quite unfortunate :(
>>>>>>
>>>>>> Yes, currently __user/__kernel macros (implemented with address_space
>>>>>> attribute) are processed by macros.
>>>>>>
>>>>>>> Now, if I understood properly, you plan to change the definition of
>>>>>>> __user and __kernel in the kernel sources in order to generate the tag
>>>>>>> compiler attributes, correct?
>>>>>>
>>>>>> Right. The original __user definition likes:
>>>>>>      # define __user         __attribute__((noderef, address_space(__user)))
>>>>>>
>>>>>> The new attribute looks like
>>>>>>      # define BTF_TYPE_TAG(value) __attribute__((btf_type_tag(#value)))
>>>>>>      #  define __user        BTF_TYPE_TAG(user)
>>>>> Ok I see.  So the kernel will stop using sparse attributes to
>>>>> implement
>>>>> __user and __kernel and start using compiler attributes for tags
>>>>> instead.
>>>>>
>>>>>>> Is that the reason why LLVM implements what we assume to be the
>>>>>>> sparse
>>>>>>> ordering, and not the correct GNU attributes ordering, for the tag
>>>>>>> attributes?
>>>>>>
>>>>>> Note that __user attributes apply to pointee's and not pointers.
>>>>>> Just like
>>>>>>       const int *p;
>>>>>> the 'const' is not applied to pointer 'p', but the pointee of 'p'.
>>>>>>
>>>>>> What current llvm dwarf generation with
>>>>>>       pointer
>>>>>>         <--- btf_type_tag
>>>>>> is just ONE implementation. As I said earlier, I am okay to
>>>>>> have dwarf implementation like
>>>>>>       p->btf_type_tag->const->int.
>>>>>> If you can propose an implementation like this in dwarf. I can propose
>>>>>> to change implementation in llvm.
>>>>> I think we are miscommunicating.
>>>>> Looks like there is a divergence on what attributes apply to what
>>>>> language entities between the sparse compiler and GCC/LLVM.  How to
>>>>> represent that in DWARF is a different matter.
>>>>> For this example:
>>>>>      int __typetag1 * __typetag2 __typetag3 * g;
>>>>> a) GCC associates __typetag1 with the pointer-to-pointer-to-int.
>>>>> b) LLVM associates __typetag1 to pointer-to-int.
>>>>> Where:
>>>>> a) Is the expected behavior of a compiler attributes, as documented
>>>>> in
>>>>>       the GCC manual.
>>>>> b) Is presumably what the sparse compiler expects, but _not_ the
>>>>>       ordering expected for a compiler GNU attribute.
>>>>> So, if the kernel source __user and __kernel annotations (which
>>>>> currently expand to sparse attributes) follow the sparse ordering, and
>>>>> you want to implement __user and __kernel in terms of compiler
>>>>> attributes instead (the annotation attributes) then you will have to:
>>>>> 1) Fix LLVM to implement the usual ordering for these attributes and
>>>>> 2) fix the kernel sources to use that ordering
>>>>> [Incidentally, the same applies to another "ex-sparse" attribute you
>>>>>     have in the kernel and also implemented in LLVM with a weird ordering:
>>>>>     the address_space attribute.]
>>>>> For 2), it may be possible to write a coccinnelle script to generate
>>>>> the
>>>>> patch...
>>>>
>>>> I don't think (2) (to change kernel source for different attr ordering)
>>>> will work. So the only thing we can do is in compiler/pahole except
>>>> macro replacement in kernel.
>>> I looked at sparse and its parser.  Wanted to be sure the ordering
>>> it
>>> uses to interpret sparse annotations (such as address_space, alignment,
>>> etc) is definitely _not_ the same ordering used by __attribute__ in C
>>> compilers.
>>> It is very different indeed and the same can be said about how
>>> sparse
>>> interprets other modifiers like `const': in sparse both `int const *foo'
>>> and `int *const foo' parse to a constant pointer to int, for example.
>>> I am not to judge how sparse handles its annotations.  It may be
>>> very
>>> well and pertinent for its particular purpose.
>>> But I am not sure if it is reasonable to expect C compilers to
>>> implement
>>> certain type __attributes__ to parse differently, just because it
>>> happens these attributes are reused from sparse annotations in a
>>> particular program (in this case the kernel.)  The debug_annotate_decl
>>> and debug_annotate_type attributes are not even intended to be
>>> kernel-specific.
>>> So, if changing the kernel sources is not an option (why btw, other
>>> than
>>> being a PITA?) at this point I really don't know what else to suggest :/
>>> Any suggestion from the front-end people?
>>
>> Just want to understand the overall picture. So gcc can still emit
>> BTF properly with btf_type_tag right? The issue we are talking about
>> here is about the dwarf, right?
> 
> If by "properly" you mean how sparse handles its annotations, then not
> really.
> 
> The issue we are talking about is rather a language-level one: to what
> entity/type the compiler attribute applies.
> 
> So, for:
> 
>    int __attribute__((debug_annotate_decl("user"))) *foo;
> 
> GCC will apply the attribute to the int type, following the rules for
> type attributes (sparse would apply the annotation to the *int type
> instead).  The emitted debug info (be it DWARF or BTF) will reflect
> that, no more no less :/

I don't know what does this 'apply the attribute to the int' mean.
In current clang implementation it means the following dwarf chains
from right to left
   variable 'foo'
     type: ptr
       base type: attr_type: attr
                     underlying type: int

So the type chain is foo -> ptr -> attr -> int

> 
>> If this is the case, we might have
>> a partial solution here.
>>    - gcc emits BTF for vmlinux
> 
> Note that for emitting BTF for vmlinux we would need support in the
> linker to merge and deduplicate BTF, which at the moment we don't have.

This should be okay. pahole will merge and deduplicate btf. In pahole 
'-j' mode, each thread will convert each .o file dwarf to btf, and
then pahole will merge and deduplicate btf.

> 
>>    - gcc emits dwarf for vmlinux ignoring btf_type_tag
>>    - in pahole, vmlinux BTF is amended with some additional misc things.
>> Although there are some use cases to have btf_type_tag in dwarf, but
>> that can be workarouned with BTF + dwarf both of which are generated
>> by the compiler. Not elegent, but probably works.
>>>
>>>>> Does this make sense?
>>>>>
>>>>>>> If that is so, we have quite a problem here: I don't think we can
>>>>>>> change
>>>>>>> the way GCC handles GNU-like attributes just because the kernel sources
>>>>>>> want to hook on these __user/__kernel sparse annotations to generate the
>>>>>>> compiler tags, even if we could mayhaps get GCC to handle
>>>>>>> debug_annotate_type and debug_annotate_decl differently.  Some would say
>>>>>>> doing so would perpetuate the mistake instead of fixing it...
>>>>>>> Is my understanding correct?
>>>>>>
>>>>>> Let us just say that the btf_type_tag attribute applies to pointees.
>>>>>> Does this help?
>>>>>>
>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Maybe you can also show what dwarf debug_info looks like
>>>>>>>>> I am not sure what you mean. This is the .debug_info section as output
>>>>>>>>> by readelf -w. I did trim some information not relevant to the discussion
>>>>>>>>> such as the DW_TAG_compile_unit DIE, for brevity.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In the case of BTF, the annotations are recorded in two type kinds recently
>>>>>>>>>>> added to the BTF specification: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>>>> The above example declaration prodcues the following BTF information:
>>>>>>>>>>>
>>>>>>>>>>> [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
>>>>>>>>>>> [2] PTR '(anon)' type_id=3
>>>>>>>>>>> [3] TYPE_TAG 'typetag1' type_id=1
>>>>>>>>>>> [4] DECL_TAG 'decltag1' type_id=6 component_idx=-1
>>>>>>>>>>> [5] DECL_TAG 'decltag2' type_id=6 component_idx=-1
>>>>>>>>>>> [6] VAR 'x' type_id=2, linkage=global
>>>>>>>>>>> [7] DATASEC '.bss' size=0 vlen=1
>>>>>>>>>>> 	type_id=6 offset=0 size=8 (VAR 'x')
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> [...]
  

> On 7/14/22 8:09 AM, Jose E. Marchesi wrote:
>> Hi Yonghong.
>> 
>>> On 7/7/22 1:24 PM, Jose E. Marchesi wrote:
>>>> Hi Yonghong.
>>>>
>>>>> On 6/21/22 9:12 AM, Jose E. Marchesi wrote:
>>>>>>
>>>>>>> On 6/17/22 10:18 AM, Jose E. Marchesi wrote:
>>>>>>>> Hi Yonghong.
>>>>>>>>
>>>>>>>>> On 6/15/22 1:57 PM, David Faust wrote:
>>>>>>>>>>
>>>>>>>>>> On 6/14/22 22:53, Yonghong Song wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 6/7/22 2:43 PM, David Faust wrote:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> This patch series adds support for:
>>>>>>>>>>>>
>>>>>>>>>>>> - Two new C-language-level attributes that allow to associate (to "annotate" or
>>>>>>>>>>>>         to "tag") particular declarations and types with arbitrary strings. As
>>>>>>>>>>>>         explained below, this is intended to be used to, for example, characterize
>>>>>>>>>>>>         certain pointer types.
>>>>>>>>>>>>
>>>>>>>>>>>> - The conveyance of that information in the DWARF output in the form of a new
>>>>>>>>>>>>         DIE: DW_TAG_GNU_annotation.
>>>>>>>>>>>>
>>>>>>>>>>>> - The conveyance of that information in the BTF output in the form of two new
>>>>>>>>>>>>         kinds of BTF objects: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>>>>>
>>>>>>>>>>>> All of these facilities are being added to the eBPF ecosystem, and support for
>>>>>>>>>>>> them exists in some form in LLVM.
>>>>>>>>>>>>
>>>>>>>>>>>> Purpose
>>>>>>>>>>>> =======
>>>>>>>>>>>>
>>>>>>>>>>>> 1)  Addition of C-family language constructs (attributes) to specify free-text
>>>>>>>>>>>>           tags on certain language elements, such as struct fields.
>>>>>>>>>>>>
>>>>>>>>>>>>           The purpose of these annotations is to provide additional information about
>>>>>>>>>>>>           types, variables, and function parameters of interest to the kernel. A
>>>>>>>>>>>>           driving use case is to tag pointer types within the linux kernel and eBPF
>>>>>>>>>>>>           programs with additional semantic information, such as '__user' or '__rcu'.
>>>>>>>>>>>>
>>>>>>>>>>>>           For example, consider the linux kernel function do_execve with the
>>>>>>>>>>>>           following declaration:
>>>>>>>>>>>>
>>>>>>>>>>>>             static int do_execve(struct filename *filename,
>>>>>>>>>>>>                const char __user *const __user *__argv,
>>>>>>>>>>>>                const char __user *const __user *__envp);
>>>>>>>>>>>>
>>>>>>>>>>>>           Here, __user could be defined with these annotations to record semantic
>>>>>>>>>>>>           information about the pointer parameters (e.g., they are user-provided) in
>>>>>>>>>>>>           DWARF and BTF information. Other kernel facilites such as the eBPF verifier
>>>>>>>>>>>>           can read the tags and make use of the information.
>>>>>>>>>>>>
>>>>>>>>>>>> 2)  Conveying the tags in the generated DWARF debug info.
>>>>>>>>>>>>
>>>>>>>>>>>>           The main motivation for emitting the tags in DWARF is that the Linux kernel
>>>>>>>>>>>>           generates its BTF information via pahole, using DWARF as a source:
>>>>>>>>>>>>
>>>>>>>>>>>>               +--------+  BTF                  BTF   +----------+
>>>>>>>>>>>>               | pahole |-------> vmlinux.btf ------->| verifier |
>>>>>>>>>>>>               +--------+                             +----------+
>>>>>>>>>>>>                   ^                                        ^
>>>>>>>>>>>>                   |                                        |
>>>>>>>>>>>>             DWARF |                                    BTF |
>>>>>>>>>>>>                   |                                        |
>>>>>>>>>>>>                vmlinux                              +-------------+
>>>>>>>>>>>>                module1.ko                           | BPF program |
>>>>>>>>>>>>                module2.ko                           +-------------+
>>>>>>>>>>>>                  ...
>>>>>>>>>>>>
>>>>>>>>>>>>           This is because:
>>>>>>>>>>>>
>>>>>>>>>>>>           a)  Unlike GCC, LLVM will only generate BTF for BPF programs.
>>>>>>>>>>>>
>>>>>>>>>>>>           b)  GCC can generate BTF for whatever target with -gbtf, but there is no
>>>>>>>>>>>>               support for linking/deduplicating BTF in the linker.
>>>>>>>>>>>>
>>>>>>>>>>>>           In the scenario above, the verifier needs access to the pointer tags of
>>>>>>>>>>>>           both the kernel types/declarations (conveyed in the DWARF and translated
>>>>>>>>>>>>           to BTF by pahole) and those of the BPF program (available directly in BTF).
>>>>>>>>>>>>
>>>>>>>>>>>>           Another motivation for having the tag information in DWARF, unrelated to
>>>>>>>>>>>>           BPF and BTF, is that the drgn project (another DWARF consumer) also wants
>>>>>>>>>>>>           to benefit from these tags in order to differentiate between different
>>>>>>>>>>>>           kinds of pointers in the kernel.
>>>>>>>>>>>>
>>>>>>>>>>>> 3)  Conveying the tags in the generated BTF debug info.
>>>>>>>>>>>>
>>>>>>>>>>>>           This is easy: the main purpose of having this info in BTF is for the
>>>>>>>>>>>>           compiled eBPF programs. The kernel verifier can then access the tags
>>>>>>>>>>>>           of pointers used by the eBPF programs.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> For more information about these tags and the motivation behind them, please
>>>>>>>>>>>> refer to the following linux kernel discussions:
>>>>>>>>>>>>
>>>>>>>>>>>>         https://lore.kernel.org/bpf/20210914223004.244411-1-yhs@fb.com/
>>>>>>>>>>>>         https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
>>>>>>>>>>>>         https://lore.kernel.org/bpf/20211112012604.1504583-1-yhs@fb.com/
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Implementation Overview
>>>>>>>>>>>> =======================
>>>>>>>>>>>>
>>>>>>>>>>>> To enable these annotations, two new C language attributes are added:
>>>>>>>>>>>> __attribute__((debug_annotate_decl("foo"))) and
>>>>>>>>>>>> __attribute__((debug_annotate_type("bar"))). Both attributes accept a single
>>>>>>>>>>>> arbitrary string constant argument, which will be recorded in the generated
>>>>>>>>>>>> DWARF and/or BTF debug information. They have no effect on code generation.
>>>>>>>>>>>>
>>>>>>>>>>>> Note that we are not using the same attribute names as LLVM (btf_decl_tag and
>>>>>>>>>>>> btf_type_tag, respectively). While these attributes are functionally very
>>>>>>>>>>>> similar, they have grown beyond purely BTF-specific uses, so inclusion of "btf"
>>>>>>>>>>>> in the attribute name seems misleading.
>>>>>>>>>>>>
>>>>>>>>>>>> DWARF support is enabled via a new DW_TAG_GNU_annotation. When generating DWARF,
>>>>>>>>>>>> declarations and types will be checked for the corresponding attributes. If
>>>>>>>>>>>> present, a DW_TAG_GNU_annotation DIE will be created as a child of the DIE for
>>>>>>>>>>>> the annotated type or declaration, one for each tag. These DIEs link the
>>>>>>>>>>>> arbitrary tag value to the item they annotate.
>>>>>>>>>>>>
>>>>>>>>>>>> For example, the following variable declaration:
>>>>>>>>>>>>
>>>>>>>>>>>>         #define __typetag1 __attribute__((debug_annotate_type ("typetag1")))
>>>>>>>>>>>>
>>>>>>>>>>>>         #define __decltag1 __attribute__((debug_annotate_decl ("decltag1")))
>>>>>>>>>>>>         #define __decltag2 __attribute__((debug_annotate_decl ("decltag2")))
>>>>>>>>>>>>
>>>>>>>>>>>>         int * __typetag1 x __decltag1 __decltag2;
>>>>>>>>>>>
>>>>>>>>>>> Based on the above example
>>>>>>>>>>>               static int do_execve(struct filename *filename,
>>>>>>>>>>>                 const char __user *const __user *__argv,
>>>>>>>>>>>                 const char __user *const __user *__envp);
>>>>>>>>>>>
>>>>>>>>>>> Should the above example should be the below?
>>>>>>>>>>>           int __typetag1 * x __decltag1 __decltag2
>>>>>>>>>>>
>>>>>>>>>> This example is not related to the one above. It is just meant to
>>>>>>>>>> show the behavior of both attributes. My apologies for not making
>>>>>>>>>> that clear.
>>>>>>>>>
>>>>>>>>> Okay, it should be fine if the dwarf debug_info is shown.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Produces the following DWARF information:
>>>>>>>>>>>>
>>>>>>>>>>>>        <1><1e>: Abbrev Number: 3 (DW_TAG_variable)
>>>>>>>>>>>>           <1f>   DW_AT_name        : x
>>>>>>>>>>>>           <21>   DW_AT_decl_file   : 1
>>>>>>>>>>>>           <22>   DW_AT_decl_line   : 7
>>>>>>>>>>>>           <23>   DW_AT_decl_column : 18
>>>>>>>>>>>>           <24>   DW_AT_type        : <0x49>
>>>>>>>>>>>>           <28>   DW_AT_external    : 1
>>>>>>>>>>>>           <28>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
>>>>>>>>>>>>           <32>   DW_AT_sibling     : <0x49>
>>>>>>>>>>>>        <2><36>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>>>> <37> DW_AT_name : (indirect string, offset: 0xd6):
>>> debug_annotate_decl
>>>>>>>>>>>>           <3b>   DW_AT_const_value : (indirect string, offset: 0xcd): decltag2
>>>>>>>>>>>>        <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>>>> <40> DW_AT_name : (indirect string, offset: 0xd6):
>>> debug_annotate_decl
>>>>>>>>>>>>           <44>   DW_AT_const_value : (indirect string, offset: 0x0): decltag1
>>>>>>>>>>>>        <2><48>: Abbrev Number: 0
>>>>>>>>>>>>        <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type)
>>>>>>>>>>>>           <4a>   DW_AT_byte_size   : 8
>>>>>>>>>>>>           <4b>   DW_AT_type        : <0x5d>
>>>>>>>>>>>>           <4f>   DW_AT_sibling     : <0x5d>
>>>>>>>>>>>>        <2><53>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>>>> <54> DW_AT_name : (indirect string, offset: 0x9):
>>> debug_annotate_type
>>>>>>>>>>>>           <58>   DW_AT_const_value : (indirect string, offset: 0x1d): typetag1
>>>>>>>>>>>>        <2><5c>: Abbrev Number: 0
>>>>>>>>>>>>        <1><5d>: Abbrev Number: 5 (DW_TAG_base_type)
>>>>>>>>>>>>           <5e>   DW_AT_byte_size   : 4
>>>>>>>>>>>>           <5f>   DW_AT_encoding    : 5	(signed)
>>>>>>>>>>>>           <60>   DW_AT_name        : int
>>>>>>>>>>>>        <1><64>: Abbrev Number: 0
>>>>>>>>>
>>>>>>>>> This shows the info in .debug_abbrev. What I mean is to
>>>>>>>>> show the related info in .debug_info section which seems more useful to
>>>>>>>>> understand the relationships between different tags. Maybe this is due
>>>>>>>>> to that I am not fully understanding what <1>/<2> means in <1><49> and
>>>>>>>>> <2><53> etc.
>>>>>>>> I think that dump actually shows .debug_info, with the abbrevs
>>>>>>>> expanded...
>>>>>>>> Anyway, it seems to us that the root of this problem is the fact the
>>>>>>>> kernel sparse annotations, such as address_space(__user), are:
>>>>>>>> 1) To be processed by an external kernel-specific tool (
>>>>>>>>        https://sparse.docs.kernel.org/en/latest/annotations.html) and not a
>>>>>>>>        C compiler, and therefore,
>>>>>>>> 2) Not quite the same than compiler attributes (despite the way they
>>>>>>>>        look.)  In particular, they seem to assume an ordering different than
>>>>>>>>        of GNU attributes: in some cases given the same written order, they
>>>>>>>>        refer to different things!.  Which is quite unfortunate :(
>>>>>>>
>>>>>>> Yes, currently __user/__kernel macros (implemented with address_space
>>>>>>> attribute) are processed by macros.
>>>>>>>
>>>>>>>> Now, if I understood properly, you plan to change the definition of
>>>>>>>> __user and __kernel in the kernel sources in order to generate the tag
>>>>>>>> compiler attributes, correct?
>>>>>>>
>>>>>>> Right. The original __user definition likes:
>>>>>>>      # define __user         __attribute__((noderef, address_space(__user)))
>>>>>>>
>>>>>>> The new attribute looks like
>>>>>>>      # define BTF_TYPE_TAG(value) __attribute__((btf_type_tag(#value)))
>>>>>>>      #  define __user        BTF_TYPE_TAG(user)
>>>>>> Ok I see.  So the kernel will stop using sparse attributes to
>>>>>> implement
>>>>>> __user and __kernel and start using compiler attributes for tags
>>>>>> instead.
>>>>>>
>>>>>>>> Is that the reason why LLVM implements what we assume to be the
>>>>>>>> sparse
>>>>>>>> ordering, and not the correct GNU attributes ordering, for the tag
>>>>>>>> attributes?
>>>>>>>
>>>>>>> Note that __user attributes apply to pointee's and not pointers.
>>>>>>> Just like
>>>>>>>       const int *p;
>>>>>>> the 'const' is not applied to pointer 'p', but the pointee of 'p'.
>>>>>>>
>>>>>>> What current llvm dwarf generation with
>>>>>>>       pointer
>>>>>>>         <--- btf_type_tag
>>>>>>> is just ONE implementation. As I said earlier, I am okay to
>>>>>>> have dwarf implementation like
>>>>>>>       p->btf_type_tag->const->int.
>>>>>>> If you can propose an implementation like this in dwarf. I can propose
>>>>>>> to change implementation in llvm.
>>>>>> I think we are miscommunicating.
>>>>>> Looks like there is a divergence on what attributes apply to what
>>>>>> language entities between the sparse compiler and GCC/LLVM.  How to
>>>>>> represent that in DWARF is a different matter.
>>>>>> For this example:
>>>>>>      int __typetag1 * __typetag2 __typetag3 * g;
>>>>>> a) GCC associates __typetag1 with the pointer-to-pointer-to-int.
>>>>>> b) LLVM associates __typetag1 to pointer-to-int.
>>>>>> Where:
>>>>>> a) Is the expected behavior of a compiler attributes, as documented
>>>>>> in
>>>>>>       the GCC manual.
>>>>>> b) Is presumably what the sparse compiler expects, but _not_ the
>>>>>>       ordering expected for a compiler GNU attribute.
>>>>>> So, if the kernel source __user and __kernel annotations (which
>>>>>> currently expand to sparse attributes) follow the sparse ordering, and
>>>>>> you want to implement __user and __kernel in terms of compiler
>>>>>> attributes instead (the annotation attributes) then you will have to:
>>>>>> 1) Fix LLVM to implement the usual ordering for these attributes and
>>>>>> 2) fix the kernel sources to use that ordering
>>>>>> [Incidentally, the same applies to another "ex-sparse" attribute you
>>>>>>     have in the kernel and also implemented in LLVM with a weird ordering:
>>>>>>     the address_space attribute.]
>>>>>> For 2), it may be possible to write a coccinnelle script to generate
>>>>>> the
>>>>>> patch...
>>>>>
>>>>> I don't think (2) (to change kernel source for different attr ordering)
>>>>> will work. So the only thing we can do is in compiler/pahole except
>>>>> macro replacement in kernel.
>>>> I looked at sparse and its parser.  Wanted to be sure the ordering
>>>> it
>>>> uses to interpret sparse annotations (such as address_space, alignment,
>>>> etc) is definitely _not_ the same ordering used by __attribute__ in C
>>>> compilers.
>>>> It is very different indeed and the same can be said about how
>>>> sparse
>>>> interprets other modifiers like `const': in sparse both `int const *foo'
>>>> and `int *const foo' parse to a constant pointer to int, for example.
>>>> I am not to judge how sparse handles its annotations.  It may be
>>>> very
>>>> well and pertinent for its particular purpose.
>>>> But I am not sure if it is reasonable to expect C compilers to
>>>> implement
>>>> certain type __attributes__ to parse differently, just because it
>>>> happens these attributes are reused from sparse annotations in a
>>>> particular program (in this case the kernel.)  The debug_annotate_decl
>>>> and debug_annotate_type attributes are not even intended to be
>>>> kernel-specific.
>>>> So, if changing the kernel sources is not an option (why btw, other
>>>> than
>>>> being a PITA?) at this point I really don't know what else to suggest :/
>>>> Any suggestion from the front-end people?
>>>
>>> Just want to understand the overall picture. So gcc can still emit
>>> BTF properly with btf_type_tag right? The issue we are talking about
>>> here is about the dwarf, right?
>> If by "properly" you mean how sparse handles its annotations, then
>> not
>> really.
>> The issue we are talking about is rather a language-level one: to
>> what
>> entity/type the compiler attribute applies.
>> So, for:
>>    int __attribute__((debug_annotate_decl("user"))) *foo;
>> GCC will apply the attribute to the int type, following the rules
>> for
>> type attributes (sparse would apply the annotation to the *int type
>> instead).  The emitted debug info (be it DWARF or BTF) will reflect
>> that, no more no less :/
>
> I don't know what does this 'apply the attribute to the int' mean.
> In current clang implementation it means the following dwarf chains
> from right to left
>   variable 'foo'
>     type: ptr
>       base type: attr_type: attr
>                     underlying type: int
>
> So the type chain is foo -> ptr -> attr -> int

Urgh sorry Yonghong, that was a bad example where there is no
divergence.  At this point I find myself confused regarding the sparse,
clang and GCC attribute issue (I have so many dumps around from all
three tools in several formats) so I better recap on this before
creating further confusion.

Will be back to you soon.

>>> If this is the case, we might have
>>> a partial solution here.
>>>    - gcc emits BTF for vmlinux
>> Note that for emitting BTF for vmlinux we would need support in the
>> linker to merge and deduplicate BTF, which at the moment we don't have.
>
> This should be okay. pahole will merge and deduplicate btf. In pahole
> '-j' mode, each thread will convert each .o file dwarf to btf, and
> then pahole will merge and deduplicate btf.

Thats nice.  If LLVM supported generating BTF for any target (I believe
you got patches for that) you could even skip the dwarf->BTF translation
step alltogether with both LLVM and GCC kernel builds :)

>
>> 
>>>    - gcc emits dwarf for vmlinux ignoring btf_type_tag
>>>    - in pahole, vmlinux BTF is amended with some additional misc things.
>>> Although there are some use cases to have btf_type_tag in dwarf, but
>>> that can be workarouned with BTF + dwarf both of which are generated
>>> by the compiler. Not elegent, but probably works.
>>>>
>>>>>> Does this make sense?
>>>>>>
>>>>>>>> If that is so, we have quite a problem here: I don't think we can
>>>>>>>> change
>>>>>>>> the way GCC handles GNU-like attributes just because the kernel sources
>>>>>>>> want to hook on these __user/__kernel sparse annotations to generate the
>>>>>>>> compiler tags, even if we could mayhaps get GCC to handle
>>>>>>>> debug_annotate_type and debug_annotate_decl differently.  Some would say
>>>>>>>> doing so would perpetuate the mistake instead of fixing it...
>>>>>>>> Is my understanding correct?
>>>>>>>
>>>>>>> Let us just say that the btf_type_tag attribute applies to pointees.
>>>>>>> Does this help?
>>>>>>>
>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Maybe you can also show what dwarf debug_info looks like
>>>>>>>>>> I am not sure what you mean. This is the .debug_info section as output
>>>>>>>>>> by readelf -w. I did trim some information not relevant to the discussion
>>>>>>>>>> such as the DW_TAG_compile_unit DIE, for brevity.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> In the case of BTF, the annotations are recorded in two type kinds recently
>>>>>>>>>>>> added to the BTF specification: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>>>>> The above example declaration prodcues the following BTF information:
>>>>>>>>>>>>
>>>>>>>>>>>> [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
>>>>>>>>>>>> [2] PTR '(anon)' type_id=3
>>>>>>>>>>>> [3] TYPE_TAG 'typetag1' type_id=1
>>>>>>>>>>>> [4] DECL_TAG 'decltag1' type_id=6 component_idx=-1
>>>>>>>>>>>> [5] DECL_TAG 'decltag2' type_id=6 component_idx=-1
>>>>>>>>>>>> [6] VAR 'x' type_id=2, linkage=global
>>>>>>>>>>>> [7] DATASEC '.bss' size=0 vlen=1
>>>>>>>>>>>> 	type_id=6 offset=0 size=8 (VAR 'x')
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> [...]
  

On 7/15/22 7:17 AM, Jose E. Marchesi wrote:
> 
>> On 7/14/22 8:09 AM, Jose E. Marchesi wrote:
>>> Hi Yonghong.
>>>
>>>> On 7/7/22 1:24 PM, Jose E. Marchesi wrote:
>>>>> Hi Yonghong.
>>>>>
>>>>>> On 6/21/22 9:12 AM, Jose E. Marchesi wrote:
>>>>>>>
>>>>>>>> On 6/17/22 10:18 AM, Jose E. Marchesi wrote:
>>>>>>>>> Hi Yonghong.
>>>>>>>>>
>>>>>>>>>> On 6/15/22 1:57 PM, David Faust wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 6/14/22 22:53, Yonghong Song wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 6/7/22 2:43 PM, David Faust wrote:
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> This patch series adds support for:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Two new C-language-level attributes that allow to associate (to "annotate" or
>>>>>>>>>>>>>          to "tag") particular declarations and types with arbitrary strings. As
>>>>>>>>>>>>>          explained below, this is intended to be used to, for example, characterize
>>>>>>>>>>>>>          certain pointer types.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - The conveyance of that information in the DWARF output in the form of a new
>>>>>>>>>>>>>          DIE: DW_TAG_GNU_annotation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - The conveyance of that information in the BTF output in the form of two new
>>>>>>>>>>>>>          kinds of BTF objects: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>>>>>>
>>>>>>>>>>>>> All of these facilities are being added to the eBPF ecosystem, and support for
>>>>>>>>>>>>> them exists in some form in LLVM.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Purpose
>>>>>>>>>>>>> =======
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1)  Addition of C-family language constructs (attributes) to specify free-text
>>>>>>>>>>>>>            tags on certain language elements, such as struct fields.
>>>>>>>>>>>>>
>>>>>>>>>>>>>            The purpose of these annotations is to provide additional information about
>>>>>>>>>>>>>            types, variables, and function parameters of interest to the kernel. A
>>>>>>>>>>>>>            driving use case is to tag pointer types within the linux kernel and eBPF
>>>>>>>>>>>>>            programs with additional semantic information, such as '__user' or '__rcu'.
>>>>>>>>>>>>>
>>>>>>>>>>>>>            For example, consider the linux kernel function do_execve with the
>>>>>>>>>>>>>            following declaration:
>>>>>>>>>>>>>
>>>>>>>>>>>>>              static int do_execve(struct filename *filename,
>>>>>>>>>>>>>                 const char __user *const __user *__argv,
>>>>>>>>>>>>>                 const char __user *const __user *__envp);
>>>>>>>>>>>>>
>>>>>>>>>>>>>            Here, __user could be defined with these annotations to record semantic
>>>>>>>>>>>>>            information about the pointer parameters (e.g., they are user-provided) in
>>>>>>>>>>>>>            DWARF and BTF information. Other kernel facilites such as the eBPF verifier
>>>>>>>>>>>>>            can read the tags and make use of the information.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2)  Conveying the tags in the generated DWARF debug info.
>>>>>>>>>>>>>
>>>>>>>>>>>>>            The main motivation for emitting the tags in DWARF is that the Linux kernel
>>>>>>>>>>>>>            generates its BTF information via pahole, using DWARF as a source:
>>>>>>>>>>>>>
>>>>>>>>>>>>>                +--------+  BTF                  BTF   +----------+
>>>>>>>>>>>>>                | pahole |-------> vmlinux.btf ------->| verifier |
>>>>>>>>>>>>>                +--------+                             +----------+
>>>>>>>>>>>>>                    ^                                        ^
>>>>>>>>>>>>>                    |                                        |
>>>>>>>>>>>>>              DWARF |                                    BTF |
>>>>>>>>>>>>>                    |                                        |
>>>>>>>>>>>>>                 vmlinux                              +-------------+
>>>>>>>>>>>>>                 module1.ko                           | BPF program |
>>>>>>>>>>>>>                 module2.ko                           +-------------+
>>>>>>>>>>>>>                   ...
>>>>>>>>>>>>>
>>>>>>>>>>>>>            This is because:
>>>>>>>>>>>>>
>>>>>>>>>>>>>            a)  Unlike GCC, LLVM will only generate BTF for BPF programs.
>>>>>>>>>>>>>
>>>>>>>>>>>>>            b)  GCC can generate BTF for whatever target with -gbtf, but there is no
>>>>>>>>>>>>>                support for linking/deduplicating BTF in the linker.
>>>>>>>>>>>>>
>>>>>>>>>>>>>            In the scenario above, the verifier needs access to the pointer tags of
>>>>>>>>>>>>>            both the kernel types/declarations (conveyed in the DWARF and translated
>>>>>>>>>>>>>            to BTF by pahole) and those of the BPF program (available directly in BTF).
>>>>>>>>>>>>>
>>>>>>>>>>>>>            Another motivation for having the tag information in DWARF, unrelated to
>>>>>>>>>>>>>            BPF and BTF, is that the drgn project (another DWARF consumer) also wants
>>>>>>>>>>>>>            to benefit from these tags in order to differentiate between different
>>>>>>>>>>>>>            kinds of pointers in the kernel.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 3)  Conveying the tags in the generated BTF debug info.
>>>>>>>>>>>>>
>>>>>>>>>>>>>            This is easy: the main purpose of having this info in BTF is for the
>>>>>>>>>>>>>            compiled eBPF programs. The kernel verifier can then access the tags
>>>>>>>>>>>>>            of pointers used by the eBPF programs.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> For more information about these tags and the motivation behind them, please
>>>>>>>>>>>>> refer to the following linux kernel discussions:
>>>>>>>>>>>>>
>>>>>>>>>>>>>          https://lore.kernel.org/bpf/20210914223004.244411-1-yhs@fb.com/
>>>>>>>>>>>>>          https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
>>>>>>>>>>>>>          https://lore.kernel.org/bpf/20211112012604.1504583-1-yhs@fb.com/
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Implementation Overview
>>>>>>>>>>>>> =======================
>>>>>>>>>>>>>
>>>>>>>>>>>>> To enable these annotations, two new C language attributes are added:
>>>>>>>>>>>>> __attribute__((debug_annotate_decl("foo"))) and
>>>>>>>>>>>>> __attribute__((debug_annotate_type("bar"))). Both attributes accept a single
>>>>>>>>>>>>> arbitrary string constant argument, which will be recorded in the generated
>>>>>>>>>>>>> DWARF and/or BTF debug information. They have no effect on code generation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Note that we are not using the same attribute names as LLVM (btf_decl_tag and
>>>>>>>>>>>>> btf_type_tag, respectively). While these attributes are functionally very
>>>>>>>>>>>>> similar, they have grown beyond purely BTF-specific uses, so inclusion of "btf"
>>>>>>>>>>>>> in the attribute name seems misleading.
>>>>>>>>>>>>>
>>>>>>>>>>>>> DWARF support is enabled via a new DW_TAG_GNU_annotation. When generating DWARF,
>>>>>>>>>>>>> declarations and types will be checked for the corresponding attributes. If
>>>>>>>>>>>>> present, a DW_TAG_GNU_annotation DIE will be created as a child of the DIE for
>>>>>>>>>>>>> the annotated type or declaration, one for each tag. These DIEs link the
>>>>>>>>>>>>> arbitrary tag value to the item they annotate.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For example, the following variable declaration:
>>>>>>>>>>>>>
>>>>>>>>>>>>>          #define __typetag1 __attribute__((debug_annotate_type ("typetag1")))
>>>>>>>>>>>>>
>>>>>>>>>>>>>          #define __decltag1 __attribute__((debug_annotate_decl ("decltag1")))
>>>>>>>>>>>>>          #define __decltag2 __attribute__((debug_annotate_decl ("decltag2")))
>>>>>>>>>>>>>
>>>>>>>>>>>>>          int * __typetag1 x __decltag1 __decltag2;
>>>>>>>>>>>>
>>>>>>>>>>>> Based on the above example
>>>>>>>>>>>>                static int do_execve(struct filename *filename,
>>>>>>>>>>>>                  const char __user *const __user *__argv,
>>>>>>>>>>>>                  const char __user *const __user *__envp);
>>>>>>>>>>>>
>>>>>>>>>>>> Should the above example should be the below?
>>>>>>>>>>>>            int __typetag1 * x __decltag1 __decltag2
>>>>>>>>>>>>
>>>>>>>>>>> This example is not related to the one above. It is just meant to
>>>>>>>>>>> show the behavior of both attributes. My apologies for not making
>>>>>>>>>>> that clear.
>>>>>>>>>>
>>>>>>>>>> Okay, it should be fine if the dwarf debug_info is shown.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Produces the following DWARF information:
>>>>>>>>>>>>>
>>>>>>>>>>>>>         <1><1e>: Abbrev Number: 3 (DW_TAG_variable)
>>>>>>>>>>>>>            <1f>   DW_AT_name        : x
>>>>>>>>>>>>>            <21>   DW_AT_decl_file   : 1
>>>>>>>>>>>>>            <22>   DW_AT_decl_line   : 7
>>>>>>>>>>>>>            <23>   DW_AT_decl_column : 18
>>>>>>>>>>>>>            <24>   DW_AT_type        : <0x49>
>>>>>>>>>>>>>            <28>   DW_AT_external    : 1
>>>>>>>>>>>>>            <28>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0 	(DW_OP_addr: 0)
>>>>>>>>>>>>>            <32>   DW_AT_sibling     : <0x49>
>>>>>>>>>>>>>         <2><36>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>>>>> <37> DW_AT_name : (indirect string, offset: 0xd6):
>>>> debug_annotate_decl
>>>>>>>>>>>>>            <3b>   DW_AT_const_value : (indirect string, offset: 0xcd): decltag2
>>>>>>>>>>>>>         <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>>>>> <40> DW_AT_name : (indirect string, offset: 0xd6):
>>>> debug_annotate_decl
>>>>>>>>>>>>>            <44>   DW_AT_const_value : (indirect string, offset: 0x0): decltag1
>>>>>>>>>>>>>         <2><48>: Abbrev Number: 0
>>>>>>>>>>>>>         <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type)
>>>>>>>>>>>>>            <4a>   DW_AT_byte_size   : 8
>>>>>>>>>>>>>            <4b>   DW_AT_type        : <0x5d>
>>>>>>>>>>>>>            <4f>   DW_AT_sibling     : <0x5d>
>>>>>>>>>>>>>         <2><53>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>>>>>>>>>> <54> DW_AT_name : (indirect string, offset: 0x9):
>>>> debug_annotate_type
>>>>>>>>>>>>>            <58>   DW_AT_const_value : (indirect string, offset: 0x1d): typetag1
>>>>>>>>>>>>>         <2><5c>: Abbrev Number: 0
>>>>>>>>>>>>>         <1><5d>: Abbrev Number: 5 (DW_TAG_base_type)
>>>>>>>>>>>>>            <5e>   DW_AT_byte_size   : 4
>>>>>>>>>>>>>            <5f>   DW_AT_encoding    : 5	(signed)
>>>>>>>>>>>>>            <60>   DW_AT_name        : int
>>>>>>>>>>>>>         <1><64>: Abbrev Number: 0
>>>>>>>>>>
>>>>>>>>>> This shows the info in .debug_abbrev. What I mean is to
>>>>>>>>>> show the related info in .debug_info section which seems more useful to
>>>>>>>>>> understand the relationships between different tags. Maybe this is due
>>>>>>>>>> to that I am not fully understanding what <1>/<2> means in <1><49> and
>>>>>>>>>> <2><53> etc.
>>>>>>>>> I think that dump actually shows .debug_info, with the abbrevs
>>>>>>>>> expanded...
>>>>>>>>> Anyway, it seems to us that the root of this problem is the fact the
>>>>>>>>> kernel sparse annotations, such as address_space(__user), are:
>>>>>>>>> 1) To be processed by an external kernel-specific tool (
>>>>>>>>>         https://sparse.docs.kernel.org/en/latest/annotations.html) and not a
>>>>>>>>>         C compiler, and therefore,
>>>>>>>>> 2) Not quite the same than compiler attributes (despite the way they
>>>>>>>>>         look.)  In particular, they seem to assume an ordering different than
>>>>>>>>>         of GNU attributes: in some cases given the same written order, they
>>>>>>>>>         refer to different things!.  Which is quite unfortunate :(
>>>>>>>>
>>>>>>>> Yes, currently __user/__kernel macros (implemented with address_space
>>>>>>>> attribute) are processed by macros.
>>>>>>>>
>>>>>>>>> Now, if I understood properly, you plan to change the definition of
>>>>>>>>> __user and __kernel in the kernel sources in order to generate the tag
>>>>>>>>> compiler attributes, correct?
>>>>>>>>
>>>>>>>> Right. The original __user definition likes:
>>>>>>>>       # define __user         __attribute__((noderef, address_space(__user)))
>>>>>>>>
>>>>>>>> The new attribute looks like
>>>>>>>>       # define BTF_TYPE_TAG(value) __attribute__((btf_type_tag(#value)))
>>>>>>>>       #  define __user        BTF_TYPE_TAG(user)
>>>>>>> Ok I see.  So the kernel will stop using sparse attributes to
>>>>>>> implement
>>>>>>> __user and __kernel and start using compiler attributes for tags
>>>>>>> instead.
>>>>>>>
>>>>>>>>> Is that the reason why LLVM implements what we assume to be the
>>>>>>>>> sparse
>>>>>>>>> ordering, and not the correct GNU attributes ordering, for the tag
>>>>>>>>> attributes?
>>>>>>>>
>>>>>>>> Note that __user attributes apply to pointee's and not pointers.
>>>>>>>> Just like
>>>>>>>>        const int *p;
>>>>>>>> the 'const' is not applied to pointer 'p', but the pointee of 'p'.
>>>>>>>>
>>>>>>>> What current llvm dwarf generation with
>>>>>>>>        pointer
>>>>>>>>          <--- btf_type_tag
>>>>>>>> is just ONE implementation. As I said earlier, I am okay to
>>>>>>>> have dwarf implementation like
>>>>>>>>        p->btf_type_tag->const->int.
>>>>>>>> If you can propose an implementation like this in dwarf. I can propose
>>>>>>>> to change implementation in llvm.
>>>>>>> I think we are miscommunicating.
>>>>>>> Looks like there is a divergence on what attributes apply to what
>>>>>>> language entities between the sparse compiler and GCC/LLVM.  How to
>>>>>>> represent that in DWARF is a different matter.
>>>>>>> For this example:
>>>>>>>       int __typetag1 * __typetag2 __typetag3 * g;
>>>>>>> a) GCC associates __typetag1 with the pointer-to-pointer-to-int.
>>>>>>> b) LLVM associates __typetag1 to pointer-to-int.
>>>>>>> Where:
>>>>>>> a) Is the expected behavior of a compiler attributes, as documented
>>>>>>> in
>>>>>>>        the GCC manual.
>>>>>>> b) Is presumably what the sparse compiler expects, but _not_ the
>>>>>>>        ordering expected for a compiler GNU attribute.
>>>>>>> So, if the kernel source __user and __kernel annotations (which
>>>>>>> currently expand to sparse attributes) follow the sparse ordering, and
>>>>>>> you want to implement __user and __kernel in terms of compiler
>>>>>>> attributes instead (the annotation attributes) then you will have to:
>>>>>>> 1) Fix LLVM to implement the usual ordering for these attributes and
>>>>>>> 2) fix the kernel sources to use that ordering
>>>>>>> [Incidentally, the same applies to another "ex-sparse" attribute you
>>>>>>>      have in the kernel and also implemented in LLVM with a weird ordering:
>>>>>>>      the address_space attribute.]
>>>>>>> For 2), it may be possible to write a coccinnelle script to generate
>>>>>>> the
>>>>>>> patch...
>>>>>>
>>>>>> I don't think (2) (to change kernel source for different attr ordering)
>>>>>> will work. So the only thing we can do is in compiler/pahole except
>>>>>> macro replacement in kernel.
>>>>> I looked at sparse and its parser.  Wanted to be sure the ordering
>>>>> it
>>>>> uses to interpret sparse annotations (such as address_space, alignment,
>>>>> etc) is definitely _not_ the same ordering used by __attribute__ in C
>>>>> compilers.
>>>>> It is very different indeed and the same can be said about how
>>>>> sparse
>>>>> interprets other modifiers like `const': in sparse both `int const *foo'
>>>>> and `int *const foo' parse to a constant pointer to int, for example.
>>>>> I am not to judge how sparse handles its annotations.  It may be
>>>>> very
>>>>> well and pertinent for its particular purpose.
>>>>> But I am not sure if it is reasonable to expect C compilers to
>>>>> implement
>>>>> certain type __attributes__ to parse differently, just because it
>>>>> happens these attributes are reused from sparse annotations in a
>>>>> particular program (in this case the kernel.)  The debug_annotate_decl
>>>>> and debug_annotate_type attributes are not even intended to be
>>>>> kernel-specific.
>>>>> So, if changing the kernel sources is not an option (why btw, other
>>>>> than
>>>>> being a PITA?) at this point I really don't know what else to suggest :/
>>>>> Any suggestion from the front-end people?
>>>>
>>>> Just want to understand the overall picture. So gcc can still emit
>>>> BTF properly with btf_type_tag right? The issue we are talking about
>>>> here is about the dwarf, right?
>>> If by "properly" you mean how sparse handles its annotations, then
>>> not
>>> really.
>>> The issue we are talking about is rather a language-level one: to
>>> what
>>> entity/type the compiler attribute applies.
>>> So, for:
>>>     int __attribute__((debug_annotate_decl("user"))) *foo;
>>> GCC will apply the attribute to the int type, following the rules
>>> for
>>> type attributes (sparse would apply the annotation to the *int type
>>> instead).  The emitted debug info (be it DWARF or BTF) will reflect
>>> that, no more no less :/
>>
>> I don't know what does this 'apply the attribute to the int' mean.
>> In current clang implementation it means the following dwarf chains
>> from right to left
>>    variable 'foo'
>>      type: ptr
>>        base type: attr_type: attr
>>                      underlying type: int
>>
>> So the type chain is foo -> ptr -> attr -> int
> 
> Urgh sorry Yonghong, that was a bad example where there is no
> divergence.  At this point I find myself confused regarding the sparse,
> clang and GCC attribute issue (I have so many dumps around from all
> three tools in several formats) so I better recap on this before
> creating further confusion.
> 
> Will be back to you soon.
> 
>>>> If this is the case, we might have
>>>> a partial solution here.
>>>>     - gcc emits BTF for vmlinux
>>> Note that for emitting BTF for vmlinux we would need support in the
>>> linker to merge and deduplicate BTF, which at the moment we don't have.
>>
>> This should be okay. pahole will merge and deduplicate btf. In pahole
>> '-j' mode, each thread will convert each .o file dwarf to btf, and
>> then pahole will merge and deduplicate btf.
> 
> Thats nice.  If LLVM supported generating BTF for any target (I believe
> you got patches for that) you could even skip the dwarf->BTF translation
> step alltogether with both LLVM and GCC kernel builds :)

Yes. This will be a potential future work to generate BTF for all 
targets (at least targets supported by linux kernel) with clang.
I indeed did some experiments before in this area...

But we need pahole regardless as pahole will do some further
processing based vmlinux.o file. So for the time being, if gcc
can generate both BTF (with debug_annotate_decl) and dwarf (without
debug_annotate_decl), pahole should be able to work it out
so we do have a path forward if necessary.

> 
>>
>>>
>>>>     - gcc emits dwarf for vmlinux ignoring btf_type_tag
>>>>     - in pahole, vmlinux BTF is amended with some additional misc things.
>>>> Although there are some use cases to have btf_type_tag in dwarf, but
>>>> that can be workarouned with BTF + dwarf both of which are generated
>>>> by the compiler. Not elegent, but probably works.
>>>>>
>>>>>>> Does this make sense?
>>>>>>>
>>>>>>>>> If that is so, we have quite a problem here: I don't think we can
>>>>>>>>> change
>>>>>>>>> the way GCC handles GNU-like attributes just because the kernel sources
>>>>>>>>> want to hook on these __user/__kernel sparse annotations to generate the
>>>>>>>>> compiler tags, even if we could mayhaps get GCC to handle
>>>>>>>>> debug_annotate_type and debug_annotate_decl differently.  Some would say
>>>>>>>>> doing so would perpetuate the mistake instead of fixing it...
>>>>>>>>> Is my understanding correct?
>>>>>>>>
>>>>>>>> Let us just say that the btf_type_tag attribute applies to pointees.
>>>>>>>> Does this help?
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Maybe you can also show what dwarf debug_info looks like
>>>>>>>>>>> I am not sure what you mean. This is the .debug_info section as output
>>>>>>>>>>> by readelf -w. I did trim some information not relevant to the discussion
>>>>>>>>>>> such as the DW_TAG_compile_unit DIE, for brevity.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the case of BTF, the annotations are recorded in two type kinds recently
>>>>>>>>>>>>> added to the BTF specification: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>>>>>>>>>> The above example declaration prodcues the following BTF information:
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
>>>>>>>>>>>>> [2] PTR '(anon)' type_id=3
>>>>>>>>>>>>> [3] TYPE_TAG 'typetag1' type_id=1
>>>>>>>>>>>>> [4] DECL_TAG 'decltag1' type_id=6 component_idx=-1
>>>>>>>>>>>>> [5] DECL_TAG 'decltag2' type_id=6 component_idx=-1
>>>>>>>>>>>>> [6] VAR 'x' type_id=2, linkage=global
>>>>>>>>>>>>> [7] DATASEC '.bss' size=0 vlen=1
>>>>>>>>>>>>> 	type_id=6 offset=0 size=8 (VAR 'x')
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> [...]
  

Hi, Jose and David,

Any progress on implement debug_annotate attribute in gcc?

Thanks,

Yonghong


On 6/15/22 3:56 PM, Yonghong Song wrote:
> 
> 
> On 6/15/22 1:57 PM, David Faust wrote:
>>
>>
>> On 6/14/22 22:53, Yonghong Song wrote:
>>>
>>>
>>> On 6/7/22 2:43 PM, David Faust wrote:
>>>> Hello,
>>>>
>>>> This patch series adds support for:
>>>>
>>>> - Two new C-language-level attributes that allow to associate (to 
>>>> "annotate" or
>>>>     to "tag") particular declarations and types with arbitrary 
>>>> strings. As
>>>>     explained below, this is intended to be used to, for example, 
>>>> characterize
>>>>     certain pointer types.
>>>>
>>>> - The conveyance of that information in the DWARF output in the form 
>>>> of a new
>>>>     DIE: DW_TAG_GNU_annotation.
>>>>
>>>> - The conveyance of that information in the BTF output in the form 
>>>> of two new
>>>>     kinds of BTF objects: BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
>>>>
>>>> All of these facilities are being added to the eBPF ecosystem, and 
>>>> support for
>>>> them exists in some form in LLVM.
>>>>
>>>> Purpose
>>>> =======
>>>>
>>>> 1)  Addition of C-family language constructs (attributes) to specify 
>>>> free-text
>>>>       tags on certain language elements, such as struct fields.
>>>>
>>>>       The purpose of these annotations is to provide additional 
>>>> information about
>>>>       types, variables, and function parameters of interest to the 
>>>> kernel. A
>>>>       driving use case is to tag pointer types within the linux 
>>>> kernel and eBPF
>>>>       programs with additional semantic information, such as 
>>>> '__user' or '__rcu'.
>>>>
>>>>       For example, consider the linux kernel function do_execve with 
>>>> the
>>>>       following declaration:
>>>>
>>>>         static int do_execve(struct filename *filename,
>>>>            const char __user *const __user *__argv,
>>>>            const char __user *const __user *__envp);
>>>>
>>>>       Here, __user could be defined with these annotations to record 
>>>> semantic
>>>>       information about the pointer parameters (e.g., they are 
>>>> user-provided) in
>>>>       DWARF and BTF information. Other kernel facilites such as the 
>>>> eBPF verifier
>>>>       can read the tags and make use of the information.
>>>>
>>>> 2)  Conveying the tags in the generated DWARF debug info.
>>>>
>>>>       The main motivation for emitting the tags in DWARF is that the 
>>>> Linux kernel
>>>>       generates its BTF information via pahole, using DWARF as a 
>>>> source:
>>>>
>>>>           +--------+  BTF                  BTF   +----------+
>>>>           | pahole |-------> vmlinux.btf ------->| verifier |
>>>>           +--------+                             +----------+
>>>>               ^                                        ^
>>>>               |                                        |
>>>>         DWARF |                                    BTF |
>>>>               |                                        |
>>>>            vmlinux                              +-------------+
>>>>            module1.ko                           | BPF program |
>>>>            module2.ko                           +-------------+
>>>>              ...
>>>>
>>>>       This is because:
>>>>
>>>>       a)  Unlike GCC, LLVM will only generate BTF for BPF programs.
>>>>
>>>>       b)  GCC can generate BTF for whatever target with -gbtf, but 
>>>> there is no
>>>>           support for linking/deduplicating BTF in the linker.
>>>>
>>>>       In the scenario above, the verifier needs access to the 
>>>> pointer tags of
>>>>       both the kernel types/declarations (conveyed in the DWARF and 
>>>> translated
>>>>       to BTF by pahole) and those of the BPF program (available 
>>>> directly in BTF).
>>>>
>>>>       Another motivation for having the tag information in DWARF, 
>>>> unrelated to
>>>>       BPF and BTF, is that the drgn project (another DWARF consumer) 
>>>> also wants
>>>>       to benefit from these tags in order to differentiate between 
>>>> different
>>>>       kinds of pointers in the kernel.
>>>>
>>>> 3)  Conveying the tags in the generated BTF debug info.
>>>>
>>>>       This is easy: the main purpose of having this info in BTF is 
>>>> for the
>>>>       compiled eBPF programs. The kernel verifier can then access 
>>>> the tags
>>>>       of pointers used by the eBPF programs.
>>>>
>>>>
>>>> For more information about these tags and the motivation behind 
>>>> them, please
>>>> refer to the following linux kernel discussions:
>>>>
>>>>     https://lore.kernel.org/bpf/20210914223004.244411-1-yhs@fb.com/
>>>>     https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
>>>>     https://lore.kernel.org/bpf/20211112012604.1504583-1-yhs@fb.com/
>>>>
>>>>
>>>> Implementation Overview
>>>> =======================
>>>>
>>>> To enable these annotations, two new C language attributes are added:
>>>> __attribute__((debug_annotate_decl("foo"))) and
>>>> __attribute__((debug_annotate_type("bar"))). Both attributes accept 
>>>> a single
>>>> arbitrary string constant argument, which will be recorded in the 
>>>> generated
>>>> DWARF and/or BTF debug information. They have no effect on code 
>>>> generation.
>>>>
>>>> Note that we are not using the same attribute names as LLVM 
>>>> (btf_decl_tag and
>>>> btf_type_tag, respectively). While these attributes are functionally 
>>>> very
>>>> similar, they have grown beyond purely BTF-specific uses, so 
>>>> inclusion of "btf"
>>>> in the attribute name seems misleading.
>>>>
>>>> DWARF support is enabled via a new DW_TAG_GNU_annotation. When 
>>>> generating DWARF,
>>>> declarations and types will be checked for the corresponding 
>>>> attributes. If
>>>> present, a DW_TAG_GNU_annotation DIE will be created as a child of 
>>>> the DIE for
>>>> the annotated type or declaration, one for each tag. These DIEs link 
>>>> the
>>>> arbitrary tag value to the item they annotate.
>>>>
>>>> For example, the following variable declaration:
>>>>
>>>>     #define __typetag1 __attribute__((debug_annotate_type 
>>>> ("typetag1")))
>>>>
>>>>     #define __decltag1 __attribute__((debug_annotate_decl 
>>>> ("decltag1")))
>>>>     #define __decltag2 __attribute__((debug_annotate_decl 
>>>> ("decltag2")))
>>>>
>>>>     int * __typetag1 x __decltag1 __decltag2;
>>>
>>> Based on the above example
>>>           static int do_execve(struct filename *filename,
>>>             const char __user *const __user *__argv,
>>>             const char __user *const __user *__envp);
>>>
>>> Should the above example should be the below?
>>>       int __typetag1 * x __decltag1 __decltag2
>>>
>>
>> This example is not related to the one above. It is just meant to
>> show the behavior of both attributes. My apologies for not making
>> that clear.
> 
> Okay, it should be fine if the dwarf debug_info is shown.
> 
>>
>>>>
>>>> Produces the following DWARF information:
>>>>
>>>>    <1><1e>: Abbrev Number: 3 (DW_TAG_variable)
>>>>       <1f>   DW_AT_name        : x
>>>>       <21>   DW_AT_decl_file   : 1
>>>>       <22>   DW_AT_decl_line   : 7
>>>>       <23>   DW_AT_decl_column : 18
>>>>       <24>   DW_AT_type        : <0x49>
>>>>       <28>   DW_AT_external    : 1
>>>>       <28>   DW_AT_location    : 9 byte block: 3 0 0 0 0 0 0 0 0     
>>>> (DW_OP_addr: 0)
>>>>       <32>   DW_AT_sibling     : <0x49>
>>>>    <2><36>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>       <37>   DW_AT_name        : (indirect string, offset: 0xd6): 
>>>> debug_annotate_decl
>>>>       <3b>   DW_AT_const_value : (indirect string, offset: 0xcd): 
>>>> decltag2
>>>>    <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>       <40>   DW_AT_name        : (indirect string, offset: 0xd6): 
>>>> debug_annotate_decl
>>>>       <44>   DW_AT_const_value : (indirect string, offset: 0x0): 
>>>> decltag1
>>>>    <2><48>: Abbrev Number: 0
>>>>    <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type)
>>>>       <4a>   DW_AT_byte_size   : 8
>>>>       <4b>   DW_AT_type        : <0x5d>
>>>>       <4f>   DW_AT_sibling     : <0x5d>
>>>>    <2><53>: Abbrev Number: 1 (User TAG value: 0x6000)
>>>>       <54>   DW_AT_name        : (indirect string, offset: 0x9): 
>>>> debug_annotate_type
>>>>       <58>   DW_AT_const_value : (indirect string, offset: 0x1d): 
>>>> typetag1
>>>>    <2><5c>: Abbrev Number: 0
>>>>    <1><5d>: Abbrev Number: 5 (DW_TAG_base_type)
>>>>       <5e>   DW_AT_byte_size   : 4
>>>>       <5f>   DW_AT_encoding    : 5    (signed)
>>>>       <60>   DW_AT_name        : int
>>>>    <1><64>: Abbrev Number: 0
> 
> This shows the info in .debug_abbrev. What I mean is to
> show the related info in .debug_info section which seems more useful to
> understand the relationships between different tags. Maybe this is due 
> to that I am not fully understanding what <1>/<2> means in <1><49> and 
> <2><53> etc.
> 
>>>
>>> Maybe you can also show what dwarf debug_info looks like
>> I am not sure what you mean. This is the .debug_info section as output
>> by readelf -w. I did trim some information not relevant to the discussion
>> such as the DW_TAG_compile_unit DIE, for brevity.
>>
>>>
>>>>
>>>> In the case of BTF, the annotations are recorded in two type kinds 
>>>> recently
>>>> added to the BTF specification: BTF_KIND_DECL_TAG and 
>>>> BTF_KIND_TYPE_TAG.
>>>> The above example declaration prodcues the following BTF information:
>>>>
>>>> [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
>>>> [2] PTR '(anon)' type_id=3
>>>> [3] TYPE_TAG 'typetag1' type_id=1
>>>> [4] DECL_TAG 'decltag1' type_id=6 component_idx=-1
>>>> [5] DECL_TAG 'decltag2' type_id=6 component_idx=-1
>>>> [6] VAR 'x' type_id=2, linkage=global
>>>> [7] DATASEC '.bss' size=0 vlen=1
>>>>     type_id=6 offset=0 size=8 (VAR 'x')
>>>>
>>>>
>>> [...]