[RFC,00/13] Proposal for a new NT_X86_CPUID core dump note

  One of the shortcomings of the previous XSAVE patch series is that it
depends on heuristics based on the total XSAVE register set size and
XCR0 mask to infer layouts of the various register blocks for core
dumps.  This series introduces a new x86-specific core dump note
intended to supplant these heuristics by storing the raw CPUID leaves
describing the XSAVE layout in core dumps.

This series proposes a new core dump note, NT_X86_CPUID (0x205), which
contains an array of structures.  Each structure describes an invidual
CPUID sub-leaf containing both the inputs to CPUID (%eax and %ecx) and
the outputs (%eax, %ebx, %ecx, and %edx) in a format roughly matching
the follow C structure:

struct cpuid_leaf
{
    uint32_t leaf;
    uint32_t subleaf;
    uint32_t eax;
    uint32_t ebx;
    uint32_t ecx;
    uint32_t edx;
};

This format is not XSAVE-specific and implementations could choose to
add additional CPUID leaves to this structure if needed in the future.
Consumers of this note should lookup the value of required leaves and
ignore any unneeded leaves.

An alternate approach might be to write out a more XSAVE-specific note
that is an array containing the offset and size of each XSAVE region.

Note that either approach would enable storing XSAVE notes in the
"compact" format at some point in the future.

This series adds support for reading/writing the note to binutils as
well as suport for parsing and generating the note in GDB.  It also
hooks this into both the FreeBSD and Linux x86 architectures in GDB to
read the XSAVE layout from this note when present, and to write out a
note when generating a core via `gcore'.  I've done some limited
testing on FreeBSD/amd64 and Linux/x86-64, but it could probably use
some more testing on Linux in particular.  (I know Simon has an AMD
machine with a layout not handled by the current heuristics for
example.)

For the gcore side, a new TARGET_OBJECT_X86_CPUID is used to fetch the
current note contents from a native target.  There is still one gap
even with this patch series which is that if you are connected to a
remote target (e.g. gdbserver), we currently do not have a known XSAVE
layout to use when writing out a core via `gcore'.  One option that
would close this gap would be to extend the remote protocol to permit
reading this new object from a debug server.  The remote target could
then implement fetching this object and also make use of this object
to implement the target::fetch_x86_xsave_layout method which would
close that gap.  Another possibility would be to just pick a "known"
XSAVE format that matches one of the heuristics.

The series is available from git@github.com:bsdjhb/gdb.git on the
`nt_x86_cpuid' branch.

I also have an implementation of this core dump note available for
FreeBSD's kernel, though I won't merge it until we've collectively
settled on the format: https://reviews.freebsd.org/D42136

Things I have not done and could use help with:

- Implementation for the Linux kernel

- Coordination with folks from LLDB

John Baldwin (13):
  binutils: Support for the NT_X86_CPUID core dump note
  i387-tdep: Add function to read XSAVE layout from NT_X86_CPUID
  gdb: Use NT_X86_CPUID in x86 FreeBSD architectures to read XSAVE
    layouts
  gdb: Use NT_X86_CPUID in x86 FreeBSD architectures to read XSAVE
    layouts
  nat/x86-cpuid.h: Remove non-x86 fallbacks
  nat/x86-cpuid: Add a function to build the contents of a NT_X86_CPUID
    note
  x86_elf_make_cpuid_note: Helper routine to build NT_X86_CPUID ELF note
  x86-fbsd-nat: Support fetching TARGET_OBJECT_X86_CPUID objects
  fbsd-tdep: Export fbsd_make_corefile_notes
  {amd64,i386}-fbsd-tdep: Include NT_X86_CPUID notes in core dumps from
    gcore
  x86-linux-nat: Support fetching TARGET_OBJECT_X86_CPUID objects
  linux-tdep: Export linux_make_corefile_notes
  {amd64,i386}-linux-tdep: Include NT_X86_CPUID notes in core dumps from
    gcore

 bfd/elf-bfd.h          |   2 +
 bfd/elf.c              |  35 +++++++++++
 binutils/readelf.c     |   2 +
 gdb/amd64-fbsd-tdep.c  |   1 +
 gdb/amd64-linux-tdep.c |   1 +
 gdb/configure.nat      |  13 ++--
 gdb/fbsd-tdep.c        |   5 +-
 gdb/fbsd-tdep.h        |   7 +++
 gdb/i386-fbsd-tdep.c   |  18 +++++-
 gdb/i386-fbsd-tdep.h   |   7 +++
 gdb/i386-linux-tdep.c  |  18 +++++-
 gdb/i386-linux-tdep.h  |   7 +++
 gdb/i387-tdep.c        | 132 +++++++++++++++++++++++++++++++++++++++++
 gdb/i387-tdep.h        |   8 +++
 gdb/linux-tdep.c       |   5 +-
 gdb/linux-tdep.h       |   7 +++
 gdb/nat/x86-cpuid.c    |  91 ++++++++++++++++++++++++++++
 gdb/nat/x86-cpuid.h    |  29 +++------
 gdb/target.h           |   2 +
 gdb/x86-fbsd-nat.c     |  37 ++++++++++++
 gdb/x86-fbsd-nat.h     |   9 +++
 gdb/x86-linux-nat.c    |  37 ++++++++++++
 gdb/x86-linux-nat.h    |   9 +++
 gdb/x86-tdep.c         |  22 +++++++
 gdb/x86-tdep.h         |   9 +++
 include/elf/common.h   |   2 +
 26 files changed, 480 insertions(+), 35 deletions(-)
 create mode 100644 gdb/nat/x86-cpuid.c
  

[Public]

Thanks for doing this, John. I am yet to go through these patches, but I wanted to let you know that for the Linux kernel, we have a patch for creating the core dump .note section which is being reviewed internally at this point. The patch follows the same structure layout proposed by you here. We would cross check to confirm that the proposed Linux patch modifications are in line with your FreeBSD changes.

Regards,
Jini.

>>-----Original Message-----
>>From: John Baldwin <jhb@FreeBSD.org>
>>Sent: Tuesday, October 10, 2023 12:06 AM
>>To: gdb-patches@sourceware.org
>>Cc: Willgerodt; Felix <felix.willgerodt@intel.com>; George; George, Jini Susan
>><JiniSusan.George@amd.com>; Simon Marchi <simon.marchi@polymtl.ca>
>>Subject: [RFC 00/13] Proposal for a new NT_X86_CPUID core dump note
>>
>>Caution: This message originated from an External Source. Use proper caution
>>when opening attachments, clicking links, or responding.
>>
>>
>>One of the shortcomings of the previous XSAVE patch series is that it depends on
>>heuristics based on the total XSAVE register set size and
>>XCR0 mask to infer layouts of the various register blocks for core dumps.  This
>>series introduces a new x86-specific core dump note intended to supplant these
>>heuristics by storing the raw CPUID leaves describing the XSAVE layout in core
>>dumps.
>>
>>This series proposes a new core dump note, NT_X86_CPUID (0x205), which
>>contains an array of structures.  Each structure describes an invidual CPUID sub-
>>leaf containing both the inputs to CPUID (%eax and %ecx) and the outputs
>>(%eax, %ebx, %ecx, and %edx) in a format roughly matching the follow C
>>structure:
>>
>>struct cpuid_leaf
>>{
>>    uint32_t leaf;
>>    uint32_t subleaf;
>>    uint32_t eax;
>>    uint32_t ebx;
>>    uint32_t ecx;
>>    uint32_t edx;
>>};
>>
>>This format is not XSAVE-specific and implementations could choose to add
>>additional CPUID leaves to this structure if needed in the future.
>>Consumers of this note should lookup the value of required leaves and ignore
>>any unneeded leaves.
>>
>>An alternate approach might be to write out a more XSAVE-specific note that is
>>an array containing the offset and size of each XSAVE region.
>>
>>Note that either approach would enable storing XSAVE notes in the "compact"
>>format at some point in the future.
>>
>>This series adds support for reading/writing the note to binutils as well as suport
>>for parsing and generating the note in GDB.  It also hooks this into both the
>>FreeBSD and Linux x86 architectures in GDB to read the XSAVE layout from this
>>note when present, and to write out a note when generating a core via `gcore'.
>>I've done some limited testing on FreeBSD/amd64 and Linux/x86-64, but it could
>>probably use some more testing on Linux in particular.  (I know Simon has an
>>AMD machine with a layout not handled by the current heuristics for
>>example.)
>>
>>For the gcore side, a new TARGET_OBJECT_X86_CPUID is used to fetch the
>>current note contents from a native target.  There is still one gap even with this
>>patch series which is that if you are connected to a remote target (e.g.
>>gdbserver), we currently do not have a known XSAVE layout to use when writing
>>out a core via `gcore'.  One option that would close this gap would be to extend
>>the remote protocol to permit reading this new object from a debug server.  The
>>remote target could then implement fetching this object and also make use of
>>this object to implement the target::fetch_x86_xsave_layout method which
>>would close that gap.  Another possibility would be to just pick a "known"
>>XSAVE format that matches one of the heuristics.
>>
>>The series is available from git@github.com:bsdjhb/gdb.git on the
>>`nt_x86_cpuid' branch.
>>
>>I also have an implementation of this core dump note available for FreeBSD's
>>kernel, though I won't merge it until we've collectively settled on the format:
>>https://reviews.freebsd.org/D42136
>>
>>Things I have not done and could use help with:
>>
>>- Implementation for the Linux kernel
>>
>>- Coordination with folks from LLDB
>>
>>John Baldwin (13):
>>  binutils: Support for the NT_X86_CPUID core dump note
>>  i387-tdep: Add function to read XSAVE layout from NT_X86_CPUID
>>  gdb: Use NT_X86_CPUID in x86 FreeBSD architectures to read XSAVE
>>    layouts
>>  gdb: Use NT_X86_CPUID in x86 FreeBSD architectures to read XSAVE
>>    layouts
>>  nat/x86-cpuid.h: Remove non-x86 fallbacks
>>  nat/x86-cpuid: Add a function to build the contents of a NT_X86_CPUID
>>    note
>>  x86_elf_make_cpuid_note: Helper routine to build NT_X86_CPUID ELF note
>>  x86-fbsd-nat: Support fetching TARGET_OBJECT_X86_CPUID objects
>>  fbsd-tdep: Export fbsd_make_corefile_notes
>>  {amd64,i386}-fbsd-tdep: Include NT_X86_CPUID notes in core dumps from
>>    gcore
>>  x86-linux-nat: Support fetching TARGET_OBJECT_X86_CPUID objects
>>  linux-tdep: Export linux_make_corefile_notes
>>  {amd64,i386}-linux-tdep: Include NT_X86_CPUID notes in core dumps from
>>    gcore
>>
>> bfd/elf-bfd.h          |   2 +
>> bfd/elf.c              |  35 +++++++++++
>> binutils/readelf.c     |   2 +
>> gdb/amd64-fbsd-tdep.c  |   1 +
>> gdb/amd64-linux-tdep.c |   1 +
>> gdb/configure.nat      |  13 ++--
>> gdb/fbsd-tdep.c        |   5 +-
>> gdb/fbsd-tdep.h        |   7 +++
>> gdb/i386-fbsd-tdep.c   |  18 +++++-
>> gdb/i386-fbsd-tdep.h   |   7 +++
>> gdb/i386-linux-tdep.c  |  18 +++++-
>> gdb/i386-linux-tdep.h  |   7 +++
>> gdb/i387-tdep.c        | 132 +++++++++++++++++++++++++++++++++++++++++
>> gdb/i387-tdep.h        |   8 +++
>> gdb/linux-tdep.c       |   5 +-
>> gdb/linux-tdep.h       |   7 +++
>> gdb/nat/x86-cpuid.c    |  91 ++++++++++++++++++++++++++++
>> gdb/nat/x86-cpuid.h    |  29 +++------
>> gdb/target.h           |   2 +
>> gdb/x86-fbsd-nat.c     |  37 ++++++++++++
>> gdb/x86-fbsd-nat.h     |   9 +++
>> gdb/x86-linux-nat.c    |  37 ++++++++++++
>> gdb/x86-linux-nat.h    |   9 +++
>> gdb/x86-tdep.c         |  22 +++++++
>> gdb/x86-tdep.h         |   9 +++
>> include/elf/common.h   |   2 +
>> 26 files changed, 480 insertions(+), 35 deletions(-)  create mode 100644
>>gdb/nat/x86-cpuid.c
>>
>>--
>>2.41.0
  

On 2023-10-09 14:36, John Baldwin wrote:
> One of the shortcomings of the previous XSAVE patch series is that it
> depends on heuristics based on the total XSAVE register set size and
> XCR0 mask to infer layouts of the various register blocks for core
> dumps.  This series introduces a new x86-specific core dump note
> intended to supplant these heuristics by storing the raw CPUID leaves
> describing the XSAVE layout in core dumps.
> 
> This series proposes a new core dump note, NT_X86_CPUID (0x205), which
> contains an array of structures.  Each structure describes an invidual
> CPUID sub-leaf containing both the inputs to CPUID (%eax and %ecx) and
> the outputs (%eax, %ebx, %ecx, and %edx) in a format roughly matching
> the follow C structure:
> 
> struct cpuid_leaf
> {
>     uint32_t leaf;
>     uint32_t subleaf;
>     uint32_t eax;
>     uint32_t ebx;
>     uint32_t ecx;
>     uint32_t edx;
> };
> 
> This format is not XSAVE-specific and implementations could choose to
> add additional CPUID leaves to this structure if needed in the future.
> Consumers of this note should lookup the value of required leaves and
> ignore any unneeded leaves.
> 
> An alternate approach might be to write out a more XSAVE-specific note
> that is an array containing the offset and size of each XSAVE region.
> 
> Note that either approach would enable storing XSAVE notes in the
> "compact" format at some point in the future.
> 
> This series adds support for reading/writing the note to binutils as
> well as suport for parsing and generating the note in GDB.  It also
> hooks this into both the FreeBSD and Linux x86 architectures in GDB to
> read the XSAVE layout from this note when present, and to write out a
> note when generating a core via `gcore'.  I've done some limited
> testing on FreeBSD/amd64 and Linux/x86-64, but it could probably use
> some more testing on Linux in particular.  (I know Simon has an AMD
> machine with a layout not handled by the current heuristics for
> example.)

I suppose you are referring to this one, which has support for PKRU but
where PKRU is not enabled (for a reason I don't know), leading to the
total sizes being different (832 vs 896):

   XSAVE features (0xd/0):
      XCR0 valid bit field mask               = 0x0000000000000207
         x87 state                            = true
         SSE state                            = true
         AVX state                            = true
         MPX BNDREGS                          = false
         MPX BNDCSR                           = false
         AVX-512 opmask                       = false
         AVX-512 ZMM_Hi256                    = false
         AVX-512 Hi16_ZMM                     = false
         PKRU state                           = true
         XTILECFG state                       = false
         XTILEDATA state                      = false
      bytes required by fields in XCR0        = 0x00000340 (832)
      bytes required by XSAVE/XRSTOR area     = 0x00000380 (896)

That machine's layout is handled by the heuristic since commit:

  gdb/x86: use size of XSAVE area of enabled features
  https://gitlab.com/gnutools/binutils-gdb/-/commit/054f25955c2b77f6e21073bfdd70a60e9df1ffe7

But anyhow, I can use it for testing.

> For the gcore side, a new TARGET_OBJECT_X86_CPUID is used to fetch the
> current note contents from a native target.  There is still one gap
> even with this patch series which is that if you are connected to a
> remote target (e.g. gdbserver), we currently do not have a known XSAVE
> layout to use when writing out a core via `gcore'.  One option that
> would close this gap would be to extend the remote protocol to permit
> reading this new object from a debug server.  The remote target could
> then implement fetching this object and also make use of this object
> to implement the target::fetch_x86_xsave_layout method which would
> close that gap.  Another possibility would be to just pick a "known"
> XSAVE format that matches one of the heuristics.

I think it would make sense to send the CPUID info over the wire.  Can
it be extra data in the target description XML?  Otherwise, a new
qxfer packet I suppose.

If the current GDBserver doesn't support sending the CPUID info, I'm not
sure I would make the remote target write a CPUID note based on a
heuristic.  It has pros and cons, we can talk about that when we get
there.

Simon
  

On 2023-10-09 14:36, John Baldwin wrote:
> One of the shortcomings of the previous XSAVE patch series is that it
> depends on heuristics based on the total XSAVE register set size and
> XCR0 mask to infer layouts of the various register blocks for core
> dumps.  This series introduces a new x86-specific core dump note
> intended to supplant these heuristics by storing the raw CPUID leaves
> describing the XSAVE layout in core dumps.
> 
> This series proposes a new core dump note, NT_X86_CPUID (0x205), which
> contains an array of structures.  Each structure describes an invidual
> CPUID sub-leaf containing both the inputs to CPUID (%eax and %ecx) and
> the outputs (%eax, %ebx, %ecx, and %edx) in a format roughly matching
> the follow C structure:
> 
> struct cpuid_leaf
> {
>     uint32_t leaf;
>     uint32_t subleaf;
>     uint32_t eax;
>     uint32_t ebx;
>     uint32_t ecx;
>     uint32_t edx;
> };
> 
> This format is not XSAVE-specific and implementations could choose to
> add additional CPUID leaves to this structure if needed in the future.
> Consumers of this note should lookup the value of required leaves and
> ignore any unneeded leaves.
> 
> An alternate approach might be to write out a more XSAVE-specific note
> that is an array containing the offset and size of each XSAVE region.

Something I thought about: I think there are asymmetrical x86 processors
now, with "big" and "little" cores, not sure how they call them.  For
them, is it possible for the XSAVE layout to be different per core, for
instance some cores supporting AVX512 and some cores not?  And
therefore, will we eventually need to include CPUID / XSAVE information
for more than one CPU core type in the core file notes?

Simon
  

On 10/12/23 7:33 AM, Simon Marchi wrote:
> 
> 
> On 2023-10-09 14:36, John Baldwin wrote:
>> One of the shortcomings of the previous XSAVE patch series is that it
>> depends on heuristics based on the total XSAVE register set size and
>> XCR0 mask to infer layouts of the various register blocks for core
>> dumps.  This series introduces a new x86-specific core dump note
>> intended to supplant these heuristics by storing the raw CPUID leaves
>> describing the XSAVE layout in core dumps.
>>
>> This series proposes a new core dump note, NT_X86_CPUID (0x205), which
>> contains an array of structures.  Each structure describes an invidual
>> CPUID sub-leaf containing both the inputs to CPUID (%eax and %ecx) and
>> the outputs (%eax, %ebx, %ecx, and %edx) in a format roughly matching
>> the follow C structure:
>>
>> struct cpuid_leaf
>> {
>>      uint32_t leaf;
>>      uint32_t subleaf;
>>      uint32_t eax;
>>      uint32_t ebx;
>>      uint32_t ecx;
>>      uint32_t edx;
>> };
>>
>> This format is not XSAVE-specific and implementations could choose to
>> add additional CPUID leaves to this structure if needed in the future.
>> Consumers of this note should lookup the value of required leaves and
>> ignore any unneeded leaves.
>>
>> An alternate approach might be to write out a more XSAVE-specific note
>> that is an array containing the offset and size of each XSAVE region.
> 
> Something I thought about: I think there are asymmetrical x86 processors
> now, with "big" and "little" cores, not sure how they call them.  For
> them, is it possible for the XSAVE layout to be different per core, for
> instance some cores supporting AVX512 and some cores not?  And
> therefore, will we eventually need to include CPUID / XSAVE information
> for more than one CPU core type in the core file notes?

(The Intel names are "P" (performance) and "E" (energy-efficient) cores
btw)

I have no idea if they use the same or different XSAVE layouts.  It would
seem to be a royal pain if they did as everytime a user thread migrated
between cores the OS would have to convert the XSAVE block from one
layout to the other.  FreeBSD does not have any notion of multiple
layouts today and just assumes that the XSAVE layout is uniform across
all cores in a system.  I believe Linux's kernel does the same from my
reading.
  

[AMD Official Use Only - General]

I think that even if the xsave layout remains uniform across the cores of a system, since we are trying to design an extensible .note section which can possibly hold all kinds of CPUID information, we might want to consider various scenarios wherein the CPUID information might differ across cores (esp for big.LITTLE/(P/E)), like the cache information, perhaps ? It might be more prudent to include the coretype information also in such cases ?

Rgds
Jini.

>>-----Original Message-----
>>From: John Baldwin <jhb@FreeBSD.org>
>>Sent: Thursday, October 12, 2023 10:48 PM
>>To: Simon Marchi <simon.marchi@polymtl.ca>; gdb-patches@sourceware.org
>>Cc: Felix <felix.willgerodt@intel.com>; George, Jini Susan
>><JiniSusan.George@amd.com>
>>Subject: Re: [RFC 00/13] Proposal for a new NT_X86_CPUID core dump note
>>
>>Caution: This message originated from an External Source. Use proper caution
>>when opening attachments, clicking links, or responding.
>>
>>
>>On 10/12/23 7:33 AM, Simon Marchi wrote:
>>>
>>>
>>> On 2023-10-09 14:36, John Baldwin wrote:
>>>> One of the shortcomings of the previous XSAVE patch series is that it
>>>> depends on heuristics based on the total XSAVE register set size and
>>>> XCR0 mask to infer layouts of the various register blocks for core
>>>> dumps.  This series introduces a new x86-specific core dump note
>>>> intended to supplant these heuristics by storing the raw CPUID leaves
>>>> describing the XSAVE layout in core dumps.
>>>>
>>>> This series proposes a new core dump note, NT_X86_CPUID (0x205),
>>>> which contains an array of structures.  Each structure describes an
>>>> invidual CPUID sub-leaf containing both the inputs to CPUID (%eax and
>>>> %ecx) and the outputs (%eax, %ebx, %ecx, and %edx) in a format
>>>> roughly matching the follow C structure:
>>>>
>>>> struct cpuid_leaf
>>>> {
>>>>      uint32_t leaf;
>>>>      uint32_t subleaf;
>>>>      uint32_t eax;
>>>>      uint32_t ebx;
>>>>      uint32_t ecx;
>>>>      uint32_t edx;
>>>> };
>>>>
>>>> This format is not XSAVE-specific and implementations could choose to
>>>> add additional CPUID leaves to this structure if needed in the future.
>>>> Consumers of this note should lookup the value of required leaves and
>>>> ignore any unneeded leaves.
>>>>
>>>> An alternate approach might be to write out a more XSAVE-specific
>>>> note that is an array containing the offset and size of each XSAVE region.
>>>
>>> Something I thought about: I think there are asymmetrical x86
>>> processors now, with "big" and "little" cores, not sure how they call
>>> them.  For them, is it possible for the XSAVE layout to be different
>>> per core, for instance some cores supporting AVX512 and some cores
>>> not?  And therefore, will we eventually need to include CPUID / XSAVE
>>> information for more than one CPU core type in the core file notes?
>>
>>(The Intel names are "P" (performance) and "E" (energy-efficient) cores
>>btw)
>>
>>I have no idea if they use the same or different XSAVE layouts.  It would seem to
>>be a royal pain if they did as everytime a user thread migrated between cores
>>the OS would have to convert the XSAVE block from one layout to the other.
>>FreeBSD does not have any notion of multiple layouts today and just assumes
>>that the XSAVE layout is uniform across all cores in a system.  I believe Linux's
>>kernel does the same from my reading.
>>
>>--
>>John Baldwin
  

On 10/13/23 2:38 AM, George, Jini Susan wrote:
> [AMD Official Use Only - General]
> 
> I think that even if the xsave layout remains uniform across the cores of a system, since we are trying to design an extensible .note section which can possibly hold all kinds of CPUID information, we might want to consider various scenarios wherein the CPUID information might differ across cores (esp for big.LITTLE/(P/E)), like the cache information, perhaps ? It might be more prudent to include the coretype information also in such cases ?

It's certainly occurred to me that it might be prudent to include the full
complement of CPUID leaves even in the initial version of this note should
that information prove useful in the future.

However, I'm not quite sure how we should "name" the different CPUID leaf
sets in this case (e.g. the set for E cores vs the set for P cores).  In
particular, it's not quite clear to me how many sets future systems might
have?  One route is to use additional NT_ values for different sets,
(e.g. a NT_X86_CPUID_E_CORE or some such if the "default" set is for the
P core).  We could also embed an identifier at the start of a note, e.g.
a 32-bit integer, and dump one note per set, so you have always have a note
for set 0, but in a system with E cores you might have a set 1 where the
convention would be P cores are set 0, and E cores are set 1.  Future
systems with multiple core types would have to decide what type of mapping
to use between core types and set IDs.  One issue with the second approach
is if you want these notes accessible via ptrace() and not just in cores,
it wouldn't be clear how to request the leaves for set N if they all share
the NT_* value.

> Rgds
> Jini.
> 
>>> -----Original Message-----
>>> From: John Baldwin <jhb@FreeBSD.org>
>>> Sent: Thursday, October 12, 2023 10:48 PM
>>> To: Simon Marchi <simon.marchi@polymtl.ca>; gdb-patches@sourceware.org
>>> Cc: Felix <felix.willgerodt@intel.com>; George, Jini Susan
>>> <JiniSusan.George@amd.com>
>>> Subject: Re: [RFC 00/13] Proposal for a new NT_X86_CPUID core dump note
>>>
>>> Caution: This message originated from an External Source. Use proper caution
>>> when opening attachments, clicking links, or responding.
>>>
>>>
>>> On 10/12/23 7:33 AM, Simon Marchi wrote:
>>>>
>>>>
>>>> On 2023-10-09 14:36, John Baldwin wrote:
>>>>> One of the shortcomings of the previous XSAVE patch series is that it
>>>>> depends on heuristics based on the total XSAVE register set size and
>>>>> XCR0 mask to infer layouts of the various register blocks for core
>>>>> dumps.  This series introduces a new x86-specific core dump note
>>>>> intended to supplant these heuristics by storing the raw CPUID leaves
>>>>> describing the XSAVE layout in core dumps.
>>>>>
>>>>> This series proposes a new core dump note, NT_X86_CPUID (0x205),
>>>>> which contains an array of structures.  Each structure describes an
>>>>> invidual CPUID sub-leaf containing both the inputs to CPUID (%eax and
>>>>> %ecx) and the outputs (%eax, %ebx, %ecx, and %edx) in a format
>>>>> roughly matching the follow C structure:
>>>>>
>>>>> struct cpuid_leaf
>>>>> {
>>>>>       uint32_t leaf;
>>>>>       uint32_t subleaf;
>>>>>       uint32_t eax;
>>>>>       uint32_t ebx;
>>>>>       uint32_t ecx;
>>>>>       uint32_t edx;
>>>>> };
>>>>>
>>>>> This format is not XSAVE-specific and implementations could choose to
>>>>> add additional CPUID leaves to this structure if needed in the future.
>>>>> Consumers of this note should lookup the value of required leaves and
>>>>> ignore any unneeded leaves.
>>>>>
>>>>> An alternate approach might be to write out a more XSAVE-specific
>>>>> note that is an array containing the offset and size of each XSAVE region.
>>>>
>>>> Something I thought about: I think there are asymmetrical x86
>>>> processors now, with "big" and "little" cores, not sure how they call
>>>> them.  For them, is it possible for the XSAVE layout to be different
>>>> per core, for instance some cores supporting AVX512 and some cores
>>>> not?  And therefore, will we eventually need to include CPUID / XSAVE
>>>> information for more than one CPU core type in the core file notes?
>>>
>>> (The Intel names are "P" (performance) and "E" (energy-efficient) cores
>>> btw)
>>>
>>> I have no idea if they use the same or different XSAVE layouts.  It would seem to
>>> be a royal pain if they did as everytime a user thread migrated between cores
>>> the OS would have to convert the XSAVE block from one layout to the other.
>>> FreeBSD does not have any notion of multiple layouts today and just assumes
>>> that the XSAVE layout is uniform across all cores in a system.  I believe Linux's
>>> kernel does the same from my reading.
>>>
>>> --
>>> John Baldwin
>
  

[AMD Official Use Only - General]

I think it might be better to go with different NT_ values for the P and the E cores. We will also need to put in additional mapping information as to which core/ cpu id belongs to which core set (P or E).

Thanks
Jini.

>>-----Original Message-----
>>From: John Baldwin <jhb@FreeBSD.org>
>>Sent: Tuesday, October 17, 2023 6:06 AM
>>To: George, Jini Susan <JiniSusan.George@amd.com>; Simon Marchi
>><simon.marchi@polymtl.ca>; gdb-patches@sourceware.org
>>Cc: Felix <felix.willgerodt@intel.com>; Balasubrmanian, Vignesh
>><Vignesh.Balasubrmanian@amd.com>
>>Subject: Re: [RFC 00/13] Proposal for a new NT_X86_CPUID core dump note
>>
>>Caution: This message originated from an External Source. Use proper caution
>>when opening attachments, clicking links, or responding.
>>
>>
>>On 10/13/23 2:38 AM, George, Jini Susan wrote:
>>> [AMD Official Use Only - General]
>>>
>>> I think that even if the xsave layout remains uniform across the cores of a
>>system, since we are trying to design an extensible .note section which can
>>possibly hold all kinds of CPUID information, we might want to consider various
>>scenarios wherein the CPUID information might differ across cores (esp for
>>big.LITTLE/(P/E)), like the cache information, perhaps ? It might be more prudent
>>to include the coretype information also in such cases ?
>>
>>It's certainly occurred to me that it might be prudent to include the full
>>complement of CPUID leaves even in the initial version of this note should that
>>information prove useful in the future.
>>
>>However, I'm not quite sure how we should "name" the different CPUID leaf sets
>>in this case (e.g. the set for E cores vs the set for P cores).  In particular, it's not
>>quite clear to me how many sets future systems might have?  One route is to use
>>additional NT_ values for different sets, (e.g. a NT_X86_CPUID_E_CORE or some
>>such if the "default" set is for the P core).  We could also embed an identifier at
>>the start of a note, e.g.
>>a 32-bit integer, and dump one note per set, so you have always have a note for
>>set 0, but in a system with E cores you might have a set 1 where the convention
>>would be P cores are set 0, and E cores are set 1.  Future systems with multiple
>>core types would have to decide what type of mapping to use between core
>>types and set IDs.  One issue with the second approach is if you want these
>>notes accessible via ptrace() and not just in cores, it wouldn't be clear how to
>>request the leaves for set N if they all share the NT_* value.
>>
>>> Rgds
>>> Jini.
>>>
>>>>> -----Original Message-----
>>>>> From: John Baldwin <jhb@FreeBSD.org>
>>>>> Sent: Thursday, October 12, 2023 10:48 PM
>>>>> To: Simon Marchi <simon.marchi@polymtl.ca>;
>>>>> gdb-patches@sourceware.org
>>>>> Cc: Felix <felix.willgerodt@intel.com>; George, Jini Susan
>>>>> <JiniSusan.George@amd.com>
>>>>> Subject: Re: [RFC 00/13] Proposal for a new NT_X86_CPUID core dump
>>>>> note
>>>>>
>>>>> Caution: This message originated from an External Source. Use proper
>>>>> caution when opening attachments, clicking links, or responding.
>>>>>
>>>>>
>>>>> On 10/12/23 7:33 AM, Simon Marchi wrote:
>>>>>>
>>>>>>
>>>>>> On 2023-10-09 14:36, John Baldwin wrote:
>>>>>>> One of the shortcomings of the previous XSAVE patch series is that
>>>>>>> it depends on heuristics based on the total XSAVE register set
>>>>>>> size and
>>>>>>> XCR0 mask to infer layouts of the various register blocks for core
>>>>>>> dumps.  This series introduces a new x86-specific core dump note
>>>>>>> intended to supplant these heuristics by storing the raw CPUID
>>>>>>> leaves describing the XSAVE layout in core dumps.
>>>>>>>
>>>>>>> This series proposes a new core dump note, NT_X86_CPUID (0x205),
>>>>>>> which contains an array of structures.  Each structure describes
>>>>>>> an invidual CPUID sub-leaf containing both the inputs to CPUID
>>>>>>> (%eax and
>>>>>>> %ecx) and the outputs (%eax, %ebx, %ecx, and %edx) in a format
>>>>>>> roughly matching the follow C structure:
>>>>>>>
>>>>>>> struct cpuid_leaf
>>>>>>> {
>>>>>>>       uint32_t leaf;
>>>>>>>       uint32_t subleaf;
>>>>>>>       uint32_t eax;
>>>>>>>       uint32_t ebx;
>>>>>>>       uint32_t ecx;
>>>>>>>       uint32_t edx;
>>>>>>> };
>>>>>>>
>>>>>>> This format is not XSAVE-specific and implementations could choose
>>>>>>> to add additional CPUID leaves to this structure if needed in the future.
>>>>>>> Consumers of this note should lookup the value of required leaves
>>>>>>> and ignore any unneeded leaves.
>>>>>>>
>>>>>>> An alternate approach might be to write out a more XSAVE-specific
>>>>>>> note that is an array containing the offset and size of each XSAVE region.
>>>>>>
>>>>>> Something I thought about: I think there are asymmetrical x86
>>>>>> processors now, with "big" and "little" cores, not sure how they
>>>>>> call them.  For them, is it possible for the XSAVE layout to be
>>>>>> different per core, for instance some cores supporting AVX512 and
>>>>>> some cores not?  And therefore, will we eventually need to include
>>>>>> CPUID / XSAVE information for more than one CPU core type in the core
>>file notes?
>>>>>
>>>>> (The Intel names are "P" (performance) and "E" (energy-efficient)
>>>>> cores
>>>>> btw)
>>>>>
>>>>> I have no idea if they use the same or different XSAVE layouts.  It
>>>>> would seem to be a royal pain if they did as everytime a user thread
>>>>> migrated between cores the OS would have to convert the XSAVE block
>>from one layout to the other.
>>>>> FreeBSD does not have any notion of multiple layouts today and just
>>>>> assumes that the XSAVE layout is uniform across all cores in a
>>>>> system.  I believe Linux's kernel does the same from my reading.
>>>>>
>>>>> --
>>>>> John Baldwin
>>>
>>
>>--
>>John Baldwin
  

On 10/26/23 9:18 AM, George, Jini Susan wrote:
> [AMD Official Use Only - General]
> 
> I think it might be better to go with different NT_ values for the P and the E cores. We will also need to put in additional mapping information as to which core/ cpu id belongs to which core set (P or E).

Hmm, do you mean mapping thread (LWP) IDs to specific core sets?
Right now core dumps don't really include the mapping of threads
to individual CPU cores, in part because threads can migrate
between cores (subject to administrative and application constraints).
I'm not really sure there is a use case for storing that information.

One way you could store that would be to have a per-thread note
storing the logical CPU mask (or cpuset, not sure which term Linux
uses, FreeBSD uses cpuset) for each thread, but then you'd need
perhaps some other note mapping logical CPU IDs to other CPU data.
(For example, if you did have a separate NT_X86_CPUID_E_CORE, you
could have some other note that was a short table mapping CPU IDs
to CPUID notes (e.g. cpu 0 is NT_X86_CPUID, cpu 1 is NT_X86_CPUID,
cpu 2 is NT_X86_CPUID_E_CORE).

However, I'm not sure it makes sense to store all of that now without
a use case.  I do think though that this does show there are ways to
extend the information in the future should a use case arrive.

> Thanks
> Jini.
> 
>>> -----Original Message-----
>>> From: John Baldwin <jhb@FreeBSD.org>
>>> Sent: Tuesday, October 17, 2023 6:06 AM
>>> To: George, Jini Susan <JiniSusan.George@amd.com>; Simon Marchi
>>> <simon.marchi@polymtl.ca>; gdb-patches@sourceware.org
>>> Cc: Felix <felix.willgerodt@intel.com>; Balasubrmanian, Vignesh
>>> <Vignesh.Balasubrmanian@amd.com>
>>> Subject: Re: [RFC 00/13] Proposal for a new NT_X86_CPUID core dump note
>>>
>>> Caution: This message originated from an External Source. Use proper caution
>>> when opening attachments, clicking links, or responding.
>>>
>>>
>>> On 10/13/23 2:38 AM, George, Jini Susan wrote:
>>>> [AMD Official Use Only - General]
>>>>
>>>> I think that even if the xsave layout remains uniform across the cores of a
>>> system, since we are trying to design an extensible .note section which can
>>> possibly hold all kinds of CPUID information, we might want to consider various
>>> scenarios wherein the CPUID information might differ across cores (esp for
>>> big.LITTLE/(P/E)), like the cache information, perhaps ? It might be more prudent
>>> to include the coretype information also in such cases ?
>>>
>>> It's certainly occurred to me that it might be prudent to include the full
>>> complement of CPUID leaves even in the initial version of this note should that
>>> information prove useful in the future.
>>>
>>> However, I'm not quite sure how we should "name" the different CPUID leaf sets
>>> in this case (e.g. the set for E cores vs the set for P cores).  In particular, it's not
>>> quite clear to me how many sets future systems might have?  One route is to use
>>> additional NT_ values for different sets, (e.g. a NT_X86_CPUID_E_CORE or some
>>> such if the "default" set is for the P core).  We could also embed an identifier at
>>> the start of a note, e.g.
>>> a 32-bit integer, and dump one note per set, so you have always have a note for
>>> set 0, but in a system with E cores you might have a set 1 where the convention
>>> would be P cores are set 0, and E cores are set 1.  Future systems with multiple
>>> core types would have to decide what type of mapping to use between core
>>> types and set IDs.  One issue with the second approach is if you want these
>>> notes accessible via ptrace() and not just in cores, it wouldn't be clear how to
>>> request the leaves for set N if they all share the NT_* value.
>>>
>>>> Rgds
>>>> Jini.
>>>>
>>>>>> -----Original Message-----
>>>>>> From: John Baldwin <jhb@FreeBSD.org>
>>>>>> Sent: Thursday, October 12, 2023 10:48 PM
>>>>>> To: Simon Marchi <simon.marchi@polymtl.ca>;
>>>>>> gdb-patches@sourceware.org
>>>>>> Cc: Felix <felix.willgerodt@intel.com>; George, Jini Susan
>>>>>> <JiniSusan.George@amd.com>
>>>>>> Subject: Re: [RFC 00/13] Proposal for a new NT_X86_CPUID core dump
>>>>>> note
>>>>>>
>>>>>> Caution: This message originated from an External Source. Use proper
>>>>>> caution when opening attachments, clicking links, or responding.
>>>>>>
>>>>>>
>>>>>> On 10/12/23 7:33 AM, Simon Marchi wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2023-10-09 14:36, John Baldwin wrote:
>>>>>>>> One of the shortcomings of the previous XSAVE patch series is that
>>>>>>>> it depends on heuristics based on the total XSAVE register set
>>>>>>>> size and
>>>>>>>> XCR0 mask to infer layouts of the various register blocks for core
>>>>>>>> dumps.  This series introduces a new x86-specific core dump note
>>>>>>>> intended to supplant these heuristics by storing the raw CPUID
>>>>>>>> leaves describing the XSAVE layout in core dumps.
>>>>>>>>
>>>>>>>> This series proposes a new core dump note, NT_X86_CPUID (0x205),
>>>>>>>> which contains an array of structures.  Each structure describes
>>>>>>>> an invidual CPUID sub-leaf containing both the inputs to CPUID
>>>>>>>> (%eax and
>>>>>>>> %ecx) and the outputs (%eax, %ebx, %ecx, and %edx) in a format
>>>>>>>> roughly matching the follow C structure:
>>>>>>>>
>>>>>>>> struct cpuid_leaf
>>>>>>>> {
>>>>>>>>        uint32_t leaf;
>>>>>>>>        uint32_t subleaf;
>>>>>>>>        uint32_t eax;
>>>>>>>>        uint32_t ebx;
>>>>>>>>        uint32_t ecx;
>>>>>>>>        uint32_t edx;
>>>>>>>> };
>>>>>>>>
>>>>>>>> This format is not XSAVE-specific and implementations could choose
>>>>>>>> to add additional CPUID leaves to this structure if needed in the future.
>>>>>>>> Consumers of this note should lookup the value of required leaves
>>>>>>>> and ignore any unneeded leaves.
>>>>>>>>
>>>>>>>> An alternate approach might be to write out a more XSAVE-specific
>>>>>>>> note that is an array containing the offset and size of each XSAVE region.
>>>>>>>
>>>>>>> Something I thought about: I think there are asymmetrical x86
>>>>>>> processors now, with "big" and "little" cores, not sure how they
>>>>>>> call them.  For them, is it possible for the XSAVE layout to be
>>>>>>> different per core, for instance some cores supporting AVX512 and
>>>>>>> some cores not?  And therefore, will we eventually need to include
>>>>>>> CPUID / XSAVE information for more than one CPU core type in the core
>>> file notes?
>>>>>>
>>>>>> (The Intel names are "P" (performance) and "E" (energy-efficient)
>>>>>> cores
>>>>>> btw)
>>>>>>
>>>>>> I have no idea if they use the same or different XSAVE layouts.  It
>>>>>> would seem to be a royal pain if they did as everytime a user thread
>>>>>> migrated between cores the OS would have to convert the XSAVE block
>> >from one layout to the other.
>>>>>> FreeBSD does not have any notion of multiple layouts today and just
>>>>>> assumes that the XSAVE layout is uniform across all cores in a
>>>>>> system.  I believe Linux's kernel does the same from my reading.
>>>>>>
>>>>>> --
>>>>>> John Baldwin
>>>>
>>>
>>> --
>>> John Baldwin
>
  

[Public]

If tomorrow, we have to deal with varying xsave layouts/ cache or any other information for different coresets, then we would need the mapping from LWPIDs to coresets.

>One way you could store that would be to have a per-thread note storing the
>logical CPU mask (or cpuset, not sure which term Linux uses, FreeBSD uses
>cpuset) for each thread, but then you'd need perhaps some other note mapping
>logical CPU IDs to other CPU data.
>(For example, if you did have a separate NT_X86_CPUID_E_CORE, you could
>have some other note that was a short table mapping CPU IDs to CPUID notes
>(e.g. cpu 0 is NT_X86_CPUID, cpu 1 is NT_X86_CPUID, cpu 2 is
>NT_X86_CPUID_E_CORE).

I was thinking of having the per-thread note storing the last executed cpu for the thread -- not sure if the cpuset will be needed, but I could be missing something. But yes, we will need 3 extra note types for P/E scenarios. (the per process E note, the per process cpu ID to coreset mapping note and the per thread last executed cpu/ cpuset note).

So, yes, since we can add other required information as need be as extra notes in the future, I feel none of this should hinder the current patch with going through.

Thanks,
Jini.

>>-----Original Message-----
>>From: John Baldwin <jhb@FreeBSD.org>
>>Sent: Friday, October 27, 2023 8:23 AM
>>To: George, Jini Susan <JiniSusan.George@amd.com>; Simon Marchi
>><simon.marchi@polymtl.ca>; gdb-patches@sourceware.org
>>Cc: Felix <felix.willgerodt@intel.com>; Balasubrmanian, Vignesh
>><Vignesh.Balasubrmanian@amd.com>
>>Subject: Re: [RFC 00/13] Proposal for a new NT_X86_CPUID core dump note
>>
>>Caution: This message originated from an External Source. Use proper caution
>>when opening attachments, clicking links, or responding.
>>
>>
>>On 10/26/23 9:18 AM, George, Jini Susan wrote:
>>> [AMD Official Use Only - General]
>>>
>>> I think it might be better to go with different NT_ values for the P and the E
>>cores. We will also need to put in additional mapping information as to which
>>core/ cpu id belongs to which core set (P or E).
>>
>>Hmm, do you mean mapping thread (LWP) IDs to specific core sets?
>>Right now core dumps don't really include the mapping of threads to individual
>>CPU cores, in part because threads can migrate between cores (subject to
>>administrative and application constraints).
>>I'm not really sure there is a use case for storing that information.
>>
>>One way you could store that would be to have a per-thread note storing the
>>logical CPU mask (or cpuset, not sure which term Linux uses, FreeBSD uses
>>cpuset) for each thread, but then you'd need perhaps some other note mapping
>>logical CPU IDs to other CPU data.
>>(For example, if you did have a separate NT_X86_CPUID_E_CORE, you could
>>have some other note that was a short table mapping CPU IDs to CPUID notes
>>(e.g. cpu 0 is NT_X86_CPUID, cpu 1 is NT_X86_CPUID, cpu 2 is
>>NT_X86_CPUID_E_CORE).
>>
>>However, I'm not sure it makes sense to store all of that now without a use
>>case.  I do think though that this does show there are ways to extend the
>>information in the future should a use case arrive.
>>
>>> Thanks
>>> Jini.
>>>
>>>>> -----Original Message-----
>>>>> From: John Baldwin <jhb@FreeBSD.org>
>>>>> Sent: Tuesday, October 17, 2023 6:06 AM
>>>>> To: George, Jini Susan <JiniSusan.George@amd.com>; Simon Marchi
>>>>> <simon.marchi@polymtl.ca>; gdb-patches@sourceware.org
>>>>> Cc: Felix <felix.willgerodt@intel.com>; Balasubrmanian, Vignesh
>>>>> <Vignesh.Balasubrmanian@amd.com>
>>>>> Subject: Re: [RFC 00/13] Proposal for a new NT_X86_CPUID core dump
>>>>> note
>>>>>
>>>>> Caution: This message originated from an External Source. Use proper
>>>>> caution when opening attachments, clicking links, or responding.
>>>>>
>>>>>
>>>>> On 10/13/23 2:38 AM, George, Jini Susan wrote:
>>>>>> [AMD Official Use Only - General]
>>>>>>
>>>>>> I think that even if the xsave layout remains uniform across the
>>>>>> cores of a
>>>>> system, since we are trying to design an extensible .note section
>>>>> which can possibly hold all kinds of CPUID information, we might
>>>>> want to consider various scenarios wherein the CPUID information
>>>>> might differ across cores (esp for big.LITTLE/(P/E)), like the cache
>>>>> information, perhaps ? It might be more prudent to include the coretype
>>information also in such cases ?
>>>>>
>>>>> It's certainly occurred to me that it might be prudent to include
>>>>> the full complement of CPUID leaves even in the initial version of
>>>>> this note should that information prove useful in the future.
>>>>>
>>>>> However, I'm not quite sure how we should "name" the different CPUID
>>>>> leaf sets in this case (e.g. the set for E cores vs the set for P
>>>>> cores).  In particular, it's not quite clear to me how many sets
>>>>> future systems might have?  One route is to use additional NT_
>>>>> values for different sets, (e.g. a NT_X86_CPUID_E_CORE or some such
>>>>> if the "default" set is for the P core).  We could also embed an identifier at
>>the start of a note, e.g.
>>>>> a 32-bit integer, and dump one note per set, so you have always have
>>>>> a note for set 0, but in a system with E cores you might have a set
>>>>> 1 where the convention would be P cores are set 0, and E cores are
>>>>> set 1.  Future systems with multiple core types would have to decide
>>>>> what type of mapping to use between core types and set IDs.  One
>>>>> issue with the second approach is if you want these notes accessible
>>>>> via ptrace() and not just in cores, it wouldn't be clear how to request the
>>leaves for set N if they all share the NT_* value.
>>>>>
>>>>>> Rgds
>>>>>> Jini.
>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: John Baldwin <jhb@FreeBSD.org>
>>>>>>>> Sent: Thursday, October 12, 2023 10:48 PM
>>>>>>>> To: Simon Marchi <simon.marchi@polymtl.ca>;
>>>>>>>> gdb-patches@sourceware.org
>>>>>>>> Cc: Felix <felix.willgerodt@intel.com>; George, Jini Susan
>>>>>>>> <JiniSusan.George@amd.com>
>>>>>>>> Subject: Re: [RFC 00/13] Proposal for a new NT_X86_CPUID core
>>>>>>>> dump note
>>>>>>>>
>>>>>>>> Caution: This message originated from an External Source. Use
>>>>>>>> proper caution when opening attachments, clicking links, or responding.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/12/23 7:33 AM, Simon Marchi wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2023-10-09 14:36, John Baldwin wrote:
>>>>>>>>>> One of the shortcomings of the previous XSAVE patch series is
>>>>>>>>>> that it depends on heuristics based on the total XSAVE register
>>>>>>>>>> set size and
>>>>>>>>>> XCR0 mask to infer layouts of the various register blocks for
>>>>>>>>>> core dumps.  This series introduces a new x86-specific core
>>>>>>>>>> dump note intended to supplant these heuristics by storing the
>>>>>>>>>> raw CPUID leaves describing the XSAVE layout in core dumps.
>>>>>>>>>>
>>>>>>>>>> This series proposes a new core dump note, NT_X86_CPUID
>>>>>>>>>> (0x205), which contains an array of structures.  Each structure
>>>>>>>>>> describes an invidual CPUID sub-leaf containing both the inputs
>>>>>>>>>> to CPUID (%eax and
>>>>>>>>>> %ecx) and the outputs (%eax, %ebx, %ecx, and %edx) in a format
>>>>>>>>>> roughly matching the follow C structure:
>>>>>>>>>>
>>>>>>>>>> struct cpuid_leaf
>>>>>>>>>> {
>>>>>>>>>>        uint32_t leaf;
>>>>>>>>>>        uint32_t subleaf;
>>>>>>>>>>        uint32_t eax;
>>>>>>>>>>        uint32_t ebx;
>>>>>>>>>>        uint32_t ecx;
>>>>>>>>>>        uint32_t edx;
>>>>>>>>>> };
>>>>>>>>>>
>>>>>>>>>> This format is not XSAVE-specific and implementations could
>>>>>>>>>> choose to add additional CPUID leaves to this structure if needed in
>>the future.
>>>>>>>>>> Consumers of this note should lookup the value of required
>>>>>>>>>> leaves and ignore any unneeded leaves.
>>>>>>>>>>
>>>>>>>>>> An alternate approach might be to write out a more
>>>>>>>>>> XSAVE-specific note that is an array containing the offset and size of
>>each XSAVE region.
>>>>>>>>>
>>>>>>>>> Something I thought about: I think there are asymmetrical x86
>>>>>>>>> processors now, with "big" and "little" cores, not sure how they
>>>>>>>>> call them.  For them, is it possible for the XSAVE layout to be
>>>>>>>>> different per core, for instance some cores supporting AVX512
>>>>>>>>> and some cores not?  And therefore, will we eventually need to
>>>>>>>>> include CPUID / XSAVE information for more than one CPU core
>>>>>>>>> type in the core
>>>>> file notes?
>>>>>>>>
>>>>>>>> (The Intel names are "P" (performance) and "E" (energy-efficient)
>>>>>>>> cores
>>>>>>>> btw)
>>>>>>>>
>>>>>>>> I have no idea if they use the same or different XSAVE layouts.
>>>>>>>> It would seem to be a royal pain if they did as everytime a user
>>>>>>>> thread migrated between cores the OS would have to convert the
>>>>>>>> XSAVE block
>>>> >from one layout to the other.
>>>>>>>> FreeBSD does not have any notion of multiple layouts today and
>>>>>>>> just assumes that the XSAVE layout is uniform across all cores in
>>>>>>>> a system.  I believe Linux's kernel does the same from my reading.
>>>>>>>>
>>>>>>>> --
>>>>>>>> John Baldwin
>>>>>>
>>>>>
>>>>> --
>>>>> John Baldwin
>>>
>>
>>--
>>John Baldwin
  

On 10/27/23 4:11 AM, George, Jini Susan wrote:
> [Public]
> 
> If tomorrow, we have to deal with varying xsave layouts/ cache or any other information for different coresets, then we would need the mapping from LWPIDs to coresets.
> 
>> One way you could store that would be to have a per-thread note storing the
>> logical CPU mask (or cpuset, not sure which term Linux uses, FreeBSD uses
>> cpuset) for each thread, but then you'd need perhaps some other note mapping
>> logical CPU IDs to other CPU data.
>> (For example, if you did have a separate NT_X86_CPUID_E_CORE, you could
>> have some other note that was a short table mapping CPU IDs to CPUID notes
>> (e.g. cpu 0 is NT_X86_CPUID, cpu 1 is NT_X86_CPUID, cpu 2 is
>> NT_X86_CPUID_E_CORE).
> 
> I was thinking of having the per-thread note storing the last executed cpu for the thread -- not sure if the cpuset will be needed, but I could be missing something. But yes, we will need 3 extra note types for P/E scenarios. (the per process E note, the per process cpu ID to coreset mapping note and the per thread last executed cpu/ cpuset note).
> 
> So, yes, since we can add other required information as need be as extra notes in the future, I feel none of this should hinder the current patch with going through.

Ok, sounds good.  Please cc me once you post the Linux kernel patches.

I opened a thread over on the LLVM discourse with the LLDB folks on this topic and
the only feedback so far has been positive:

https://discourse.llvm.org/t/proposal-for-a-new-nt-x86-cpuid-core-dump-note/74458