RFC: x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ #21265]
Commit Message
On Thu, Oct 19, 2017 at 2:55 PM, Carlos O'Donell <carlos@redhat.com> wrote:
> On 10/19/2017 10:41 AM, H.J. Lu wrote:
>> In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
>> mask and bound registers. It simplifies _dl_runtime_resolve and supports
>> different calling conventions. ld.so code size is reduced by more than
>> 1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
>> than saving and restoring vector and bound registers individually.
>>
>> Latency for _dl_runtime_resolve to lookup the function, foo, from one
>> shared library plus libc.so:
>>
>> Before After Change
>>
>> Westmere (SSE)/fxsave 345 866 151%
>> IvyBridge (AVX)/xsave 420 643 53%
>> Haswell (AVX)/xsave 713 1252 75%
>> Skylake (AVX+MPX)/xsavec 559 719 28%
>> Skylake (AVX512+MPX)/xsavec 145 272 87%
>
> This is a good baseline, but as you note, the change may not be observable
> in any real world programs.
>
> The case I made to David Kreitzer here:
> https://sourceware.org/ml/libc-alpha/2017-03/msg00430.html
> ~~~
> ... Alternatively a more detailed performance analysis of
> the impact on applications that don't use __regcall is required before adding
> instructions to the hot path of the average application (or removing their use
> in _dl_runtime_resolve since that penalizes the dynamic loader for all applications
> on hardware that supports those vector registers).
> ~~~
>
>> This is the worst case where portion of time spent for saving and
>> restoring registers is bigger than majority of cases. With smaller
>> _dl_runtime_resolve code size, overall performance impact is negligible.
>>
>> On IvyBridge, differences in build and test time of binutils with lazy
>> binding GCC and binutils are noises. On Westmere, differences in
>> bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
>> binutils are also noises.
> Do you have any statistics on the timing for large applications that
> use a lot of libraries? I don't see gcc, binutils, or glibc as indicative
> of the complexity of shared libraries in terms of loaded shared libraries.
_dl_runtime_resolve is only called once when an external function is
called the first time. Many shared libraries isn't a problem unless
all execution
time is spent in _dl_runtime_resolve. I don't believe this is a
typical behavior.
> Something like libreoffice's soffice.bin has 142 DSOs, or chrome's
> 103 DSOs. It might be hard to measure if the lazy resolution is impacting
> the performance or if you are hitting some other performance boundary, but
> a black-box test showing performance didn't get *worse* for startup and
> exit, would mean it isn't the bottlneck (but might be some day). To test
> this you should be able to use libreoffice's CLI arguments to batch process
> some files and time that (or the --cat files option).
My machines run Fedora 26, which default to DT_BIND_NOW. Both
libreoffice and chrom are marked with:
0x000000000000001e (FLAGS) BIND_NOW
_dl_runtime_resolve isn't used.
> If we can show that the above latency is in the noise for real applications
> using many DSOs, then it makes your case better for supporting the alternate
> calling conventions.
>
Here is the updated patch which updates xsave state size for
GLIBC_TUNABLES=glibc.tune.hwcaps=-XSAVEC_Usable
Comments
On 10/19/2017 03:36 PM, H.J. Lu wrote:
>> Do you have any statistics on the timing for large applications
>> that use a lot of libraries? I don't see gcc, binutils, or glibc as
>> indicative of the complexity of shared libraries in terms of loaded
>> shared libraries.
>
> _dl_runtime_resolve is only called once when an external function is
> called the first time. Many shared libraries isn't a problem unless
> all execution time is spent in _dl_runtime_resolve. I don't believe
> this is a typical behavior.
When you have many shared libraries, you are constantly calling
_dl_runtime_resolve as the application features are first being used,
and the question I have is "What kind of additional latency does this
look like for an application with a lot of DSOs and a lot of external
functions?"
I understand that you *have* tested the raw latency of the call itself,
but it's not clear how that relates to real-world performance. I would
like to see a real look at some application to see how it operates.
>> If we can show that the above latency is in the noise for real
>> applications using many DSOs, then it makes your case better for
>> supporting the alternate calling conventions.
>>
>
> Here is the updated patch which updates xsave state size for
> GLIBC_TUNABLES=glibc.tune.hwcaps=-XSAVEC_Usable
OK.
Is the purpose of the tunable to disable or enable using xsave and
allow an application to get back the performance they might have lost?
If we are going to recommend this to users, we should add another
tunable that is easier to use and document that.
e.g. glibc.tune.x86_optimize_???call=1 (disables xsave usage,
defaults to 0).
On 2017.10.19 at 15:36 -0700, H.J. Lu wrote:
> On Thu, Oct 19, 2017 at 2:55 PM, Carlos O'Donell <carlos@redhat.com> wrote:
> > On 10/19/2017 10:41 AM, H.J. Lu wrote:
> >> In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
> >> mask and bound registers. It simplifies _dl_runtime_resolve and supports
> >> different calling conventions. ld.so code size is reduced by more than
> >> 1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
> >> than saving and restoring vector and bound registers individually.
> >>
> >> Latency for _dl_runtime_resolve to lookup the function, foo, from one
> >> shared library plus libc.so:
> >>
> >> Before After Change
> >>
> >> Westmere (SSE)/fxsave 345 866 151%
> >> IvyBridge (AVX)/xsave 420 643 53%
> >> Haswell (AVX)/xsave 713 1252 75%
> >> Skylake (AVX+MPX)/xsavec 559 719 28%
> >> Skylake (AVX512+MPX)/xsavec 145 272 87%
> >
> > This is a good baseline, but as you note, the change may not be observable
> > in any real world programs.
> >
> > The case I made to David Kreitzer here:
> > https://sourceware.org/ml/libc-alpha/2017-03/msg00430.html
> > ~~~
> > ... Alternatively a more detailed performance analysis of
> > the impact on applications that don't use __regcall is required before adding
> > instructions to the hot path of the average application (or removing their use
> > in _dl_runtime_resolve since that penalizes the dynamic loader for all applications
> > on hardware that supports those vector registers).
> > ~~~
> >
> >> This is the worst case where portion of time spent for saving and
> >> restoring registers is bigger than majority of cases. With smaller
> >> _dl_runtime_resolve code size, overall performance impact is negligible.
> >>
> >> On IvyBridge, differences in build and test time of binutils with lazy
> >> binding GCC and binutils are noises. On Westmere, differences in
> >> bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
> >> binutils are also noises.
> > Do you have any statistics on the timing for large applications that
> > use a lot of libraries? I don't see gcc, binutils, or glibc as indicative
> > of the complexity of shared libraries in terms of loaded shared libraries.
>
> _dl_runtime_resolve is only called once when an external function is
> called the first time. Many shared libraries isn't a problem unless
> all execution
> time is spent in _dl_runtime_resolve. I don't believe this is a
> typical behavior.
>
> > Something like libreoffice's soffice.bin has 142 DSOs, or chrome's
> > 103 DSOs. It might be hard to measure if the lazy resolution is impacting
> > the performance or if you are hitting some other performance boundary, but
> > a black-box test showing performance didn't get *worse* for startup and
> > exit, would mean it isn't the bottlneck (but might be some day). To test
> > this you should be able to use libreoffice's CLI arguments to batch process
> > some files and time that (or the --cat files option).
I did some testing on my old SSE only machine and everything is in the
noise. For example:
~ % ldd /usr/lib64/libreoffice/program/soffice.bin | wc -l
105
~ % hardening-check /usr/lib64/libreoffice/program/soffice.bin
/usr/lib64/libreoffice/program/soffice.bin:
Position Independent Executable: no, normal executable!
Stack protected: no, not found!
Fortify Source functions: no, not found!
Read-only relocations: yes
Immediate binding: no, not found!
(with H.J.'s patch)
Performance counter stats for '/var/tmp/glibc-build/elf/ld.so /usr/lib64/libreoffice/program/soffice.bin --convert-to pdf kandide.odt' (4 runs):
2463.681675 task-clock (msec) # 1.040 CPUs utilized ( +- 0.06% )
414 context-switches # 0.168 K/sec ( +- 8.88% )
10 cpu-migrations # 0.004 K/sec ( +- 11.98% )
28,227 page-faults # 0.011 M/sec ( +- 0.04% )
7,823,762,346 cycles # 3.176 GHz ( +- 0.15% ) (67.30%)
1,360,335,356 stalled-cycles-frontend # 17.39% frontend cycles idle ( +- 0.51% ) (66.78%)
2,090,675,875 stalled-cycles-backend # 26.72% backend cycles idle ( +- 1.02% ) (66.70%)
8,984,501,079 instructions # 1.15 insn per cycle
# 0.23 stalled cycles per insn ( +- 0.11% ) (66.96%)
1,866,843,047 branches # 757.745 M/sec ( +- 0.28% ) (67.25%)
73,973,482 branch-misses # 3.96% of all branches ( +- 0.15% ) (67.37%)
2.368775642 seconds time elapsed ( +- 0.21% )
(without)
Performance counter stats for '/usr/lib64/libreoffice/program/soffice.bin --convert-to pdf kandide.odt' (4 runs):
2467.698417 task-clock (msec) # 1.040 CPUs utilized ( +- 0.23% )
540 context-switches # 0.219 K/sec ( +- 17.02% )
12 cpu-migrations # 0.005 K/sec ( +- 14.85% )
28,245 page-faults # 0.011 M/sec ( +- 0.02% )
7,806,607,838 cycles # 3.164 GHz ( +- 0.09% ) (67.06%)
1,338,588,952 stalled-cycles-frontend # 17.15% frontend cycles idle ( +- 0.30% ) (66.99%)
2,103,802,012 stalled-cycles-backend # 26.95% backend cycles idle ( +- 0.77% ) (66.92%)
9,012,688,271 instructions # 1.15 insn per cycle
# 0.23 stalled cycles per insn ( +- 0.14% ) (67.02%)
1,870,634,478 branches # 758.048 M/sec ( +- 0.31% ) (67.19%)
73,921,605 branch-misses # 3.95% of all branches ( +- 0.13% ) (67.08%)
2.373621006 seconds time elapsed ( +- 0.27% )
Compile times using clang, that was built with shared libs, also don't
change at all.
On Thu, Oct 19, 2017 at 9:30 PM, Carlos O'Donell <carlos@redhat.com> wrote:
> On 10/19/2017 03:36 PM, H.J. Lu wrote:
>>> Do you have any statistics on the timing for large applications
>>> that use a lot of libraries? I don't see gcc, binutils, or glibc as
>>> indicative of the complexity of shared libraries in terms of loaded
>>> shared libraries.
>>
>> _dl_runtime_resolve is only called once when an external function is
>> called the first time. Many shared libraries isn't a problem unless
>> all execution time is spent in _dl_runtime_resolve. I don't believe
>> this is a typical behavior.
>
> When you have many shared libraries, you are constantly calling
> _dl_runtime_resolve as the application features are first being used,
> and the question I have is "What kind of additional latency does this
> look like for an application with a lot of DSOs and a lot of external
> functions?"
When there are many DSOs, it takes more time to lookup a symbol
and time to save/restore vector registers becomes noise. The only
case when time to save/restore vector registers becomes non-trivial is
1. There are a few DSOs so that symbol lookup takes fewer cycles. And
2. There are many external function calls which are executed only once. And
3. These external functions take very few cycles.
I can create such a testcase. But I don't think it is a typical case.
> I understand that you *have* tested the raw latency of the call itself,
> but it's not clear how that relates to real-world performance. I would
> like to see a real look at some application to see how it operates.
>
>>> If we can show that the above latency is in the noise for real
>>> applications using many DSOs, then it makes your case better for
>>> supporting the alternate calling conventions.
>>>
>>
>> Here is the updated patch which updates xsave state size for
>> GLIBC_TUNABLES=glibc.tune.hwcaps=-XSAVEC_Usable
>
> OK.
>
> Is the purpose of the tunable to disable or enable using xsave and
> allow an application to get back the performance they might have lost?
No. This disables XSAVEC and uses XSAVE instead. XSAVEC takes
less space and should be faster.
> If we are going to recommend this to users, we should add another
> tunable that is easier to use and document that.
>
> e.g. glibc.tune.x86_optimize_???call=1 (disables xsave usage,
> defaults to 0).
We can add this on top of what we have now. But it will make text
size of ld.so bigger, which pollute cache. Increased cycles to save/restore
vector registers with fxsave/xsave/xsavec is just noise.
On Fri, Oct 20, 2017 at 12:24 AM, Markus Trippelsdorf
<markus@trippelsdorf.de> wrote:
> On 2017.10.19 at 15:36 -0700, H.J. Lu wrote:
>> On Thu, Oct 19, 2017 at 2:55 PM, Carlos O'Donell <carlos@redhat.com> wrote:
>> > On 10/19/2017 10:41 AM, H.J. Lu wrote:
>> >> In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
>> >> mask and bound registers. It simplifies _dl_runtime_resolve and supports
>> >> different calling conventions. ld.so code size is reduced by more than
>> >> 1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
>> >> than saving and restoring vector and bound registers individually.
>> >>
>> >> Latency for _dl_runtime_resolve to lookup the function, foo, from one
>> >> shared library plus libc.so:
>> >>
>> >> Before After Change
>> >>
>> >> Westmere (SSE)/fxsave 345 866 151%
>> >> IvyBridge (AVX)/xsave 420 643 53%
>> >> Haswell (AVX)/xsave 713 1252 75%
>> >> Skylake (AVX+MPX)/xsavec 559 719 28%
>> >> Skylake (AVX512+MPX)/xsavec 145 272 87%
>> >
>> > This is a good baseline, but as you note, the change may not be observable
>> > in any real world programs.
>> >
>> > The case I made to David Kreitzer here:
>> > https://sourceware.org/ml/libc-alpha/2017-03/msg00430.html
>> > ~~~
>> > ... Alternatively a more detailed performance analysis of
>> > the impact on applications that don't use __regcall is required before adding
>> > instructions to the hot path of the average application (or removing their use
>> > in _dl_runtime_resolve since that penalizes the dynamic loader for all applications
>> > on hardware that supports those vector registers).
>> > ~~~
>> >
>> >> This is the worst case where portion of time spent for saving and
>> >> restoring registers is bigger than majority of cases. With smaller
>> >> _dl_runtime_resolve code size, overall performance impact is negligible.
>> >>
>> >> On IvyBridge, differences in build and test time of binutils with lazy
>> >> binding GCC and binutils are noises. On Westmere, differences in
>> >> bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
>> >> binutils are also noises.
>> > Do you have any statistics on the timing for large applications that
>> > use a lot of libraries? I don't see gcc, binutils, or glibc as indicative
>> > of the complexity of shared libraries in terms of loaded shared libraries.
>>
>> _dl_runtime_resolve is only called once when an external function is
>> called the first time. Many shared libraries isn't a problem unless
>> all execution
>> time is spent in _dl_runtime_resolve. I don't believe this is a
>> typical behavior.
>>
>> > Something like libreoffice's soffice.bin has 142 DSOs, or chrome's
>> > 103 DSOs. It might be hard to measure if the lazy resolution is impacting
>> > the performance or if you are hitting some other performance boundary, but
>> > a black-box test showing performance didn't get *worse* for startup and
>> > exit, would mean it isn't the bottlneck (but might be some day). To test
>> > this you should be able to use libreoffice's CLI arguments to batch process
>> > some files and time that (or the --cat files option).
>
> I did some testing on my old SSE only machine and everything is in the
> noise. For example:
>
> ~ % ldd /usr/lib64/libreoffice/program/soffice.bin | wc -l
> 105
> ~ % hardening-check /usr/lib64/libreoffice/program/soffice.bin
> /usr/lib64/libreoffice/program/soffice.bin:
> Position Independent Executable: no, normal executable!
> Stack protected: no, not found!
> Fortify Source functions: no, not found!
> Read-only relocations: yes
> Immediate binding: no, not found!
I have
[hjl@gnu-6 tmp]$ readelf -d /usr/lib64/libreoffice/program/soffice.bin
Dynamic section at offset 0xdb8 contains 27 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libuno_sal.so.3]
0x0000000000000001 (NEEDED) Shared library: [libsofficeapp.so]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000f (RPATH) Library rpath: [$ORIGIN]
0x000000000000000c (INIT) 0x710
0x000000000000000d (FINI) 0x904
0x0000000000000019 (INIT_ARRAY) 0x200da0
0x000000000000001b (INIT_ARRAYSZ) 8 (bytes)
0x000000000000001a (FINI_ARRAY) 0x200da8
0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
0x000000006ffffef5 (GNU_HASH) 0x298
0x0000000000000005 (STRTAB) 0x478
0x0000000000000006 (SYMTAB) 0x2e0
0x000000000000000a (STRSZ) 301 (bytes)
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000015 (DEBUG) 0x0
0x0000000000000003 (PLTGOT) 0x200fa8
0x0000000000000007 (RELA) 0x608
0x0000000000000008 (RELASZ) 264 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x0000000000000018 (BIND_NOW)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _dl_runtime_resolve isn't
used at all.
0x000000006ffffffb (FLAGS_1) Flags: NOW ORIGIN PIE
0x000000006ffffffe (VERNEED) 0x5c8
0x000000006fffffff (VERNEEDNUM) 2
0x000000006ffffff0 (VERSYM) 0x5a6
0x000000006ffffff9 (RELACOUNT) 3
0x0000000000000000 (NULL) 0x0
[hjl@gnu-6 tmp]$
> (with H.J.'s patch)
> Performance counter stats for '/var/tmp/glibc-build/elf/ld.so /usr/lib64/libreoffice/program/soffice.bin --convert-to pdf kandide.odt' (4 runs):
>
> 2463.681675 task-clock (msec) # 1.040 CPUs utilized ( +- 0.06% )
> 414 context-switches # 0.168 K/sec ( +- 8.88% )
> 10 cpu-migrations # 0.004 K/sec ( +- 11.98% )
> 28,227 page-faults # 0.011 M/sec ( +- 0.04% )
> 7,823,762,346 cycles # 3.176 GHz ( +- 0.15% ) (67.30%)
> 1,360,335,356 stalled-cycles-frontend # 17.39% frontend cycles idle ( +- 0.51% ) (66.78%)
> 2,090,675,875 stalled-cycles-backend # 26.72% backend cycles idle ( +- 1.02% ) (66.70%)
> 8,984,501,079 instructions # 1.15 insn per cycle
> # 0.23 stalled cycles per insn ( +- 0.11% ) (66.96%)
> 1,866,843,047 branches # 757.745 M/sec ( +- 0.28% ) (67.25%)
> 73,973,482 branch-misses # 3.96% of all branches ( +- 0.15% ) (67.37%)
>
> 2.368775642 seconds time elapsed ( +- 0.21% )
>
> (without)
> Performance counter stats for '/usr/lib64/libreoffice/program/soffice.bin --convert-to pdf kandide.odt' (4 runs):
>
> 2467.698417 task-clock (msec) # 1.040 CPUs utilized ( +- 0.23% )
> 540 context-switches # 0.219 K/sec ( +- 17.02% )
> 12 cpu-migrations # 0.005 K/sec ( +- 14.85% )
> 28,245 page-faults # 0.011 M/sec ( +- 0.02% )
> 7,806,607,838 cycles # 3.164 GHz ( +- 0.09% ) (67.06%)
> 1,338,588,952 stalled-cycles-frontend # 17.15% frontend cycles idle ( +- 0.30% ) (66.99%)
> 2,103,802,012 stalled-cycles-backend # 26.95% backend cycles idle ( +- 0.77% ) (66.92%)
> 9,012,688,271 instructions # 1.15 insn per cycle
> # 0.23 stalled cycles per insn ( +- 0.14% ) (67.02%)
> 1,870,634,478 branches # 758.048 M/sec ( +- 0.31% ) (67.19%)
> 73,921,605 branch-misses # 3.95% of all branches ( +- 0.13% ) (67.08%)
>
> 2.373621006 seconds time elapsed ( +- 0.27% )
>
>
> Compile times using clang, that was built with shared libs, also don't
> change at all.
>
> --
> Markus
On 2017.10.20 at 04:11 -0700, H.J. Lu wrote:
> On Fri, Oct 20, 2017 at 12:24 AM, Markus Trippelsdorf
> <markus@trippelsdorf.de> wrote:
> > On 2017.10.19 at 15:36 -0700, H.J. Lu wrote:
> >> On Thu, Oct 19, 2017 at 2:55 PM, Carlos O'Donell <carlos@redhat.com> wrote:
> >> > On 10/19/2017 10:41 AM, H.J. Lu wrote:
> >> >> In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
> >> >> mask and bound registers. It simplifies _dl_runtime_resolve and supports
> >> >> different calling conventions. ld.so code size is reduced by more than
> >> >> 1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
> >> >> than saving and restoring vector and bound registers individually.
> >> >>
> >> >> Latency for _dl_runtime_resolve to lookup the function, foo, from one
> >> >> shared library plus libc.so:
> >> >>
> >> >> Before After Change
> >> >>
> >> >> Westmere (SSE)/fxsave 345 866 151%
> >> >> IvyBridge (AVX)/xsave 420 643 53%
> >> >> Haswell (AVX)/xsave 713 1252 75%
> >> >> Skylake (AVX+MPX)/xsavec 559 719 28%
> >> >> Skylake (AVX512+MPX)/xsavec 145 272 87%
> >> >
> >> > This is a good baseline, but as you note, the change may not be observable
> >> > in any real world programs.
> >> >
> >> > The case I made to David Kreitzer here:
> >> > https://sourceware.org/ml/libc-alpha/2017-03/msg00430.html
> >> > ~~~
> >> > ... Alternatively a more detailed performance analysis of
> >> > the impact on applications that don't use __regcall is required before adding
> >> > instructions to the hot path of the average application (or removing their use
> >> > in _dl_runtime_resolve since that penalizes the dynamic loader for all applications
> >> > on hardware that supports those vector registers).
> >> > ~~~
> >> >
> >> >> This is the worst case where portion of time spent for saving and
> >> >> restoring registers is bigger than majority of cases. With smaller
> >> >> _dl_runtime_resolve code size, overall performance impact is negligible.
> >> >>
> >> >> On IvyBridge, differences in build and test time of binutils with lazy
> >> >> binding GCC and binutils are noises. On Westmere, differences in
> >> >> bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
> >> >> binutils are also noises.
> >> > Do you have any statistics on the timing for large applications that
> >> > use a lot of libraries? I don't see gcc, binutils, or glibc as indicative
> >> > of the complexity of shared libraries in terms of loaded shared libraries.
> >>
> >> _dl_runtime_resolve is only called once when an external function is
> >> called the first time. Many shared libraries isn't a problem unless
> >> all execution
> >> time is spent in _dl_runtime_resolve. I don't believe this is a
> >> typical behavior.
> >>
> >> > Something like libreoffice's soffice.bin has 142 DSOs, or chrome's
> >> > 103 DSOs. It might be hard to measure if the lazy resolution is impacting
> >> > the performance or if you are hitting some other performance boundary, but
> >> > a black-box test showing performance didn't get *worse* for startup and
> >> > exit, would mean it isn't the bottlneck (but might be some day). To test
> >> > this you should be able to use libreoffice's CLI arguments to batch process
> >> > some files and time that (or the --cat files option).
> >
> > I did some testing on my old SSE only machine and everything is in the
> > noise. For example:
> >
> > ~ % ldd /usr/lib64/libreoffice/program/soffice.bin | wc -l
> > 105
> > ~ % hardening-check /usr/lib64/libreoffice/program/soffice.bin
> > /usr/lib64/libreoffice/program/soffice.bin:
> > Position Independent Executable: no, normal executable!
> > Stack protected: no, not found!
> > Fortify Source functions: no, not found!
> > Read-only relocations: yes
> > Immediate binding: no, not found!
>
> I have
>
> [hjl@gnu-6 tmp]$ readelf -d /usr/lib64/libreoffice/program/soffice.bin
>
> Dynamic section at offset 0xdb8 contains 27 entries:
> Tag Type Name/Value
> 0x0000000000000001 (NEEDED) Shared library: [libuno_sal.so.3]
> 0x0000000000000001 (NEEDED) Shared library: [libsofficeapp.so]
> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
> 0x000000000000000f (RPATH) Library rpath: [$ORIGIN]
> 0x000000000000000c (INIT) 0x710
> 0x000000000000000d (FINI) 0x904
> 0x0000000000000019 (INIT_ARRAY) 0x200da0
> 0x000000000000001b (INIT_ARRAYSZ) 8 (bytes)
> 0x000000000000001a (FINI_ARRAY) 0x200da8
> 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
> 0x000000006ffffef5 (GNU_HASH) 0x298
> 0x0000000000000005 (STRTAB) 0x478
> 0x0000000000000006 (SYMTAB) 0x2e0
> 0x000000000000000a (STRSZ) 301 (bytes)
> 0x000000000000000b (SYMENT) 24 (bytes)
> 0x0000000000000015 (DEBUG) 0x0
> 0x0000000000000003 (PLTGOT) 0x200fa8
> 0x0000000000000007 (RELA) 0x608
> 0x0000000000000008 (RELASZ) 264 (bytes)
> 0x0000000000000009 (RELAENT) 24 (bytes)
> 0x0000000000000018 (BIND_NOW)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _dl_runtime_resolve isn't
> used at all.
Yes. That is why I posted the hardening-check output:
"Immediate binding: no, not found!" means that "-z lazy" was used in my
case.
On Fri, Oct 20, 2017 at 4:16 AM, Markus Trippelsdorf
<markus@trippelsdorf.de> wrote:
> On 2017.10.20 at 04:11 -0700, H.J. Lu wrote:
>> On Fri, Oct 20, 2017 at 12:24 AM, Markus Trippelsdorf
>> <markus@trippelsdorf.de> wrote:
>> > On 2017.10.19 at 15:36 -0700, H.J. Lu wrote:
>> >> On Thu, Oct 19, 2017 at 2:55 PM, Carlos O'Donell <carlos@redhat.com> wrote:
>> >> > On 10/19/2017 10:41 AM, H.J. Lu wrote:
>> >> >> In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
>> >> >> mask and bound registers. It simplifies _dl_runtime_resolve and supports
>> >> >> different calling conventions. ld.so code size is reduced by more than
>> >> >> 1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
>> >> >> than saving and restoring vector and bound registers individually.
>> >> >>
>> >> >> Latency for _dl_runtime_resolve to lookup the function, foo, from one
>> >> >> shared library plus libc.so:
>> >> >>
>> >> >> Before After Change
>> >> >>
>> >> >> Westmere (SSE)/fxsave 345 866 151%
>> >> >> IvyBridge (AVX)/xsave 420 643 53%
>> >> >> Haswell (AVX)/xsave 713 1252 75%
>> >> >> Skylake (AVX+MPX)/xsavec 559 719 28%
>> >> >> Skylake (AVX512+MPX)/xsavec 145 272 87%
>> >> >
>> >> > This is a good baseline, but as you note, the change may not be observable
>> >> > in any real world programs.
>> >> >
>> >> > The case I made to David Kreitzer here:
>> >> > https://sourceware.org/ml/libc-alpha/2017-03/msg00430.html
>> >> > ~~~
>> >> > ... Alternatively a more detailed performance analysis of
>> >> > the impact on applications that don't use __regcall is required before adding
>> >> > instructions to the hot path of the average application (or removing their use
>> >> > in _dl_runtime_resolve since that penalizes the dynamic loader for all applications
>> >> > on hardware that supports those vector registers).
>> >> > ~~~
>> >> >
>> >> >> This is the worst case where portion of time spent for saving and
>> >> >> restoring registers is bigger than majority of cases. With smaller
>> >> >> _dl_runtime_resolve code size, overall performance impact is negligible.
>> >> >>
>> >> >> On IvyBridge, differences in build and test time of binutils with lazy
>> >> >> binding GCC and binutils are noises. On Westmere, differences in
>> >> >> bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
>> >> >> binutils are also noises.
>> >> > Do you have any statistics on the timing for large applications that
>> >> > use a lot of libraries? I don't see gcc, binutils, or glibc as indicative
>> >> > of the complexity of shared libraries in terms of loaded shared libraries.
>> >>
>> >> _dl_runtime_resolve is only called once when an external function is
>> >> called the first time. Many shared libraries isn't a problem unless
>> >> all execution
>> >> time is spent in _dl_runtime_resolve. I don't believe this is a
>> >> typical behavior.
>> >>
>> >> > Something like libreoffice's soffice.bin has 142 DSOs, or chrome's
>> >> > 103 DSOs. It might be hard to measure if the lazy resolution is impacting
>> >> > the performance or if you are hitting some other performance boundary, but
>> >> > a black-box test showing performance didn't get *worse* for startup and
>> >> > exit, would mean it isn't the bottlneck (but might be some day). To test
>> >> > this you should be able to use libreoffice's CLI arguments to batch process
>> >> > some files and time that (or the --cat files option).
>> >
>> > I did some testing on my old SSE only machine and everything is in the
>> > noise. For example:
>> >
>> > ~ % ldd /usr/lib64/libreoffice/program/soffice.bin | wc -l
>> > 105
>> > ~ % hardening-check /usr/lib64/libreoffice/program/soffice.bin
>> > /usr/lib64/libreoffice/program/soffice.bin:
>> > Position Independent Executable: no, normal executable!
>> > Stack protected: no, not found!
>> > Fortify Source functions: no, not found!
>> > Read-only relocations: yes
>> > Immediate binding: no, not found!
>>
>> I have
>>
>> [hjl@gnu-6 tmp]$ readelf -d /usr/lib64/libreoffice/program/soffice.bin
>>
>> Dynamic section at offset 0xdb8 contains 27 entries:
>> Tag Type Name/Value
>> 0x0000000000000001 (NEEDED) Shared library: [libuno_sal.so.3]
>> 0x0000000000000001 (NEEDED) Shared library: [libsofficeapp.so]
>> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
>> 0x000000000000000f (RPATH) Library rpath: [$ORIGIN]
>> 0x000000000000000c (INIT) 0x710
>> 0x000000000000000d (FINI) 0x904
>> 0x0000000000000019 (INIT_ARRAY) 0x200da0
>> 0x000000000000001b (INIT_ARRAYSZ) 8 (bytes)
>> 0x000000000000001a (FINI_ARRAY) 0x200da8
>> 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)
>> 0x000000006ffffef5 (GNU_HASH) 0x298
>> 0x0000000000000005 (STRTAB) 0x478
>> 0x0000000000000006 (SYMTAB) 0x2e0
>> 0x000000000000000a (STRSZ) 301 (bytes)
>> 0x000000000000000b (SYMENT) 24 (bytes)
>> 0x0000000000000015 (DEBUG) 0x0
>> 0x0000000000000003 (PLTGOT) 0x200fa8
>> 0x0000000000000007 (RELA) 0x608
>> 0x0000000000000008 (RELASZ) 264 (bytes)
>> 0x0000000000000009 (RELAENT) 24 (bytes)
>> 0x0000000000000018 (BIND_NOW)
>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _dl_runtime_resolve isn't
>> used at all.
>
> Yes. That is why I posted the hardening-check output:
> "Immediate binding: no, not found!" means that "-z lazy" was used in my
> case.
>
Great. Performance impact of my patch is just noise. On Ivy Bridge, for
binutils build and check with lazy binding GCC, as and ld, I got
Before
191.83user 24.37system 0:51.83elapsed 417%CPU (0avgtext+0avgdata
145800maxresident)k
108.09user 37.37system 1:48.58elapsed 133%CPU (0avgtext+0avgdata
2098644maxresident)k
After
191.68user 24.06system 0:51.94elapsed 415%CPU (0avgtext+0avgdata
145852maxresident)k
107.52user 37.22system 1:45.87elapsed 136%CPU (0avgtext+0avgdata
2098712maxresident)k
On 10/20/2017 01:09 PM, H.J. Lu wrote:
> When there are many DSOs, it takes more time to lookup a symbol
> and time to save/restore vector registers becomes noise. The only
> case when time to save/restore vector registers becomes non-trivial is
>
> 1. There are a few DSOs so that symbol lookup takes fewer cycles. And
> 2. There are many external function calls which are executed only once. And
> 3. These external functions take very few cycles.
>
> I can create such a testcase. But I don't think it is a typical case.
Completely agree. Basically, a program which is affected would have to
(a) call many functions, (b) with short symbol lookup chains, and (c) do
very little actual work. This seems to be a very unlikely scenario.
I have a test case. GCC scales poorly with many function calls, and it
is difficult to get --export-dynamic to work with recent GCC/binutils.
I will try to run it on various machines.
Thanks,
Florian
On 10/20/2017 02:58 PM, Florian Weimer wrote:
> On 10/20/2017 01:09 PM, H.J. Lu wrote:
>> When there are many DSOs, it takes more time to lookup a symbol
>> and time to save/restore vector registers becomes noise. The only
>> case when time to save/restore vector registers becomes non-trivial is
>>
>> 1. There are a few DSOs so that symbol lookup takes fewer cycles. And
>> 2. There are many external function calls which are executed only
>> once. And
>> 3. These external functions take very few cycles.
>>
>> I can create such a testcase. But I don't think it is a typical case.
>
> Completely agree. Basically, a program which is affected would have to
> (a) call many functions, (b) with short symbol lookup chains, and (c) do
> very little actual work. This seems to be a very unlikely scenario.
>
> I have a test case. GCC scales poorly with many function calls, and it
> is difficult to get --export-dynamic to work with recent GCC/binutils. I
> will try to run it on various machines.
LD_DEBUG=statistics shows this:
9506:
9506: runtime linker statistics:
9506: total startup time in dynamic loader: 19960074 cycles
9506: time needed for relocation: 19105814 cycles
(95.7%)
9506: number of relocations: 87
9506: number of relocations from cache: 3
9506: number of relative relocations: 1226
9506: time needed to load objects: 701382 cycles
(3.5%)
9506:
9506: runtime linker statistics:
9506: final number of relocations: 781589
9506: final number of relocations from cache: 3
This is a main program which contains 1,500 function calls. The
functions are defined in a single DSO, and each function calls 520 other
functions, giving a total number of 781,500 relocations from the test.
On my laptop (Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz), I get this
(ten runs, real time measured in seconds):
> t.test(prev_laptop, after_laptop)
Welch Two Sample t-test
data: prev_laptop and after_laptop
t = -14.932, df = 18, p-value = 1.392e-11
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.05749145 -0.04330855
sample estimates:
mean of x mean of y
0.2345 0.2849
So it's definitely not in the noise. The penalty appears to be around
65ns per relocation.
I said in the past that we should use XSAVE in the trampoline, so that
we do not have to touch the dynamic linker for each new CPU generation,
and I think that alone is worth the slight additional cost.
I'll check a few additional machines over the coming hours.
Note that XSAVE will still not allow us to support *arbitrary* calling
conventions, so we shouldn't advertise it as such. But hopefully, it
will be sufficient to get the ABI-violating binaries mentioned in bug
21265 back into working order.
Thanks,
Florian
On 10/20/2017 03:21 PM, Florian Weimer wrote:
> LD_DEBUG=statistics shows this:
>
> 9506:
> 9506: runtime linker statistics:
> 9506: total startup time in dynamic loader: 19960074 cycles
> 9506: time needed for relocation: 19105814 cycles
> (95.7%)
> 9506: number of relocations: 87
> 9506: number of relocations from cache: 3
> 9506: number of relative relocations: 1226
> 9506: time needed to load objects: 701382 cycles
> (3.5%)
> 9506:
> 9506: runtime linker statistics:
> 9506: final number of relocations: 781589
> 9506: final number of relocations from cache: 3
>
> This is a main program which contains 1,500 function calls. The
> functions are defined in a single DSO, and each function calls 520 other
> functions, giving a total number of 781,500 relocations from the test.
>
> On my laptop (Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz), I get this
> (ten runs, real time measured in seconds):
>
> > t.test(prev_laptop, after_laptop)
>
> Welch Two Sample t-test
>
> data: prev_laptop and after_laptop
> t = -14.932, df = 18, p-value = 1.392e-11
> alternative hypothesis: true difference in means is not equal to 0
> 95 percent confidence interval:
> -0.05749145 -0.04330855
> sample estimates:
> mean of x mean of y
> 0.2345 0.2849
>
> So it's definitely not in the noise. The penalty appears to be around
> 65ns per relocation.
I did some more benchmarks. Ryzen takes a bit of a hit (100ns or ~30%).
Purley looks very good (35ns or ~5%). The outlier is KNL, with 750ns
added per relocation (at lower clock rates admittedly) and ~50% longer
relocation times overall. (The caveat is that this is lab hardware,
which may or may not match production silicon.)
I still think these numbers are okay. To put this into perspective, the
total number of relocations that are processed when running yum, a
non-trivial Python application, is less than 22,000. So there is an
expected additional startup overhead of about 16.5ms for KNL, but that
is still in the noise for a simple yum command such as “yum repolist”.
Any other ideas what else we could benchmark?
Thanks,
Florian
* Florian Weimer:
> On 10/20/2017 03:21 PM, Florian Weimer wrote:
>> LD_DEBUG=statistics shows this:
>>
>> 9506:
>> 9506: runtime linker statistics:
>> 9506: total startup time in dynamic loader: 19960074 cycles
>> 9506: time needed for relocation: 19105814 cycles
>> (95.7%)
>> 9506: number of relocations: 87
>> 9506: number of relocations from cache: 3
>> 9506: number of relative relocations: 1226
>> 9506: time needed to load objects: 701382 cycles
>> (3.5%)
>> 9506:
>> 9506: runtime linker statistics:
>> 9506: final number of relocations: 781589
>> 9506: final number of relocations from cache: 3
>>
>> This is a main program which contains 1,500 function calls. The
>> functions are defined in a single DSO, and each function calls 520 other
>> functions, giving a total number of 781,500 relocations from the test.
>>
>> On my laptop (Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz), I get this
>> (ten runs, real time measured in seconds):
>>
>> > t.test(prev_laptop, after_laptop)
>>
>> Welch Two Sample t-test
>>
>> data: prev_laptop and after_laptop
>> t = -14.932, df = 18, p-value = 1.392e-11
>> alternative hypothesis: true difference in means is not equal to 0
>> 95 percent confidence interval:
>> -0.05749145 -0.04330855
>> sample estimates:
>> mean of x mean of y
>> 0.2345 0.2849
>>
>> So it's definitely not in the noise. The penalty appears to be around
>> 65ns per relocation.
>
> I did some more benchmarks. Ryzen takes a bit of a hit (100ns or ~30%).
> Purley looks very good (35ns or ~5%). The outlier is KNL, with 750ns
> added per relocation (at lower clock rates admittedly) and ~50% longer
> relocation times overall. (The caveat is that this is lab hardware,
> which may or may not match production silicon.)
I found another piece of perhaps interesting hardware, an Intel(R)
Core(TM) m7-6Y75. The performance hit is around 3% or 16ns. This is
a lower-power mobile CPU in a tablet. So even there, the numbers are
good.
On 10/20/2017 09:31 AM, Florian Weimer wrote:
> * Florian Weimer:
>
>> On 10/20/2017 03:21 PM, Florian Weimer wrote:
>>> LD_DEBUG=statistics shows this:
>>>
>>> 9506:
>>> 9506: runtime linker statistics:
>>> 9506: total startup time in dynamic loader: 19960074 cycles
>>> 9506: time needed for relocation: 19105814 cycles
>>> (95.7%)
>>> 9506: number of relocations: 87
>>> 9506: number of relocations from cache: 3
>>> 9506: number of relative relocations: 1226
>>> 9506: time needed to load objects: 701382 cycles
>>> (3.5%)
>>> 9506:
>>> 9506: runtime linker statistics:
>>> 9506: final number of relocations: 781589
>>> 9506: final number of relocations from cache: 3
>>>
>>> This is a main program which contains 1,500 function calls. The
>>> functions are defined in a single DSO, and each function calls 520 other
>>> functions, giving a total number of 781,500 relocations from the test.
>>>
>>> On my laptop (Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz), I get this
>>> (ten runs, real time measured in seconds):
>>>
>>> > t.test(prev_laptop, after_laptop)
>>>
>>> Welch Two Sample t-test
>>>
>>> data: prev_laptop and after_laptop
>>> t = -14.932, df = 18, p-value = 1.392e-11
>>> alternative hypothesis: true difference in means is not equal to 0
>>> 95 percent confidence interval:
>>> -0.05749145 -0.04330855
>>> sample estimates:
>>> mean of x mean of y
>>> 0.2345 0.2849
>>>
>>> So it's definitely not in the noise. The penalty appears to be around
>>> 65ns per relocation.
>>
>> I did some more benchmarks. Ryzen takes a bit of a hit (100ns or ~30%).
>> Purley looks very good (35ns or ~5%). The outlier is KNL, with 750ns
>> added per relocation (at lower clock rates admittedly) and ~50% longer
>> relocation times overall. (The caveat is that this is lab hardware,
>> which may or may not match production silicon.)
>
> I found another piece of perhaps interesting hardware, an Intel(R)
> Core(TM) m7-6Y75. The performance hit is around 3% or 16ns. This is
> a lower-power mobile CPU in a tablet. So even there, the numbers are
> good.
Florian, H.J, and Markus,
Thank you very much for doing some more thourough tests on the patch set.
H.J.,
Thank you for all the work on this and for moving this forward so that
we can support more of the user applications on GNU/Linux.
The patch looks good to me with the minor nit that you should use ALIGN_UP
to make it clear you're doing an alignment operation.
OK with that change.
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
On 10/19/2017 03:36 PM, H.J. Lu wrote:
> [BZ #21265]
> * sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
> New.
> * sysdeps/x86/cpu-features.c (get_common_indeces): Set
> xsave_state_size, xsave_state_full_size and
> bit_arch_XSAVEC_Usable if needed.
> (init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
> and bit_arch_Use_dl_runtime_resolve_opt.
> * sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
> Removed.
> (bit_arch_Use_dl_runtime_resolve_slow): Likewise.
> (bit_arch_Prefer_No_AVX512): Updated.
> (bit_arch_MathVec_Prefer_No_AVX512): Likewise.
> (bit_arch_XSAVEC_Usable): New.
> (STATE_SAVE_OFFSET): Likewise.
> (STATE_SAVE_MASK): Likewise.
> [__ASSEMBLER__]: Include <cpu-features-offsets.h>.
> (cpu_features): Add xsave_state_size and xsave_state_full_size.
> (index_arch_Use_dl_runtime_resolve_opt): Removed.
> (index_arch_Use_dl_runtime_resolve_slow): Likewise.
> (index_arch_XSAVEC_Usable): New.
> * sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
> Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
> * sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
> is enabled.
> * sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
> Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
> _dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
> _dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
> with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
> _dl_runtime_resolve_xsavec.
> * sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
> Removed.
> (DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
> instead of VEC_SIZE.
> (REGISTER_SAVE_BND0): Removed.
> (REGISTER_SAVE_BND1): Likewise.
> (REGISTER_SAVE_BND3): Likewise.
> (REGISTER_SAVE_RAX): Always defined to 0.
> (VMOV): Removed.
> (_dl_runtime_resolve_avx): Likewise.
> (_dl_runtime_resolve_avx_slow): Likewise.
> (_dl_runtime_resolve_avx_opt): Likewise.
> (_dl_runtime_resolve_avx512): Likewise.
> (_dl_runtime_resolve_avx512_opt): Likewise.
> (_dl_runtime_resolve_sse): Likewise.
> (_dl_runtime_resolve_sse_vex): Likewise.
> (USE_FXSAVE): New.
> (_dl_runtime_resolve_fxsave): Likewise.
> (USE_XSAVE): Likewise.
> (_dl_runtime_resolve_xsave): Likewise.
> (USE_XSAVEC): Likewise.
> (_dl_runtime_resolve_xsavec): Likewise.
> * sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
> Removed.
> (_dl_runtime_resolve_avx512_opt): Likewise.
> (_dl_runtime_resolve_avx): Likewise.
> (_dl_runtime_resolve_avx_opt): Likewise.
> (_dl_runtime_resolve_sse): Likewise.
> (_dl_runtime_resolve_sse_vex): Likewise.
> (_dl_runtime_resolve_fxsave): New.
> (_dl_runtime_resolve_xsave): Likewise.
> (_dl_runtime_resolve_xsavec): Likewise.
> ---
> sysdeps/x86/cpu-features-offsets.sym | 1 +
> sysdeps/x86/cpu-features.c | 87 +++++++++---
> sysdeps/x86/cpu-features.h | 34 ++++-
> sysdeps/x86/cpu-tunables.c | 17 ++-
> sysdeps/x86_64/Makefile | 4 +
> sysdeps/x86_64/dl-machine.h | 38 ++---
> sysdeps/x86_64/dl-trampoline.S | 87 ++++--------
> sysdeps/x86_64/dl-trampoline.h | 266 ++++++++++-------------------------
> 8 files changed, 228 insertions(+), 306 deletions(-)
>
> diff --git a/sysdeps/x86/cpu-features-offsets.sym b/sysdeps/x86/cpu-features-offsets.sym
> index f6739fae81..33dd094e37 100644
> --- a/sysdeps/x86/cpu-features-offsets.sym
> +++ b/sysdeps/x86/cpu-features-offsets.sym
> @@ -15,6 +15,7 @@ CPUID_ECX_OFFSET offsetof (struct cpuid_registers, ecx)
> CPUID_EDX_OFFSET offsetof (struct cpuid_registers, edx)
> FAMILY_OFFSET offsetof (struct cpu_features, family)
> MODEL_OFFSET offsetof (struct cpu_features, model)
> +XSAVE_STATE_SIZE_OFFSET offsetof (struct cpu_features, xsave_state_size)
> FEATURE_OFFSET offsetof (struct cpu_features, feature)
> FEATURE_SIZE sizeof (unsigned int)
>
> diff --git a/sysdeps/x86/cpu-features.c b/sysdeps/x86/cpu-features.c
> index 332b0f0d4a..6a5034f3c7 100644
> --- a/sysdeps/x86/cpu-features.c
> +++ b/sysdeps/x86/cpu-features.c
#include <libc-pointer-arith.h>
> @@ -103,6 +103,76 @@ get_common_indeces (struct cpu_features *cpu_features,
> }
> }
> }
> +
> + /* For _dl_runtime_resolve, set xsave_state_size to xsave area
> + size + integer register save size and align it to 64 bytes. */
OK.
> + if (cpu_features->max_cpuid >= 0xd)
> + {
> + unsigned int eax, ebx, ecx, edx;
> +
> + __cpuid_count (0xd, 0, eax, ebx, ecx, edx);
> + if (ebx != 0)
> + {
> + unsigned int xsave_state_full_size
> + = (ebx + STATE_SAVE_OFFSET + 63) & -64;
Use ALIGN_UP.
> +
> + cpu_features->xsave_state_size
> + = xsave_state_full_size;
> + cpu_features->xsave_state_full_size
> + = xsave_state_full_size;
> +
> + __cpuid_count (0xd, 1, eax, ebx, ecx, edx);
> +
> + /* Check if XSAVEC is available. */
> + if ((eax & (1 << 1)) != 0)
> + {
> + unsigned int xstate_comp_offsets[32];
> + unsigned int xstate_comp_sizes[32];
> + unsigned int i;
> +
> + xstate_comp_offsets[0] = 0;
> + xstate_comp_offsets[1] = 160;
> + xstate_comp_offsets[2] = 576;
> + xstate_comp_sizes[0] = 160;
> + xstate_comp_sizes[1] = 256;
> +
> + for (i = 2; i < 32; i++)
> + {
> + if ((STATE_SAVE_MASK & (1 << i)) != 0)
> + {
> + __cpuid_count (0xd, i, eax, ebx, ecx, edx);
> + xstate_comp_sizes[i] = eax;
> + }
> + else
> + {
> + ecx = 0;
> + xstate_comp_sizes[i] = 0;
OK.
> + }
> +
> + if (i > 2)
> + {
> + xstate_comp_offsets[i]
> + = (xstate_comp_offsets[i - 1]
> + + xstate_comp_sizes[i -1]);
> + if ((ecx & (1 << 1)) != 0)
> + xstate_comp_offsets[i]
> + = (xstate_comp_offsets[i] + 63) & -64;
> + }
> + }
> +
> + /* Use XSAVEC. */
> + unsigned int size
> + = xstate_comp_offsets[31] + xstate_comp_sizes[31];
> + if (size)
> + {
> + cpu_features->xsave_state_size
> + = (size + STATE_SAVE_OFFSET + 63) & -64;
Use ALIGN_UP.
> + cpu_features->feature[index_arch_XSAVEC_Usable]
> + |= bit_arch_XSAVEC_Usable;
OK.
> + }
> + }
> + }
> + }
> }
> }
>
> @@ -242,23 +312,6 @@ init_cpu_features (struct cpu_features *cpu_features)
> else
> cpu_features->feature[index_arch_Prefer_No_AVX512]
> |= bit_arch_Prefer_No_AVX512;
> -
> - /* To avoid SSE transition penalty, use _dl_runtime_resolve_slow.
> - If XGETBV suports ECX == 1, use _dl_runtime_resolve_opt.
> - Use _dl_runtime_resolve_opt only with AVX512F since it is
> - slower than _dl_runtime_resolve_slow with AVX. */
> - cpu_features->feature[index_arch_Use_dl_runtime_resolve_slow]
> - |= bit_arch_Use_dl_runtime_resolve_slow;
> - if (CPU_FEATURES_ARCH_P (cpu_features, AVX512F_Usable)
> - && cpu_features->max_cpuid >= 0xd)
> - {
> - unsigned int eax;
> -
> - __cpuid_count (0xd, 1, eax, ebx, ecx, edx);
> - if ((eax & (1 << 2)) != 0)
> - cpu_features->feature[index_arch_Use_dl_runtime_resolve_opt]
> - |= bit_arch_Use_dl_runtime_resolve_opt;
> - }
> }
> /* This spells out "AuthenticAMD". */
> else if (ebx == 0x68747541 && ecx == 0x444d4163 && edx == 0x69746e65)
> diff --git a/sysdeps/x86/cpu-features.h b/sysdeps/x86/cpu-features.h
> index a032a2e168..b7f7898d11 100644
> --- a/sysdeps/x86/cpu-features.h
> +++ b/sysdeps/x86/cpu-features.h
> @@ -37,10 +37,9 @@
> #define bit_arch_Prefer_No_VZEROUPPER (1 << 17)
> #define bit_arch_Fast_Unaligned_Copy (1 << 18)
> #define bit_arch_Prefer_ERMS (1 << 19)
> -#define bit_arch_Use_dl_runtime_resolve_opt (1 << 20)
> -#define bit_arch_Use_dl_runtime_resolve_slow (1 << 21)
> -#define bit_arch_Prefer_No_AVX512 (1 << 22)
> -#define bit_arch_MathVec_Prefer_No_AVX512 (1 << 23)
> +#define bit_arch_Prefer_No_AVX512 (1 << 20)
> +#define bit_arch_MathVec_Prefer_No_AVX512 (1 << 21)
> +#define bit_arch_XSAVEC_Usable (1 << 22)
OK.
>
> /* CPUID Feature flags. */
>
> @@ -91,8 +90,18 @@
> /* The current maximum size of the feature integer bit array. */
> #define FEATURE_INDEX_MAX 1
>
> -#ifndef __ASSEMBLER__
> +/* Offset for fxsave/xsave area used by _dl_runtime_resolve. Also need
> + space to preserve RCX, RDX, RSI, RDI, R8, R9 and RAX. It must be
> + aligned to 16 bytes for fxsave and 64 bytes for xsave. */
> +#define STATE_SAVE_OFFSET (8 * 7 + 8)
>
> +/* Save SSE, AVX, AVX512, mask and bound registers. */
> +#define STATE_SAVE_MASK \
> + ((1 << 1) | (1 << 2) | (1 << 3) | (1 << 5) | (1 << 6) | (1 << 7))
> +
> +#ifdef __ASSEMBLER__
> +# include <cpu-features-offsets.h>
> +#else /* __ASSEMBLER__ */
> enum
> {
> COMMON_CPUID_INDEX_1 = 0,
> @@ -121,6 +130,18 @@ struct cpu_features
> } cpuid[COMMON_CPUID_INDEX_MAX];
> unsigned int family;
> unsigned int model;
> + /* The state size for XSAVEC or XSAVE. The type must be unsigned long
> + int so that we use
> +
> + sub xsave_state_size_offset(%rip) %RSP_LP
> +
> + in _dl_runtime_resolve. */
> + unsigned long int xsave_state_size;
> + /* The full state size for XSAVE when XSAVEC is disabled by
> +
> + GLIBC_TUNABLES=glibc.tune.hwcaps=-XSAVEC_Usable
> + */
> + unsigned int xsave_state_full_size;
> unsigned int feature[FEATURE_INDEX_MAX];
> /* Data cache size for use in memory and string routines, typically
> L1 size. */
> @@ -237,10 +258,9 @@ extern const struct cpu_features *__get_cpu_features (void)
> # define index_arch_Prefer_No_VZEROUPPER FEATURE_INDEX_1
> # define index_arch_Fast_Unaligned_Copy FEATURE_INDEX_1
> # define index_arch_Prefer_ERMS FEATURE_INDEX_1
> -# define index_arch_Use_dl_runtime_resolve_opt FEATURE_INDEX_1
> -# define index_arch_Use_dl_runtime_resolve_slow FEATURE_INDEX_1
> # define index_arch_Prefer_No_AVX512 FEATURE_INDEX_1
> # define index_arch_MathVec_Prefer_No_AVX512 FEATURE_INDEX_1
> +# define index_arch_XSAVEC_Usable FEATURE_INDEX_1
OK.
>
> #endif /* !__ASSEMBLER__ */
>
> diff --git a/sysdeps/x86/cpu-tunables.c b/sysdeps/x86/cpu-tunables.c
> index ec72d86f08..dcd0165f2e 100644
> --- a/sysdeps/x86/cpu-tunables.c
> +++ b/sysdeps/x86/cpu-tunables.c
> @@ -242,6 +242,16 @@ TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *valp)
> Slow_SSE4_2, SSE4_2,
> disable, 11);
> break;
> + case 13:
> + if (disable)
> + {
> + /* Update xsave_state_size to XSAVE state size. */
> + cpu_features->xsave_state_size
> + = cpu_features->xsave_state_full_size;
> + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features,
> + XSAVEC_Usable, 13);
> + }
OK.
> + break;
> case 14:
> if (disable)
> {
> @@ -317,13 +327,6 @@ TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *valp)
> disable, 26);
> }
> break;
> - case 27:
> - {
> - CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features,
> - Use_dl_runtime_resolve_slow,
> - disable, 27);
> - }
> - break;
> }
> p += len + 1;
> }
> diff --git a/sysdeps/x86_64/Makefile b/sysdeps/x86_64/Makefile
> index 12d4737240..9f1562f1b2 100644
> --- a/sysdeps/x86_64/Makefile
> +++ b/sysdeps/x86_64/Makefile
> @@ -55,6 +55,10 @@ CFLAGS-tst-quad2pie.c = $(PIE-ccflag)
> tests += tst-x86_64-1
> modules-names += x86_64/tst-x86_64mod-1
> LDFLAGS-tst-x86_64mod-1.so = -Wl,-soname,tst-x86_64mod-1.so
> +ifneq (no,$(have-tunables))
> +# Test the state size for XSAVE when XSAVEC is disabled.
> +tst-x86_64-1-ENV = GLIBC_TUNABLES=glibc.tune.hwcaps=-XSAVEC_Usable
> +endif
OK. Thanks for adding a test with the tunable!
>
> $(objpfx)tst-x86_64-1: $(objpfx)x86_64/tst-x86_64mod-1.so
>
> diff --git a/sysdeps/x86_64/dl-machine.h b/sysdeps/x86_64/dl-machine.h
> index 6a04cbcdc9..905a37a5cc 100644
> --- a/sysdeps/x86_64/dl-machine.h
> +++ b/sysdeps/x86_64/dl-machine.h
> @@ -66,12 +66,9 @@ static inline int __attribute__ ((unused, always_inline))
> elf_machine_runtime_setup (struct link_map *l, int lazy, int profile)
> {
> Elf64_Addr *got;
> - extern void _dl_runtime_resolve_sse (ElfW(Word)) attribute_hidden;
> - extern void _dl_runtime_resolve_avx (ElfW(Word)) attribute_hidden;
> - extern void _dl_runtime_resolve_avx_slow (ElfW(Word)) attribute_hidden;
> - extern void _dl_runtime_resolve_avx_opt (ElfW(Word)) attribute_hidden;
> - extern void _dl_runtime_resolve_avx512 (ElfW(Word)) attribute_hidden;
> - extern void _dl_runtime_resolve_avx512_opt (ElfW(Word)) attribute_hidden;
> + extern void _dl_runtime_resolve_fxsave (ElfW(Word)) attribute_hidden;
> + extern void _dl_runtime_resolve_xsave (ElfW(Word)) attribute_hidden;
> + extern void _dl_runtime_resolve_xsavec (ElfW(Word)) attribute_hidden;
OK.
> extern void _dl_runtime_profile_sse (ElfW(Word)) attribute_hidden;
> extern void _dl_runtime_profile_avx (ElfW(Word)) attribute_hidden;
> extern void _dl_runtime_profile_avx512 (ElfW(Word)) attribute_hidden;
> @@ -120,29 +117,14 @@ elf_machine_runtime_setup (struct link_map *l, int lazy, int profile)
> /* This function will get called to fix up the GOT entry
> indicated by the offset on the stack, and then jump to
> the resolved address. */
> - if (HAS_ARCH_FEATURE (AVX512F_Usable))
> - {
> - if (HAS_ARCH_FEATURE (Use_dl_runtime_resolve_opt))
> - *(ElfW(Addr) *) (got + 2)
> - = (ElfW(Addr)) &_dl_runtime_resolve_avx512_opt;
> - else
> - *(ElfW(Addr) *) (got + 2)
> - = (ElfW(Addr)) &_dl_runtime_resolve_avx512;
> - }
> - else if (HAS_ARCH_FEATURE (AVX_Usable))
> - {
> - if (HAS_ARCH_FEATURE (Use_dl_runtime_resolve_opt))
> - *(ElfW(Addr) *) (got + 2)
> - = (ElfW(Addr)) &_dl_runtime_resolve_avx_opt;
> - else if (HAS_ARCH_FEATURE (Use_dl_runtime_resolve_slow))
> - *(ElfW(Addr) *) (got + 2)
> - = (ElfW(Addr)) &_dl_runtime_resolve_avx_slow;
> - else
> - *(ElfW(Addr) *) (got + 2)
> - = (ElfW(Addr)) &_dl_runtime_resolve_avx;
> - }
> + if (GLRO(dl_x86_cpu_features).xsave_state_size != 0)
> + *(ElfW(Addr) *) (got + 2)
> + = (HAS_ARCH_FEATURE (XSAVEC_Usable)
> + ? (ElfW(Addr)) &_dl_runtime_resolve_xsavec
> + : (ElfW(Addr)) &_dl_runtime_resolve_xsave);
OK.
> else
> - *(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_resolve_sse;
> + *(ElfW(Addr) *) (got + 2)
> + = (ElfW(Addr)) &_dl_runtime_resolve_fxsave;
> }
> }
>
> diff --git a/sysdeps/x86_64/dl-trampoline.S b/sysdeps/x86_64/dl-trampoline.S
> index c14c61aa58..a645572e44 100644
> --- a/sysdeps/x86_64/dl-trampoline.S
> +++ b/sysdeps/x86_64/dl-trampoline.S
> @@ -34,41 +34,24 @@
> # define DL_STACK_ALIGNMENT 8
> #endif
>
> -#ifndef DL_RUNTIME_UNALIGNED_VEC_SIZE
> -/* The maximum size in bytes of unaligned vector load and store in the
> - dynamic linker. Since SSE optimized memory/string functions with
> - aligned SSE register load and store are used in the dynamic linker,
> - we must set this to 8 so that _dl_runtime_resolve_sse will align the
> - stack before calling _dl_fixup. */
> -# define DL_RUNTIME_UNALIGNED_VEC_SIZE 8
> -#endif
> -
> -/* True if _dl_runtime_resolve should align stack to VEC_SIZE bytes. */
> +/* True if _dl_runtime_resolve should align stack for STATE_SAVE or align
> + stack to 16 bytes before calling _dl_fixup. */
> #define DL_RUNTIME_RESOLVE_REALIGN_STACK \
> - (VEC_SIZE > DL_STACK_ALIGNMENT \
> - && VEC_SIZE > DL_RUNTIME_UNALIGNED_VEC_SIZE)
> -
> -/* Align vector register save area to 16 bytes. */
> -#define REGISTER_SAVE_VEC_OFF 0
> + (STATE_SAVE_ALIGNMENT > DL_STACK_ALIGNMENT \
> + || 16 > DL_STACK_ALIGNMENT)
OK.
>
> /* Area on stack to save and restore registers used for parameter
> passing when calling _dl_fixup. */
> #ifdef __ILP32__
> -# define REGISTER_SAVE_RAX (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 8)
> # define PRESERVE_BND_REGS_PREFIX
> #else
> -/* Align bound register save area to 16 bytes. */
> -# define REGISTER_SAVE_BND0 (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 8)
> -# define REGISTER_SAVE_BND1 (REGISTER_SAVE_BND0 + 16)
> -# define REGISTER_SAVE_BND2 (REGISTER_SAVE_BND1 + 16)
> -# define REGISTER_SAVE_BND3 (REGISTER_SAVE_BND2 + 16)
> -# define REGISTER_SAVE_RAX (REGISTER_SAVE_BND3 + 16)
> # ifdef HAVE_MPX_SUPPORT
> # define PRESERVE_BND_REGS_PREFIX bnd
> # else
> # define PRESERVE_BND_REGS_PREFIX .byte 0xf2
> # endif
> #endif
> +#define REGISTER_SAVE_RAX 0
> #define REGISTER_SAVE_RCX (REGISTER_SAVE_RAX + 8)
> #define REGISTER_SAVE_RDX (REGISTER_SAVE_RCX + 8)
> #define REGISTER_SAVE_RSI (REGISTER_SAVE_RDX + 8)
> @@ -80,68 +63,56 @@
>
> #define VEC_SIZE 64
> #define VMOVA vmovdqa64
> -#if DL_RUNTIME_RESOLVE_REALIGN_STACK || VEC_SIZE <= DL_STACK_ALIGNMENT
> -# define VMOV vmovdqa64
> -#else
> -# define VMOV vmovdqu64
> -#endif
> #define VEC(i) zmm##i
> -#define _dl_runtime_resolve _dl_runtime_resolve_avx512
> #define _dl_runtime_profile _dl_runtime_profile_avx512
> #include "dl-trampoline.h"
> -#undef _dl_runtime_resolve
> #undef _dl_runtime_profile
> #undef VEC
> -#undef VMOV
> #undef VMOVA
> #undef VEC_SIZE
>
> #define VEC_SIZE 32
> #define VMOVA vmovdqa
> -#if DL_RUNTIME_RESOLVE_REALIGN_STACK || VEC_SIZE <= DL_STACK_ALIGNMENT
> -# define VMOV vmovdqa
> -#else
> -# define VMOV vmovdqu
> -#endif
> #define VEC(i) ymm##i
> -#define _dl_runtime_resolve _dl_runtime_resolve_avx
> -#define _dl_runtime_resolve_opt _dl_runtime_resolve_avx_opt
> #define _dl_runtime_profile _dl_runtime_profile_avx
> #include "dl-trampoline.h"
> -#undef _dl_runtime_resolve
> -#undef _dl_runtime_resolve_opt
> #undef _dl_runtime_profile
> #undef VEC
> -#undef VMOV
> #undef VMOVA
> #undef VEC_SIZE
>
> /* movaps/movups is 1-byte shorter. */
> #define VEC_SIZE 16
> #define VMOVA movaps
> -#if DL_RUNTIME_RESOLVE_REALIGN_STACK || VEC_SIZE <= DL_STACK_ALIGNMENT
> -# define VMOV movaps
> -#else
> -# define VMOV movups
> -#endif
> #define VEC(i) xmm##i
> -#define _dl_runtime_resolve _dl_runtime_resolve_sse
> #define _dl_runtime_profile _dl_runtime_profile_sse
> #undef RESTORE_AVX
> #include "dl-trampoline.h"
> -#undef _dl_runtime_resolve
> #undef _dl_runtime_profile
> -#undef VMOV
> +#undef VEC
> #undef VMOVA
> +#undef VEC_SIZE
>
> -/* Used by _dl_runtime_resolve_avx_opt/_dl_runtime_resolve_avx512_opt
> - to preserve the full vector registers with zero upper bits. */
> -#define VMOVA vmovdqa
> -#if DL_RUNTIME_RESOLVE_REALIGN_STACK || VEC_SIZE <= DL_STACK_ALIGNMENT
> -# define VMOV vmovdqa
> -#else
> -# define VMOV vmovdqu
> -#endif
> -#define _dl_runtime_resolve _dl_runtime_resolve_sse_vex
> -#define _dl_runtime_resolve_opt _dl_runtime_resolve_avx512_opt
> +#define USE_FXSAVE
> +#define STATE_SAVE_ALIGNMENT 16
> +#define _dl_runtime_resolve _dl_runtime_resolve_fxsave
> +#include "dl-trampoline.h"
> +#undef _dl_runtime_resolve
> +#undef USE_FXSAVE
> +#undef STATE_SAVE_ALIGNMENT
> +
> +#define USE_XSAVE
> +#define STATE_SAVE_ALIGNMENT 64
> +#define _dl_runtime_resolve _dl_runtime_resolve_xsave
> +#include "dl-trampoline.h"
> +#undef _dl_runtime_resolve
> +#undef USE_XSAVE
> +#undef STATE_SAVE_ALIGNMENT
> +
> +#define USE_XSAVEC
> +#define STATE_SAVE_ALIGNMENT 64
> +#define _dl_runtime_resolve _dl_runtime_resolve_xsavec
> #include "dl-trampoline.h"
> +#undef _dl_runtime_resolve
> +#undef USE_XSAVEC
> +#undef STATE_SAVE_ALIGNMENT
OK.
> diff --git a/sysdeps/x86_64/dl-trampoline.h b/sysdeps/x86_64/dl-trampoline.h
> index 8db24c16ac..dfd7e4b803 100644
> --- a/sysdeps/x86_64/dl-trampoline.h
> +++ b/sysdeps/x86_64/dl-trampoline.h
> @@ -16,140 +16,47 @@
> License along with the GNU C Library; if not, see
> <http://www.gnu.org/licenses/>. */
>
> -#undef REGISTER_SAVE_AREA_RAW
> -#ifdef __ILP32__
> -/* X32 saves RCX, RDX, RSI, RDI, R8 and R9 plus RAX as well as VEC0 to
> - VEC7. */
> -# define REGISTER_SAVE_AREA_RAW (8 * 7 + VEC_SIZE * 8)
> -#else
> -/* X86-64 saves RCX, RDX, RSI, RDI, R8 and R9 plus RAX as well as
> - BND0, BND1, BND2, BND3 and VEC0 to VEC7. */
> -# define REGISTER_SAVE_AREA_RAW (8 * 7 + 16 * 4 + VEC_SIZE * 8)
> -#endif
> + .text
> +#ifdef _dl_runtime_resolve
>
> -#undef REGISTER_SAVE_AREA
> -#undef LOCAL_STORAGE_AREA
> -#undef BASE
> -#if DL_RUNTIME_RESOLVE_REALIGN_STACK
> -# define REGISTER_SAVE_AREA (REGISTER_SAVE_AREA_RAW + 8)
> -/* Local stack area before jumping to function address: RBX. */
> -# define LOCAL_STORAGE_AREA 8
> -# define BASE rbx
> -# if (REGISTER_SAVE_AREA % VEC_SIZE) != 0
> -# error REGISTER_SAVE_AREA must be multples of VEC_SIZE
> -# endif
> -#else
> -# define REGISTER_SAVE_AREA REGISTER_SAVE_AREA_RAW
> -/* Local stack area before jumping to function address: All saved
> - registers. */
> -# define LOCAL_STORAGE_AREA REGISTER_SAVE_AREA
> -# define BASE rsp
> -# if (REGISTER_SAVE_AREA % 16) != 8
> -# error REGISTER_SAVE_AREA must be odd multples of 8
> +# undef REGISTER_SAVE_AREA
> +# undef LOCAL_STORAGE_AREA
> +# undef BASE
> +
> +# if (STATE_SAVE_ALIGNMENT % 16) != 0
> +# error STATE_SAVE_ALIGNMENT must be multples of 16
> # endif
> -#endif
>
> - .text
> -#ifdef _dl_runtime_resolve_opt
> -/* Use the smallest vector registers to preserve the full YMM/ZMM
> - registers to avoid SSE transition penalty. */
> -
> -# if VEC_SIZE == 32
> -/* Check if the upper 128 bits in %ymm0 - %ymm7 registers are non-zero
> - and preserve %xmm0 - %xmm7 registers with the zero upper bits. Since
> - there is no SSE transition penalty on AVX512 processors which don't
> - support XGETBV with ECX == 1, _dl_runtime_resolve_avx512_slow isn't
> - provided. */
> - .globl _dl_runtime_resolve_avx_slow
> - .hidden _dl_runtime_resolve_avx_slow
> - .type _dl_runtime_resolve_avx_slow, @function
> - .align 16
> -_dl_runtime_resolve_avx_slow:
> - cfi_startproc
> - cfi_adjust_cfa_offset(16) # Incorporate PLT
> - vorpd %ymm0, %ymm1, %ymm8
> - vorpd %ymm2, %ymm3, %ymm9
> - vorpd %ymm4, %ymm5, %ymm10
> - vorpd %ymm6, %ymm7, %ymm11
> - vorpd %ymm8, %ymm9, %ymm9
> - vorpd %ymm10, %ymm11, %ymm10
> - vpcmpeqd %xmm8, %xmm8, %xmm8
> - vorpd %ymm9, %ymm10, %ymm10
> - vptest %ymm10, %ymm8
> - # Preserve %ymm0 - %ymm7 registers if the upper 128 bits of any
> - # %ymm0 - %ymm7 registers aren't zero.
> - PRESERVE_BND_REGS_PREFIX
> - jnc _dl_runtime_resolve_avx
> - # Use vzeroupper to avoid SSE transition penalty.
> - vzeroupper
> - # Preserve %xmm0 - %xmm7 registers with the zero upper 128 bits
> - # when the upper 128 bits of %ymm0 - %ymm7 registers are zero.
> - PRESERVE_BND_REGS_PREFIX
> - jmp _dl_runtime_resolve_sse_vex
> - cfi_adjust_cfa_offset(-16) # Restore PLT adjustment
> - cfi_endproc
> - .size _dl_runtime_resolve_avx_slow, .-_dl_runtime_resolve_avx_slow
> +# if (STATE_SAVE_OFFSET % STATE_SAVE_ALIGNMENT) != 0
> +# error STATE_SAVE_OFFSET must be multples of STATE_SAVE_ALIGNMENT
> # endif
>
> -/* Use XGETBV with ECX == 1 to check which bits in vector registers are
> - non-zero and only preserve the non-zero lower bits with zero upper
> - bits. */
> - .globl _dl_runtime_resolve_opt
> - .hidden _dl_runtime_resolve_opt
> - .type _dl_runtime_resolve_opt, @function
> - .align 16
> -_dl_runtime_resolve_opt:
> - cfi_startproc
> - cfi_adjust_cfa_offset(16) # Incorporate PLT
> - pushq %rax
> - cfi_adjust_cfa_offset(8)
> - cfi_rel_offset(%rax, 0)
> - pushq %rcx
> - cfi_adjust_cfa_offset(8)
> - cfi_rel_offset(%rcx, 0)
> - pushq %rdx
> - cfi_adjust_cfa_offset(8)
> - cfi_rel_offset(%rdx, 0)
> - movl $1, %ecx
> - xgetbv
> - movl %eax, %r11d
> - popq %rdx
> - cfi_adjust_cfa_offset(-8)
> - cfi_restore (%rdx)
> - popq %rcx
> - cfi_adjust_cfa_offset(-8)
> - cfi_restore (%rcx)
> - popq %rax
> - cfi_adjust_cfa_offset(-8)
> - cfi_restore (%rax)
> -# if VEC_SIZE == 32
> - # For YMM registers, check if YMM state is in use.
> - andl $bit_YMM_state, %r11d
> - # Preserve %xmm0 - %xmm7 registers with the zero upper 128 bits if
> - # YMM state isn't in use.
> - PRESERVE_BND_REGS_PREFIX
> - jz _dl_runtime_resolve_sse_vex
> -# elif VEC_SIZE == 16
> - # For ZMM registers, check if YMM state and ZMM state are in
> - # use.
> - andl $(bit_YMM_state | bit_ZMM0_15_state), %r11d
> - cmpl $bit_YMM_state, %r11d
> - # Preserve %zmm0 - %zmm7 registers if ZMM state is in use.
> - PRESERVE_BND_REGS_PREFIX
> - jg _dl_runtime_resolve_avx512
> - # Preserve %ymm0 - %ymm7 registers with the zero upper 256 bits if
> - # ZMM state isn't in use.
> - PRESERVE_BND_REGS_PREFIX
> - je _dl_runtime_resolve_avx
> - # Preserve %xmm0 - %xmm7 registers with the zero upper 384 bits if
> - # neither YMM state nor ZMM state are in use.
> +# if DL_RUNTIME_RESOLVE_REALIGN_STACK
> +/* Local stack area before jumping to function address: RBX. */
> +# define LOCAL_STORAGE_AREA 8
> +# define BASE rbx
> +# ifdef USE_FXSAVE
> +/* Use fxsave to save XMM registers. */
> +# define REGISTER_SAVE_AREA (512 + STATE_SAVE_OFFSET)
> +# if (REGISTER_SAVE_AREA % 16) != 0
> +# error REGISTER_SAVE_AREA must be multples of 16
> +# endif
> +# endif
> # else
> -# error Unsupported VEC_SIZE!
> +# ifndef USE_FXSAVE
> +# error USE_FXSAVE must be defined
> +# endif
> +/* Use fxsave to save XMM registers. */
> +# define REGISTER_SAVE_AREA (512 + STATE_SAVE_OFFSET + 8)
> +/* Local stack area before jumping to function address: All saved
> + registers. */
> +# define LOCAL_STORAGE_AREA REGISTER_SAVE_AREA
> +# define BASE rsp
> +# if (REGISTER_SAVE_AREA % 16) != 8
> +# error REGISTER_SAVE_AREA must be odd multples of 8
> +# endif
> # endif
> - cfi_adjust_cfa_offset(-16) # Restore PLT adjustment
> - cfi_endproc
> - .size _dl_runtime_resolve_opt, .-_dl_runtime_resolve_opt
> -#endif
> +
> .globl _dl_runtime_resolve
> .hidden _dl_runtime_resolve
> .type _dl_runtime_resolve, @function
> @@ -157,21 +64,29 @@ _dl_runtime_resolve_opt:
> cfi_startproc
> _dl_runtime_resolve:
> cfi_adjust_cfa_offset(16) # Incorporate PLT
> -#if DL_RUNTIME_RESOLVE_REALIGN_STACK
> -# if LOCAL_STORAGE_AREA != 8
> -# error LOCAL_STORAGE_AREA must be 8
> -# endif
> +# if DL_RUNTIME_RESOLVE_REALIGN_STACK
> +# if LOCAL_STORAGE_AREA != 8
> +# error LOCAL_STORAGE_AREA must be 8
> +# endif
OK.
> pushq %rbx # push subtracts stack by 8.
> cfi_adjust_cfa_offset(8)
> cfi_rel_offset(%rbx, 0)
> mov %RSP_LP, %RBX_LP
> cfi_def_cfa_register(%rbx)
> - and $-VEC_SIZE, %RSP_LP
> -#endif
> + and $-STATE_SAVE_ALIGNMENT, %RSP_LP
> +# endif
> +# ifdef REGISTER_SAVE_AREA
> sub $REGISTER_SAVE_AREA, %RSP_LP
> -#if !DL_RUNTIME_RESOLVE_REALIGN_STACK
> +# if !DL_RUNTIME_RESOLVE_REALIGN_STACK
> cfi_adjust_cfa_offset(REGISTER_SAVE_AREA)
> -#endif
> +# endif
> +# else
> +# if IS_IN (rtld)
> + sub _rtld_local_ro+RTLD_GLOBAL_RO_DL_X86_CPU_FEATURES_OFFSET+XSAVE_STATE_SIZE_OFFSET(%rip), %RSP_LP
> +# else
> + sub _dl_x86_cpu_features+XSAVE_STATE_SIZE_OFFSET(%rip), %RSP_LP
> +# endif
> +# endif
OK. Allocate stack space of the required size to save the results.
> # Preserve registers otherwise clobbered.
> movq %rax, REGISTER_SAVE_RAX(%rsp)
> movq %rcx, REGISTER_SAVE_RCX(%rsp)
> @@ -180,59 +95,42 @@ _dl_runtime_resolve:
> movq %rdi, REGISTER_SAVE_RDI(%rsp)
> movq %r8, REGISTER_SAVE_R8(%rsp)
> movq %r9, REGISTER_SAVE_R9(%rsp)
> - VMOV %VEC(0), (REGISTER_SAVE_VEC_OFF)(%rsp)
> - VMOV %VEC(1), (REGISTER_SAVE_VEC_OFF + VEC_SIZE)(%rsp)
> - VMOV %VEC(2), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 2)(%rsp)
> - VMOV %VEC(3), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 3)(%rsp)
> - VMOV %VEC(4), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 4)(%rsp)
> - VMOV %VEC(5), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 5)(%rsp)
> - VMOV %VEC(6), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 6)(%rsp)
> - VMOV %VEC(7), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 7)(%rsp)
> -#ifndef __ILP32__
> - # We also have to preserve bound registers. These are nops if
> - # Intel MPX isn't available or disabled.
> -# ifdef HAVE_MPX_SUPPORT
> - bndmov %bnd0, REGISTER_SAVE_BND0(%rsp)
> - bndmov %bnd1, REGISTER_SAVE_BND1(%rsp)
> - bndmov %bnd2, REGISTER_SAVE_BND2(%rsp)
> - bndmov %bnd3, REGISTER_SAVE_BND3(%rsp)
> +# ifdef USE_FXSAVE
> + fxsave STATE_SAVE_OFFSET(%rsp)
OK.
> # else
> -# if REGISTER_SAVE_BND0 == 0
> - .byte 0x66,0x0f,0x1b,0x04,0x24
> + movl $STATE_SAVE_MASK, %eax
> + xorl %edx, %edx
> + # Clear the XSAVE Header.
> +# ifdef USE_XSAVE
> + movq %rdx, (STATE_SAVE_OFFSET + 512)(%rsp)
> + movq %rdx, (STATE_SAVE_OFFSET + 512 + 8)(%rsp)
> +# endif
> + movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 2)(%rsp)
> + movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 3)(%rsp)
> + movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 4)(%rsp)
> + movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 5)(%rsp)
> + movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 6)(%rsp)
> + movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 7)(%rsp)
> +# ifdef USE_XSAVE
> + xsave STATE_SAVE_OFFSET(%rsp)
OK.
> # else
> - .byte 0x66,0x0f,0x1b,0x44,0x24,REGISTER_SAVE_BND0
> + xsavec STATE_SAVE_OFFSET(%rsp)
OK.
> # endif
> - .byte 0x66,0x0f,0x1b,0x4c,0x24,REGISTER_SAVE_BND1
> - .byte 0x66,0x0f,0x1b,0x54,0x24,REGISTER_SAVE_BND2
> - .byte 0x66,0x0f,0x1b,0x5c,0x24,REGISTER_SAVE_BND3
> # endif
> -#endif
> # Copy args pushed by PLT in register.
> # %rdi: link_map, %rsi: reloc_index
> mov (LOCAL_STORAGE_AREA + 8)(%BASE), %RSI_LP
> mov LOCAL_STORAGE_AREA(%BASE), %RDI_LP
> call _dl_fixup # Call resolver.
> mov %RAX_LP, %R11_LP # Save return value
> -#ifndef __ILP32__
> - # Restore bound registers. These are nops if Intel MPX isn't
> - # avaiable or disabled.
> -# ifdef HAVE_MPX_SUPPORT
> - bndmov REGISTER_SAVE_BND3(%rsp), %bnd3
> - bndmov REGISTER_SAVE_BND2(%rsp), %bnd2
> - bndmov REGISTER_SAVE_BND1(%rsp), %bnd1
> - bndmov REGISTER_SAVE_BND0(%rsp), %bnd0
> + # Get register content back.
> +# ifdef USE_FXSAVE
> + fxrstor STATE_SAVE_OFFSET(%rsp)
> # else
> - .byte 0x66,0x0f,0x1a,0x5c,0x24,REGISTER_SAVE_BND3
> - .byte 0x66,0x0f,0x1a,0x54,0x24,REGISTER_SAVE_BND2
> - .byte 0x66,0x0f,0x1a,0x4c,0x24,REGISTER_SAVE_BND1
> -# if REGISTER_SAVE_BND0 == 0
> - .byte 0x66,0x0f,0x1a,0x04,0x24
> -# else
> - .byte 0x66,0x0f,0x1a,0x44,0x24,REGISTER_SAVE_BND0
> -# endif
> + movl $STATE_SAVE_MASK, %eax
> + xorl %edx, %edx
> + xrstor STATE_SAVE_OFFSET(%rsp)
OK.
> # endif
> -#endif
> - # Get register content back.
> movq REGISTER_SAVE_R9(%rsp), %r9
> movq REGISTER_SAVE_R8(%rsp), %r8
> movq REGISTER_SAVE_RDI(%rsp), %rdi
> @@ -240,20 +138,12 @@ _dl_runtime_resolve:
> movq REGISTER_SAVE_RDX(%rsp), %rdx
> movq REGISTER_SAVE_RCX(%rsp), %rcx
> movq REGISTER_SAVE_RAX(%rsp), %rax
> - VMOV (REGISTER_SAVE_VEC_OFF)(%rsp), %VEC(0)
> - VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE)(%rsp), %VEC(1)
> - VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 2)(%rsp), %VEC(2)
> - VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 3)(%rsp), %VEC(3)
> - VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 4)(%rsp), %VEC(4)
> - VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 5)(%rsp), %VEC(5)
> - VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 6)(%rsp), %VEC(6)
> - VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 7)(%rsp), %VEC(7)
OK. Don't need these any more.
> -#if DL_RUNTIME_RESOLVE_REALIGN_STACK
> +# if DL_RUNTIME_RESOLVE_REALIGN_STACK
> mov %RBX_LP, %RSP_LP
> cfi_def_cfa_register(%rsp)
> movq (%rsp), %rbx
> cfi_restore(%rbx)
> -#endif
> +# endif
> # Adjust stack(PLT did 2 pushes)
> add $(LOCAL_STORAGE_AREA + 16), %RSP_LP
> cfi_adjust_cfa_offset(-(LOCAL_STORAGE_AREA + 16))
> @@ -262,11 +152,9 @@ _dl_runtime_resolve:
> jmp *%r11 # Jump to function address.
> cfi_endproc
> .size _dl_runtime_resolve, .-_dl_runtime_resolve
> +#endif
>
>
> -/* To preserve %xmm0 - %xmm7 registers, dl-trampoline.h is included
> - twice, for _dl_runtime_resolve_sse and _dl_runtime_resolve_sse_vex.
> - But we don't need another _dl_runtime_profile for XMM registers. */
OK.
> #if !defined PROF && defined _dl_runtime_profile
> # if (LR_VECTOR_OFFSET % VEC_SIZE) != 0
> # error LR_VECTOR_OFFSET must be multples of VEC_SIZE
> -- 2.13.6
From 282a76afb190e7908a70b8762f28eb809459595e Mon Sep 17 00:00:00 2001
From: "H.J. Lu" <hjl.tools@gmail.com>
Date: Thu, 23 Mar 2017 08:21:52 -0700
Subject: [PATCH] x86-64: Use fxsave/xsave/xsavec in _dl_runtime_resolve [BZ
#21265]
In _dl_runtime_resolve, use fxsave/xsave/xsavec to preserve all vector,
mask and bound registers. It simplifies _dl_runtime_resolve and supports
different calling conventions. ld.so code size is reduced by more than
1 KB. However, use fxsave/xsave/xsavec takes a little bit more cycles
than saving and restoring vector and bound registers individually.
Latency for _dl_runtime_resolve to lookup the function, foo, from one
shared library plus libc.so:
Before After Change
Westmere (SSE)/fxsave 345 866 151%
IvyBridge (AVX)/xsave 420 643 53%
Haswell (AVX)/xsave 713 1252 75%
Skylake (AVX+MPX)/xsavec 559 719 28%
Skylake (AVX512+MPX)/xsavec 145 272 87%
Ryzen (AVX)/xsavec 280 553 97%
This is the worst case where portion of time spent for saving and
restoring registers is bigger than majority of cases. With smaller
_dl_runtime_resolve code size, overall performance impact is negligible.
On IvyBridge, differences in build and test time of binutils with lazy
binding GCC and binutils are noises. On Westmere, differences in
bootstrap and "makc check" time of GCC 7 with lazy binding GCC and
binutils are also noises.
[BZ #21265]
* sysdeps/x86/cpu-features-offsets.sym (XSAVE_STATE_SIZE_OFFSET):
New.
* sysdeps/x86/cpu-features.c (get_common_indeces): Set
xsave_state_size, xsave_state_full_size and
bit_arch_XSAVEC_Usable if needed.
(init_cpu_features): Remove bit_arch_Use_dl_runtime_resolve_slow
and bit_arch_Use_dl_runtime_resolve_opt.
* sysdeps/x86/cpu-features.h (bit_arch_Use_dl_runtime_resolve_opt):
Removed.
(bit_arch_Use_dl_runtime_resolve_slow): Likewise.
(bit_arch_Prefer_No_AVX512): Updated.
(bit_arch_MathVec_Prefer_No_AVX512): Likewise.
(bit_arch_XSAVEC_Usable): New.
(STATE_SAVE_OFFSET): Likewise.
(STATE_SAVE_MASK): Likewise.
[__ASSEMBLER__]: Include <cpu-features-offsets.h>.
(cpu_features): Add xsave_state_size and xsave_state_full_size.
(index_arch_Use_dl_runtime_resolve_opt): Removed.
(index_arch_Use_dl_runtime_resolve_slow): Likewise.
(index_arch_XSAVEC_Usable): New.
* sysdeps/x86/cpu-tunables.c (TUNABLE_CALLBACK (set_hwcaps)):
Support XSAVEC_Usable. Remove Use_dl_runtime_resolve_slow.
* sysdeps/x86_64/Makefile (tst-x86_64-1-ENV): New if tunables
is enabled.
* sysdeps/x86_64/dl-machine.h (elf_machine_runtime_setup):
Replace _dl_runtime_resolve_sse, _dl_runtime_resolve_avx,
_dl_runtime_resolve_avx_slow, _dl_runtime_resolve_avx_opt,
_dl_runtime_resolve_avx512 and _dl_runtime_resolve_avx512_opt
with _dl_runtime_resolve_fxsave, _dl_runtime_resolve_xsave and
_dl_runtime_resolve_xsavec.
* sysdeps/x86_64/dl-trampoline.S (DL_RUNTIME_UNALIGNED_VEC_SIZE):
Removed.
(DL_RUNTIME_RESOLVE_REALIGN_STACK): Check STATE_SAVE_ALIGNMENT
instead of VEC_SIZE.
(REGISTER_SAVE_BND0): Removed.
(REGISTER_SAVE_BND1): Likewise.
(REGISTER_SAVE_BND3): Likewise.
(REGISTER_SAVE_RAX): Always defined to 0.
(VMOV): Removed.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_slow): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_avx512): Likewise.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(USE_FXSAVE): New.
(_dl_runtime_resolve_fxsave): Likewise.
(USE_XSAVE): Likewise.
(_dl_runtime_resolve_xsave): Likewise.
(USE_XSAVEC): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
* sysdeps/x86_64/dl-trampoline.h (_dl_runtime_resolve_avx512):
Removed.
(_dl_runtime_resolve_avx512_opt): Likewise.
(_dl_runtime_resolve_avx): Likewise.
(_dl_runtime_resolve_avx_opt): Likewise.
(_dl_runtime_resolve_sse): Likewise.
(_dl_runtime_resolve_sse_vex): Likewise.
(_dl_runtime_resolve_fxsave): New.
(_dl_runtime_resolve_xsave): Likewise.
(_dl_runtime_resolve_xsavec): Likewise.
---
sysdeps/x86/cpu-features-offsets.sym | 1 +
sysdeps/x86/cpu-features.c | 87 +++++++++---
sysdeps/x86/cpu-features.h | 34 ++++-
sysdeps/x86/cpu-tunables.c | 17 ++-
sysdeps/x86_64/Makefile | 4 +
sysdeps/x86_64/dl-machine.h | 38 ++---
sysdeps/x86_64/dl-trampoline.S | 87 ++++--------
sysdeps/x86_64/dl-trampoline.h | 266 ++++++++++-------------------------
8 files changed, 228 insertions(+), 306 deletions(-)
@@ -15,6 +15,7 @@ CPUID_ECX_OFFSET offsetof (struct cpuid_registers, ecx)
CPUID_EDX_OFFSET offsetof (struct cpuid_registers, edx)
FAMILY_OFFSET offsetof (struct cpu_features, family)
MODEL_OFFSET offsetof (struct cpu_features, model)
+XSAVE_STATE_SIZE_OFFSET offsetof (struct cpu_features, xsave_state_size)
FEATURE_OFFSET offsetof (struct cpu_features, feature)
FEATURE_SIZE sizeof (unsigned int)
@@ -103,6 +103,76 @@ get_common_indeces (struct cpu_features *cpu_features,
}
}
}
+
+ /* For _dl_runtime_resolve, set xsave_state_size to xsave area
+ size + integer register save size and align it to 64 bytes. */
+ if (cpu_features->max_cpuid >= 0xd)
+ {
+ unsigned int eax, ebx, ecx, edx;
+
+ __cpuid_count (0xd, 0, eax, ebx, ecx, edx);
+ if (ebx != 0)
+ {
+ unsigned int xsave_state_full_size
+ = (ebx + STATE_SAVE_OFFSET + 63) & -64;
+
+ cpu_features->xsave_state_size
+ = xsave_state_full_size;
+ cpu_features->xsave_state_full_size
+ = xsave_state_full_size;
+
+ __cpuid_count (0xd, 1, eax, ebx, ecx, edx);
+
+ /* Check if XSAVEC is available. */
+ if ((eax & (1 << 1)) != 0)
+ {
+ unsigned int xstate_comp_offsets[32];
+ unsigned int xstate_comp_sizes[32];
+ unsigned int i;
+
+ xstate_comp_offsets[0] = 0;
+ xstate_comp_offsets[1] = 160;
+ xstate_comp_offsets[2] = 576;
+ xstate_comp_sizes[0] = 160;
+ xstate_comp_sizes[1] = 256;
+
+ for (i = 2; i < 32; i++)
+ {
+ if ((STATE_SAVE_MASK & (1 << i)) != 0)
+ {
+ __cpuid_count (0xd, i, eax, ebx, ecx, edx);
+ xstate_comp_sizes[i] = eax;
+ }
+ else
+ {
+ ecx = 0;
+ xstate_comp_sizes[i] = 0;
+ }
+
+ if (i > 2)
+ {
+ xstate_comp_offsets[i]
+ = (xstate_comp_offsets[i - 1]
+ + xstate_comp_sizes[i -1]);
+ if ((ecx & (1 << 1)) != 0)
+ xstate_comp_offsets[i]
+ = (xstate_comp_offsets[i] + 63) & -64;
+ }
+ }
+
+ /* Use XSAVEC. */
+ unsigned int size
+ = xstate_comp_offsets[31] + xstate_comp_sizes[31];
+ if (size)
+ {
+ cpu_features->xsave_state_size
+ = (size + STATE_SAVE_OFFSET + 63) & -64;
+ cpu_features->feature[index_arch_XSAVEC_Usable]
+ |= bit_arch_XSAVEC_Usable;
+ }
+ }
+ }
+ }
}
}
@@ -242,23 +312,6 @@ init_cpu_features (struct cpu_features *cpu_features)
else
cpu_features->feature[index_arch_Prefer_No_AVX512]
|= bit_arch_Prefer_No_AVX512;
-
- /* To avoid SSE transition penalty, use _dl_runtime_resolve_slow.
- If XGETBV suports ECX == 1, use _dl_runtime_resolve_opt.
- Use _dl_runtime_resolve_opt only with AVX512F since it is
- slower than _dl_runtime_resolve_slow with AVX. */
- cpu_features->feature[index_arch_Use_dl_runtime_resolve_slow]
- |= bit_arch_Use_dl_runtime_resolve_slow;
- if (CPU_FEATURES_ARCH_P (cpu_features, AVX512F_Usable)
- && cpu_features->max_cpuid >= 0xd)
- {
- unsigned int eax;
-
- __cpuid_count (0xd, 1, eax, ebx, ecx, edx);
- if ((eax & (1 << 2)) != 0)
- cpu_features->feature[index_arch_Use_dl_runtime_resolve_opt]
- |= bit_arch_Use_dl_runtime_resolve_opt;
- }
}
/* This spells out "AuthenticAMD". */
else if (ebx == 0x68747541 && ecx == 0x444d4163 && edx == 0x69746e65)
@@ -37,10 +37,9 @@
#define bit_arch_Prefer_No_VZEROUPPER (1 << 17)
#define bit_arch_Fast_Unaligned_Copy (1 << 18)
#define bit_arch_Prefer_ERMS (1 << 19)
-#define bit_arch_Use_dl_runtime_resolve_opt (1 << 20)
-#define bit_arch_Use_dl_runtime_resolve_slow (1 << 21)
-#define bit_arch_Prefer_No_AVX512 (1 << 22)
-#define bit_arch_MathVec_Prefer_No_AVX512 (1 << 23)
+#define bit_arch_Prefer_No_AVX512 (1 << 20)
+#define bit_arch_MathVec_Prefer_No_AVX512 (1 << 21)
+#define bit_arch_XSAVEC_Usable (1 << 22)
/* CPUID Feature flags. */
@@ -91,8 +90,18 @@
/* The current maximum size of the feature integer bit array. */
#define FEATURE_INDEX_MAX 1
-#ifndef __ASSEMBLER__
+/* Offset for fxsave/xsave area used by _dl_runtime_resolve. Also need
+ space to preserve RCX, RDX, RSI, RDI, R8, R9 and RAX. It must be
+ aligned to 16 bytes for fxsave and 64 bytes for xsave. */
+#define STATE_SAVE_OFFSET (8 * 7 + 8)
+/* Save SSE, AVX, AVX512, mask and bound registers. */
+#define STATE_SAVE_MASK \
+ ((1 << 1) | (1 << 2) | (1 << 3) | (1 << 5) | (1 << 6) | (1 << 7))
+
+#ifdef __ASSEMBLER__
+# include <cpu-features-offsets.h>
+#else /* __ASSEMBLER__ */
enum
{
COMMON_CPUID_INDEX_1 = 0,
@@ -121,6 +130,18 @@ struct cpu_features
} cpuid[COMMON_CPUID_INDEX_MAX];
unsigned int family;
unsigned int model;
+ /* The state size for XSAVEC or XSAVE. The type must be unsigned long
+ int so that we use
+
+ sub xsave_state_size_offset(%rip) %RSP_LP
+
+ in _dl_runtime_resolve. */
+ unsigned long int xsave_state_size;
+ /* The full state size for XSAVE when XSAVEC is disabled by
+
+ GLIBC_TUNABLES=glibc.tune.hwcaps=-XSAVEC_Usable
+ */
+ unsigned int xsave_state_full_size;
unsigned int feature[FEATURE_INDEX_MAX];
/* Data cache size for use in memory and string routines, typically
L1 size. */
@@ -237,10 +258,9 @@ extern const struct cpu_features *__get_cpu_features (void)
# define index_arch_Prefer_No_VZEROUPPER FEATURE_INDEX_1
# define index_arch_Fast_Unaligned_Copy FEATURE_INDEX_1
# define index_arch_Prefer_ERMS FEATURE_INDEX_1
-# define index_arch_Use_dl_runtime_resolve_opt FEATURE_INDEX_1
-# define index_arch_Use_dl_runtime_resolve_slow FEATURE_INDEX_1
# define index_arch_Prefer_No_AVX512 FEATURE_INDEX_1
# define index_arch_MathVec_Prefer_No_AVX512 FEATURE_INDEX_1
+# define index_arch_XSAVEC_Usable FEATURE_INDEX_1
#endif /* !__ASSEMBLER__ */
@@ -242,6 +242,16 @@ TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *valp)
Slow_SSE4_2, SSE4_2,
disable, 11);
break;
+ case 13:
+ if (disable)
+ {
+ /* Update xsave_state_size to XSAVE state size. */
+ cpu_features->xsave_state_size
+ = cpu_features->xsave_state_full_size;
+ CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features,
+ XSAVEC_Usable, 13);
+ }
+ break;
case 14:
if (disable)
{
@@ -317,13 +327,6 @@ TUNABLE_CALLBACK (set_hwcaps) (tunable_val_t *valp)
disable, 26);
}
break;
- case 27:
- {
- CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features,
- Use_dl_runtime_resolve_slow,
- disable, 27);
- }
- break;
}
p += len + 1;
}
@@ -55,6 +55,10 @@ CFLAGS-tst-quad2pie.c = $(PIE-ccflag)
tests += tst-x86_64-1
modules-names += x86_64/tst-x86_64mod-1
LDFLAGS-tst-x86_64mod-1.so = -Wl,-soname,tst-x86_64mod-1.so
+ifneq (no,$(have-tunables))
+# Test the state size for XSAVE when XSAVEC is disabled.
+tst-x86_64-1-ENV = GLIBC_TUNABLES=glibc.tune.hwcaps=-XSAVEC_Usable
+endif
$(objpfx)tst-x86_64-1: $(objpfx)x86_64/tst-x86_64mod-1.so
@@ -66,12 +66,9 @@ static inline int __attribute__ ((unused, always_inline))
elf_machine_runtime_setup (struct link_map *l, int lazy, int profile)
{
Elf64_Addr *got;
- extern void _dl_runtime_resolve_sse (ElfW(Word)) attribute_hidden;
- extern void _dl_runtime_resolve_avx (ElfW(Word)) attribute_hidden;
- extern void _dl_runtime_resolve_avx_slow (ElfW(Word)) attribute_hidden;
- extern void _dl_runtime_resolve_avx_opt (ElfW(Word)) attribute_hidden;
- extern void _dl_runtime_resolve_avx512 (ElfW(Word)) attribute_hidden;
- extern void _dl_runtime_resolve_avx512_opt (ElfW(Word)) attribute_hidden;
+ extern void _dl_runtime_resolve_fxsave (ElfW(Word)) attribute_hidden;
+ extern void _dl_runtime_resolve_xsave (ElfW(Word)) attribute_hidden;
+ extern void _dl_runtime_resolve_xsavec (ElfW(Word)) attribute_hidden;
extern void _dl_runtime_profile_sse (ElfW(Word)) attribute_hidden;
extern void _dl_runtime_profile_avx (ElfW(Word)) attribute_hidden;
extern void _dl_runtime_profile_avx512 (ElfW(Word)) attribute_hidden;
@@ -120,29 +117,14 @@ elf_machine_runtime_setup (struct link_map *l, int lazy, int profile)
/* This function will get called to fix up the GOT entry
indicated by the offset on the stack, and then jump to
the resolved address. */
- if (HAS_ARCH_FEATURE (AVX512F_Usable))
- {
- if (HAS_ARCH_FEATURE (Use_dl_runtime_resolve_opt))
- *(ElfW(Addr) *) (got + 2)
- = (ElfW(Addr)) &_dl_runtime_resolve_avx512_opt;
- else
- *(ElfW(Addr) *) (got + 2)
- = (ElfW(Addr)) &_dl_runtime_resolve_avx512;
- }
- else if (HAS_ARCH_FEATURE (AVX_Usable))
- {
- if (HAS_ARCH_FEATURE (Use_dl_runtime_resolve_opt))
- *(ElfW(Addr) *) (got + 2)
- = (ElfW(Addr)) &_dl_runtime_resolve_avx_opt;
- else if (HAS_ARCH_FEATURE (Use_dl_runtime_resolve_slow))
- *(ElfW(Addr) *) (got + 2)
- = (ElfW(Addr)) &_dl_runtime_resolve_avx_slow;
- else
- *(ElfW(Addr) *) (got + 2)
- = (ElfW(Addr)) &_dl_runtime_resolve_avx;
- }
+ if (GLRO(dl_x86_cpu_features).xsave_state_size != 0)
+ *(ElfW(Addr) *) (got + 2)
+ = (HAS_ARCH_FEATURE (XSAVEC_Usable)
+ ? (ElfW(Addr)) &_dl_runtime_resolve_xsavec
+ : (ElfW(Addr)) &_dl_runtime_resolve_xsave);
else
- *(ElfW(Addr) *) (got + 2) = (ElfW(Addr)) &_dl_runtime_resolve_sse;
+ *(ElfW(Addr) *) (got + 2)
+ = (ElfW(Addr)) &_dl_runtime_resolve_fxsave;
}
}
@@ -34,41 +34,24 @@
# define DL_STACK_ALIGNMENT 8
#endif
-#ifndef DL_RUNTIME_UNALIGNED_VEC_SIZE
-/* The maximum size in bytes of unaligned vector load and store in the
- dynamic linker. Since SSE optimized memory/string functions with
- aligned SSE register load and store are used in the dynamic linker,
- we must set this to 8 so that _dl_runtime_resolve_sse will align the
- stack before calling _dl_fixup. */
-# define DL_RUNTIME_UNALIGNED_VEC_SIZE 8
-#endif
-
-/* True if _dl_runtime_resolve should align stack to VEC_SIZE bytes. */
+/* True if _dl_runtime_resolve should align stack for STATE_SAVE or align
+ stack to 16 bytes before calling _dl_fixup. */
#define DL_RUNTIME_RESOLVE_REALIGN_STACK \
- (VEC_SIZE > DL_STACK_ALIGNMENT \
- && VEC_SIZE > DL_RUNTIME_UNALIGNED_VEC_SIZE)
-
-/* Align vector register save area to 16 bytes. */
-#define REGISTER_SAVE_VEC_OFF 0
+ (STATE_SAVE_ALIGNMENT > DL_STACK_ALIGNMENT \
+ || 16 > DL_STACK_ALIGNMENT)
/* Area on stack to save and restore registers used for parameter
passing when calling _dl_fixup. */
#ifdef __ILP32__
-# define REGISTER_SAVE_RAX (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 8)
# define PRESERVE_BND_REGS_PREFIX
#else
-/* Align bound register save area to 16 bytes. */
-# define REGISTER_SAVE_BND0 (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 8)
-# define REGISTER_SAVE_BND1 (REGISTER_SAVE_BND0 + 16)
-# define REGISTER_SAVE_BND2 (REGISTER_SAVE_BND1 + 16)
-# define REGISTER_SAVE_BND3 (REGISTER_SAVE_BND2 + 16)
-# define REGISTER_SAVE_RAX (REGISTER_SAVE_BND3 + 16)
# ifdef HAVE_MPX_SUPPORT
# define PRESERVE_BND_REGS_PREFIX bnd
# else
# define PRESERVE_BND_REGS_PREFIX .byte 0xf2
# endif
#endif
+#define REGISTER_SAVE_RAX 0
#define REGISTER_SAVE_RCX (REGISTER_SAVE_RAX + 8)
#define REGISTER_SAVE_RDX (REGISTER_SAVE_RCX + 8)
#define REGISTER_SAVE_RSI (REGISTER_SAVE_RDX + 8)
@@ -80,68 +63,56 @@
#define VEC_SIZE 64
#define VMOVA vmovdqa64
-#if DL_RUNTIME_RESOLVE_REALIGN_STACK || VEC_SIZE <= DL_STACK_ALIGNMENT
-# define VMOV vmovdqa64
-#else
-# define VMOV vmovdqu64
-#endif
#define VEC(i) zmm##i
-#define _dl_runtime_resolve _dl_runtime_resolve_avx512
#define _dl_runtime_profile _dl_runtime_profile_avx512
#include "dl-trampoline.h"
-#undef _dl_runtime_resolve
#undef _dl_runtime_profile
#undef VEC
-#undef VMOV
#undef VMOVA
#undef VEC_SIZE
#define VEC_SIZE 32
#define VMOVA vmovdqa
-#if DL_RUNTIME_RESOLVE_REALIGN_STACK || VEC_SIZE <= DL_STACK_ALIGNMENT
-# define VMOV vmovdqa
-#else
-# define VMOV vmovdqu
-#endif
#define VEC(i) ymm##i
-#define _dl_runtime_resolve _dl_runtime_resolve_avx
-#define _dl_runtime_resolve_opt _dl_runtime_resolve_avx_opt
#define _dl_runtime_profile _dl_runtime_profile_avx
#include "dl-trampoline.h"
-#undef _dl_runtime_resolve
-#undef _dl_runtime_resolve_opt
#undef _dl_runtime_profile
#undef VEC
-#undef VMOV
#undef VMOVA
#undef VEC_SIZE
/* movaps/movups is 1-byte shorter. */
#define VEC_SIZE 16
#define VMOVA movaps
-#if DL_RUNTIME_RESOLVE_REALIGN_STACK || VEC_SIZE <= DL_STACK_ALIGNMENT
-# define VMOV movaps
-#else
-# define VMOV movups
-#endif
#define VEC(i) xmm##i
-#define _dl_runtime_resolve _dl_runtime_resolve_sse
#define _dl_runtime_profile _dl_runtime_profile_sse
#undef RESTORE_AVX
#include "dl-trampoline.h"
-#undef _dl_runtime_resolve
#undef _dl_runtime_profile
-#undef VMOV
+#undef VEC
#undef VMOVA
+#undef VEC_SIZE
-/* Used by _dl_runtime_resolve_avx_opt/_dl_runtime_resolve_avx512_opt
- to preserve the full vector registers with zero upper bits. */
-#define VMOVA vmovdqa
-#if DL_RUNTIME_RESOLVE_REALIGN_STACK || VEC_SIZE <= DL_STACK_ALIGNMENT
-# define VMOV vmovdqa
-#else
-# define VMOV vmovdqu
-#endif
-#define _dl_runtime_resolve _dl_runtime_resolve_sse_vex
-#define _dl_runtime_resolve_opt _dl_runtime_resolve_avx512_opt
+#define USE_FXSAVE
+#define STATE_SAVE_ALIGNMENT 16
+#define _dl_runtime_resolve _dl_runtime_resolve_fxsave
+#include "dl-trampoline.h"
+#undef _dl_runtime_resolve
+#undef USE_FXSAVE
+#undef STATE_SAVE_ALIGNMENT
+
+#define USE_XSAVE
+#define STATE_SAVE_ALIGNMENT 64
+#define _dl_runtime_resolve _dl_runtime_resolve_xsave
+#include "dl-trampoline.h"
+#undef _dl_runtime_resolve
+#undef USE_XSAVE
+#undef STATE_SAVE_ALIGNMENT
+
+#define USE_XSAVEC
+#define STATE_SAVE_ALIGNMENT 64
+#define _dl_runtime_resolve _dl_runtime_resolve_xsavec
#include "dl-trampoline.h"
+#undef _dl_runtime_resolve
+#undef USE_XSAVEC
+#undef STATE_SAVE_ALIGNMENT
@@ -16,140 +16,47 @@
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */
-#undef REGISTER_SAVE_AREA_RAW
-#ifdef __ILP32__
-/* X32 saves RCX, RDX, RSI, RDI, R8 and R9 plus RAX as well as VEC0 to
- VEC7. */
-# define REGISTER_SAVE_AREA_RAW (8 * 7 + VEC_SIZE * 8)
-#else
-/* X86-64 saves RCX, RDX, RSI, RDI, R8 and R9 plus RAX as well as
- BND0, BND1, BND2, BND3 and VEC0 to VEC7. */
-# define REGISTER_SAVE_AREA_RAW (8 * 7 + 16 * 4 + VEC_SIZE * 8)
-#endif
+ .text
+#ifdef _dl_runtime_resolve
-#undef REGISTER_SAVE_AREA
-#undef LOCAL_STORAGE_AREA
-#undef BASE
-#if DL_RUNTIME_RESOLVE_REALIGN_STACK
-# define REGISTER_SAVE_AREA (REGISTER_SAVE_AREA_RAW + 8)
-/* Local stack area before jumping to function address: RBX. */
-# define LOCAL_STORAGE_AREA 8
-# define BASE rbx
-# if (REGISTER_SAVE_AREA % VEC_SIZE) != 0
-# error REGISTER_SAVE_AREA must be multples of VEC_SIZE
-# endif
-#else
-# define REGISTER_SAVE_AREA REGISTER_SAVE_AREA_RAW
-/* Local stack area before jumping to function address: All saved
- registers. */
-# define LOCAL_STORAGE_AREA REGISTER_SAVE_AREA
-# define BASE rsp
-# if (REGISTER_SAVE_AREA % 16) != 8
-# error REGISTER_SAVE_AREA must be odd multples of 8
+# undef REGISTER_SAVE_AREA
+# undef LOCAL_STORAGE_AREA
+# undef BASE
+
+# if (STATE_SAVE_ALIGNMENT % 16) != 0
+# error STATE_SAVE_ALIGNMENT must be multples of 16
# endif
-#endif
- .text
-#ifdef _dl_runtime_resolve_opt
-/* Use the smallest vector registers to preserve the full YMM/ZMM
- registers to avoid SSE transition penalty. */
-
-# if VEC_SIZE == 32
-/* Check if the upper 128 bits in %ymm0 - %ymm7 registers are non-zero
- and preserve %xmm0 - %xmm7 registers with the zero upper bits. Since
- there is no SSE transition penalty on AVX512 processors which don't
- support XGETBV with ECX == 1, _dl_runtime_resolve_avx512_slow isn't
- provided. */
- .globl _dl_runtime_resolve_avx_slow
- .hidden _dl_runtime_resolve_avx_slow
- .type _dl_runtime_resolve_avx_slow, @function
- .align 16
-_dl_runtime_resolve_avx_slow:
- cfi_startproc
- cfi_adjust_cfa_offset(16) # Incorporate PLT
- vorpd %ymm0, %ymm1, %ymm8
- vorpd %ymm2, %ymm3, %ymm9
- vorpd %ymm4, %ymm5, %ymm10
- vorpd %ymm6, %ymm7, %ymm11
- vorpd %ymm8, %ymm9, %ymm9
- vorpd %ymm10, %ymm11, %ymm10
- vpcmpeqd %xmm8, %xmm8, %xmm8
- vorpd %ymm9, %ymm10, %ymm10
- vptest %ymm10, %ymm8
- # Preserve %ymm0 - %ymm7 registers if the upper 128 bits of any
- # %ymm0 - %ymm7 registers aren't zero.
- PRESERVE_BND_REGS_PREFIX
- jnc _dl_runtime_resolve_avx
- # Use vzeroupper to avoid SSE transition penalty.
- vzeroupper
- # Preserve %xmm0 - %xmm7 registers with the zero upper 128 bits
- # when the upper 128 bits of %ymm0 - %ymm7 registers are zero.
- PRESERVE_BND_REGS_PREFIX
- jmp _dl_runtime_resolve_sse_vex
- cfi_adjust_cfa_offset(-16) # Restore PLT adjustment
- cfi_endproc
- .size _dl_runtime_resolve_avx_slow, .-_dl_runtime_resolve_avx_slow
+# if (STATE_SAVE_OFFSET % STATE_SAVE_ALIGNMENT) != 0
+# error STATE_SAVE_OFFSET must be multples of STATE_SAVE_ALIGNMENT
# endif
-/* Use XGETBV with ECX == 1 to check which bits in vector registers are
- non-zero and only preserve the non-zero lower bits with zero upper
- bits. */
- .globl _dl_runtime_resolve_opt
- .hidden _dl_runtime_resolve_opt
- .type _dl_runtime_resolve_opt, @function
- .align 16
-_dl_runtime_resolve_opt:
- cfi_startproc
- cfi_adjust_cfa_offset(16) # Incorporate PLT
- pushq %rax
- cfi_adjust_cfa_offset(8)
- cfi_rel_offset(%rax, 0)
- pushq %rcx
- cfi_adjust_cfa_offset(8)
- cfi_rel_offset(%rcx, 0)
- pushq %rdx
- cfi_adjust_cfa_offset(8)
- cfi_rel_offset(%rdx, 0)
- movl $1, %ecx
- xgetbv
- movl %eax, %r11d
- popq %rdx
- cfi_adjust_cfa_offset(-8)
- cfi_restore (%rdx)
- popq %rcx
- cfi_adjust_cfa_offset(-8)
- cfi_restore (%rcx)
- popq %rax
- cfi_adjust_cfa_offset(-8)
- cfi_restore (%rax)
-# if VEC_SIZE == 32
- # For YMM registers, check if YMM state is in use.
- andl $bit_YMM_state, %r11d
- # Preserve %xmm0 - %xmm7 registers with the zero upper 128 bits if
- # YMM state isn't in use.
- PRESERVE_BND_REGS_PREFIX
- jz _dl_runtime_resolve_sse_vex
-# elif VEC_SIZE == 16
- # For ZMM registers, check if YMM state and ZMM state are in
- # use.
- andl $(bit_YMM_state | bit_ZMM0_15_state), %r11d
- cmpl $bit_YMM_state, %r11d
- # Preserve %zmm0 - %zmm7 registers if ZMM state is in use.
- PRESERVE_BND_REGS_PREFIX
- jg _dl_runtime_resolve_avx512
- # Preserve %ymm0 - %ymm7 registers with the zero upper 256 bits if
- # ZMM state isn't in use.
- PRESERVE_BND_REGS_PREFIX
- je _dl_runtime_resolve_avx
- # Preserve %xmm0 - %xmm7 registers with the zero upper 384 bits if
- # neither YMM state nor ZMM state are in use.
+# if DL_RUNTIME_RESOLVE_REALIGN_STACK
+/* Local stack area before jumping to function address: RBX. */
+# define LOCAL_STORAGE_AREA 8
+# define BASE rbx
+# ifdef USE_FXSAVE
+/* Use fxsave to save XMM registers. */
+# define REGISTER_SAVE_AREA (512 + STATE_SAVE_OFFSET)
+# if (REGISTER_SAVE_AREA % 16) != 0
+# error REGISTER_SAVE_AREA must be multples of 16
+# endif
+# endif
# else
-# error Unsupported VEC_SIZE!
+# ifndef USE_FXSAVE
+# error USE_FXSAVE must be defined
+# endif
+/* Use fxsave to save XMM registers. */
+# define REGISTER_SAVE_AREA (512 + STATE_SAVE_OFFSET + 8)
+/* Local stack area before jumping to function address: All saved
+ registers. */
+# define LOCAL_STORAGE_AREA REGISTER_SAVE_AREA
+# define BASE rsp
+# if (REGISTER_SAVE_AREA % 16) != 8
+# error REGISTER_SAVE_AREA must be odd multples of 8
+# endif
# endif
- cfi_adjust_cfa_offset(-16) # Restore PLT adjustment
- cfi_endproc
- .size _dl_runtime_resolve_opt, .-_dl_runtime_resolve_opt
-#endif
+
.globl _dl_runtime_resolve
.hidden _dl_runtime_resolve
.type _dl_runtime_resolve, @function
@@ -157,21 +64,29 @@ _dl_runtime_resolve_opt:
cfi_startproc
_dl_runtime_resolve:
cfi_adjust_cfa_offset(16) # Incorporate PLT
-#if DL_RUNTIME_RESOLVE_REALIGN_STACK
-# if LOCAL_STORAGE_AREA != 8
-# error LOCAL_STORAGE_AREA must be 8
-# endif
+# if DL_RUNTIME_RESOLVE_REALIGN_STACK
+# if LOCAL_STORAGE_AREA != 8
+# error LOCAL_STORAGE_AREA must be 8
+# endif
pushq %rbx # push subtracts stack by 8.
cfi_adjust_cfa_offset(8)
cfi_rel_offset(%rbx, 0)
mov %RSP_LP, %RBX_LP
cfi_def_cfa_register(%rbx)
- and $-VEC_SIZE, %RSP_LP
-#endif
+ and $-STATE_SAVE_ALIGNMENT, %RSP_LP
+# endif
+# ifdef REGISTER_SAVE_AREA
sub $REGISTER_SAVE_AREA, %RSP_LP
-#if !DL_RUNTIME_RESOLVE_REALIGN_STACK
+# if !DL_RUNTIME_RESOLVE_REALIGN_STACK
cfi_adjust_cfa_offset(REGISTER_SAVE_AREA)
-#endif
+# endif
+# else
+# if IS_IN (rtld)
+ sub _rtld_local_ro+RTLD_GLOBAL_RO_DL_X86_CPU_FEATURES_OFFSET+XSAVE_STATE_SIZE_OFFSET(%rip), %RSP_LP
+# else
+ sub _dl_x86_cpu_features+XSAVE_STATE_SIZE_OFFSET(%rip), %RSP_LP
+# endif
+# endif
# Preserve registers otherwise clobbered.
movq %rax, REGISTER_SAVE_RAX(%rsp)
movq %rcx, REGISTER_SAVE_RCX(%rsp)
@@ -180,59 +95,42 @@ _dl_runtime_resolve:
movq %rdi, REGISTER_SAVE_RDI(%rsp)
movq %r8, REGISTER_SAVE_R8(%rsp)
movq %r9, REGISTER_SAVE_R9(%rsp)
- VMOV %VEC(0), (REGISTER_SAVE_VEC_OFF)(%rsp)
- VMOV %VEC(1), (REGISTER_SAVE_VEC_OFF + VEC_SIZE)(%rsp)
- VMOV %VEC(2), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 2)(%rsp)
- VMOV %VEC(3), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 3)(%rsp)
- VMOV %VEC(4), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 4)(%rsp)
- VMOV %VEC(5), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 5)(%rsp)
- VMOV %VEC(6), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 6)(%rsp)
- VMOV %VEC(7), (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 7)(%rsp)
-#ifndef __ILP32__
- # We also have to preserve bound registers. These are nops if
- # Intel MPX isn't available or disabled.
-# ifdef HAVE_MPX_SUPPORT
- bndmov %bnd0, REGISTER_SAVE_BND0(%rsp)
- bndmov %bnd1, REGISTER_SAVE_BND1(%rsp)
- bndmov %bnd2, REGISTER_SAVE_BND2(%rsp)
- bndmov %bnd3, REGISTER_SAVE_BND3(%rsp)
+# ifdef USE_FXSAVE
+ fxsave STATE_SAVE_OFFSET(%rsp)
# else
-# if REGISTER_SAVE_BND0 == 0
- .byte 0x66,0x0f,0x1b,0x04,0x24
+ movl $STATE_SAVE_MASK, %eax
+ xorl %edx, %edx
+ # Clear the XSAVE Header.
+# ifdef USE_XSAVE
+ movq %rdx, (STATE_SAVE_OFFSET + 512)(%rsp)
+ movq %rdx, (STATE_SAVE_OFFSET + 512 + 8)(%rsp)
+# endif
+ movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 2)(%rsp)
+ movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 3)(%rsp)
+ movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 4)(%rsp)
+ movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 5)(%rsp)
+ movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 6)(%rsp)
+ movq %rdx, (STATE_SAVE_OFFSET + 512 + 8 * 7)(%rsp)
+# ifdef USE_XSAVE
+ xsave STATE_SAVE_OFFSET(%rsp)
# else
- .byte 0x66,0x0f,0x1b,0x44,0x24,REGISTER_SAVE_BND0
+ xsavec STATE_SAVE_OFFSET(%rsp)
# endif
- .byte 0x66,0x0f,0x1b,0x4c,0x24,REGISTER_SAVE_BND1
- .byte 0x66,0x0f,0x1b,0x54,0x24,REGISTER_SAVE_BND2
- .byte 0x66,0x0f,0x1b,0x5c,0x24,REGISTER_SAVE_BND3
# endif
-#endif
# Copy args pushed by PLT in register.
# %rdi: link_map, %rsi: reloc_index
mov (LOCAL_STORAGE_AREA + 8)(%BASE), %RSI_LP
mov LOCAL_STORAGE_AREA(%BASE), %RDI_LP
call _dl_fixup # Call resolver.
mov %RAX_LP, %R11_LP # Save return value
-#ifndef __ILP32__
- # Restore bound registers. These are nops if Intel MPX isn't
- # avaiable or disabled.
-# ifdef HAVE_MPX_SUPPORT
- bndmov REGISTER_SAVE_BND3(%rsp), %bnd3
- bndmov REGISTER_SAVE_BND2(%rsp), %bnd2
- bndmov REGISTER_SAVE_BND1(%rsp), %bnd1
- bndmov REGISTER_SAVE_BND0(%rsp), %bnd0
+ # Get register content back.
+# ifdef USE_FXSAVE
+ fxrstor STATE_SAVE_OFFSET(%rsp)
# else
- .byte 0x66,0x0f,0x1a,0x5c,0x24,REGISTER_SAVE_BND3
- .byte 0x66,0x0f,0x1a,0x54,0x24,REGISTER_SAVE_BND2
- .byte 0x66,0x0f,0x1a,0x4c,0x24,REGISTER_SAVE_BND1
-# if REGISTER_SAVE_BND0 == 0
- .byte 0x66,0x0f,0x1a,0x04,0x24
-# else
- .byte 0x66,0x0f,0x1a,0x44,0x24,REGISTER_SAVE_BND0
-# endif
+ movl $STATE_SAVE_MASK, %eax
+ xorl %edx, %edx
+ xrstor STATE_SAVE_OFFSET(%rsp)
# endif
-#endif
- # Get register content back.
movq REGISTER_SAVE_R9(%rsp), %r9
movq REGISTER_SAVE_R8(%rsp), %r8
movq REGISTER_SAVE_RDI(%rsp), %rdi
@@ -240,20 +138,12 @@ _dl_runtime_resolve:
movq REGISTER_SAVE_RDX(%rsp), %rdx
movq REGISTER_SAVE_RCX(%rsp), %rcx
movq REGISTER_SAVE_RAX(%rsp), %rax
- VMOV (REGISTER_SAVE_VEC_OFF)(%rsp), %VEC(0)
- VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE)(%rsp), %VEC(1)
- VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 2)(%rsp), %VEC(2)
- VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 3)(%rsp), %VEC(3)
- VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 4)(%rsp), %VEC(4)
- VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 5)(%rsp), %VEC(5)
- VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 6)(%rsp), %VEC(6)
- VMOV (REGISTER_SAVE_VEC_OFF + VEC_SIZE * 7)(%rsp), %VEC(7)
-#if DL_RUNTIME_RESOLVE_REALIGN_STACK
+# if DL_RUNTIME_RESOLVE_REALIGN_STACK
mov %RBX_LP, %RSP_LP
cfi_def_cfa_register(%rsp)
movq (%rsp), %rbx
cfi_restore(%rbx)
-#endif
+# endif
# Adjust stack(PLT did 2 pushes)
add $(LOCAL_STORAGE_AREA + 16), %RSP_LP
cfi_adjust_cfa_offset(-(LOCAL_STORAGE_AREA + 16))
@@ -262,11 +152,9 @@ _dl_runtime_resolve:
jmp *%r11 # Jump to function address.
cfi_endproc
.size _dl_runtime_resolve, .-_dl_runtime_resolve
+#endif
-/* To preserve %xmm0 - %xmm7 registers, dl-trampoline.h is included
- twice, for _dl_runtime_resolve_sse and _dl_runtime_resolve_sse_vex.
- But we don't need another _dl_runtime_profile for XMM registers. */
#if !defined PROF && defined _dl_runtime_profile
# if (LR_VECTOR_OFFSET % VEC_SIZE) != 0
# error LR_VECTOR_OFFSET must be multples of VEC_SIZE
--
2.13.6