[2/4] S390: Use own tbegin macro instead of __builtin_tbegin.
Commit Message
This patch defines __libc_tbegin, __libc_tend, __libc_tabort and
__libc_tx_nesting_depth in htm.h which replaces the direct usage of
equivalent gcc builtins.
We have to use an own inline assembly instead of __builtin_tbegin,
as tbegin has to filter program interruptions which can't be done with
the builtin. Before this change, e.g. a segmentation fault within a
transaction, leads to a coredump where the instruction pointer points
behind the tbegin instruction instead of real failing one.
Now the transaction aborts and the code should be reexecuted by the
fallback path without transactions. The segmentation fault will
produce a coredump with the real failing instruction.
The fpc is not saved before starting the transaction. If e.g. the
rounging mode is changed and the transaction is aborting afterwards,
the builtin will not restore the fpc. This is now done with the
__libc_tbegin macro.
Now the call saved fprs have to be saved / restored in the
__libc_tbegin macro. Using the gcc builtin had forced the saving /
restoring of fprs at begin / end of e.g. __lll_lock_elision function.
The new macro saves these fprs before tbegin instruction and only
restores them on a transaction abort. Restoring is not needed on
a successfully started transaction.
The used inline assembly does not clobber the fprs / vrs!
Clobbering the latter ones would force the compiler to save / restore
the call saved fprs as those overlap with the vrs, but they only
need to be restored if the transaction fails. Thus the user of the
tbegin macros has to compile the file / function with -msoft-float.
It prevents gcc from using fprs / vrs.
ChangeLog:
* sysdeps/unix/sysv/linux/s390/Makefile (elision-CFLAGS):
Add -msoft-float.
* sysdeps/unix/sysv/linux/s390/htm.h: New File.
* sysdeps/unix/sysv/linux/s390/elision-lock.c:
Use __libc_t* transaction macros instead of __builtin_t*.
* sysdeps/unix/sysv/linux/s390/elision-trylock.c: Likewise.
* sysdeps/unix/sysv/linux/s390/elision-unlock.c: Likewise.
---
sysdeps/unix/sysv/linux/s390/Makefile | 2 +-
sysdeps/unix/sysv/linux/s390/elision-lock.c | 16 +--
sysdeps/unix/sysv/linux/s390/elision-trylock.c | 16 +--
sysdeps/unix/sysv/linux/s390/elision-unlock.c | 6 +-
sysdeps/unix/sysv/linux/s390/htm.h | 149 +++++++++++++++++++++++++
5 files changed, 164 insertions(+), 25 deletions(-)
create mode 100644 sysdeps/unix/sysv/linux/s390/htm.h
Comments
On 12/06/2016 02:51 PM, Stefan Liebler wrote:
> The used inline assembly does not clobber the fprs / vrs!
> Clobbering the latter ones would force the compiler to save / restore
> the call saved fprs as those overlap with the vrs, but they only
> need to be restored if the transaction fails. Thus the user of the
> tbegin macros has to compile the file / function with -msoft-float.
> It prevents gcc from using fprs / vrs.
Is it possible to disable the new macros if the file is not compiled
with -msoft-float?
Thanks,
Florian
On 12/22/2016 01:54 PM, Florian Weimer wrote:
> On 12/06/2016 02:51 PM, Stefan Liebler wrote:
>
>> The used inline assembly does not clobber the fprs / vrs!
>> Clobbering the latter ones would force the compiler to save / restore
>> the call saved fprs as those overlap with the vrs, but they only
>> need to be restored if the transaction fails. Thus the user of the
>> tbegin macros has to compile the file / function with -msoft-float.
>> It prevents gcc from using fprs / vrs.
>
> Is it possible to disable the new macros if the file is not compiled
> with -msoft-float?
Those macros/builtins are only used for the mutex lock elision code in
elision-*.c files. This code does not use float computation. The
compiler flag is used to prevent the compiler to perform optimizations
which use fprs or vrs. At the moment gcc performs no such optimizations.
If you want to use such macros/builtins in other files you'll at least
have to add -mhtm compiler flag to be able to use those builtins.
Bye
Stefan
On Tue, 2016-12-06 at 14:51 +0100, Stefan Liebler wrote:
> We have to use an own inline assembly instead of __builtin_tbegin,
> as tbegin has to filter program interruptions which can't be done with
> the builtin. Before this change, e.g. a segmentation fault within a
> transaction, leads to a coredump where the instruction pointer points
> behind the tbegin instruction instead of real failing one.
> Now the transaction aborts and the code should be reexecuted by the
> fallback path without transactions. The segmentation fault will
> produce a coredump with the real failing instruction.
I'm not sure that this is always a good thing to do. It is nicer for
programmers if they can get an instruction pointer that points to where
the faulty access happened. However:
(1) This can mask failures present in an application, because the
fallback path is not guaranteed to operate on the same data. It may
never run into this problem. Can inconsistencies arise due to that?
For example, is a masked segfault still visible through performance
counters or something like that?
(2) This introduces a facility to probe memory for being accessible or
not, considering that you say it masks segfaults. It seems that this
probing may not be visible to the same extent as possible if a signal
handler were installed. Is this relevant from a security perspective?
It may not be given that perhaps the user can do that anyway through
direct usage of transactions, or perhaps because the user would have to
be able to check whether the program is currently executing in a
transaction or not.
It might be good to at least mention this in a comment in the code.
> +++ b/sysdeps/unix/sysv/linux/s390/htm.h
> @@ -0,0 +1,149 @@
> +/* Shared HTM header. Work around false transactional execution facility
> + intrinsics.
> +
> + Copyright (C) 2016 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <http://www.gnu.org/licenses/>. */
> +
> +#ifndef _HTM_H
> +#define _HTM_H 1
> +
> +#include <htmintrin.h>
> +
> +#ifdef __s390x__
> +# define TX_FPRS_BYTES 64
> +# define TX_SAVE_FPRS \
> + " std %%f8, 0(%[R_FPRS])\n\t" \
> + " std %%f9, 8(%[R_FPRS])\n\t" \
> + " std %%f10, 16(%[R_FPRS])\n\t" \
> + " std %%f11, 24(%[R_FPRS])\n\t" \
> + " std %%f12, 32(%[R_FPRS])\n\t" \
> + " std %%f13, 40(%[R_FPRS])\n\t" \
> + " std %%f14, 48(%[R_FPRS])\n\t" \
> + " std %%f15, 56(%[R_FPRS])\n\t"
> +
> +# define TX_RESTORE_FPRS \
> + " ld %%f8, 0(%[R_FPRS])\n\t" \
> + " ld %%f9, 8(%[R_FPRS])\n\t" \
> + " ld %%f10, 16(%[R_FPRS])\n\t" \
> + " ld %%f11, 24(%[R_FPRS])\n\t" \
> + " ld %%f12, 32(%[R_FPRS])\n\t" \
> + " ld %%f13, 40(%[R_FPRS])\n\t" \
> + " ld %%f14, 48(%[R_FPRS])\n\t" \
> + " ld %%f15, 56(%[R_FPRS])\n\t"
> +
> +#else
> +
> +# define TX_FPRS_BYTES 16
> +# define TX_SAVE_FPRS \
> + " std %%f4, 0(%[R_FPRS])\n\t" \
> + " std %%f6, 8(%[R_FPRS])\n\t"
> +
> +# define TX_RESTORE_FPRS \
> + " ld %%f4, 0(%[R_FPRS])\n\t" \
> + " ld %%f6, 8(%[R_FPRS])\n\t"
> +
> +#endif /* ! __s390x__ */
> +
> +/* Use own inline assembly instead of __builtin_tbegin, as tbegin
> + has to filter program interruptions which can't be done with the builtin.
> + Now the fprs have to be saved / restored here, too.
> + The fpc is also not saved / restored with the builtin.
> + The used inline assembly does not clobber the volatile fprs / vrs!
> + Clobbering the latter ones would force the compiler to save / restore
> + the call saved fprs as those overlap with the vrs, but they only need to be
> + restored if the transaction fails but not if the transaction is successfully
> + started. Thus the user of the tbegin macros in this header file has to
> + compile the file / function with -msoft-float. It prevents gcc from using
> + fprs / vrs. */
> +#define __libc_tbegin(tdb) \
> + ({ int __ret; \
> + int __fpc; \
> + char __fprs[TX_FPRS_BYTES]; \
> + __asm__ __volatile__ (".machine push\n\t" \
> + ".machinemode \"zarch_nohighgprs\"\n\t" \
> + ".machine \"all\"\n\t" \
> + /* Save state at the outermost transaction. \
> + As extracting nesting depth is expensive \
> + on at least zEC12, save fprs at inner \
> + transactions, too. \
> + The fpc and fprs are saved here as they \
> + are not saved by tbegin. There exist no \
> + call-saved vrs, thus they are not saved \
> + here. */ \
> + " efpc %[R_FPC]\n\t" \
> + TX_SAVE_FPRS \
> + /* Begin transaction: save all gprs, allow \
> + ar modification and fp operations. Some \
> + program-interruptions (e.g. a null \
> + pointer access) are filtered and the \
> + trancsaction will abort. In this case \
> + the normal lock path will execute it \
> + again and result in a core dump wich does \
> + now show at tbegin but the real executed \
> + instruction. */ \
> + " tbegin 0, 0xFF0E\n\t" \
> + /* Branch away in abort case (this is the \
> + prefered sequence. See PoP in chapter 5 \
> + Transactional-Execution Facility \
> + Operation). */ \
> + " jnz 0f\n\t" \
> + /* Transaction has successfully started. */ \
> + " lhi %[R_RET], 0\n\t" \
> + " j 1f\n\t" \
> + /* Transaction has aborted. Now we are at \
> + the outermost transaction. Restore fprs \
> + and fpc. */ \
> + "0: ipm %[R_RET]\n\t" \
> + " srl %[R_RET], 28\n\t" \
> + " sfpc %[R_FPC]\n\t" \
> + TX_RESTORE_FPRS \
> + "1:\n\t" \
> + ".machine pop\n" \
> + : [R_RET] "=&d" (__ret), \
> + [R_FPC] "=&d" (__fpc) \
> + : [R_FPRS] "a" (__fprs) \
> + : "cc", "memory"); \
> + __ret; \
> + })
> +
> +/* These builtins are correct. Use them. */
Is "correct" the right word here? I suppose they are always correct,
it's just that you want something different for glibc internally.
On 01/10/2017 05:34 PM, Torvald Riegel wrote:
> (2) This introduces a facility to probe memory for being accessible or
> not, considering that you say it masks segfaults. It seems that this
> probing may not be visible to the same extent as possible if a signal
> handler were installed. Is this relevant from a security perspective?
If the fallback implementation has essentially the same behavior, I
don't think there is a transaction-specific security problem.
One thing to check is if anything in the transaction memory code writes
unprotected function pointers/code addresses to memory. I'm not
familiar with z Systems machine code, so I don't know if that's the case.
For example, it would be problematic to store the address of the
transaction abort handler in a TLS variable.
>> + /* Begin transaction: save all gprs, allow \
>> + ar modification and fp operations. Some \
>> + program-interruptions (e.g. a null \
>> + pointer access) are filtered and the \
>> + trancsaction will abort. In this case \
Typo: “transaction”
Thanks,
Florian
On Thu, 2017-01-12 at 16:45 +0100, Florian Weimer wrote:
> On 01/10/2017 05:34 PM, Torvald Riegel wrote:
>
> > (2) This introduces a facility to probe memory for being accessible or
> > not, considering that you say it masks segfaults. It seems that this
> > probing may not be visible to the same extent as possible if a signal
> > handler were installed. Is this relevant from a security perspective?
>
> If the fallback implementation has essentially the same behavior, I
> don't think there is a transaction-specific security problem.
We don't know what the fallback implementation in the user code does.
It can detect whether it is running in a HW transaction and run
different code depending on that.
There are different approaches to what HTMs do when transactions run
into segfaults and the like. IIRC, Intel masks them all, so
transactions aborts before the segfault "materializes". AMD's old ASF
proposal did not mask segfaults (or normal faults).
I'm not quite sure whether the amount of probing that you could do with
this patch on s390 would be substantially different than what would be
possible on Intel's RTM (or our lock elision implementation for TSX).
On 01/13/2017 12:28 PM, Torvald Riegel wrote:
> On Thu, 2017-01-12 at 16:45 +0100, Florian Weimer wrote:
>> On 01/10/2017 05:34 PM, Torvald Riegel wrote:
>>
>>> (2) This introduces a facility to probe memory for being accessible or
>>> not, considering that you say it masks segfaults. It seems that this
>>> probing may not be visible to the same extent as possible if a signal
>>> handler were installed. Is this relevant from a security perspective?
>>
>> If the fallback implementation has essentially the same behavior, I
>> don't think there is a transaction-specific security problem.
>
> We don't know what the fallback implementation in the user code does.
> It can detect whether it is running in a HW transaction and run
> different code depending on that.
Is this really supported? I assumed that if you acquire a lock (which
might use elision), hardware transactions are off limits because glibc
might require their use internally.
Thanks,
Florian
On Fri, 2017-01-13 at 15:50 +0100, Florian Weimer wrote:
> On 01/13/2017 12:28 PM, Torvald Riegel wrote:
> > On Thu, 2017-01-12 at 16:45 +0100, Florian Weimer wrote:
> >> On 01/10/2017 05:34 PM, Torvald Riegel wrote:
> >>
> >>> (2) This introduces a facility to probe memory for being accessible or
> >>> not, considering that you say it masks segfaults. It seems that this
> >>> probing may not be visible to the same extent as possible if a signal
> >>> handler were installed. Is this relevant from a security perspective?
> >>
> >> If the fallback implementation has essentially the same behavior, I
> >> don't think there is a transaction-specific security problem.
> >
> > We don't know what the fallback implementation in the user code does.
> > It can detect whether it is running in a HW transaction and run
> > different code depending on that.
>
> Is this really supported? I assumed that if you acquire a lock (which
> might use elision), hardware transactions are off limits because glibc
> might require their use internally.
You can check whether you run in a transaction. That's no guarantee
that that transaction has been started by the elision code, but you can
check that nonetheless.
HW transactions must be usable regardless of whether something else uses
elision or not. One of course shouldn't just commit transactions that
one did not start, though.
Anyway, I'm not sure whether this gives user code more means to exploit
or probe something, or not. Maybe I was just too cautious here.
On 01/13/2017 04:32 PM, Torvald Riegel wrote:
> Anyway, I'm not sure whether this gives user code more means to exploit
> or probe something, or not. Maybe I was just too cautious here.
Well, on a certain level, it's no worse than fork. :)
I'm not too worried about it (even though fork is of course abused in
this way, it used to be the main reason why Windows and Linux exploit
development were so different).
Florian
On 01/13/2017 04:32 PM, Torvald Riegel wrote:
> On Fri, 2017-01-13 at 15:50 +0100, Florian Weimer wrote:
>> On 01/13/2017 12:28 PM, Torvald Riegel wrote:
>>> On Thu, 2017-01-12 at 16:45 +0100, Florian Weimer wrote:
>>>> On 01/10/2017 05:34 PM, Torvald Riegel wrote:
>>>>
>>>>> (2) This introduces a facility to probe memory for being accessible or
>>>>> not, considering that you say it masks segfaults. It seems that this
>>>>> probing may not be visible to the same extent as possible if a signal
>>>>> handler were installed. Is this relevant from a security perspective?
>>>>
>>>> If the fallback implementation has essentially the same behavior, I
>>>> don't think there is a transaction-specific security problem.
>>>
>>> We don't know what the fallback implementation in the user code does.
>>> It can detect whether it is running in a HW transaction and run
>>> different code depending on that.
Yes, the user can detect if it is running within a transaction (e.g.
with the __builtin_tx_nesting_depth like used in elision_trylock).
With non transactional stores, he could also store something even if the
transaction aborts.
>>
>> Is this really supported? I assumed that if you acquire a lock (which
>> might use elision), hardware transactions are off limits because glibc
>> might require their use internally.
If someone starts a transaction within a transaction, then the nesting
depth is incremented. If the outermost transaction is ended, the changes
are committed.
>
> You can check whether you run in a transaction. That's no guarantee
> that that transaction has been started by the elision code, but you can
> check that nonetheless.
>
> HW transactions must be usable regardless of whether something else uses
> elision or not. One of course shouldn't just commit transactions that
> one did not start, though.
You are right, but this is possible.
>
> Anyway, I'm not sure whether this gives user code more means to exploit
> or probe something, or not. Maybe I was just too cautious here.
>
>
That's correct, you can probe something.
But if someone wants to do that, he can also start a transaction
directly without pthread_mutex_lock.
On Fri, 2017-01-13 at 17:22 +0100, Stefan Liebler wrote:
> On 01/13/2017 04:32 PM, Torvald Riegel wrote:
> > Anyway, I'm not sure whether this gives user code more means to exploit
> > or probe something, or not. Maybe I was just too cautious here.
> >
> >
> That's correct, you can probe something.
> But if someone wants to do that, he can also start a transaction
> directly without pthread_mutex_lock.
True, but it might be easier for an exploit to set up a call to existing
code than to either build binary code that starts transactions or find
suitable binary code elsewhere.
@@ -22,7 +22,7 @@ ifeq ($(enable-lock-elision),yes)
libpthread-sysdep_routines += elision-lock elision-unlock elision-timed \
elision-trylock
-elision-CFLAGS = -mhtm
+elision-CFLAGS = -mhtm -msoft-float
CFLAGS-elision-lock.c = $(elision-CFLAGS)
CFLAGS-elision-timed.c = $(elision-CFLAGS)
CFLAGS-elision-trylock.c = $(elision-CFLAGS)
@@ -19,7 +19,7 @@
#include <pthread.h>
#include <pthreadP.h>
#include <lowlevellock.h>
-#include <htmintrin.h>
+#include <htm.h>
#include <elision-conf.h>
#include <stdint.h>
@@ -60,27 +60,23 @@ __lll_lock_elision (int *futex, short *adapt_count, EXTRAARG int private)
goto use_lock;
}
- __asm__ volatile (".machinemode \"zarch_nohighgprs\"\n\t"
- ".machine \"all\""
- : : : "memory");
-
int try_tbegin;
for (try_tbegin = aconf.try_tbegin;
try_tbegin > 0;
try_tbegin--)
{
- unsigned status;
+ int status;
if (__builtin_expect
- ((status = __builtin_tbegin((void *)0)) == _HTM_TBEGIN_STARTED, 1))
+ ((status = __libc_tbegin ((void *) 0)) == _HTM_TBEGIN_STARTED, 1))
{
if (*futex == 0)
return 0;
/* Lock was busy. Fall back to normal locking. */
- if (__builtin_expect (__builtin_tx_nesting_depth (), 1))
+ if (__builtin_expect (__libc_tx_nesting_depth (), 1))
{
/* In a non-nested transaction there is no need to abort,
which is expensive. */
- __builtin_tend ();
+ __libc_tend ();
/* Don't try to use transactions for the next couple of times.
See above for why relaxed MO is sufficient. */
if (aconf.skip_lock_busy > 0)
@@ -100,7 +96,7 @@ __lll_lock_elision (int *futex, short *adapt_count, EXTRAARG int private)
because using the default lock with the inner mutex
would abort the outer transaction.
*/
- __builtin_tabort (_HTM_FIRST_USER_ABORT_CODE | 1);
+ __libc_tabort (_HTM_FIRST_USER_ABORT_CODE | 1);
}
}
else
@@ -19,7 +19,7 @@
#include <pthread.h>
#include <pthreadP.h>
#include <lowlevellock.h>
-#include <htmintrin.h>
+#include <htm.h>
#include <elision-conf.h>
#define aconf __elision_aconf
@@ -30,15 +30,11 @@
int
__lll_trylock_elision (int *futex, short *adapt_count)
{
- __asm__ __volatile__ (".machinemode \"zarch_nohighgprs\"\n\t"
- ".machine \"all\""
- : : : "memory");
-
/* Implement POSIX semantics by forbiding nesting elided trylocks.
Sorry. After the abort the code is re-executed
non transactional and if the lock was already locked
return an error. */
- if (__builtin_tx_nesting_depth () > 0)
+ if (__libc_tx_nesting_depth () > 0)
{
/* Note that this abort may terminate an outermost transaction that
was created outside glibc.
@@ -46,7 +42,7 @@ __lll_trylock_elision (int *futex, short *adapt_count)
them to use the default lock instead of retrying transactions
until their try_tbegin is zero.
*/
- __builtin_tabort (_HTM_FIRST_USER_ABORT_CODE | 1);
+ __libc_tabort (_HTM_FIRST_USER_ABORT_CODE | 1);
}
/* Only try a transaction if it's worth it. See __lll_lock_elision for
@@ -54,17 +50,17 @@ __lll_trylock_elision (int *futex, short *adapt_count)
just a hint. */
if (atomic_load_relaxed (adapt_count) <= 0)
{
- unsigned status;
+ int status;
if (__builtin_expect
- ((status = __builtin_tbegin ((void *)0)) == _HTM_TBEGIN_STARTED, 1))
+ ((status = __libc_tbegin ((void *) 0)) == _HTM_TBEGIN_STARTED, 1))
{
if (*futex == 0)
return 0;
/* Lock was busy. Fall back to normal locking. */
/* Since we are in a non-nested transaction there is no need to abort,
which is expensive. */
- __builtin_tend ();
+ __libc_tend ();
/* Note: Changing the adapt_count here might abort a transaction on a
different cpu, but that could happen anyway when the futex is
acquired, so there's no need to check the nesting depth here.
@@ -18,6 +18,7 @@
#include <pthreadP.h>
#include <lowlevellock.h>
+#include <htm.h>
int
__lll_unlock_elision(int *futex, int private)
@@ -27,10 +28,7 @@ __lll_unlock_elision(int *futex, int private)
have closed the transaction, but that is impossible to detect reliably. */
if (*futex == 0)
{
- __asm__ volatile (".machinemode \"zarch_nohighgprs\"\n\t"
- ".machine \"all\""
- : : : "memory");
- __builtin_tend();
+ __libc_tend ();
}
else
lll_unlock ((*futex), private);
new file mode 100644
@@ -0,0 +1,149 @@
+/* Shared HTM header. Work around false transactional execution facility
+ intrinsics.
+
+ Copyright (C) 2016 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _HTM_H
+#define _HTM_H 1
+
+#include <htmintrin.h>
+
+#ifdef __s390x__
+# define TX_FPRS_BYTES 64
+# define TX_SAVE_FPRS \
+ " std %%f8, 0(%[R_FPRS])\n\t" \
+ " std %%f9, 8(%[R_FPRS])\n\t" \
+ " std %%f10, 16(%[R_FPRS])\n\t" \
+ " std %%f11, 24(%[R_FPRS])\n\t" \
+ " std %%f12, 32(%[R_FPRS])\n\t" \
+ " std %%f13, 40(%[R_FPRS])\n\t" \
+ " std %%f14, 48(%[R_FPRS])\n\t" \
+ " std %%f15, 56(%[R_FPRS])\n\t"
+
+# define TX_RESTORE_FPRS \
+ " ld %%f8, 0(%[R_FPRS])\n\t" \
+ " ld %%f9, 8(%[R_FPRS])\n\t" \
+ " ld %%f10, 16(%[R_FPRS])\n\t" \
+ " ld %%f11, 24(%[R_FPRS])\n\t" \
+ " ld %%f12, 32(%[R_FPRS])\n\t" \
+ " ld %%f13, 40(%[R_FPRS])\n\t" \
+ " ld %%f14, 48(%[R_FPRS])\n\t" \
+ " ld %%f15, 56(%[R_FPRS])\n\t"
+
+#else
+
+# define TX_FPRS_BYTES 16
+# define TX_SAVE_FPRS \
+ " std %%f4, 0(%[R_FPRS])\n\t" \
+ " std %%f6, 8(%[R_FPRS])\n\t"
+
+# define TX_RESTORE_FPRS \
+ " ld %%f4, 0(%[R_FPRS])\n\t" \
+ " ld %%f6, 8(%[R_FPRS])\n\t"
+
+#endif /* ! __s390x__ */
+
+/* Use own inline assembly instead of __builtin_tbegin, as tbegin
+ has to filter program interruptions which can't be done with the builtin.
+ Now the fprs have to be saved / restored here, too.
+ The fpc is also not saved / restored with the builtin.
+ The used inline assembly does not clobber the volatile fprs / vrs!
+ Clobbering the latter ones would force the compiler to save / restore
+ the call saved fprs as those overlap with the vrs, but they only need to be
+ restored if the transaction fails but not if the transaction is successfully
+ started. Thus the user of the tbegin macros in this header file has to
+ compile the file / function with -msoft-float. It prevents gcc from using
+ fprs / vrs. */
+#define __libc_tbegin(tdb) \
+ ({ int __ret; \
+ int __fpc; \
+ char __fprs[TX_FPRS_BYTES]; \
+ __asm__ __volatile__ (".machine push\n\t" \
+ ".machinemode \"zarch_nohighgprs\"\n\t" \
+ ".machine \"all\"\n\t" \
+ /* Save state at the outermost transaction. \
+ As extracting nesting depth is expensive \
+ on at least zEC12, save fprs at inner \
+ transactions, too. \
+ The fpc and fprs are saved here as they \
+ are not saved by tbegin. There exist no \
+ call-saved vrs, thus they are not saved \
+ here. */ \
+ " efpc %[R_FPC]\n\t" \
+ TX_SAVE_FPRS \
+ /* Begin transaction: save all gprs, allow \
+ ar modification and fp operations. Some \
+ program-interruptions (e.g. a null \
+ pointer access) are filtered and the \
+ trancsaction will abort. In this case \
+ the normal lock path will execute it \
+ again and result in a core dump wich does \
+ now show at tbegin but the real executed \
+ instruction. */ \
+ " tbegin 0, 0xFF0E\n\t" \
+ /* Branch away in abort case (this is the \
+ prefered sequence. See PoP in chapter 5 \
+ Transactional-Execution Facility \
+ Operation). */ \
+ " jnz 0f\n\t" \
+ /* Transaction has successfully started. */ \
+ " lhi %[R_RET], 0\n\t" \
+ " j 1f\n\t" \
+ /* Transaction has aborted. Now we are at \
+ the outermost transaction. Restore fprs \
+ and fpc. */ \
+ "0: ipm %[R_RET]\n\t" \
+ " srl %[R_RET], 28\n\t" \
+ " sfpc %[R_FPC]\n\t" \
+ TX_RESTORE_FPRS \
+ "1:\n\t" \
+ ".machine pop\n" \
+ : [R_RET] "=&d" (__ret), \
+ [R_FPC] "=&d" (__fpc) \
+ : [R_FPRS] "a" (__fprs) \
+ : "cc", "memory"); \
+ __ret; \
+ })
+
+/* These builtins are correct. Use them. */
+#define __libc_tend() \
+ ({ __asm__ __volatile__ (".machine push\n\t" \
+ ".machinemode \"zarch_nohighgprs\"\n\t" \
+ ".machine \"all\"\n\t"); \
+ int __ret = __builtin_tend (); \
+ __asm__ __volatile__ (".machine pop"); \
+ __ret; \
+ })
+
+#define __libc_tabort(abortcode) \
+ __asm__ __volatile__ (".machine push\n\t" \
+ ".machinemode \"zarch_nohighgprs\"\n\t" \
+ ".machine \"all\"\n\t"); \
+ __builtin_tabort (abortcode); \
+ __asm__ __volatile__ (".machine pop")
+
+#define __libc_tx_nesting_depth() \
+ ({ __asm__ __volatile__ (".machine push\n\t" \
+ ".machinemode \"zarch_nohighgprs\"\n\t" \
+ ".machine \"all\"\n\t"); \
+ int __ret = __builtin_tx_nesting_depth (); \
+ __asm__ __volatile__ (".machine pop"); \
+ __ret; \
+ })
+
+#endif