[08/20] alpha: fold null and difference tests in strcmp aligned loop
Checks
| Context |
Check |
Description |
| redhat-pt-bot/TryBot-apply_patch |
success
|
Patch applied to master at the time it was sent
|
| linaro-tcwg-bot/tcwg_glibc_build--master-arm |
success
|
Build passed
|
| linaro-tcwg-bot/tcwg_glibc_build--master-aarch64 |
success
|
Build passed
|
| linaro-tcwg-bot/tcwg_glibc_check--master-aarch64 |
success
|
Test passed
|
Commit Message
The co-aligned main loop tested for a difference (xor; bne) and for the
terminating null (cmpbge; beq) with two separate branches per quadword.
On EV6/EV7 the loop is issue-bound and right at the one-branch-per-cycle
limit, so the second branch is a bottleneck.
Combine the two into a single syndrome -- "or" of the difference bits and
the null mask -- so one branch closes the loop and the difference test for
the first word moves out into the head. The loop still loads the next
word only after the current one tests clean (a zero syndrome implies no
null, so the string continues and the next word is mapped), so it never
reads past the terminating null onto an unmapped page.
bench-strcmp on EV7 (21364): co-aligned compares ~6-9% faster across
lengths that stay in L1, neutral once memory-bound; the mutually
misaligned path is unchanged. Passes string/test-strcmp.
---
sysdeps/alpha/strcmp.S | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
Comments
On 11/08/26 22:19, Matt Turner wrote:
> The co-aligned main loop tested for a difference (xor; bne) and for the
> terminating null (cmpbge; beq) with two separate branches per quadword.
> On EV6/EV7 the loop is issue-bound and right at the one-branch-per-cycle
> limit, so the second branch is a bottleneck.
>
> Combine the two into a single syndrome -- "or" of the difference bits and
> the null mask -- so one branch closes the loop and the difference test for
> the first word moves out into the head. The loop still loads the next
> word only after the current one tests clean (a zero syndrome implies no
> null, so the string continues and the next word is mapped), so it never
> reads past the terminating null onto an unmapped page.
>
> bench-strcmp on EV7 (21364): co-aligned compares ~6-9% faster across
> lengths that stay in L1, neutral once memory-bound; the mutually
> misaligned path is unchanged. Passes string/test-strcmp.
The alpha strcmp.S is another potential candidate to be removed, the
generic implementation contains all the optimizations bits (word aligned
loops, word unaligned loop, and mismatch/EOS tails) and will use all
the require alpha instruction to speed them up.
> ---
> sysdeps/alpha/strcmp.S | 24 +++++++++++++++++-------
> 1 file changed, 17 insertions(+), 7 deletions(-)
>
> diff --git ./sysdeps/alpha/strcmp.S ./sysdeps/alpha/strcmp.S
> index c7f092d1b7..6454b7aab0 100644
> --- ./sysdeps/alpha/strcmp.S
> +++ ./sysdeps/alpha/strcmp.S
> @@ -53,22 +53,32 @@ $aligned:
> ornot t0, t3, t0 # .. e1 :
> cmpbge zero, t1, t7 # e0 : bits set iff null found
> bne t7, $eos # e1 (zdb)
> + xor t0, t1, t2 # e0 : difference in the first word?
> + bne t2, $wordcmp # .. e1 (zdb)
> +
> + /* Aligned compare main loop. Fold the difference test (xor) and the
> + null test (cmpbge) into a single syndrome with "or", so one branch
> + per quadword closes the loop instead of two. Each load is gated by
> + the previous word being clean -- a zero syndrome means no null, so
> + the string continues and the next word is mapped -- hence we never
> + read past the terminating null onto an unmapped page.
>
> - /* Aligned compare main loop.
> On entry to this basic block:
> t0 == an s1 word.
> - t1 == an s2 word not containing a null. */
> + t1 == an s2 word, equal to t0 and not containing a null. */
>
> $a_loop:
> - xor t0, t1, t2 # e0 :
> - bne t2, $wordcmp # .. e1 (zdb)
> ldq_u t1, 8(a1) # e0 :
> ldq_u t0, 8(a0) # .. e1 :
> addq a1, 8, a1 # e0 :
> addq a0, 8, a0 # .. e1 :
> - cmpbge zero, t1, t7 # e0 :
> - beq t7, $a_loop # .. e1 (zdb)
> - br $eos # e1 :
> + xor t0, t1, t2 # e0 : bytes that differ
> + cmpbge zero, t1, t7 # .. e1 : bits set iff null found
> + or t2, t7, t8 # e0 : syndrome = difference | null
> + beq t8, $a_loop # .. e1 (zdb) : clean word, keep going
> +
> + bne t7, $eos # e0 : null present (handles diff-before-null)
> + br $wordcmp # .. e1 : a pure difference, no null
>
> /* The two strings are not co-aligned. Align s1 and cope. */
>
@@ -53,22 +53,32 @@ $aligned:
ornot t0, t3, t0 # .. e1 :
cmpbge zero, t1, t7 # e0 : bits set iff null found
bne t7, $eos # e1 (zdb)
+ xor t0, t1, t2 # e0 : difference in the first word?
+ bne t2, $wordcmp # .. e1 (zdb)
+
+ /* Aligned compare main loop. Fold the difference test (xor) and the
+ null test (cmpbge) into a single syndrome with "or", so one branch
+ per quadword closes the loop instead of two. Each load is gated by
+ the previous word being clean -- a zero syndrome means no null, so
+ the string continues and the next word is mapped -- hence we never
+ read past the terminating null onto an unmapped page.
- /* Aligned compare main loop.
On entry to this basic block:
t0 == an s1 word.
- t1 == an s2 word not containing a null. */
+ t1 == an s2 word, equal to t0 and not containing a null. */
$a_loop:
- xor t0, t1, t2 # e0 :
- bne t2, $wordcmp # .. e1 (zdb)
ldq_u t1, 8(a1) # e0 :
ldq_u t0, 8(a0) # .. e1 :
addq a1, 8, a1 # e0 :
addq a0, 8, a0 # .. e1 :
- cmpbge zero, t1, t7 # e0 :
- beq t7, $a_loop # .. e1 (zdb)
- br $eos # e1 :
+ xor t0, t1, t2 # e0 : bytes that differ
+ cmpbge zero, t1, t7 # .. e1 : bits set iff null found
+ or t2, t7, t8 # e0 : syndrome = difference | null
+ beq t8, $a_loop # .. e1 (zdb) : clean word, keep going
+
+ bne t7, $eos # e0 : null present (handles diff-before-null)
+ br $wordcmp # .. e1 : a pure difference, no null
/* The two strings are not co-aligned. Align s1 and cope. */