From patchwork Wed Aug 12 01:19:40 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141148 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 372E94B99F55 for ; Wed, 12 Aug 2026 01:22:19 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 372E94B99F55 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=V4jN/bA+ X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x112a.google.com (mail-yw1-x112a.google.com [IPv6:2607:f8b0:4864:20::112a]) by sourceware.org (Postfix) with ESMTPS id EB2A24BA2E2A for ; Wed, 12 Aug 2026 01:20:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EB2A24BA2E2A Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org EB2A24BA2E2A Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::112a ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497606; cv=none; b=Ovk/d6Oth0H0gaI6SDstTwqEUM5ztT6pXLnsRSdSnhI30s1Ezu4HyIQFz2aPza3Me6tLZCb2lVX1FgdqNevTRoVUAuIohAbMyTpZqOA0myYnfbYEAVDpaqBQR3zs83S3Xv6XkqRHpgGrtALb4HC0bhT5tIPevHqzfki3C5tc3ms= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497606; c=relaxed/simple; bh=+FcqB2xMOHrYgTlx4PkYsXfmW7dJ+VbA1mwaNzA0uQQ=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=NmS2612whWLxZ5cNWiZI7gdlf5SNC5ayR36YaAY3XUb9T1oQT1NnakPYdKHqXbEIpIV+sX3eOWFBGKVLTlu9OwRLOpQjHUCbSoxlVGSgLR6Lu8UPad4T+NgTqpgrtYXu7WFz7a9pXE33zyKHaaK3kC6OVlMRLzNDTZga24n+7tY= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=V4jN/bA+ DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org EB2A24BA2E2A Received: by mail-yw1-x112a.google.com with SMTP id 00721157ae682-8228ed0081fso6790837b3.2 for ; Tue, 11 Aug 2026 18:20:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497605; x=1787102405; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6FfQvm6c5iW58XuFOa/z3jVt5WuOCpA8I7kR7TGTOwM=; b=V4jN/bA+ByZWlb82KyjASbXrYT+A48n0aILHqwsBtjqD3KVYJftDIrptL29D5O6Ul9 rcRAQNMtwPyvP2f/XZEsG0mSJuticKSY+rIRajSpa2AJhtcg2PqkAGmikbqT+9PEKMaf 2KMLaAXFpxkNUh162dKfyoQNDICZvdH9zRE8jWHcgsEj60FElzfiLmCpHPs+zZCPaClq quPrtuZhreVfcAZIweztguAKqgyU2zKhZyTDbjuEMw6GD45ENBJsycZN4samUEOrLGO1 xiPxvySubvg8/Q5gmPRa0IYoD+Es5kTRs6O3CWpR/eyvdCXRORRbHMM7dkw4PPpZpgE0 XLlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497605; x=1787102405; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6FfQvm6c5iW58XuFOa/z3jVt5WuOCpA8I7kR7TGTOwM=; b=BAoRP2byeCpqrIFNPKXsYf5KzsMfxcX71C7KVvNgRQjN1DCrRhDb8ujMp/W85MIePW kjcISABxZ/LRmHXeLAtooo17tkQuTZJ0H5wdH3ocK9ZqUWmoLSNNEig/pRH6J7FHHPjA CVzPsqAvmVZM/7PQLpnXWzBHlZSu5u/ayisxOW9ACSdwxD6/g8Xjrh1ejcGXt2+mie26 cjlCTaOlMPVYgAVq9Ua0xWqBVXZEX5JCZ9WNa11ar1/CfLX0tT9T4DxO3VnxxEc+ucJ0 G364KbPTWHe6yUHd/lOXjY8v4lSKyAIIrp/XgEZsk4hMjjoJVkS8eoDff5PfACeNw85I 26Ug== X-Gm-Message-State: AOJu0Yxh1zb96bxgHu1iJPR9wmVBm/d/a0RQlPDC4ozoPbttH6omMxcn ThzZ1SuBW0GW2gJTWIXkT9oCQ8b/1pg6cn0jh0ZtPFADMvKhnU9B79RijU9iJC3Y8VI= X-Gm-Gg: AR+sD11kBpOR51X/ZnAePna6pbydcVsCGFquc/4fIFv2MPONDDtnJMg5AxavrF601Qm LXQ9/uh3UimZcWBCcQ+x9BIOzUqFjIntiGEJNmYWAX9XlThlJ1KDxW07ZpKxrnVkscV3/mv0RUq bXN9QGjCkPYTNUCIOBCkG/MCcHlUaVNCoiF07VgLRKgT/sETeQK1AkTnZ/Bg1zdEsLnC7qzRWPL QzTy+SC3P1FAI9iOCsL3qHqGhmKGUH8yLXnHhmYpQ+N/q3RAZy2DxvRwgjaIDE7qxn9tao9MKQ9 fO4WbCtD1Z5z96tsKLLgOvYrs3kfv5UJWTn5fUIUxNSwmnP3JHUV0Pz+p5hfkDHxTxs98lQfBBI mttiONu9cSV4vpFB6vj6WWEBGx6WRl6Z98B744AT+BHc8wkrbfegGMFX+JNDT9UxgqcN9lklIsy wznkNlnpRptYhGIKalSWXldXNl2MJcS5yWLVAmox9e5JF03/4oQ4RnveJlul0= X-Received: by 2002:a05:690c:e28a:10b0:81f:653d:4992 with SMTP id 00721157ae682-83105c13c13mr8763067b3.13.1786497605193; Tue, 11 Aug 2026 18:20:05 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a9631445sm6321657b3.29.2026.08.11.18.20.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:04 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 03/20] alpha: use prefetch-with-modify-intent in memcpy on EV7 Date: Tue, 11 Aug 2026 21:19:40 -0400 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, LOTS_OF_MONEY, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The unrolled loop in the ev6 memcpy issues wh64 one trip ahead of the stores to avoid read-for-ownership on whole-line stores into the destination. That is tuned for the 21264 (EV6, EV67, EV68) with an external Bcache. On the 21364 (EV7), with its on-chip memory controller and much longer memory latency, one line is far too close: the hint neither overlaps the fill nor avoids the Mbox replay traps it provokes in the store stream. The loop also never prefetches the source, leaning entirely on software pipelining to hide load latency. Dispatch on IMPLVER (2 for the 21264 family, 3 for the 21364 family) and give EV7 its own unrolled loop that software-prefetches both streams six cache lines ahead of the current block: the source with a plain read prefetch (LDL to R31) and the destination with a prefetch with modify intent (LDS to F31, a ReadBlkMod that allocates the line writeable). Both are reads only, so prefetching past the end of the region is harmless and no overrun guard is needed. The distance is not critical: 4 to 12 lines all measure the same on EV7. The 21264 code path is unchanged. Measured on an AlphaServer ES47 (EV7, 1.3GHz), cycles/call: size cold before cold after hot before hot after 1024 3510 3050 -13% 585 246 -58% 16384 55299 46369 -16% 8602 4054 -53% 65536 327197 291205 -11% 61712 27244 -56% 262144 676953 535414 -21% 248933 98661 -60% The large hot-cache win comes from removing wh64: it was the source of hundreds of millions of Mbox replay traps (perf -e r4) in the store stream, which the read-only prefetch path does not provoke. --- sysdeps/alpha/alphaev6/memcpy.S | 61 ++++++++++++++++++++++++++++++++- 1 file changed, 60 insertions(+), 1 deletion(-) diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S index 04bf2422e3..7f5799ecac 100644 --- ./sysdeps/alpha/alphaev6/memcpy.S +++ ./sysdeps/alpha/alphaev6/memcpy.S @@ -85,9 +85,29 @@ $single_head_quad: bne $1, $single_head_quad # U : still not fully aligned $do_unroll: - addq $16, 64, $7 # E : Initial (+1 trip) wh64 address + /* + * Dispatch on the CPU implementation. The 21264 (EV6, EV67, EV68) + * issues wh64 one trip ahead of the stores to avoid read-for-ownership + * on whole-line stores into the destination. That distance is far too + * short for the 21364 (EV7), whose on-chip memory controller has a much + * longer latency: there wh64 neither overlaps the fill nor avoids the + * Mbox replay traps it provokes in the store stream. EV7 takes its own + * unrolled loop that software-prefetches both streams several lines + * ahead -- the source with a plain read prefetch (LDL to R31) and the + * destination with a prefetch-with-modify-intent (LDS to F31, a + * ReadBlkMod that allocates the line writeable). Both are reads only, + * so prefetching past the end of the region is harmless and no overrun + * guard is needed. IMPLVER returns 2 for the 21264 family and 3 for + * the 21364 family. + */ cmple $18, 127, $1 # E : Can we go through the unrolled loop? bne $1, $tail_quads # U : Nope + implver $2 # E : 2 = 21264 (EV6/67/68), 3 = 21364 + subq $2, 3, $2 # E : EV7 -> 0 + + beq $2, $ev7_unroll # U : EV7 takes the prefetch-modify path + addq $16, 64, $7 # E : Initial (+1 trip) wh64 address + nop # E : nop # E : $unroll_body: @@ -142,6 +162,45 @@ $unroll_body: stq $3, -8($16) # L : bytes 24..31 nop # E : beq $1, $unroll_body + br $31, $tail_quads # U : 1..15 trailing quads/bytes + + /* + * EV7 (21364): 64-byte unrolled copy loop that software-prefetches six + * cache lines (384 bytes) ahead of the current block -- the source with + * a read prefetch and the destination with modify intent. The distance + * is not critical: anything from 4 to 12 lines measures the same on EV7. + * Both src and dest are 0mod8 here (dest is 0mod64), and the prefetches + * only read, so running past the end of the region is harmless. + * $16 - dest, $17 - src, $18 - bytes left, $1..$7,$22 scratch. + */ + .align 4 +$ev7_unroll: + lds $f31, 6*64($16) # L : dest prefetch w/ modify intent + ldl $31, 6*64($17) # L : src prefetch (read) + ldq $1, 0($17) # L : bytes 0..7 + ldq $2, 8($17) # L : bytes 8..15 + ldq $3, 16($17) # L : bytes 16..23 + ldq $4, 24($17) # L : bytes 24..31 + ldq $5, 32($17) # L : bytes 32..39 + ldq $6, 40($17) # L : bytes 40..47 + ldq $7, 48($17) # L : bytes 48..55 + ldq $22, 56($17) # L : bytes 56..63 + + stq $1, 0($16) # L : + stq $2, 8($16) # L : + stq $3, 16($16) # L : + stq $4, 24($16) # L : + stq $5, 32($16) # L : + stq $6, 40($16) # L : + stq $7, 48($16) # L : + stq $22, 56($16) # L : + + subq $18, 64, $18 # E : count -= 64 + addq $17, 64, $17 # E : src += 64 + addq $16, 64, $16 # E : dest += 64 + cmple $18, 63, $1 # E : less than one more full trip? + beq $1, $ev7_unroll # U : keep going + /* fall through to $tail_quads for the 1..63 byte remainder */ $tail_quads: $no_unroll: