From patchwork Wed Aug 12 01:19:41 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141151 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E001F4BAE7FF for ; Wed, 12 Aug 2026 01:22:45 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E001F4BAE7FF Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=Leiw7JHl X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb133.google.com (mail-yx1-xb133.google.com [IPv6:2607:f8b0:4864:20::b133]) by sourceware.org (Postfix) with ESMTPS id A872D4B99F54 for ; Wed, 12 Aug 2026 01:20:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A872D4B99F54 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org A872D4B99F54 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b133 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497607; cv=none; b=L0JxelQ1NsksKMVn/i1q5YxgOwMjJ7DCdjLB7HBiLMmNYFhsdwlld+IKCvE4hUehDnzL/1GAnFsP2JVlkQYMisL2odrlD6LuYOwkRPtBSgjfT8BdxEElO4xG5wAWipgRNPZoLbn5gm9u0yVqiTd5MYEaDtDEwDQNfvpn/PoXVY8= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497607; c=relaxed/simple; bh=hdmvmijq7CSF3s3DnxSPTqh4a3DLd6z0/oD7SRBlByI=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=lY+8FZ8gZ+lnRC4ZVroP+oSFGVI5EqupFBDTWSL6UIwx7jtlbGzg+5rgAFUDaW+TeF7a1GeTI5IxD5Spo05i6ibv4jzho6pgie2cDh2MWKpsxh4sgYyAKWSY45XSQRjtqCc87KnEGW18DYXqoRCNPzUNPV8y3T30DX4mZKqV/+8= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=Leiw7JHl DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A872D4B99F54 Received: by mail-yx1-xb133.google.com with SMTP id 956f58d0204a3-66804667af2so498099d50.0 for ; Tue, 11 Aug 2026 18:20:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497607; x=1787102407; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=kUwz4wZg7ZCR62D+xpqGgmRz+CF2/WFs1n94TAJEVbY=; b=Leiw7JHlky58co0BoCWktmoN+b6un6ExSshyiq8ionbUbi18RZesspC6IIgUSAP4mQ KDZ0EXJbiSIBf0D0hSNHXAkSeWspNB4NPz84CYq9Jso6lU4HUK98RvlPhUG51SgunpnI x1PzvTKzv/HDmw/lsHL5X/mbcWsqLcRw+aIv638mTxzL4k0HeZU++NchR8KM9mAM3RgR qqz7GlYX2xBBAo3FC8TVu4m3SO+K9N8Duoc5f7V3qMBRwEeEqfJupg7htNjthyV9uBk3 pyHWu0IK7at3c/ln7OkTtovEX4ug3fMGW8lTUfUAR1RPafIl84BSn5cV0UjpfhsnjZNa iElg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497607; x=1787102407; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=kUwz4wZg7ZCR62D+xpqGgmRz+CF2/WFs1n94TAJEVbY=; b=oFFhUxgJ62ArpwHe+yZ2Cld+tYnP5aaxclRzzgSCc+IGWfzu058gdQ7lnzb72nnsLP edrIRZ53If8hy/TyKmX+gO/CODXl5BbsBvR6sECq/V6thWr+LXY+jS5XWTAnBrSJ8vVZ piouwpbNycDoQBgsGwmAPdvUToNxJ5T5G2UyOAFyEyrt3EtTgKohP4m8o98GGKb2CnHt a1tRynDg7KvbfBHitpyHLfb6FSnMimfl1p5szOD7vkjKvzs27L6p8LJXI2e4mUHpvhTn MGpUT4g/RBSpSRXgSuqZCp+hzsc7mYtViVnwoM/stgEx5A0mEow+0Opk3lM2BPWZVsQ+ EOHQ== X-Gm-Message-State: AOJu0YxjvjWdeGE3+KukYoCa5CZjQGBYdyn4z1JvAz7e2zFaIljPqllG HhfmylCNpQnJoHLyRtEniT9xWVjXqnaVLNFABZXJ5Y7foCzlNXaDeeHTHHHP/nx16pM= X-Gm-Gg: AR+sD13DbfPntxHJASStNtN555OyP4/dUJ8Hr2CY9ydGScTem2Qzx0u/nKTM5+WrsSy +EWe7CVj/CLFjbKsaOclBF8wQknkEbjzGtCjOU1tHT+VUHjOXYuH53XmKR/Xs1Asew7wxAslaz9 cIWXpaxVQCRa7+OwuNquKAZT6CUbRSsqqRqOjck3L+paTw7HGOOsNDFbCzTzTa+4Qyc6k/DNAZ3 8CYdpHAlmFqWU4lskOu75HiF7UF+kJMt60mb2m+Ue+OydW2sqD7ffOAYDE46rLQfLjkJbp6Tsif g/o6FKWwuunOtICIWH63YUcndXF240wiMMICCSCTx6dFkFoATm0aLV+XkSoO9FqqmeV04DqmzEr gt/jDF9zy/ssxG5ixjnS9Pm/HcZcsspftQpf5DHqZV44ilsK+KX5z2kGFXe31iRWC5lQMk7dnke 2LW+afJ73Os/KqHDNFMZcadvUXiSjyCKg7VvOrzFVImLowYtcOFHdZ74Si6Uc= X-Received: by 2002:a05:690e:c41:b0:667:ca27:479e with SMTP id 956f58d0204a3-66b34112f6bmr739604d50.8.1786497606892; Tue, 11 Aug 2026 18:20:06 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66b1ef5f935sm2086925d50.1.2026.08.11.18.20.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:06 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 04/20] alpha: issue the ev6 memcpy wh64 two lines ahead Date: Tue, 11 Aug 2026 21:19:41 -0400 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, LOTS_OF_MONEY, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The 21264 memcpy loop issued its wh64 write hint one 64-byte line ahead of the stores that consume it, which is too close to be much use. wh64 has to claim the line before the stores reach it and, unlike a load prefetch, there is no fill latency for a late hint to hide behind -- a late hint collides with the store stream and provokes Mbox replay traps instead. The same mistake costs memset a factor of three; here it costs less, because the loads of this loop already put some distance between the hint and the stores that consume it. Issue it two lines ahead. Measured on an EV68CB (DS15, 2MB off-chip Bcache), cold copies improve by 6-8% from 4KB upwards, and 1KB is unchanged. Two lines rather than four: the gain past two lines is under 1%, while the stretch at the end of the region that has to fall back to hinting the current block grows with the distance and makes short copies slower. The curve at 512 bytes runs 702 cycles at one line, 760 at two, 863 at three and 870 at four; at 256KB it is 324k, 301k, 300k and 298k. Two lines is the only depth that does not lose ground at the short end. The CWG asks for at least two blocks of lead per stream, and notes that short trip count loops may want less than the 8/n blocks its MAF rule suggests, which is exactly this tradeoff. Both guards move with the distance, because wh64 makes the entire 64-byte block UNPREDICTABLE and a hint past the end of the destination would corrupt memory. The steady-state hint consumed at trip N+1 covers [$16+128, $16+192), so that trip needs 192 bytes left and the trip that computes it needs 256. The loop is only known to have 128 bytes left on entry, so the initial hint needs its own check -- it is the one hint that can overrun where the steady-state one cannot -- and falls back to naming the first block. Verified on an EV68CB against the C memcpy over every size class from 0 to 64KB at 72 destination alignments, with 512-byte guard regions either side checked for stray writes. --- sysdeps/alpha/alphaev6/memcpy.S | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S index 7f5799ecac..cf5c97c1de 100644 --- ./sysdeps/alpha/alphaev6/memcpy.S +++ ./sysdeps/alpha/alphaev6/memcpy.S @@ -87,9 +87,16 @@ $single_head_quad: $do_unroll: /* * Dispatch on the CPU implementation. The 21264 (EV6, EV67, EV68) - * issues wh64 one trip ahead of the stores to avoid read-for-ownership - * on whole-line stores into the destination. That distance is far too - * short for the 21364 (EV7), whose on-chip memory controller has a much + * issues wh64 two lines ahead of the stores to avoid read-for-ownership + * on whole-line stores into the destination. wh64 has to claim the line + * before the stores reach it and, unlike a load prefetch, there is no + * fill latency for a late hint to hide behind, so a one-line lead is too + * short to do much good. Going deeper than two costs more than it + * returns here: the loads of this loop already supply some slack, so the + * gain past two lines is under 1%, while the stretch at the end of the + * region that has to fall back to hinting the current block grows and + * makes short copies slower. memset, which has no load stream to hide + * behind, needs four. wh64 is far too * longer latency: there wh64 neither overlaps the fill nor avoids the * Mbox replay traps it provokes in the store stream. EV7 takes its own * unrolled loop that software-prefetches both streams several lines @@ -106,9 +113,17 @@ $do_unroll: subq $2, 3, $2 # E : EV7 -> 0 beq $2, $ev7_unroll # U : EV7 takes the prefetch-modify path - addq $16, 64, $7 # E : Initial (+1 trip) wh64 address - nop # E : - nop # E : + + /* + * Initial wh64 target, two lines (128 bytes) ahead of the first block. + * The loop is only known to have 128 bytes left at this point, so the + * first hint can overrun where the steady-state one cannot; if there is + * not room for it, hint the first block instead. wh64 makes the whole + * 64-byte block UNPREDICTABLE, so an overrunning hint corrupts memory. + */ + addq $16, 128, $7 # E : initial (+2 line) wh64 address + subq $18, 192, $2 # E : room for [$16+128, $16+192)? + cmovlt $2, $16, $7 # E : no -- hint the first block instead $unroll_body: wh64 ($7) # L1 : memory subsystem hint: 64 bytes at @@ -134,7 +149,11 @@ $unroll_body: stq $4, 8($16) # L : bytes 8..15 stq $5, 16($16) # L : bytes 16..23 - subq $18, 192, $2 # E : At least two more trips to go? + lda $2, -256($18) # E : room for the +2 line hint next trip? + # it covers [$16+128, $16+192) there, + # so that trip needs 192 and this one + # needs 256 (256 exceeds the subq + # literal field, hence lda) nop # E : stq $3, 24($16) # L : bytes 24..31