From patchwork Wed Aug 12 01:19:38 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141145 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D6FC74BAE7F6 for ; Wed, 12 Aug 2026 01:20:50 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D6FC74BAE7F6 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=EB2AL30H X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb129.google.com (mail-yx1-xb129.google.com [IPv6:2607:f8b0:4864:20::b129]) by sourceware.org (Postfix) with ESMTPS id 71C044BA23FA for ; Wed, 12 Aug 2026 01:20:01 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 71C044BA23FA Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 71C044BA23FA Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b129 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497601; cv=none; b=aPiuM1+SSmi7f8fG91LFOKUTfrwqD2Cgnc1obqrhfTsYZ2iLADevuRgJf6UTlTPpdDxQJhdm6LebqBEmCnH1znxffzOtq4DkLz9SWGyvmn55f4whfkxrj2w+ZRz44UO63WmmPWyT0Z39haPBCxIBYfQs5wUihsw3KiCZHacTWYg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497601; c=relaxed/simple; bh=cYG/9LAv0DvyotaR4Hqlkc+2336ipPVrAHkoEX4qrAo=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=v5ytpOsUhjIgtWX8jMgBjxa5rm9I1F/ir4OidkbKxWPqX6+K4v/NATJNkyNBk3l6g3pJRg/jv0MFID+q0SvOrepDNgt+Vi+G6qxLeFvhLXj5i3MbtM2DboxTALVmR4yrUEfqr4Wi8KAmQ0ktFMZgHXW8E/iYqMgu27jHx09QZvM= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=EB2AL30H DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 71C044BA23FA Received: by mail-yx1-xb129.google.com with SMTP id 956f58d0204a3-6688acd1a51so764657d50.3 for ; Tue, 11 Aug 2026 18:20:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497601; x=1787102401; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Guo0w532EWlyrXcw2wnGIhpaDpCVAsfNIK+x4FnQBUg=; b=EB2AL30HY77ZqVcAn3KIRj9App/fHT092Besprs2inelVEnTVowoxz7hnHTMz6pycv YGFHtFX6xgznp8JU0h5szGcvy5g9alZxFk3AXnfHhShRpeC2yGMf0DLfDo/C4bfzbL7Q z2bxceJN6tHCNccj3skUE2UFq5LPs+zmxN6r21wNmyadnOK+FU8IBdIlmT3EqrOBInOI aizY+wgKdsMjZDpftfWnOXJUGNuIFK6W3SBSpcvNbXx3eNYypISnOaRSBT0KYViMHl9k NAT0EuPlQiju2BODEnOp0OXepkCt63tFxZn6pOwzKCACrBRO1DL25F+yPGk8vqWsvEJX 1dNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497601; x=1787102401; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Guo0w532EWlyrXcw2wnGIhpaDpCVAsfNIK+x4FnQBUg=; b=dJDUgLmQ/u1WHOZQ1+jUEXZfo0GcnE257tdQu65zr6Ib4x8TtZZ3K0MzZDJ7vuR1eq EKW0JD+cgRXqqbXMTVKlB7SnA2F1E4hdChNKP3koBaaO0YGZpuFjvJHxMkyglkq8Nptp KgqdbIc3Wgj7YsYpcmhDr46pA4DOZjGAVGrN9D8QGEiYPvo/jq7lIRIEeASGWem6vcNV Sju/M3vfZiNfmXc3dROHgJBH4RoNFVuQY4R6CPEUUnHLncT4ISnKz/sPd4ytf5qrxZsb GIBdWs2BfQdQMyOZ9sIfiynD80BClmVtJ5/+0BCd/MCxPvTWycbcYfIgm96tYDrRSf2x F/cQ== X-Gm-Message-State: AOJu0YxJi+dW1meGmCnOVkvW0wBzmi07V2T/0alznum6QGsnL8+R64/N 70QV08uMRvPB0d3+ATSGOzfEm+iZiIthQ8wL/EW9yDbXK6k6WorCo3rvJu3Jx+Dmp+0= X-Gm-Gg: AR+sD13cr9HVqQIC5XFniXGAXmlqMy2NRz0W0rSsdGd27OVXQzcDaOmoK3IXutmYbpD yOMcn6ZoxigXb6tLvXPE7cu4cs7LRo+Qh4qQDuq/0z1V+wYCE4ZWe4nGW/6My5zGr3AudfVRVMn acP1ujeqJEq0coklnFmKQXCQsJpwmSY4GShLYkhsMHqO29MpUROP3OdYjW/Wijsg4Cmp61Mt4Hw 5oPZr2fHmBek8f718eG47vbJVDcQ0hZMZhCwKQpv5fED2ulhe7o68gR5Gqz4RK95QxKiWliVglb WuaDHc6kPUwqthwFNmrVHxNiJHB+7y6JauBecRYXjJmkOZR5r7MLmeNQf8sN7E+YZN2NsjjZkFN 8uU7jit1nNm9ubcg0mZI552/TftzLe5OyvsPNvlR6o/0xiEgBKCfr3PoG04BwpwAq0Vtc9DkAHm zqPweUzoYAm/9LTlxIqWmWxAU6397A9vyNi79rTocaNQpZA3zxn3yKCEpcNJn3kUdhRicMkQ== X-Received: by 2002:a05:690e:130e:b0:668:9ea4:775d with SMTP id 956f58d0204a3-66b3434d335mr758078d50.37.1786497600707; Tue, 11 Aug 2026 18:20:00 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66b1ef961a1sm2094575d50.8.2026.08.11.18.19.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:00 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 01/20] alpha: use prefetch-with-modify-intent in memset on EV7 Date: Tue, 11 Aug 2026 21:19:38 -0400 Message-ID: <282b022a12959d4cb81f02aabcfcffad74c4cd06.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The unrolled loop in the ev6 memset issues wh64 two cache lines ahead of the stores to avoid read-for-ownership on whole-line stores. That is tuned for the 21264 (EV6, EV67, EV68) with an external Bcache. On the 21364 (EV7), with its on-chip memory controller and much longer memory latency, two lines is far too close: the hint neither overlaps the fill nor avoids the Mbox replay traps it provokes in the store stream. Dispatch on IMPLVER (2 for the 21264 family, 3 for the 21364 family) and give EV7 its own unrolled loop that issues a software prefetch with modify intent (LDS to F31, a ReadBlkMod that allocates the line writeable) six cache lines ahead of the stores. On EV7 this beats wh64 at every distance and, unlike wh64, only reads, so prefetching past the end of the region is harmless and no overrun guard is needed. The 21264 code path is unchanged. Measured on an AlphaServer ES47 (EV7, 1.3GHz), cycles/call: size cold before cold after hot before hot after 1024 2056 1566 -24% 220 204 4096 13167 11036 -16% 807 731 65536 111438 76719 -31% 8495 7729 262144 423219 285094 -33% 102067 54753 -46% Mbox replay traps (perf -e r4) were used to identify the wh64 prefetch distance as the bottleneck. --- sysdeps/alpha/alphaev6/memset.S | 51 +++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git ./sysdeps/alpha/alphaev6/memset.S ./sysdeps/alpha/alphaev6/memset.S index 6057274f1a..5c1e307cdf 100644 --- ./sysdeps/alpha/alphaev6/memset.S +++ ./sysdeps/alpha/alphaev6/memset.S @@ -103,6 +103,26 @@ $aligned: * $3 Number quads to write */ + /* + * Dispatch on the CPU implementation. The 21264 (EV6, EV67, EV68) uses + * wh64 two cache lines ahead to avoid read-for-ownership on whole-line + * stores. That distance is far too short for the 21364 (EV7), whose + * on-chip memory controller has a much longer latency: there the hint + * neither overlaps the fill nor avoids Mbox replay traps. Issuing a + * software prefetch-with-modify-intent (LDS to F31, a ReadBlkMod that + * allocates the line writeable) six lines ahead instead recovers a + * large cold-store win -- about -30% at >=64KB -- and, unlike wh64, + * does not zero the line, so no end-guard against overrun is needed. + * EV7 therefore takes its own unrolled loop. IMPLVER returns 2 for the + * 21264 family (EV6/67/68) and 3 for the 21364 family (EV7/EV79). + */ + subq $3, 16, $4 # E : Only try to unroll if > 128 bytes + blt $4, $loop # U : too small -- scalar tail loop + implver $2 # E : 2 = 21264 (EV6/67/68), 3 = 21364 (EV7) + subq $2, 3, $2 # E : EV7 -> 0 + + beq $2, $ev7_unroll # U : EV7 takes the prefetch-modify path + and $16, 0x3f, $2 # E : Forward work (only useful for unrolled loop) subq $3, 16, $4 # E : Only try to unroll if > 128 bytes subq $2, 0x40, $1 # E : bias counter (aligning stuff 0mod64) @@ -174,6 +194,37 @@ $do_wh64: nop nop beq $3, $no_quad # U : Might have finished already + br $31, $loop # U : 1..7 trailing quads + + /* + * EV7 (21364): 64-byte unrolled store loop that issues a software + * prefetch-with-modify-intent (LDS to F31) six cache lines (384 bytes) + * ahead of the current block. No 0mod64 pre-align is needed. The + * prefetch only reads -- it never writes -- so prefetching past the end + * of the region is harmless (a prefetch that misses the TLB or hits an + * unwriteable page is simply dropped), and no end-guard is required. + * $3 - quads left, $5 - dest (0mod8), $17 - store value, $7 scratch. + */ + .align 4 +$ev7_unroll: + lds $f31, 6*64($5) # L : prefetch w/ modify intent, six lines ahead + stq $17, 0($5) # L : + stq $17, 8($5) # L : + stq $17, 16($5) # L : + stq $17, 24($5) # L : + stq $17, 32($5) # L : + stq $17, 40($5) # L : + stq $17, 48($5) # L : + stq $17, 56($5) # L : + + subq $3, 8, $3 # E : decrement quad count + addq $5, 64, $5 # E : advance dest + subq $3, 8, $7 # E : at least 8 more quads? + bge $7, $ev7_unroll # U : + + beq $3, $no_quad # U : exact multiple, done + nop + nop .align 4 /* From patchwork Wed Aug 12 01:19:39 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141146 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 1B3614B9DB5A for ; Wed, 12 Aug 2026 01:21:03 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 1B3614B9DB5A Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=jlGPUMjy X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb12e.google.com (mail-yx1-xb12e.google.com [IPv6:2607:f8b0:4864:20::b12e]) by sourceware.org (Postfix) with ESMTPS id 5B7624BA7986 for ; Wed, 12 Aug 2026 01:20:03 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 5B7624BA7986 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 5B7624BA7986 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b12e ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497603; cv=none; b=KhuUVeF1CX8z/Ba3Pf9ZUCk+nFokuZ/ydWSWin4VhIU6THxdOQXtDvIUd4i0yVdCFSqStUv+xZmxwDRThlXGK/Jal7V0BsjU4MVaGJJx2Fpilc8QckbZ/s6iE2SqolzrsZyObyQkcabhri7wCIE7BU4DIes6TLgwoAIvahIehtg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497603; c=relaxed/simple; bh=SdvdiY1oKgd3fwtaIckKMc7RzhFJPAj+xatBiIxDQPQ=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=THpCyZuyT12tMYq2/wXuEiM4xDYHX66poklq9CJUosr/t/MvkYwhAyyPKM4eMvbKC1LuGdXoJ0oVxxwIWiOKhs/hvy+pJcYnttCsl0auombeQMSVRGUqpF1p2ALcFp68iV246HxZlCyf3kZgx5x+F7rGb0cOfHce4xTJm4Om5gQ= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=jlGPUMjy DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 5B7624BA7986 Received: by mail-yx1-xb12e.google.com with SMTP id 956f58d0204a3-66807ba2f0fso817252d50.3 for ; Tue, 11 Aug 2026 18:20:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497603; x=1787102403; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=EXhffOexjCpeUoBQBPDq/+mLWgr41okxTQJfkqs3S6Q=; b=jlGPUMjy3kCrubaxyaqgEPuIx9q6sHuFweGmgg7WMdDsgp+Zhen2flmkpvX40bjjTu w1h99rACxHOCH4ngk5+JQv6AIwLWUej8gQPZ4DzNqZAlhPhgS7OXalBqgAdjKONaot2v hA4wqhH0U+mmtipuEYAauct7eZQQ9iGU21w9xsArDR64kRm08urOP0yloDhvQlJd4PwU a658074aL0AfSsxLZQ5GocdRGpHx3aKEcOnGHAafKIqdR2mMI2V0BJlJ1CmETS4rS14R 4qBZx8af6aqBTMkVLZ8UIBiUm+8uf73h6/APMwBuaaHi3mMhWqE7PDt7pfAlwJc9WSJd oa4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497603; x=1787102403; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=EXhffOexjCpeUoBQBPDq/+mLWgr41okxTQJfkqs3S6Q=; b=WjQ9kpXl8vJhO0/O/RoPeo8a0X5IpHhrBZXGpvqXX8TXINYKNbf9v0afMJ04DS7DM4 I2rgF14KcbpdxCiUTGpbGujpzC309Su/jRZnJAM+SirpEYdZzghTuvEjlOmNU7PQ0mDf D1rTwBUNiAtp2JK0kzSj2OUSn271CUJjU62OgK5++n0zqUvVtDoT/eQkUa6rUWK3NW6j 0aKfQWlVgRAtr4jATJdOMsjT6/PuNUiiZ/+SBFX1FrRgVxid3ZbGr+XcW7uIda4ZjiR8 n3Oc9Y/lzHCwMhHxDHiAQhjUN4tSkhdMoem+OmN1fZWTJsN0EacxQxIJd3pA/PPjpQrX 2h9Q== X-Gm-Message-State: AOJu0YyOrX/0qpAiGSHUnS0xY4FxkoZqeJs4tKjnffH8eRLk4VdDT6DC ofYXbBXpZFJc8qcSD2o0IalnQ/2SH5+msr9M/U+GRCGoNm7xrTZtcp2Gq9jzv6LjJ1Q= X-Gm-Gg: AR+sD11KGg7mio6XaS3MKZHxDYT6AuVejyAIorKVspVhUKDCr3WgxWIkSr9cZw9RLxi ZmCbEgNyUcBswkF5Ug48HLOtaIRBEVZ49+Gv6UKCnjLc3ns+LLYPorHeTgFgZ8t97O7uuLpts1C SKxvvsPCR8jH3O8M5jzZMeYrcyBJv2jOmzxvGBzFovgvcWjETI7OzrME1M70R0sSUuYXW2jU3Pa z5vJGlcR42C0SA5tXWgFDaXcCdP77Z89wjHcIQQg7gdwnoya0bAqspSliNOgPW3pDidQZpx6GsJ ioAgbHNHOzmFMBO644BUqtvXjTJnYNnyHxuoO+tGl0cQ9pA/uogh2qV0BDYvi0MO0biVsdCymHH ZvY4BMNKgd2T7+DzSPtMjFwyd2RWKeHhF0bl0MWB3C2tStiUKZxN8sSYsIQPebChMM4Q9SdbYpU bGl6XkNfLUglWLYRvMVyBPLIXkPGiIwsXabaAhmaRTNfGXhTkETRb4jvOkGEs= X-Received: by 2002:a53:b787:0:b0:669:4879:867d with SMTP id 956f58d0204a3-66b34308752mr537169d50.30.1786497602623; Tue, 11 Aug 2026 18:20:02 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66b1f2843edsm2115304d50.18.2026.08.11.18.20.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:01 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 02/20] alpha: issue the ev6 memset wh64 four lines ahead Date: Tue, 11 Aug 2026 21:19:39 -0400 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The 21264 memset loop issued its wh64 write hint one 64-byte line ahead of the stores that consume it. That is far too close, and the cost is not subtle: on an EV68CB (DS15, 2MB off-chip Bcache) a cold 256KB memset ran at 453k cycles with the one-line hint against 406k with no hint issued at all, so the wh64 was not merely ineffective but a net loss. Issue it four lines (256 bytes) ahead instead. The same measurement gives 151k cycles, a 3.0x improvement, and the curve is flat from three lines out to eight, so the depth is not delicately tuned -- it only has to clear the threshold. Four lines is at least as fast as one at every size in both the hot and cold cases, so no size range regresses. This matches the CWG, which asks for at least two cache blocks of lead per stream and up to 8/n blocks for n streams against the 21264's 8-entry MAF; a one-line hint was below the architectural minimum. The mechanism is Mbox replay traps, not fill latency. Counted with perf on the 21264 (raw event r4), the one-line hint takes 69885 replays per cold 256KB call and the four-line hint takes 11467, an 83% reduction, while Bcache misses are unchanged at 4162 against 4125. No extra data is moved; the loop was simply trapping and replaying its own store stream, at close to one replay per instruction retired. wh64 has to claim the line before the stores reach it, and unlike a load prefetch there is no fill latency for a late hint to hide behind -- it just collides. The end guard moves with the distance: the hint consumed at trip N+1 covers [$5+256, $5+320), so that trip needs 40 quads remaining, and it is computed at trip N, which therefore needs 48. When fewer remain the target falls back to the block the next trip stores, which is always in range. This matters because wh64 makes the entire 64-byte block UNPREDICTABLE, so a hint that runs past the end of the region would corrupt memory. 320 does not fit the 8-bit addq literal field, so the address computation becomes an lda; both issue on the E box, so the schedule is unchanged. Verified on an EV68CB against the C memset over every size class from 0 to 64KB at 72 destination alignments, with 512-byte guard regions either side checked for stray writes. --- sysdeps/alpha/alphaev6/memset.S | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git ./sysdeps/alpha/alphaev6/memset.S ./sysdeps/alpha/alphaev6/memset.S index 5c1e307cdf..fb01baa795 100644 --- ./sysdeps/alpha/alphaev6/memset.S +++ ./sysdeps/alpha/alphaev6/memset.S @@ -158,19 +158,38 @@ $bigalign: * Scratch registers available: $7, $2, $4, $1 * We know that we'll be taking a minimum of one trip through. * CWG Section 3.7.6: do not expect a sustained store rate of > 1/cycle - * Assumes the wh64 needs to be for 2 trips through the loop in the future. - * The wh64 is issued on for the starting destination address for trip +2 - * through the loop, and if there are less than two trips left, the target - * address will be for the current trip. + * + * The wh64 is issued five trips ahead of the block it is computed on, + * so that when it is consumed at the top of the next trip it names the + * block four lines (256 bytes) beyond the stores of that trip. A + * shorter lead does not work: wh64 has to claim the line before the + * stores reach it and, unlike a load prefetch, there is no fill latency + * for a late hint to hide behind. Measured on an EV68CB (DS15, 2MB + * off-chip Bcache), a one-line lead is 453k cycles for a cold 256KB + * memset against 406k for issuing no hint at all -- i.e. it is worse + * than useless -- while four lines is 151k, a 3.0x improvement. The + * curve is flat from three lines out to eight, so the exact depth in + * that range does not matter; being under it does. + * + * If fewer than four lines remain the hint would run off the end of the + * region, and wh64 makes the whole 64-byte block UNPREDICTABLE, so the + * target falls back to the block the next trip stores. That is always + * in range and still avoids that block's read-for-ownership. + * + * Guard arithmetic: the hint consumed at trip N+1 covers + * [$5+256, $5+320), so trip N+1 needs 40 quads left; it is computed at + * trip N, which therefore needs 48. */ $do_wh64: wh64 ($4) # L1 : memory subsystem write hint - subq $3, 24, $2 # E : For determining future wh64 addresses + subq $3, 48, $2 # E : >= 48 quads left? (see guard note above) stq $17, 0($5) # L : nop # E : - addq $5, 128, $4 # E : speculative target of next wh64 + lda $4, 320($5) # E : speculative target of next wh64 (4 lines + # ahead of the trip that consumes it; 320 + # exceeds the 8-bit addq literal field) stq $17, 8($5) # L : stq $17, 16($5) # L : addq $5, 64, $7 # E : Fallback address for wh64 (== next trip addr) From patchwork Wed Aug 12 01:19:40 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141148 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 372E94B99F55 for ; Wed, 12 Aug 2026 01:22:19 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 372E94B99F55 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=V4jN/bA+ X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x112a.google.com (mail-yw1-x112a.google.com [IPv6:2607:f8b0:4864:20::112a]) by sourceware.org (Postfix) with ESMTPS id EB2A24BA2E2A for ; Wed, 12 Aug 2026 01:20:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EB2A24BA2E2A Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org EB2A24BA2E2A Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::112a ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497606; cv=none; b=Ovk/d6Oth0H0gaI6SDstTwqEUM5ztT6pXLnsRSdSnhI30s1Ezu4HyIQFz2aPza3Me6tLZCb2lVX1FgdqNevTRoVUAuIohAbMyTpZqOA0myYnfbYEAVDpaqBQR3zs83S3Xv6XkqRHpgGrtALb4HC0bhT5tIPevHqzfki3C5tc3ms= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497606; c=relaxed/simple; bh=+FcqB2xMOHrYgTlx4PkYsXfmW7dJ+VbA1mwaNzA0uQQ=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=NmS2612whWLxZ5cNWiZI7gdlf5SNC5ayR36YaAY3XUb9T1oQT1NnakPYdKHqXbEIpIV+sX3eOWFBGKVLTlu9OwRLOpQjHUCbSoxlVGSgLR6Lu8UPad4T+NgTqpgrtYXu7WFz7a9pXE33zyKHaaK3kC6OVlMRLzNDTZga24n+7tY= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=V4jN/bA+ DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org EB2A24BA2E2A Received: by mail-yw1-x112a.google.com with SMTP id 00721157ae682-8228ed0081fso6790837b3.2 for ; Tue, 11 Aug 2026 18:20:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497605; x=1787102405; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=6FfQvm6c5iW58XuFOa/z3jVt5WuOCpA8I7kR7TGTOwM=; b=V4jN/bA+ByZWlb82KyjASbXrYT+A48n0aILHqwsBtjqD3KVYJftDIrptL29D5O6Ul9 rcRAQNMtwPyvP2f/XZEsG0mSJuticKSY+rIRajSpa2AJhtcg2PqkAGmikbqT+9PEKMaf 2KMLaAXFpxkNUh162dKfyoQNDICZvdH9zRE8jWHcgsEj60FElzfiLmCpHPs+zZCPaClq quPrtuZhreVfcAZIweztguAKqgyU2zKhZyTDbjuEMw6GD45ENBJsycZN4samUEOrLGO1 xiPxvySubvg8/Q5gmPRa0IYoD+Es5kTRs6O3CWpR/eyvdCXRORRbHMM7dkw4PPpZpgE0 XLlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497605; x=1787102405; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=6FfQvm6c5iW58XuFOa/z3jVt5WuOCpA8I7kR7TGTOwM=; b=BAoRP2byeCpqrIFNPKXsYf5KzsMfxcX71C7KVvNgRQjN1DCrRhDb8ujMp/W85MIePW kjcISABxZ/LRmHXeLAtooo17tkQuTZJ0H5wdH3ocK9ZqUWmoLSNNEig/pRH6J7FHHPjA CVzPsqAvmVZM/7PQLpnXWzBHlZSu5u/ayisxOW9ACSdwxD6/g8Xjrh1ejcGXt2+mie26 cjlCTaOlMPVYgAVq9Ua0xWqBVXZEX5JCZ9WNa11ar1/CfLX0tT9T4DxO3VnxxEc+ucJ0 G364KbPTWHe6yUHd/lOXjY8v4lSKyAIIrp/XgEZsk4hMjjoJVkS8eoDff5PfACeNw85I 26Ug== X-Gm-Message-State: AOJu0Yxh1zb96bxgHu1iJPR9wmVBm/d/a0RQlPDC4ozoPbttH6omMxcn ThzZ1SuBW0GW2gJTWIXkT9oCQ8b/1pg6cn0jh0ZtPFADMvKhnU9B79RijU9iJC3Y8VI= X-Gm-Gg: AR+sD11kBpOR51X/ZnAePna6pbydcVsCGFquc/4fIFv2MPONDDtnJMg5AxavrF601Qm LXQ9/uh3UimZcWBCcQ+x9BIOzUqFjIntiGEJNmYWAX9XlThlJ1KDxW07ZpKxrnVkscV3/mv0RUq bXN9QGjCkPYTNUCIOBCkG/MCcHlUaVNCoiF07VgLRKgT/sETeQK1AkTnZ/Bg1zdEsLnC7qzRWPL QzTy+SC3P1FAI9iOCsL3qHqGhmKGUH8yLXnHhmYpQ+N/q3RAZy2DxvRwgjaIDE7qxn9tao9MKQ9 fO4WbCtD1Z5z96tsKLLgOvYrs3kfv5UJWTn5fUIUxNSwmnP3JHUV0Pz+p5hfkDHxTxs98lQfBBI mttiONu9cSV4vpFB6vj6WWEBGx6WRl6Z98B744AT+BHc8wkrbfegGMFX+JNDT9UxgqcN9lklIsy wznkNlnpRptYhGIKalSWXldXNl2MJcS5yWLVAmox9e5JF03/4oQ4RnveJlul0= X-Received: by 2002:a05:690c:e28a:10b0:81f:653d:4992 with SMTP id 00721157ae682-83105c13c13mr8763067b3.13.1786497605193; Tue, 11 Aug 2026 18:20:05 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a9631445sm6321657b3.29.2026.08.11.18.20.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:04 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 03/20] alpha: use prefetch-with-modify-intent in memcpy on EV7 Date: Tue, 11 Aug 2026 21:19:40 -0400 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, LOTS_OF_MONEY, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The unrolled loop in the ev6 memcpy issues wh64 one trip ahead of the stores to avoid read-for-ownership on whole-line stores into the destination. That is tuned for the 21264 (EV6, EV67, EV68) with an external Bcache. On the 21364 (EV7), with its on-chip memory controller and much longer memory latency, one line is far too close: the hint neither overlaps the fill nor avoids the Mbox replay traps it provokes in the store stream. The loop also never prefetches the source, leaning entirely on software pipelining to hide load latency. Dispatch on IMPLVER (2 for the 21264 family, 3 for the 21364 family) and give EV7 its own unrolled loop that software-prefetches both streams six cache lines ahead of the current block: the source with a plain read prefetch (LDL to R31) and the destination with a prefetch with modify intent (LDS to F31, a ReadBlkMod that allocates the line writeable). Both are reads only, so prefetching past the end of the region is harmless and no overrun guard is needed. The distance is not critical: 4 to 12 lines all measure the same on EV7. The 21264 code path is unchanged. Measured on an AlphaServer ES47 (EV7, 1.3GHz), cycles/call: size cold before cold after hot before hot after 1024 3510 3050 -13% 585 246 -58% 16384 55299 46369 -16% 8602 4054 -53% 65536 327197 291205 -11% 61712 27244 -56% 262144 676953 535414 -21% 248933 98661 -60% The large hot-cache win comes from removing wh64: it was the source of hundreds of millions of Mbox replay traps (perf -e r4) in the store stream, which the read-only prefetch path does not provoke. --- sysdeps/alpha/alphaev6/memcpy.S | 61 ++++++++++++++++++++++++++++++++- 1 file changed, 60 insertions(+), 1 deletion(-) diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S index 04bf2422e3..7f5799ecac 100644 --- ./sysdeps/alpha/alphaev6/memcpy.S +++ ./sysdeps/alpha/alphaev6/memcpy.S @@ -85,9 +85,29 @@ $single_head_quad: bne $1, $single_head_quad # U : still not fully aligned $do_unroll: - addq $16, 64, $7 # E : Initial (+1 trip) wh64 address + /* + * Dispatch on the CPU implementation. The 21264 (EV6, EV67, EV68) + * issues wh64 one trip ahead of the stores to avoid read-for-ownership + * on whole-line stores into the destination. That distance is far too + * short for the 21364 (EV7), whose on-chip memory controller has a much + * longer latency: there wh64 neither overlaps the fill nor avoids the + * Mbox replay traps it provokes in the store stream. EV7 takes its own + * unrolled loop that software-prefetches both streams several lines + * ahead -- the source with a plain read prefetch (LDL to R31) and the + * destination with a prefetch-with-modify-intent (LDS to F31, a + * ReadBlkMod that allocates the line writeable). Both are reads only, + * so prefetching past the end of the region is harmless and no overrun + * guard is needed. IMPLVER returns 2 for the 21264 family and 3 for + * the 21364 family. + */ cmple $18, 127, $1 # E : Can we go through the unrolled loop? bne $1, $tail_quads # U : Nope + implver $2 # E : 2 = 21264 (EV6/67/68), 3 = 21364 + subq $2, 3, $2 # E : EV7 -> 0 + + beq $2, $ev7_unroll # U : EV7 takes the prefetch-modify path + addq $16, 64, $7 # E : Initial (+1 trip) wh64 address + nop # E : nop # E : $unroll_body: @@ -142,6 +162,45 @@ $unroll_body: stq $3, -8($16) # L : bytes 24..31 nop # E : beq $1, $unroll_body + br $31, $tail_quads # U : 1..15 trailing quads/bytes + + /* + * EV7 (21364): 64-byte unrolled copy loop that software-prefetches six + * cache lines (384 bytes) ahead of the current block -- the source with + * a read prefetch and the destination with modify intent. The distance + * is not critical: anything from 4 to 12 lines measures the same on EV7. + * Both src and dest are 0mod8 here (dest is 0mod64), and the prefetches + * only read, so running past the end of the region is harmless. + * $16 - dest, $17 - src, $18 - bytes left, $1..$7,$22 scratch. + */ + .align 4 +$ev7_unroll: + lds $f31, 6*64($16) # L : dest prefetch w/ modify intent + ldl $31, 6*64($17) # L : src prefetch (read) + ldq $1, 0($17) # L : bytes 0..7 + ldq $2, 8($17) # L : bytes 8..15 + ldq $3, 16($17) # L : bytes 16..23 + ldq $4, 24($17) # L : bytes 24..31 + ldq $5, 32($17) # L : bytes 32..39 + ldq $6, 40($17) # L : bytes 40..47 + ldq $7, 48($17) # L : bytes 48..55 + ldq $22, 56($17) # L : bytes 56..63 + + stq $1, 0($16) # L : + stq $2, 8($16) # L : + stq $3, 16($16) # L : + stq $4, 24($16) # L : + stq $5, 32($16) # L : + stq $6, 40($16) # L : + stq $7, 48($16) # L : + stq $22, 56($16) # L : + + subq $18, 64, $18 # E : count -= 64 + addq $17, 64, $17 # E : src += 64 + addq $16, 64, $16 # E : dest += 64 + cmple $18, 63, $1 # E : less than one more full trip? + beq $1, $ev7_unroll # U : keep going + /* fall through to $tail_quads for the 1..63 byte remainder */ $tail_quads: $no_unroll: From patchwork Wed Aug 12 01:19:41 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141151 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E001F4BAE7FF for ; Wed, 12 Aug 2026 01:22:45 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E001F4BAE7FF Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=Leiw7JHl X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb133.google.com (mail-yx1-xb133.google.com [IPv6:2607:f8b0:4864:20::b133]) by sourceware.org (Postfix) with ESMTPS id A872D4B99F54 for ; Wed, 12 Aug 2026 01:20:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A872D4B99F54 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org A872D4B99F54 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b133 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497607; cv=none; b=L0JxelQ1NsksKMVn/i1q5YxgOwMjJ7DCdjLB7HBiLMmNYFhsdwlld+IKCvE4hUehDnzL/1GAnFsP2JVlkQYMisL2odrlD6LuYOwkRPtBSgjfT8BdxEElO4xG5wAWipgRNPZoLbn5gm9u0yVqiTd5MYEaDtDEwDQNfvpn/PoXVY8= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497607; c=relaxed/simple; bh=hdmvmijq7CSF3s3DnxSPTqh4a3DLd6z0/oD7SRBlByI=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=lY+8FZ8gZ+lnRC4ZVroP+oSFGVI5EqupFBDTWSL6UIwx7jtlbGzg+5rgAFUDaW+TeF7a1GeTI5IxD5Spo05i6ibv4jzho6pgie2cDh2MWKpsxh4sgYyAKWSY45XSQRjtqCc87KnEGW18DYXqoRCNPzUNPV8y3T30DX4mZKqV/+8= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=Leiw7JHl DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A872D4B99F54 Received: by mail-yx1-xb133.google.com with SMTP id 956f58d0204a3-66804667af2so498099d50.0 for ; Tue, 11 Aug 2026 18:20:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497607; x=1787102407; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=kUwz4wZg7ZCR62D+xpqGgmRz+CF2/WFs1n94TAJEVbY=; b=Leiw7JHlky58co0BoCWktmoN+b6un6ExSshyiq8ionbUbi18RZesspC6IIgUSAP4mQ KDZ0EXJbiSIBf0D0hSNHXAkSeWspNB4NPz84CYq9Jso6lU4HUK98RvlPhUG51SgunpnI x1PzvTKzv/HDmw/lsHL5X/mbcWsqLcRw+aIv638mTxzL4k0HeZU++NchR8KM9mAM3RgR qqz7GlYX2xBBAo3FC8TVu4m3SO+K9N8Duoc5f7V3qMBRwEeEqfJupg7htNjthyV9uBk3 pyHWu0IK7at3c/ln7OkTtovEX4ug3fMGW8lTUfUAR1RPafIl84BSn5cV0UjpfhsnjZNa iElg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497607; x=1787102407; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=kUwz4wZg7ZCR62D+xpqGgmRz+CF2/WFs1n94TAJEVbY=; b=oFFhUxgJ62ArpwHe+yZ2Cld+tYnP5aaxclRzzgSCc+IGWfzu058gdQ7lnzb72nnsLP edrIRZ53If8hy/TyKmX+gO/CODXl5BbsBvR6sECq/V6thWr+LXY+jS5XWTAnBrSJ8vVZ piouwpbNycDoQBgsGwmAPdvUToNxJ5T5G2UyOAFyEyrt3EtTgKohP4m8o98GGKb2CnHt a1tRynDg7KvbfBHitpyHLfb6FSnMimfl1p5szOD7vkjKvzs27L6p8LJXI2e4mUHpvhTn MGpUT4g/RBSpSRXgSuqZCp+hzsc7mYtViVnwoM/stgEx5A0mEow+0Opk3lM2BPWZVsQ+ EOHQ== X-Gm-Message-State: AOJu0YxjvjWdeGE3+KukYoCa5CZjQGBYdyn4z1JvAz7e2zFaIljPqllG HhfmylCNpQnJoHLyRtEniT9xWVjXqnaVLNFABZXJ5Y7foCzlNXaDeeHTHHHP/nx16pM= X-Gm-Gg: AR+sD13DbfPntxHJASStNtN555OyP4/dUJ8Hr2CY9ydGScTem2Qzx0u/nKTM5+WrsSy +EWe7CVj/CLFjbKsaOclBF8wQknkEbjzGtCjOU1tHT+VUHjOXYuH53XmKR/Xs1Asew7wxAslaz9 cIWXpaxVQCRa7+OwuNquKAZT6CUbRSsqqRqOjck3L+paTw7HGOOsNDFbCzTzTa+4Qyc6k/DNAZ3 8CYdpHAlmFqWU4lskOu75HiF7UF+kJMt60mb2m+Ue+OydW2sqD7ffOAYDE46rLQfLjkJbp6Tsif g/o6FKWwuunOtICIWH63YUcndXF240wiMMICCSCTx6dFkFoATm0aLV+XkSoO9FqqmeV04DqmzEr gt/jDF9zy/ssxG5ixjnS9Pm/HcZcsspftQpf5DHqZV44ilsK+KX5z2kGFXe31iRWC5lQMk7dnke 2LW+afJ73Os/KqHDNFMZcadvUXiSjyCKg7VvOrzFVImLowYtcOFHdZ74Si6Uc= X-Received: by 2002:a05:690e:c41:b0:667:ca27:479e with SMTP id 956f58d0204a3-66b34112f6bmr739604d50.8.1786497606892; Tue, 11 Aug 2026 18:20:06 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66b1ef5f935sm2086925d50.1.2026.08.11.18.20.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:06 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 04/20] alpha: issue the ev6 memcpy wh64 two lines ahead Date: Tue, 11 Aug 2026 21:19:41 -0400 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, LOTS_OF_MONEY, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The 21264 memcpy loop issued its wh64 write hint one 64-byte line ahead of the stores that consume it, which is too close to be much use. wh64 has to claim the line before the stores reach it and, unlike a load prefetch, there is no fill latency for a late hint to hide behind -- a late hint collides with the store stream and provokes Mbox replay traps instead. The same mistake costs memset a factor of three; here it costs less, because the loads of this loop already put some distance between the hint and the stores that consume it. Issue it two lines ahead. Measured on an EV68CB (DS15, 2MB off-chip Bcache), cold copies improve by 6-8% from 4KB upwards, and 1KB is unchanged. Two lines rather than four: the gain past two lines is under 1%, while the stretch at the end of the region that has to fall back to hinting the current block grows with the distance and makes short copies slower. The curve at 512 bytes runs 702 cycles at one line, 760 at two, 863 at three and 870 at four; at 256KB it is 324k, 301k, 300k and 298k. Two lines is the only depth that does not lose ground at the short end. The CWG asks for at least two blocks of lead per stream, and notes that short trip count loops may want less than the 8/n blocks its MAF rule suggests, which is exactly this tradeoff. Both guards move with the distance, because wh64 makes the entire 64-byte block UNPREDICTABLE and a hint past the end of the destination would corrupt memory. The steady-state hint consumed at trip N+1 covers [$16+128, $16+192), so that trip needs 192 bytes left and the trip that computes it needs 256. The loop is only known to have 128 bytes left on entry, so the initial hint needs its own check -- it is the one hint that can overrun where the steady-state one cannot -- and falls back to naming the first block. Verified on an EV68CB against the C memcpy over every size class from 0 to 64KB at 72 destination alignments, with 512-byte guard regions either side checked for stray writes. --- sysdeps/alpha/alphaev6/memcpy.S | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S index 7f5799ecac..cf5c97c1de 100644 --- ./sysdeps/alpha/alphaev6/memcpy.S +++ ./sysdeps/alpha/alphaev6/memcpy.S @@ -87,9 +87,16 @@ $single_head_quad: $do_unroll: /* * Dispatch on the CPU implementation. The 21264 (EV6, EV67, EV68) - * issues wh64 one trip ahead of the stores to avoid read-for-ownership - * on whole-line stores into the destination. That distance is far too - * short for the 21364 (EV7), whose on-chip memory controller has a much + * issues wh64 two lines ahead of the stores to avoid read-for-ownership + * on whole-line stores into the destination. wh64 has to claim the line + * before the stores reach it and, unlike a load prefetch, there is no + * fill latency for a late hint to hide behind, so a one-line lead is too + * short to do much good. Going deeper than two costs more than it + * returns here: the loads of this loop already supply some slack, so the + * gain past two lines is under 1%, while the stretch at the end of the + * region that has to fall back to hinting the current block grows and + * makes short copies slower. memset, which has no load stream to hide + * behind, needs four. wh64 is far too * longer latency: there wh64 neither overlaps the fill nor avoids the * Mbox replay traps it provokes in the store stream. EV7 takes its own * unrolled loop that software-prefetches both streams several lines @@ -106,9 +113,17 @@ $do_unroll: subq $2, 3, $2 # E : EV7 -> 0 beq $2, $ev7_unroll # U : EV7 takes the prefetch-modify path - addq $16, 64, $7 # E : Initial (+1 trip) wh64 address - nop # E : - nop # E : + + /* + * Initial wh64 target, two lines (128 bytes) ahead of the first block. + * The loop is only known to have 128 bytes left at this point, so the + * first hint can overrun where the steady-state one cannot; if there is + * not room for it, hint the first block instead. wh64 makes the whole + * 64-byte block UNPREDICTABLE, so an overrunning hint corrupts memory. + */ + addq $16, 128, $7 # E : initial (+2 line) wh64 address + subq $18, 192, $2 # E : room for [$16+128, $16+192)? + cmovlt $2, $16, $7 # E : no -- hint the first block instead $unroll_body: wh64 ($7) # L1 : memory subsystem hint: 64 bytes at @@ -134,7 +149,11 @@ $unroll_body: stq $4, 8($16) # L : bytes 8..15 stq $5, 16($16) # L : bytes 16..23 - subq $18, 192, $2 # E : At least two more trips to go? + lda $2, -256($18) # E : room for the +2 line hint next trip? + # it covers [$16+128, $16+192) there, + # so that trip needs 192 and this one + # needs 256 (256 exceeds the subq + # literal field, hence lda) nop # E : stq $3, 24($16) # L : bytes 24..31 From patchwork Wed Aug 12 01:19:42 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141153 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C51514B99F42 for ; Wed, 12 Aug 2026 01:23:54 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org C51514B99F42 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=WDoSVW1M X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x1131.google.com (mail-yw1-x1131.google.com [IPv6:2607:f8b0:4864:20::1131]) by sourceware.org (Postfix) with ESMTPS id 3C5754B99F53 for ; Wed, 12 Aug 2026 01:20:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 3C5754B99F53 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 3C5754B99F53 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::1131 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497609; cv=none; b=qJiTmQtdRiqByhec+ZXC64P8NyG+Yi9bWdtCrqmA9BrPvLpP18DtHV3dzMhwTqabd1Bdn+XzdJLM/gvVdNbsoo1bppEzOgalDTlc2Dlmx9QHPk7FmU3kQ/Hec5rfNjorh0f1kgqpAQp5Xw+4LGDpKM3kPNn4bROwQqAlm2oCBhk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497609; c=relaxed/simple; bh=A3nIMyMdVEFRngTHoglLWZV9KECLdOAycTfOduhPIwc=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=XfrqttAREmtsRNXKk1jOTOROus4P0tmQUEDHqzRkJx4LdPxE3lH2N057ETeGV53Q7xu8xAGn/yrQk5xOEkzHhoe9pZH+Sk8YzXonOaCtwb+CcorZOro0qdEcpZREiNmcSSJesQS8+HnIpqS94DxeCfcYIrwwVDjaiq0jBa50l9I= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=WDoSVW1M DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 3C5754B99F53 Received: by mail-yw1-x1131.google.com with SMTP id 00721157ae682-81ecf499af9so6760557b3.1 for ; Tue, 11 Aug 2026 18:20:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497608; x=1787102408; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=xsYsyw7H+RuD6bPbUkLqCpu9niUJzaGcQzbzDBSYYI4=; b=WDoSVW1M01zf0NZ7ZEeJCXjgUDQ+b8ebb0ropgFWm8LSX3w3q2NO5OcpXJfwaEjrb9 KgZ1jPLMQH7g/KLP+JbRSdbqTT2VPT3U8sLAJeKkwytmfSK2xhMxDFUclB3tCKu0/wMX NI0UIVerKJu3NGBbNEaCWk8AYrmLCqi8lgw55uHb3tqhaotdgRyKHU+QHCPPoYH4BdvO ipoHLR8UCxvqNM7L4Q4Kd+T/3HHnyxShX2eBmj/rSL9/e3n/pAlGN+HDYx9eY3IHjlmQ HeQcQKJ8zekk/zTkDyke9coa+LOJ1EqGSJPTsNyS0QlDakNY7DKQc3RjtFZcFPfeKdcX Nb6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497608; x=1787102408; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=xsYsyw7H+RuD6bPbUkLqCpu9niUJzaGcQzbzDBSYYI4=; b=CbyWesJfzPGb8RR0Q636Sefm9TCYXPm9M4qxBZ3EJ4kGWfHnQitroZxA7Nkrc3VYHR 2v1+5tOXfx+gmTvad69G1HfNfa8aqAb5l/lsEroTAWG8LsRZl4PyZnWwx5LfZ3+t+T9/ Ih2FSquOsLlrMuxIgsBjQuYf6EwQhzsFJDSKcv4Ec7JJgPC1djuwgB2rSxve2hJwkhaG ml9Lz2SEjBKBOdWGuFxF2WrWPofn3ZjhUtxEKFjXSFkM9wcCaOvisnoQ5+yJUdRw+StN d4+6EvZYwXIZmb7IHZW8L5zZYZ2YDTK3jh8x3irl86g3/pMPT7CQj0DmhVSm+TnIPljP GV9w== X-Gm-Message-State: AOJu0Yz4VqBf9s7AkkECacYeKF2njhm0A7+2iVDP+HDvsQd5QVWOl/DP h3smJPT26X8Ahkf7qoaKw9rnLDh9dAjXa5t5hV/u75/KCgcNihsRYBovSyPvz1uYkp4= X-Gm-Gg: AR+sD12R6Z6y/Fr4rNa4CFmgCH471pLJptwEpFFXTzRhbMlLbBmwy4OcK9e8f+GinSh FENQEriJghusrtSa98vuxgVEQEo9m0aSuxNc5OYfsh+X6oOkpJx1nLu1Vm3dH16abDIhMdKj5ix z5LtX2yn1w/UXTA+Mx1H/xg82kSQz6U5MQZ74Hkrz/7+aQNA/+ySTHeoqLv3oIe47Ulfbh+UuJR BVKGRw40gENZ6H/+hmcRkYmKszcLhDQV4zikXEC1P0rKNmQw2pvuWtvUXx54fI2kH2nXMBHMF1L quUFMag0XWwfb1sd+H12N1/haqI0p1G6yGMJ6Ypr+bl7KCfMSDccLNS+nRFTWCyooWRfwASimGC 9t3dCVe65cSIjCcLFQxHcyZI+Bntu7Sb8RP2liXn1k0ZfaAgleUYVj488iY2Pp0KNlvgPc3d/37 sgi/hnDu4I4TzNwlTGvZ8AXiQlKtvHRTGSrySriCbKn8vQ0hYAzT8dF2W0XU/qesTVfPsPbLHBL Iec+8evrA== X-Received: by 2002:a05:690c:4b8c:b0:81c:872a:5152 with SMTP id 00721157ae682-8310ba393e1mr9074367b3.30.1786497608490; Tue, 11 Aug 2026 18:20:08 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830abb2e076sm5827557b3.47.2026.08.11.18.20.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:07 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 05/20] alpha: document ev6 memcpy design constraints Date: Tue, 11 Aug 2026 21:19:42 -0400 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Record two non-obvious decisions in memcpy.S so the next reader does not spend the effort a second time. No source prefetch on EV6: adding a read prefetch on the source measured a net loss on an EV68CB -- it gave back a third of the wh64 gain on cold copies and cost 15% at 64KB hot. The loop is memory-latency-bound with loads already spread across the trip; an extra reference only competes for the 21264's 8-entry MAF. Misaligned path not unrolled: unrolling four quads to a trip cuts 38% of instructions but gains only 1.4% in cycles, with IPC falling from 0.37 to 0.23. The path stalls on the serial ldq_u -> extql/extqh -> bis chain that produces each quad; feeding it more instructions per trip does not shorten that chain. Comment only, no change to generated code. --- sysdeps/alpha/alphaev6/memcpy.S | 38 +++++++++++++++++++++++++++------ 1 file changed, 31 insertions(+), 7 deletions(-) diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S index cf5c97c1de..89324be23a 100644 --- ./sysdeps/alpha/alphaev6/memcpy.S +++ ./sysdeps/alpha/alphaev6/memcpy.S @@ -96,13 +96,23 @@ $do_unroll: * gain past two lines is under 1%, while the stretch at the end of the * region that has to fall back to hinting the current block grows and * makes short copies slower. memset, which has no load stream to hide - * behind, needs four. wh64 is far too - * longer latency: there wh64 neither overlaps the fill nor avoids the - * Mbox replay traps it provokes in the store stream. EV7 takes its own - * unrolled loop that software-prefetches both streams several lines - * ahead -- the source with a plain read prefetch (LDL to R31) and the - * destination with a prefetch-with-modify-intent (LDS to F31, a - * ReadBlkMod that allocates the line writeable). Both are reads only, + * behind, needs four. + * + * The source stream deliberately has no prefetch of its own. The CWG + * asks for one per stream, but adding a read prefetch here measured a + * net loss on an EV68CB -- it gave back a third of the wh64 gain on cold + * copies and cost 15% at 64KB hot. The loads of this loop are already + * spread across the trip, so the memory-level parallelism a prefetch + * would add is present anyway, and the extra reference only competes for + * the 21264's 8-entry MAF with the demand misses that need it. + * + * wh64 is far too short a hint for the 21364 (EV7), whose on-chip memory + * controller has a much longer latency: there wh64 neither overlaps the + * fill nor avoids the Mbox replay traps it provokes in the store stream. + * EV7 takes its own unrolled loop that software-prefetches both streams + * several lines ahead -- the source with a plain read prefetch (LDL to + * R31) and the destination with a prefetch-with-modify-intent (LDS to + * F31, a ReadBlkMod that allocates the line writeable). Both are reads only, * so prefetching past the end of the region is harmless and no overrun * guard is needed. IMPLVER returns 2 for the 21264 family and 3 for * the 21364 family. @@ -289,6 +299,20 @@ $dest_0mod8: ldq_u $3, 0($17) # L : seed (rotating load) of 8 bytes nop # E : + /* + * This loop moves a quadword a trip, about 1.4 instructions per byte + * against 0.46 for the aligned loop above, and a mismatched-alignment + * copy costs 1.84 cycles a byte on an EV68CB where an aligned one costs + * 1.13. The instruction count is not what makes it slow, so unrolling + * it does not help: a four-quad-per-trip version cut instructions by + * 38% and bought 1.4%, with IPC falling from 0.37 to 0.23. Both paths + * take the same Bcache misses -- the data moved is identical -- and + * this one takes fewer replay traps than the aligned loop, so it is + * neither memory-bound nor issue-bound. It is stalled on the serial + * ldq_u -> extql/extqh -> bis chain that produces each stored quad. + * Making it faster means breaking that dependence, not feeding it more + * instructions per trip. + */ $mis_quad: ldq_u $16, 8($17) # L : Fetch next 8 extql $3, $17, $3 # U : masking From patchwork Wed Aug 12 01:19:43 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141149 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 33E0A4B99F58 for ; Wed, 12 Aug 2026 01:22:38 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 33E0A4B99F58 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=Ki4WwAo1 X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) by sourceware.org (Postfix) with ESMTPS id EB2AE4BAE7E0 for ; Wed, 12 Aug 2026 01:20:11 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EB2AE4BAE7E0 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org EB2AE4BAE7E0 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::1134 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497612; cv=none; b=WJSM7WLOIeM81HoFzCCfIPRkYw3vjrNvMB+Wkv0aMmYZXOd5nWP837W7ElrvXCfQK2H6T0hKnEt+igXWy3C0BjDV4wYyp85h+60YJK+9wVDNML7yHWtKHnbzH1V01U09rClN8hFSVBCiUVLL4C70vEb5zBMJpEQHx4BJ9e8kQds= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497612; c=relaxed/simple; bh=JW930pgZE4X0rZ0zHsoDM63HAHQ8SAEIIFv5KXb8hKQ=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=e7cA/lPcl3vGf4I4zLkHdfDH8ntJ+OU0qNhkmqfqgZ8n3n4rHYvNgIwaXXHCIImIrgP03Nmb3dO2C7tj7h4ROZZOyZyJwLsbGCYzyP75LRYhYIBsAevb2uMJHGkszvS20di5HFfLbJGNEnUqwU9WIykPjvWAuizHmtet4lMJq5U= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=Ki4WwAo1 DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org EB2AE4BAE7E0 Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-821106b5b64so9693007b3.1 for ; Tue, 11 Aug 2026 18:20:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497611; x=1787102411; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=AnUO+BcyFy0GOQ1I5/raq9joURRMFitcPwAnjxPYYtI=; b=Ki4WwAo1fuhFecfSF2RnhEvWw23wqQPt2FJm8Qp5ZAgMDLO4A6b8oSLCP3tEFmBg9I IVOMqP09ulskJcexJRrWMsIA+5xy6cWWAFtwd2XnFfIzkT9mL4j04I0h6fyIGgrJNyqi LuOx7kxpyCzLS53GU3VKbT6i5LZdY0YONDGjN26Tb5CU5JrZaPh0PZmhU1z0uuTbFo9K kaFivN/4v4PmvSbUzRH73tahnIhv0WS7fYp7ZeUkYfQDwWxlIuCDKUErN8lg7AMeO81T lMf89w3TajteKX97bCYCRCPaeNIAuAUlWhpt2/Qb0P0wcDGB5G+0Fe/OEa6RX1qCDxUt ye7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497611; x=1787102411; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=AnUO+BcyFy0GOQ1I5/raq9joURRMFitcPwAnjxPYYtI=; b=E3dJTcc5Wz3RqwBXEJT09JRLF2IgTKFj0ezg474c7hlOJdy4LyB7U9ob3k8DIou4p4 JIy5gdeIWIBvUhhggbTVtrXYHmoIGO5PBRVZIa5W0GdfwzkcE24zMccULLwsYX81Ijb9 M96mEUpjNBtyHxmR8y0h725zwpLIYzmziIUPJXbUeOfz5shAf1mLwu0O7tPw4m5OIjBI 4KueGn8GeW6ceaYcuyP8JELK6IzMRvU0FWExtoRf3FBTGN57wigrYdRYL9GHbCguA6ue Kvl3TPh6jt56BLmSsRuvT1uewiFOTj1dkUBfreq1UwZFvZKdu9O7pb7nsZ7QqukEg2BR bU3Q== X-Gm-Message-State: AOJu0YxU01FF+cx3+BkgfQvCQhy3vNH6O6tETG5J2hhdQeHSDT42qfqi LS3XxnfPZFF+7suc28Y4TybSgjZnBmSaI4TprwNZ9sY+2D4SHjrdZTHcT1hlxtIFPyI= X-Gm-Gg: AR+sD12bWqN7A2FRRebmO+o97dj+ftcXVeJuwi2S68u4qdDjI3BjyQapeXyCixWvTcX wFayTq6Hwl/4iD0DQ5l4oDEYlh8GJ31fG6Qamg4wI7YS5rOJLXfwzc6/uPyGGZCp/cOyfZm1tCa TYZKDQKhMmTpWahNuHWbO2KTwWiwIwsMIuVKaTvKnr1W6Xu6wpI3k62+vCJcUnTVj94vOiTlxLz tZw/zJ800NMxbXBWEc+4es0Tt5HAR8/JrUmZM6854CsAJFDEPt6uvgD6ibMBLDykkWrZTdNPXKz 1l2+fppSQa9kQ0p1lWtLuRp5TbQyCQ5Vy239ou9ESZk92s1pgYGCqTRa+muwv9Jciq87A5OgfbZ YSsB/aUwaTjNC3y3T84GsnsmNLg3W/rXWgo7tIKW3h+gsr3AnZyRDvufTmboYFZECEFHMAYzcDI aoxgIl1Kbcu4XE/7kMZ2fu+15D3uDV27X2jqYvAiwPv20IjFMJVMU/xIhe4M4= X-Received: by 2002:a05:690c:6d84:b0:7ef:9fd9:db07 with SMTP id 00721157ae682-831438c1ff3mr4686277b3.12.1786497611255; Tue, 11 Aug 2026 18:20:11 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a368e86asm6286567b3.1.2026.08.11.18.20.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:09 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 06/20] alpha: finalize the ev6 memcpy wh64 address late to avoid Mbox replays Date: Tue, 11 Aug 2026 21:19:43 -0400 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org $7 holds the address the next trip's wh64 issues against, and the cmov that picks between the two-line hint and the end-of-region fallback ran in the same group as the loads near the top of the loop. Letting the hint issue that early, while the previous trip's stores are still in flight, provokes Mbox replay traps in the store stream. Move the cmov down into the group with the loop branch. The fallback address has to move from $1 to $23 to get there, because $1 is reused for the loop condition by the cmple two groups earlier; $23 is otherwise unused in this file. Instruction count is unchanged -- the cmov swaps places with a nop at either end. Measured on an EV68CB over a cold 256KB copy, this cuts Mbox replay traps (perf raw event r4 on the 21264) from roughly 1000 per call to roughly 750, and is worth 1.4-1.8% from 2KB upwards. Short copies are unaffected at 256 bytes and lose about 1% at 512. This is worth a comment because the placement looks arbitrary and reads like something to tidy up. It is load-bearing: hoisting the cmov back beside the loads, or reusing $1 so that it has to be hoisted, gives the gain back. A variant that instead added a second fallback level reached the same effect by accident -- it too pushed the last write of $7 into the final group -- but paid fifteen thousand extra instructions per 256KB copy and lost 2.3% at 256 bytes for it. Verified on an EV68CB against the C memcpy over every size class from 0 to 64KB at 72 destination alignments, with 512-byte guard regions either side checked for stray writes. --- sysdeps/alpha/alphaev6/memcpy.S | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S index 89324be23a..d5b72d4961 100644 --- ./sysdeps/alpha/alphaev6/memcpy.S +++ ./sysdeps/alpha/alphaev6/memcpy.S @@ -148,7 +148,7 @@ $unroll_body: nop # E : ldq $3, 24($17) # L : bytes 24..31 - addq $16, 64, $1 # E : fallback value for wh64 + addq $16, 64, $23 # E : fallback wh64 address (see below) nop # E : nop # E : @@ -173,8 +173,7 @@ $unroll_body: ldq $6, 0($17) # L : bytes 0..7 ldq $4, 8($17) # L : bytes 8..15 - cmovlt $2, $1, $7 # E : Latency 2, extra map slot - Use - # fallback wh64 address if < 2 more trips + nop # E : nop # E : ldq $5, 16($17) # L : bytes 16..23 @@ -187,9 +186,23 @@ $unroll_body: stq $4, -24($16) # L : bytes 8..15 cmple $18, 63, $1 # E : At least one more trip? + /* + * $7 is deliberately finalized late, in the group with the loop branch + * rather than alongside the loads above, and the fallback address lives + * in $23 so that it survives the cmple that reuses $1 for the loop + * condition. This is not cosmetic: $7 is what the next trip's wh64 + * issues against, and letting that hint issue while the previous trip's + * stores are still in flight provokes Mbox replay traps in the store + * stream. Measured on an EV68CB over a cold 256KB copy, moving the + * cmov down cuts replays (perf r4) from about 1000 per call to about + * 750 and is worth 1.4-1.8% from 2KB upwards, for no extra instructions. + * Moving it back up, or reusing $1 so that it has to move back up, + * gives up that gain. + */ stq $5, -16($16) # L : bytes 16..23 stq $3, -8($16) # L : bytes 24..31 - nop # E : + cmovlt $2, $23, $7 # E : fallback if < 2 more trips (late; see + # the note above -- do not hoist this) beq $1, $unroll_body br $31, $tail_quads # U : 1..15 trailing quads/bytes From patchwork Wed Aug 12 01:19:44 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141144 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 10FDD4BA2E03 for ; Wed, 12 Aug 2026 01:20:46 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 10FDD4BA2E03 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=S4tWitq0 X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x1136.google.com (mail-yw1-x1136.google.com [IPv6:2607:f8b0:4864:20::1136]) by sourceware.org (Postfix) with ESMTPS id 6E76E4B99F50 for ; Wed, 12 Aug 2026 01:20:14 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6E76E4B99F50 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 6E76E4B99F50 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::1136 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497614; cv=none; b=lNKAxnwvtjyq0OuFTv4ZB3ng82Qq7nU7TByjW6VaAhB8IfVT8pJB0BIF5F1HkWigzD6VS5std7BzqOxzGN6lRlMwKfb+My6AJejnh1O9R6xNzLy2OsajpXf/NroEnOxHxBxhNw00RZbC6bTYH9iQ3TlqEHHVHq4CRJP9cKMTO8A= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497614; c=relaxed/simple; bh=ziV+qQap/+58lKfYfq6YV4OjJ7xQnpM0Vwx87b4Y1PU=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=lvWTIH5cxI5bKQTavqS11a3fo84FoaylaaFqeetD69+9ZgZOey7CkWwnUW2ixOFCmpw9iqJ5kPglyd8UzX8S5TtS2urFsUGM3zEJMWkG4aQpvVLCE5OgwywWFTb8Pd+lPBkfihIorTsYHhJ7oRHoLs1qyCGBeQv9V9Hde5NhH6E= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=S4tWitq0 DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 6E76E4B99F50 Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-8201447e8cdso5804397b3.3 for ; Tue, 11 Aug 2026 18:20:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497614; x=1787102414; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=rb2krvaqc3u4iNjvVtXiJ+MM9DGKvyi5OSfAngEA4ic=; b=S4tWitq0l+hS7YvQqZC3ffybdMAMg/v3Z6aiULDmDUmpv52Jcn0KTjANq9QCM85q69 QEn2jPK1psgTsHEaSc4i6O5b34cESI3cWBfxXFF8mfZqtREdWm0oPslrTiRN7YlOOTXE 9md6kWsELSjUIHYxQUB/8Z72KTyUm21dJ7OB0frxqArWda0TK1DL7ye27OnXP3j6u5Th 5cppK2jDMJY2uN503hPeLo6nqjrAXk8+74DUXT0Q0hc76H71RDPeuzvqi0/nxiDglDGi GjSLsnADMkpeU4t6if/zPkVphdfbr3jW4275fqG0SCZ2RXm1BqwsdEDlIwRmW2LISxQj 5auA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497614; x=1787102414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=rb2krvaqc3u4iNjvVtXiJ+MM9DGKvyi5OSfAngEA4ic=; b=N5GwsxxxVIOB+IRHQxefgSqvEL0vmIK5rx/wQCpLWqzWwDA6QGfyshqvU056uyUkA+ O/mywArC20Zxm98q89MaWFRmuKGTldVamnS/DWV4+HycTZbDFqB1EbGi0DqrZk9Fd/ta Zh37ZSrzriymrVgCOUjZAgE6pcP+V3QFcu+fkHZx05sfbZByCNja6mlt6d+LccZwRsLr E1TRHDIP9KbPD/UEdaPFlqUiQxm0DZ7F9lhdEmy4RK6zhe3QozlwWID5K9az6c4Tne4a JNQuZlclUakNsXwoTgJW5oKUBaSjc2EDkgeZ5VSAbYNcjL0I0oW6RBQTfLQv8OF33XNR VJVg== X-Gm-Message-State: AOJu0YzJ/XuIGjaUXA0V1iYgphEkKHTcbJ1eZtBpCZitXhV6ftKDTZJb KOyie/TE4+uoSaHewe5CcE8KHKQGhSeBe92mbrKYV2MNf9bHpMPitoORCmFPljT/uq4= X-Gm-Gg: AR+sD108TnxD17KRB6bKYig3hMht2/Y9dxzaRXJPpNDdMFECJRf9fMe/sYtwvjbzMUg zFX2IgpZ0xgSk8V9VftcbIjOLbvXuo6403Ys/vqxUjyGUSbZMtd3kloBGs58DwAEwq4pIdC+e+7 U9nBBfowmyW3RPJ+E8djxC3qAHAd1ghZr5pPh4A5Z6pa96y/Iu2WRAYwlkhjLN5AYk5r8uNGn63 Xd9pUe7tu6DYXGqWLClHi6T/xGsbgq8a6A4/5UEP8AYNjeuJjBqfzGz//QK2nMBdkM+phmzAxab 9Dre9BFus3V4VbucAdn0HEZVq6z7OBsaGJwL3jdgzjg5viruzcZ1Uz9BLdoSYFk/O5K1ANzy192 6fBxPBpGBLQxN+GT39bwoGI07G/ebjWSr1yZ8p5LQ6h0yE/zFuJmbFHwLZYj3Z/f+7BKHrnsSyg HEYgNqBD7/SL1xZYyZwAaLC7QMfwmr1JAn3WMKa1HOfcY0z1518PLaUa5VqFw= X-Received: by 2002:a05:690c:4b8c:b0:81c:872a:5152 with SMTP id 00721157ae682-8310ba393e1mr9076307b3.30.1786497613775; Tue, 11 Aug 2026 18:20:13 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a368e875sm6275967b3.3.2026.08.11.18.20.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:12 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 07/20] alpha: deepen memchr prefetch distance for EV7 Date: Tue, 11 Aug 2026 21:19:44 -0400 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The alpha memchr already issues a read prefetch ahead of the scan, but only three cache lines out -- tuned for the 21264 (EV6/67/68) and its external Bcache. On the 21364 (EV7), whose on-chip memory controller has a much longer memory latency, three lines is too close to hide the miss. Issue the prefetch eight cache lines ahead instead (both the priming prefetches before the cacheline loop and the one inside it). This is a read hint, so on the 21264 a deeper distance is at worst dropped; no IMPLVER dispatch is needed. Measured on an AlphaServer ES47 (EV7, 1.3GHz), cold cyc/call (full scan): size before after 4096 3898 3268 1.19x 65536 61359 52271 1.17x 262144 246666 208148 1.19x The 21264 side of that claim has since been measured rather than assumed: on an EV68CB the read-prefetch distance curve for a streaming scan is flat from four cache lines out to eight, so moving from three to eight neither helps nor hurts there. An assembly rewrite was prototyped but, with the deeper prefetch, a read-hint loop measured no faster than this C. The only further gain came from a modify-intent (LDS/LDT) prefetch, which is not safe to issue on the read-only operand of a general-purpose memchr: it requests the line for ownership, which is wrong for a buffer the caller may share. It is also not the win it looks like. On an EV68CB, modify intent on a pure read stream measures 2x slower than a plain read prefetch -- for memcmp, 756446 cycles against 368989 on a cold 256KB compare -- because ReadBlkMod provokes about twice the Mbox replay traps. Bcache misses are unchanged, so the cost is not the extra coherence traffic one might expect; it is the replays. --- sysdeps/alpha/memchr.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git ./sysdeps/alpha/memchr.c ./sysdeps/alpha/memchr.c index d9115923d2..663cd08454 100644 --- ./sysdeps/alpha/memchr.c +++ ./sysdeps/alpha/memchr.c @@ -79,10 +79,15 @@ __memchr (const void *s, int xc, size_t n) /* If the block is sufficiently large, align to cacheline and prefetch. */ if (unlikely (n >= 256)) { - /* Prefetch 3 cache lines beyond the one we're working on. */ - prefetch (s_align + 8); - prefetch (s_align + 16); + /* Prefetch several cache lines beyond the one we're working on. + The 21364 (EV7), with its on-chip memory controller and much longer + memory latency than the 21264's external Bcache, needs the read + prefetch issued considerably further ahead to hide the miss; eight + lines beats the three that suit the 21264, and on the 21264 a deeper + read hint is at worst dropped. */ prefetch (s_align + 24); + prefetch (s_align + 40); + prefetch (s_align + 56); while ((word)s_align & 63) { @@ -120,10 +125,10 @@ __memchr (const void *s, int xc, size_t n) } while (0) /* While there's still lots more data to potentially be read, - continue issuing prefetches for the 4th cacheline out. */ + continue issuing prefetches for the 8th cacheline out. */ while (n >= 256) { - prefetch (s_align + 24); + prefetch (s_align + 64); CACHELINE_LOOP; } From patchwork Wed Aug 12 01:19:45 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141154 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 56B6D4BA902C for ; Wed, 12 Aug 2026 01:24:08 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 56B6D4BA902C Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=T7+i5efK X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb130.google.com (mail-yx1-xb130.google.com [IPv6:2607:f8b0:4864:20::b130]) by sourceware.org (Postfix) with ESMTPS id 006564B9DB50 for ; Wed, 12 Aug 2026 01:20:15 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 006564B9DB50 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 006564B9DB50 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b130 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497616; cv=none; b=fJj8JHo799LoDT/jZScfjxfRHu2q8hBFuWTGcKxMeObzrDV8wyVB+6XaaXES7JpjOfht3/3lgrJWtY91BAxlRnaekkxF6cSqX7kYoOJTuk5pftIrADArvBtZB94vfSxuocJ5Qy157c4INxiMUAQeTiXnXOHovLAN7jdmrppRnJw= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497616; c=relaxed/simple; bh=kAZWto6Xa5KojFu3uJJpUonduq3d6WUwWuGcNir4c1Y=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=ZBSVAjIFRVmcWz+7dCGwG373qCdaNBtYywVbUWDKFtz9WHNezxJJBmYvts+0ipJt1EwdYQpCVVc1SfsIz3hihggr28FINGfnItXtXg7yxuvk0VwbNKjOK3cLi6s0/MwtUNaCe3TCodJ/7S/FGRka8RyLcdZ/jiOG8G6AQ/TX3cA= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=T7+i5efK DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 006564B9DB50 Received: by mail-yx1-xb130.google.com with SMTP id 956f58d0204a3-6611669cd16so713908d50.0 for ; Tue, 11 Aug 2026 18:20:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497615; x=1787102415; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=dKlN/Zkoq4D+CTzl4L7+EuPVfZAT4GrtPH/zCzEcRZk=; b=T7+i5efKAt7eUXkMlNxwtgrrF1EQNZkg53EjDnOz36Pc5hVBTMFVFrjBC/ITzXnwG3 O8LnqNXRsoPKjgmkKIJmsUEqmxihahbjKMWBeMjc5f84LvloYO/Uf/x5RYtTKCsQxttE chsBGEelN593c+ie6+0JLAos1OUx9G6EFr4oVKgu9+UMONf+ESS/iZ9m773ZZjRiVAbb W21SNSJ1da/jNdusxzrNZes8fPcnoJTYMBnxULycw7c2Opz5FTWKAf0ZWVWJ6HLIViIz Crn7FJEsbe0jeqwkqYLGDA1LoFDicNuhf/fCyE73ek9gaUUGreKkL6bvCMsvy6i4Rao9 kwNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497615; x=1787102415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=dKlN/Zkoq4D+CTzl4L7+EuPVfZAT4GrtPH/zCzEcRZk=; b=HuNOsplALzy20K//4hRJagkpNM2QgK6b0M3KPaPpd36VYjBENRwfoR85ojieqGBDAw ZTfmmZr+C1cPFQPvD4YALMvZ1REOVjhdCkNbPApDqOBug46owJJTMWGdg4Yhpgiy/BPT wiG59bbB9OblVH6rVvEtT4YAK2LQdKIX2MxMemtXWnNrYKGrtiOHqM171TLsYVt3nAWT FBYSaCC5+SWgioLwggBJnC07zWxGT63YqVcIynBync6lQuZVQqAyprLbY8hoxXCpjrr1 l9HzPUEVHxTs2mElm3YKhjnaBdJ76zNVQEfelIo3IoyoVSIgerTG4kGyQbkGDCw3Yxdo 3ORw== X-Gm-Message-State: AOJu0YzqokzCmxJkwKwfeU8Ux3Wk07/GoXLhn6TvzvnwvW1n6X8XRFYg R7qb++PjuzV1JCFdDXqi5Kgook7XUnPkHV1eXRcQ+LQyghgl8KDU4bIBMKevWxHbtG8= X-Gm-Gg: AR+sD13yZGBFoagQlOPtEo9n4LKgoyodSEJVfnF99dbKue9KQgw9ZFTDHLQdjHyu1wc DtAHaspGjwDr6yAYXVF1SQEGHgdDlHfXoFSdNsZSyJMRo9K+WKWWmNS72UPoaUcvNFQyKzxFYyA tu3nGfeHg/SfJhjYcRA/7Ck/+VF4kRv9FWr7DEuxCoa3aZguxfgbZ5ks2npuhi57+sc8ZyYrnzo RCfVvpmj4KITebivLare1NjAIggtYQUa22+W4MyQjgzt1vyvWU6ufW0xQaY8M2rOUm2TLFYTP1J WWL+6WMwcBLRE4FPE4/dAgc4/ffK4qzQwOyi+eoAPVS1kCt6H5RxqRpV2WzFXfHBvF9hqagtkjj Lwq78bc8hztnkxpemhNRLRTkiavDEZbrCrjcnR/mLazeHsL7CL4Nkj76bhCf/j7qRKviE8ERHFv AxPsthUw2F0DCz6laXyVlRID43/Z+I7iimintpZTe75dWUDvZNLs15q31d40PVMstng+qcyA== X-Received: by 2002:a05:690e:c4a:b0:667:5e83:80ce with SMTP id 956f58d0204a3-66b34308a2amr740698d50.25.1786497615299; Tue, 11 Aug 2026 18:20:15 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66b1f255731sm2095868d50.16.2026.08.11.18.20.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:14 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 08/20] alpha: fold null and difference tests in strcmp aligned loop Date: Tue, 11 Aug 2026 21:19:45 -0400 Message-ID: <524c04bcf903d341b71104102b513868f626eb0d.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The co-aligned main loop tested for a difference (xor; bne) and for the terminating null (cmpbge; beq) with two separate branches per quadword. On EV6/EV7 the loop is issue-bound and right at the one-branch-per-cycle limit, so the second branch is a bottleneck. Combine the two into a single syndrome -- "or" of the difference bits and the null mask -- so one branch closes the loop and the difference test for the first word moves out into the head. The loop still loads the next word only after the current one tests clean (a zero syndrome implies no null, so the string continues and the next word is mapped), so it never reads past the terminating null onto an unmapped page. bench-strcmp on EV7 (21364): co-aligned compares ~6-9% faster across lengths that stay in L1, neutral once memory-bound; the mutually misaligned path is unchanged. Passes string/test-strcmp. --- sysdeps/alpha/strcmp.S | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git ./sysdeps/alpha/strcmp.S ./sysdeps/alpha/strcmp.S index c7f092d1b7..6454b7aab0 100644 --- ./sysdeps/alpha/strcmp.S +++ ./sysdeps/alpha/strcmp.S @@ -53,22 +53,32 @@ $aligned: ornot t0, t3, t0 # .. e1 : cmpbge zero, t1, t7 # e0 : bits set iff null found bne t7, $eos # e1 (zdb) + xor t0, t1, t2 # e0 : difference in the first word? + bne t2, $wordcmp # .. e1 (zdb) + + /* Aligned compare main loop. Fold the difference test (xor) and the + null test (cmpbge) into a single syndrome with "or", so one branch + per quadword closes the loop instead of two. Each load is gated by + the previous word being clean -- a zero syndrome means no null, so + the string continues and the next word is mapped -- hence we never + read past the terminating null onto an unmapped page. - /* Aligned compare main loop. On entry to this basic block: t0 == an s1 word. - t1 == an s2 word not containing a null. */ + t1 == an s2 word, equal to t0 and not containing a null. */ $a_loop: - xor t0, t1, t2 # e0 : - bne t2, $wordcmp # .. e1 (zdb) ldq_u t1, 8(a1) # e0 : ldq_u t0, 8(a0) # .. e1 : addq a1, 8, a1 # e0 : addq a0, 8, a0 # .. e1 : - cmpbge zero, t1, t7 # e0 : - beq t7, $a_loop # .. e1 (zdb) - br $eos # e1 : + xor t0, t1, t2 # e0 : bytes that differ + cmpbge zero, t1, t7 # .. e1 : bits set iff null found + or t2, t7, t8 # e0 : syndrome = difference | null + beq t8, $a_loop # .. e1 (zdb) : clean word, keep going + + bne t7, $eos # e0 : null present (handles diff-before-null) + br $wordcmp # .. e1 : a pure difference, no null /* The two strings are not co-aligned. Align s1 and cope. */ From patchwork Wed Aug 12 01:19:46 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141150 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 6AE3E4B9DB50 for ; Wed, 12 Aug 2026 01:22:42 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 6AE3E4B9DB50 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=hPS+yYRO X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x1129.google.com (mail-yw1-x1129.google.com [IPv6:2607:f8b0:4864:20::1129]) by sourceware.org (Postfix) with ESMTPS id A66634B99F5A for ; Wed, 12 Aug 2026 01:20:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A66634B99F5A Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org A66634B99F5A Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::1129 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497618; cv=none; b=c2clEjLyU6Bi1AKVnOJhP6PmN63+1gM7To9SecXT3xhfuvc1MDDesfxo8qsfvoV5w8tiMXxy3jUF4xarZR6uKKpmTjIZqzMXRU8tuTlrvb8aFlvZ28Ee8S2AIEBAXwypOufinU9GYBvNgsQthlxqg2zTU6sWilwwJ9KQgfc9O90= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497618; c=relaxed/simple; bh=q5rYlev+GS1vep/g42eEKRMCNIUdXJDn2FC7yrzbgho=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=XVs/YphGbKYdB/uvED5A+A76cXkgfig8D0hHQV8EzvAQr4ZwHEbrLGQn6pVSICvC+WUF1Xp1jjampI3UcIggrg4MqHVgy+6L4RjNWdkh6laQMwS01FN7xU3x5baMFBzynzBFc4e11BaSxEoJ1PmLoS/jF6IN4vv7XAZf9Fqavjg= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=hPS+yYRO DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A66634B99F5A Received: by mail-yw1-x1129.google.com with SMTP id 00721157ae682-824af9535c8so7577007b3.1 for ; Tue, 11 Aug 2026 18:20:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497618; x=1787102418; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=BWK+Y+i9wP/beTCn4RP4PFZ91fod+UCI7Jyrzk5fNbQ=; b=hPS+yYROjYrYAFRqaKrN3Ht5MF1UDMV5OGhgHQ81LupyBKzyuEhvcMz6+yncfVsyCr kI1shJx9ZbMpy8TJdniN3BmiEoTkr68Dh4m3Ww92FXhW2zD8AUM1kyP3SGx6+cyojioj xuOQUunDIewiLbJse/AXXyES8t/9wXCDTuG+kMkf1XdoEz5JoqPoXMpBdaF5QWWMorh+ JwBkJysZEurYNiA1Gs9qq4GJsrmQq6z3mZRacHig6wN6Yw5n9yv1nrkCYFgUgJ3q/dNX wiKLKclimWnk2YUwqPpM+9D8XgWje8oWuMfv2dz3D+U85NGemSnGFXW6TghYaNcezBy9 fijw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497618; x=1787102418; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=BWK+Y+i9wP/beTCn4RP4PFZ91fod+UCI7Jyrzk5fNbQ=; b=TWblhGCYKINMkq5JWRxAqSbjqouOoTXjVyUDAmTPT8fft9MzLNmhugxigh2jL3hldI 3m9yAuRmSNZSAs8mdvwcazQw7wrH9Gxp7Ykv1u2G+ez2BVQva33mMJIDsUB1p1YmjFTT mmh9NLRtARNwZTRtyDIlXjnMSkNtGvoKaEC+Ws1WDhsoe9329VMo490EV8BpxMeJ4khn J2RPxGAaOGctebxSOL7+wGlLDeENFHEdkHHBgWqcml7Bl/eeH/CdnCBdkrlDglcujBgU SVZkuJ5wuHNGpNnlNh+33w1EaP4uGWEMt9shXpzpBokzr3X3vIAPg1m0qW8ESvYiBD13 fA3Q== X-Gm-Message-State: AOJu0YyRuqRBUCVVl4tXznS9jc6O7JZ3kS3ZKqkbcx54XLv3spu0CXj1 3Jzz0OVwpIMvgd04JV9CC6kQcwc6Lch0+Oxivz5skmtu9V6qGAdBlNHTchJILO9xS7s= X-Gm-Gg: AR+sD113RuX9KH3GU3EIwvV3e8C06JnxqGYj9sE0/elfeT11J60lq8I0rpiWBhGBY9R QNE6Mdtb7lzvRZj6OGykw/zjeJoObZXe/5T5mukIMmxzlgVeMIpoj3cx98dtYLIxIis0e7O438c QIbRl4f/dk4z6iXc9eH2/2bwxFzhoNju2SCJ6b72vWCiW4F3Jegy0CS7RgvHt+qVgTWeoX13mbb Bkk/G89ayeBY7IOir1tWNCXuNfozHg7acIXnkQsVMYOj/5Y5J9RL/i1jNIpC1OTdXLAjBTWJuTB rClip7qbj09lb0fcvH+bmftzswA430CUC8aara/5I0TMiRyGQWFL7E6vmFMZJyAbhZHL0DxBI3C odZWnZLEXNltWzZ1X94KCLuOrfxS9qEKICnxwOfBnTc59UAaWkKuPy3dibfgT2uCtpSnqw4HCYa AgY5i8SWn4DokChF7cZygIUMMZilYbSeKBwCttPOHwauSbxOeXXC2Mc5xoN7E= X-Received: by 2002:a05:690c:97:b0:81e:799f:d55e with SMTP id 00721157ae682-83131aeaf4fmr6431467b3.19.1786497617886; Tue, 11 Aug 2026 18:20:17 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a378d0c9sm6465757b3.2.2026.08.11.18.20.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:17 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 09/20] alpha: add EV6/EV7 assembly memcmp Date: Tue, 11 Aug 2026 21:19:46 -0400 Message-ID: <1f888a256c172062afcbe683dee419aac5d279e3.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Alpha had no assembly memcmp and fell back to the generic C implementation. Add one for the alphaev6 directory (shared by the 21264 and 21364). When both operands share alignment, byte-align to a quadword and then compare 64 bytes per iteration, OR-accumulating the per-quad xors; a non-zero accumulator means the difference is somewhere in the block, which a byte scan from the block base pinpoints to return the correct sign. Operands of differing alignment use a byte compare. The loop issues a plain read prefetch (LDL to R31) on both streams eight cache lines ahead. On the 21364 (EV7), whose on-chip memory controller has a long memory latency, this hides the Bcache misses that otherwise stall the compare; on the 21264 it is at worst a dropped hint, so no IMPLVER dispatch is needed. A read prefetch -- rather than the modify-intent LDS/LDT used by memset/memcpy -- is deliberate: memcmp never writes its operands, so requesting the lines for ownership would only add coherence traffic on a multiprocessor. The 1 to 7 bytes left after the loops are compared with a single masked quadword rather than a byte at a time, which would cost about eight issue slots per byte and dominate any length that is not a multiple of eight. Both operands are 0mod8 by then, since the head aligned them and everything since has advanced by whole quadwords, and an aligned quadword cannot cross a page boundary, so a whole one can be loaded at either pointer even though only some of it belongs to the compare; masking the difference discards the rest. Nothing is stored, so reading past the end of the operands is not observable. A mismatch still falls into the byte loop, which is what determines which byte differed and in which direction. Measured on an AlphaServer ES47 (EV7, 1.3GHz), cold cyc/call vs the generic C: size generic asm 1024 3617 1502 2.35x 4096 14527 6423 2.24x 65536 231362 96424 2.40x 262144 916913 386234 2.37x bcmp is provided as a weak alias, as in the generic implementation. Verified on an EV68CB against the C memcmp for every length from 0 to 300 at 24 alignments, both for equal operands and with a difference planted at every position in turn, comparing the sign of the result. __memcmpeq is aliased to memcmp here too. string/memcmp.c provides that alias for the C implementation and string/memcmpeq.c is deliberately empty, so an assembly memcmp that replaces the C one has to define it, as sparc64 and s390 do. Without it an --host=alphaev6 or later build links no __memcmpeq at all, which drops a symbol the ABI lists and fails to link string/inl-tester. --- sysdeps/alpha/alphaev6/memcmp.S | 226 ++++++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 sysdeps/alpha/alphaev6/memcmp.S diff --git ./sysdeps/alpha/alphaev6/memcmp.S ./sysdeps/alpha/alphaev6/memcmp.S new file mode 100644 index 0000000000..0a3d2e11e6 --- /dev/null +++ ./sysdeps/alpha/alphaev6/memcmp.S @@ -0,0 +1,226 @@ +/* Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +/* + * EV6/EV7 memcmp. When both operands share alignment, byte-align to a + * quadword and compare 64 bytes per iteration, OR-accumulating the per-quad + * xors; a non-zero accumulator means a difference lies in the block, which a + * byte scan from the block base then pinpoints to return the correct sign. + * Operands of differing alignment fall back to a byte compare. + * + * The loop issues a plain read prefetch (LDL to R31) on both streams several + * cache lines ahead. On the 21364 (EV7), whose on-chip memory controller has + * a long latency, this hides the Bcache misses that otherwise stall the + * compare; on the 21264 it is at worst a dropped hint. A read prefetch (not + * the modify-intent LDS/LDT) is deliberate: memcmp never writes its operands, + * so requesting lines for ownership would only add coherence traffic on SMP. + * + * Temp usage: + * $0 - result + * $1,$2 - scratch + * $3,$4 - loaded data + * $5,$6 - xor / accumulator + */ + +#include + + .arch ev6 + .set noreorder + .set noat + +/* Read prefetch distance, in bytes (8 cache lines ahead). The CWG asks for + 8/n cache blocks for n streams against the 21264's 8-entry MAF, so four for + the two streams here; measured on an EV68CB the curve is flat from four + blocks out to eight, and only a distance below two costs anything. + + These must stay plain read prefetches (LDL to R31). Asking for the lines + with modify intent instead (LDS to F31) measures 2x slower on an EV68CB -- + 756446 cycles against 368989 for a cold 256KB compare -- because ReadBlkMod + provokes about twice the Mbox replay traps on a stream that is only ever + read. Bcache misses are unchanged, so it is not extra memory traffic. + ReadBlkModEN (LDT to F31) measures identical to the plain prefetch, since + the evict-next variants are 21364-only and degrade here. */ +#define PF (8 * 64) + +ENTRY(memcmp) + .prologue 0 + + mov $31, $0 # E : default result 0 (equal) + beq $18, $ret # U : n == 0 -> equal + xor $16, $17, $1 # E : do the operands share alignment? + and $1, 7, $1 # E : ... mod 8 + + bne $1, $bytewise # U : different alignment -> byte compare + and $16, 7, $1 # E : already 0mod8? + beq $1, $aligned # U : yes + nop # E : + + /* Same misalignment: byte-compare up to a 0mod8 boundary. */ +$head: + ldbu $3, 0($16) # L : a byte + ldbu $4, 0($17) # L : b byte + lda $16, 1($16) # E : a++ + lda $17, 1($17) # E : b++ + + subq $3, $4, $0 # E : difference (unsigned bytes) + bne $0, $ret # U : differ -> return sign + subq $18, 1, $18 # E : n-- + beq $18, $reteq # U : consumed all -> equal + + and $16, 7, $1 # E : aligned now? + bne $1, $head # U : keep going + +$aligned: + cmple $18, 63, $1 # E : at least one 64B block? + bne $1, $quadtail # U : no + nop # E : + nop # E : + + .align 4 +$loop: + ldl $31, PF($16) # L : read prefetch a, PF bytes ahead + ldl $31, PF($17) # L : read prefetch b + ldq $3, 0($16) # L : a[0..7] + ldq $4, 0($17) # L : b[0..7] + + xor $3, $4, $5 # E : seed the accumulator + ldq $3, 8($16) # L : + ldq $4, 8($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 16($16) # L : + ldq $4, 16($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 24($16) # L : + ldq $4, 24($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 32($16) # L : + ldq $4, 32($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 40($16) # L : + ldq $4, 40($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 48($16) # L : + ldq $4, 48($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : + ldq $3, 56($16) # L : + ldq $4, 56($17) # L : + xor $3, $4, $6 # E : + + or $5, $6, $5 # E : full 64B xor summary + bne $5, $bytewise # U : a difference is in this block + subq $18, 64, $18 # E : n -= 64 + lda $16, 64($16) # E : a += 64 + + lda $17, 64($17) # E : b += 64 + cmple $18, 63, $1 # E : another full block? + beq $1, $loop # U : + nop # E : + + /* 0..63 bytes left, still 0mod8: compare remaining quads. */ +$quadtail: + subq $18, 8, $2 # E : (n - 8) + blt $2, $tailquad # U : < 8 left -> masked quad + +$qt: + ldq $3, 0($16) # L : + ldq $4, 0($17) # L : + xor $3, $4, $5 # E : + bne $5, $bytewise # U : differ in this quad (n unchanged) + + lda $16, 8($16) # E : a += 8 + lda $17, 8($17) # E : b += 8 + mov $2, $18 # E : commit n -= 8 + subq $18, 8, $2 # E : (n - 8) + + bge $2, $qt # U : another quad + nop # E : + nop # E : + nop # E : + + /* + * 1..7 bytes left, both operands 0mod8 -- the head aligned them and + * everything since has advanced by whole quadwords. An aligned + * quadword cannot cross a page boundary, so loading a whole one at + * either pointer is safe even though only $18 bytes belong to the + * compare; masking the difference down to those bytes discards + * whatever followed. Nothing is stored, so reading past the end of + * the operands is not observable. + * + * This is only the tail. A mismatch still falls into the byte loop + * below, which is what works out which byte differed and in which + * direction, and which the block and quad loops above also branch to. + */ +$tailquad: + beq $18, $reteq # U : nothing left -> equal + nop # E : + nop # E : + nop # E : + + ldq $3, 0($16) # L : + ldq $4, 0($17) # L : + xor $3, $4, $5 # E : + mskql $5, $18, $5 # U : only the bytes being compared + + beq $5, $reteq # U : all equal + nop # E : + nop # E : + nop # E : + + /* Tail bytes, and the unaligned / locate path. On entry from a block + or quad mismatch, the first differing byte lies within the bytes + scanned here, so the loop returns its sign before running off. */ +$bytewise: + beq $18, $reteq # U : nothing left -> equal + nop # E : + +$bl: + ldbu $3, 0($16) # L : + ldbu $4, 0($17) # L : + lda $16, 1($16) # E : + lda $17, 1($17) # E : + + subq $3, $4, $0 # E : unsigned byte difference + bne $0, $ret # U : differ -> sign + subq $18, 1, $18 # E : n-- + bne $18, $bl # U : + +$reteq: + mov $31, $0 # E : equal +$ret: + ret $31, ($26), 1 # L0 : + +END(memcmp) +libc_hidden_builtin_def (memcmp) +#ifdef weak_alias +# undef bcmp +weak_alias (memcmp, bcmp) +#endif +#undef __memcmpeq +strong_alias (memcmp, __memcmpeq) +libc_hidden_def (__memcmpeq) From patchwork Wed Aug 12 01:19:47 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141155 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id EA9864BA902C for ; Wed, 12 Aug 2026 01:24:21 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org EA9864BA902C Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=LH2ZB5VW X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb12d.google.com (mail-yx1-xb12d.google.com [IPv6:2607:f8b0:4864:20::b12d]) by sourceware.org (Postfix) with ESMTPS id 474064BAE7FC for ; Wed, 12 Aug 2026 01:20:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 474064BAE7FC Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 474064BAE7FC Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b12d ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497621; cv=none; b=qJ9vi8hDnNK52utQzzxj/ULMKUkbQ8022GY87gpgFfhFllK2MPPK6hCEpAp3SdYOJanYRjFNRBtMiB3wNs6NtGuSyUvKdMqjh/5TSRfI2sjT9yq0hd05+LY92LHp+86xK26OHcnJKjmSUjEG7silAKYkr/akMqj/iEUmzIrRIoA= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497621; c=relaxed/simple; bh=xAUmbmWo2PLiRaLXRoxRTEKrYpfJ0y1/VCMvLXArPiE=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=SQsQQCrVj3jzGCwALwVO6VVD/v2tRQjVcKEtp1iPjv+aHUDqHd77wKhq942SFIkB5eMOtTGTeU5sA6P13H+rHGKtQ1Y2ujoHx/YjcWTXMmWnM3lNUvlUbkLt2lZMTwkTkrow94zuXhl4Vk4owR7wD1U0AjiGQW9YKEQEBmULyTY= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=LH2ZB5VW DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 474064BAE7FC Received: by mail-yx1-xb12d.google.com with SMTP id 956f58d0204a3-6688dfaed31so550755d50.3 for ; Tue, 11 Aug 2026 18:20:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497620; x=1787102420; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=iqx53qEE0zBqO9FfUhfzM+lsNzmRJdkYyNQnRhent30=; b=LH2ZB5VWRvZnNStuVi+2HFMg+Aj74UeBDakZw5bWpbiRNJP9vQe9DQTRdtzcgpFE7+ 5004xNhX9TgxsREoahCBo/v0XimMx15rFK4azcD/VB+QyQaxTv0VaAuKPTbrO0JtBC8M Yqxh65tHIucFew8Wx4z2YwsEGzQcRynRMkpgQoryWSiCA4LcT5YfPlT3yErcuasj9+GY 3G5JCCqk4vH/dva/YqPOeMN5Ih1w/rKgVGW9Rypy5VaN4Znlf0kz/6XNVHtWyBB22DM6 fxM45SnByBDvOG3iuFj7X8yf5sozXKkui8kW57PHdfQo3DLuUHgQamS77g1rDf3W63d6 Lt1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497620; x=1787102420; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=iqx53qEE0zBqO9FfUhfzM+lsNzmRJdkYyNQnRhent30=; b=sMi1q/wwhTLEk6W3l4CrKX+rb8jeGr0RuA/BkVHiNrqyTQdhGacvLy64w52S6mR4EO OgxJ6EW09mZWPued6wrqBqALMRxbDdcUBwgtvPp7a7tF5Ryq67XA1dIl/qep48hBxPaZ QzOJmVxp5oP1c6I50p9G6gH0NfiBTrSX5gn0Nh732QB5CJ6jC0f343LhpEgW17aEtU2y ynZusMIMSvxgmjhRLt1RIvA0/mb+CxgMOsHX6lzgqIZxEg4xvXi9tMrtPc1uPLhqfhID 4XH+4trOyXTpMIpB3frJya9U16Yqm8w1Lvp0UAo1f7P20g3D3qkhIP22iYjv/QfNNSbu /aMQ== X-Gm-Message-State: AOJu0YyLtptfF+g6mnLNiNn49ULnjYPU1djp8h6o+VmjnkgSHbJCAgtL PCFoUa3aHL2NaBDzZDvy+3cRWd8Z0W8zaCKiDQ3FcLq5rKMZGHk7x/vSUDW2BZCb0Ow= X-Gm-Gg: AR+sD13oGOQrBylQjQ9DFT+KDKHgOJg8iBTRx2/mooycni/59MSeyu1s2hKGdfifKjX BlNqowmzwOywxYxS2ZPRLQ3dbZPlYiwU45Ih6227iXMGhRrd3gqur8ilfplT+YGjRG+Lmr/MlI1 FyxjiTaTGHeruD0gf6Rdb3S7FX5WdY9AULO7dxihBoduadNxaTMPd6lp+UJ/kutNTJoyXiRaiVX LmK6qmeEEDuHW5891B5L2WM1tbR8z9YvL+1mAVg9E9y7wNRjbkGkyY5yYVIBe/9axfBnRj581mM QlZjk6FLA6m8j9UH5xZOicU+aIvu8RRqoZgcdj3Fg12IdchbaVZ5On/Gugrrw/UH+1+bHfMVsN9 4QQE9HcrKemVpjC4ER1rrWW0c4RqlSewQAffiybFhFWLGafNW1K+B0hmty/7GbYgOg4hdptbp4P usdtNjdMBO/FyF/m4NKgo/eKNckzfyU73KaWdHXuciSB4vCdd1kvLk2GvUHwk= X-Received: by 2002:a05:690e:168f:b0:666:396e:bf59 with SMTP id 956f58d0204a3-66b34108954mr750470d50.1.1786497620415; Tue, 11 Aug 2026 18:20:20 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66b1ef961a1sm2094907d50.8.2026.08.11.18.20.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:19 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 10/20] alpha: add EV6/EV7 assembly memmove Date: Tue, 11 Aug 2026 21:19:47 -0400 Message-ID: <5f894ac3615b8c12c7cb091154d688b4570b5f3f.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-9.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, LOTS_OF_MONEY, RCVD_IN_DNSWL_NONE, SCC_10_SHORT_WORD_LINES, SCC_5_SHORT_WORD_LINES, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Alpha used the generic C memmove. Add an assembly one built out of the ev6 memcpy: when the destination does not overlap the source from below -- that is, (uintptr) dst - (uintptr) src >= n, which also covers dst < src -- the copy runs forward and is exactly memcpy, with the same-alignment quadword fast path, the wh64 unrolled loop on the 21264 and the prefetch loop on the 21364, and the rotating ldq_u path for mismatched alignment. Only the genuinely overlapping case (src < dst < src+n) copies backward, and that is rare and not performance critical, so it uses a straightforward descending aligned-quadword-or-byte loop. Two details of the forward path are specific to memmove and are not inherited from memcpy, because memcpy never sees an overlapping copy. First, wh64 claims a cache line for writing without filling it from memory, which makes the whole aligned 64-byte block UNPREDICTABLE unless every byte of it is stored. In a forward overlapping copy the hint runs ahead of the stores, so it can land on source bytes that have not been read yet and destroy them before the copy consumes them. At trip k the hint covers [dst_k+128, dst_k+192) and the unread source begins at dst_k + (src-dst), so the copy is safe exactly when src-dst >= 192. Below that the copy takes the prefetch loop instead, which substitutes LDS to F31 for the wh64 -- a prefetch with modify intent that only reads, and so cannot destroy the source -- and is correct at any overlap because it reads a whole block into registers before storing any of it. Note that bound tracks the wh64 distance: it is the hint's far edge, so it has to move whenever that moves. Second, the guard sends a too-small gap to the 64-byte prefetch loop rather than to the scalar tail loop, which would cost roughly 8x on exactly the copies it catches. A Python list.remove() moves its ob_item array with a gap of a single pointer, so it always takes this path. Verified on an EV68CB against the C memmove over every size class from 0 to 64KB, at 72 destination alignments for the non-overlapping case and across source-destination gaps from 8 bytes to 1KB for the overlapping one, with 512-byte guard regions checked for stray writes. --- sysdeps/alpha/alphaev6/memmove.S | 384 +++++++++++++++++++++++++++++++ 1 file changed, 384 insertions(+) create mode 100644 sysdeps/alpha/alphaev6/memmove.S diff --git ./sysdeps/alpha/alphaev6/memmove.S ./sysdeps/alpha/alphaev6/memmove.S new file mode 100644 index 0000000000..7c0cca1391 --- /dev/null +++ ./sysdeps/alpha/alphaev6/memmove.S @@ -0,0 +1,384 @@ +/* Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +/* + * EV6/EV7 memmove. Alpha previously used the generic C memmove. + * + * When the destination does not overlap the source from below -- i.e. + * (uintptr)dst - (uintptr)src >= n, which also covers dst < src -- the copy + * runs forward and is exactly the ev6 memcpy (same-alignment quadword fast + * path with the EV7 prefetch unrolled loop, and the rotating ldq_u path for + * mismatched alignment). Only the genuinely overlapping case (src < dst < + * src+n) copies backward; that is rare and not performance critical, so it + * uses a straightforward aligned-quadword-or-byte descending loop. + * + * Temp usage mirrors memcpy ($0 result, $1..$7,$22 scratch); the backward + * path additionally keeps the running pointers in $16/$17 and uses $1,$2. + */ + +#include + + .arch ev6 + .set noreorder + .set noat + +ENTRY(memmove) + .prologue 0 + + mov $16, $0 # E : return value = dest + ble $18, $nomoredata # U : nothing to do + subq $16, $17, $1 # E : dst - src + cmpult $1, $18, $2 # E : (dst-src) < n -> overlap from below + + bne $2, $backward # U : overlapping -> copy backward + xor $16, $17, $1 # E : forward: same alignment? + and $1, 7, $1 # E : ... mod 8 + bne $1, $misaligned # U : no -> rotating path + + and $16, 7, $1 # E : both 0mod8? + beq $1, $both_0mod8 # U : yes + nop # E : + + /* Same misalignment: copy bytes until both reach a 0mod8 boundary. */ +$head_align: + ldbu $1, 0($17) # L : grab a byte + subq $18, 1, $18 # E : count-- + addq $17, 1, $17 # E : src++ + stb $1, 0($16) # L : + addq $16, 1, $16 # E : dest++ + and $16, 7, $1 # E : 0mod8 yet? + ble $18, $nomoredata # U : done? + bne $1, $head_align # U : + +$both_0mod8: + cmple $18, 127, $1 # E : Can we unroll the loop? + bne $1, $no_unroll # U : + and $16, 63, $1 # E : get mod64 alignment + beq $1, $do_unroll # U : no single quads to fiddle + +$single_head_quad: + ldq $1, 0($17) # L : get 8 bytes + subq $18, 8, $18 # E : count -= 8 + addq $17, 8, $17 # E : src += 8 + nop # E : + + stq $1, 0($16) # L : store + addq $16, 8, $16 # E : dest += 8 + and $16, 63, $1 # E : get mod64 alignment + bne $1, $single_head_quad # U : still not fully aligned + +$do_unroll: + /* + * Same IMPLVER dispatch as memcpy: the 21264 (EV6/67/68) uses wh64 two + * lines ahead; the 21364 (EV7) takes a loop that software-prefetches both + * streams (read on the source, modify intent on the destination) six + * cache lines ahead. IMPLVER returns 2 for the 21264 family, 3 for EV7. + */ + cmple $18, 127, $1 # E : Can we go through the unrolled loop? + bne $1, $tail_quads # U : Nope + implver $2 # E : 2 = 21264, 3 = 21364 + subq $2, 3, $2 # E : EV7 -> 0 + + beq $2, $ev7_unroll # U : EV7 takes the prefetch-modify path + + /* + * wh64 claims a cache line for writing without filling it from memory, + * which makes the entire 64-byte block UNPREDICTABLE unless every byte + * of it is stored. In a forward overlapping copy (dst < src) the hint + * runs ahead of the stores and can therefore land on source bytes that + * have not been read yet, destroying them before the copy consumes + * them. + * + * At trip k the hint covers [dst_k+128, dst_k+192) and the unread + * source begins at src_k = dst_k + (src-dst), so the copy is safe + * exactly when src-dst >= 192. Note this bound tracks the wh64 + * distance: it is the hint's far edge, so it has to be updated whenever + * that distance changes. When dst >= src+n (no overlap at all) src-dst + * wraps to a huge unsigned value, cmpult is false, and the wh64 path is + * used as normal. + * + * Too close a gap falls through to $ev7_unroll rather than the scalar + * $tail_quads loop. That path copies 64 bytes a trip like this one and + * merely swaps wh64 for LDS to F31, a prefetch-with-modify-intent that + * only reads and so cannot destroy the source; ReadBlkMod is + * implemented on the 21264 (it is the evict-next variants that are + * 21364-only). It reads a whole block into registers before storing + * any of it, so it is correct at any overlap. Landing on the scalar + * loop instead would cost roughly 8x on exactly the small-gap copies + * this guard catches -- a Python list.remove() moves its ob_item array + * with a gap of one pointer, so it always takes this path. + */ + subq $17, $16, $1 # E : src - dst + cmpult $1, 192, $2 # E : (src-dst) < 192 ? + bne $2, $ev7_unroll # U : too close -- use the non-wh64 loop + + /* + * Initial wh64 target, two lines ahead of the first block. Only 128 + * bytes are known to remain here, so unlike the steady-state hint this + * one can overrun; fall back to the first block when it would. + */ + addq $16, 128, $7 # E : initial (+2 line) wh64 address + subq $18, 192, $2 # E : room for [$16+128, $16+192)? + cmovlt $2, $16, $7 # E : no -- hint the first block instead + +$unroll_body: + wh64 ($7) # L1 : 64 bytes at ($7) about to be overwritten + ldq $6, 0($17) # L0 : bytes 0..7 + nop # E : + nop # E : + + ldq $4, 8($17) # L : bytes 8..15 + ldq $5, 16($17) # L : bytes 16..23 + addq $7, 64, $7 # E : Update next wh64 address + nop # E : + + ldq $3, 24($17) # L : bytes 24..31 + addq $16, 64, $23 # E : fallback wh64 address (see below) + nop # E : + nop # E : + + addq $17, 32, $17 # E : src += 32 bytes + stq $6, 0($16) # L : bytes 0..7 + nop # E : + nop # E : + + stq $4, 8($16) # L : bytes 8..15 + stq $5, 16($16) # L : bytes 16..23 + lda $2, -256($18) # E : room for the +2 line hint next trip? + # it covers [$16+128, $16+192) there, + # so that trip needs 192 and this one + # needs 256 + nop # E : + + stq $3, 24($16) # L : bytes 24..31 + addq $16, 32, $16 # E : dest += 32 bytes + nop # E : + nop # E : + + ldq $6, 0($17) # L : bytes 0..7 + ldq $4, 8($17) # L : bytes 8..15 + nop # E : + nop # E : + + ldq $5, 16($17) # L : bytes 16..23 + ldq $3, 24($17) # L : bytes 24..31 + addq $16, 32, $16 # E : dest += 32 + subq $18, 64, $18 # E : count -= 64 + + addq $17, 32, $17 # E : src += 32 + stq $6, -32($16) # L : bytes 0..7 + stq $4, -24($16) # L : bytes 8..15 + cmple $18, 63, $1 # E : At least one more trip? + + /* + * $7 is deliberately finalized late, in the group with the loop branch + * rather than alongside the loads above, and the fallback address lives + * in $23 so that it survives the cmple that reuses $1 for the loop + * condition. This is not cosmetic: $7 is what the next trip's wh64 + * issues against, and letting that hint issue while the previous trip's + * stores are still in flight provokes Mbox replay traps in the store + * stream. Measured on an EV68CB over a cold 256KB copy, moving the + * cmov down cuts replays (perf r4) from about 1000 per call to about + * 750 and is worth 1.4-1.8% from 2KB upwards, for no extra instructions. + * Moving it back up, or reusing $1 so that it has to move back up, + * gives up that gain. + */ + stq $5, -16($16) # L : bytes 16..23 + stq $3, -8($16) # L : bytes 24..31 + cmovlt $2, $23, $7 # E : fallback if < 2 more trips (late; see + # the note above -- do not hoist this) + beq $1, $unroll_body + br $31, $tail_quads # U : 1..15 trailing quads/bytes + + .align 4 +$ev7_unroll: + lds $f31, 6*64($16) # L : dest prefetch w/ modify intent + ldl $31, 6*64($17) # L : src prefetch (read) + ldq $1, 0($17) # L : bytes 0..7 + ldq $2, 8($17) # L : bytes 8..15 + ldq $3, 16($17) # L : bytes 16..23 + ldq $4, 24($17) # L : bytes 24..31 + ldq $5, 32($17) # L : bytes 32..39 + ldq $6, 40($17) # L : bytes 40..47 + ldq $7, 48($17) # L : bytes 48..55 + ldq $22, 56($17) # L : bytes 56..63 + + stq $1, 0($16) # L : + stq $2, 8($16) # L : + stq $3, 16($16) # L : + stq $4, 24($16) # L : + stq $5, 32($16) # L : + stq $6, 40($16) # L : + stq $7, 48($16) # L : + stq $22, 56($16) # L : + + subq $18, 64, $18 # E : count -= 64 + addq $17, 64, $17 # E : src += 64 + addq $16, 64, $16 # E : dest += 64 + cmple $18, 63, $1 # E : less than one more full trip? + beq $1, $ev7_unroll # U : keep going + +$tail_quads: +$no_unroll: + .align 4 + subq $18, 8, $18 # E : At least a quad left? + blt $18, $less_than_8 # U : Nope + nop # E : + nop # E : + +$move_a_quad: + ldq $1, 0($17) # L : fetch 8 + subq $18, 8, $18 # E : count -= 8 + addq $17, 8, $17 # E : src += 8 + nop # E : + + stq $1, 0($16) # L : store 8 + addq $16, 8, $16 # E : dest += 8 + bge $18, $move_a_quad # U : + nop # E : + +$less_than_8: + .align 4 + addq $18, 8, $18 # E : add back for trailing bytes + ble $18, $nomoredata # U : All-done + nop # E : + nop # E : + +$tail_bytes: + subq $18, 1, $18 # E : count-- + ldbu $1, 0($17) # L : fetch a byte + addq $17, 1, $17 # E : src++ + nop # E : + + stb $1, 0($16) # L : store a byte + addq $16, 1, $16 # E : dest++ + bgt $18, $tail_bytes # U : more to be done? + nop # E : + + ret $31, ($26), 1 # L0 : + nop # E : + nop # E : + nop # E : + +$misaligned: + mov $0, $4 # E : dest temp + and $0, 7, $1 # E : dest alignment mod8 + beq $1, $dest_0mod8 # U : life doesn't totally suck + nop + +$aligndest: + ble $18, $nomoredata # U : + ldbu $1, 0($17) # L : fetch a byte + subq $18, 1, $18 # E : count-- + addq $17, 1, $17 # E : src++ + + stb $1, 0($4) # L : store it + addq $4, 1, $4 # E : dest++ + and $4, 7, $1 # E : dest 0mod8 yet? + bne $1, $aligndest # U : go until we are aligned. + + /* Source has unknown alignment, but dest is known to be 0mod8 */ +$dest_0mod8: + subq $18, 8, $18 # E : At least a quad left? + blt $18, $misalign_tail # U : Nope + ldq_u $3, 0($17) # L : seed (rotating load) of 8 bytes + nop # E : + +$mis_quad: + ldq_u $16, 8($17) # L : Fetch next 8 + extql $3, $17, $3 # U : masking + extqh $16, $17, $1 # U : masking + bis $3, $1, $1 # E : merged bytes to store + + subq $18, 8, $18 # E : count -= 8 + addq $17, 8, $17 # E : src += 8 + stq $1, 0($4) # L : store 8 (aligned) + mov $16, $3 # E : "rotate" source data + + addq $4, 8, $4 # E : dest += 8 + bge $18, $mis_quad # U : More quads to move + nop + nop + +$misalign_tail: + addq $18, 8, $18 # E : account for tail stuff + ble $18, $nomoredata # U : + nop + nop + +$misalign_byte: + ldbu $1, 0($17) # L : fetch 1 + subq $18, 1, $18 # E : count-- + addq $17, 1, $17 # E : src++ + nop # E : + + stb $1, 0($4) # L : store + addq $4, 1, $4 # E : dest++ + bgt $18, $misalign_byte # U : more to go? + nop + +$nomoredata: + ret $31, ($26), 1 # L0 : + + /* + * Overlapping move (src < dst < src+n): copy from the high end down. + * Aligned quadwords when the two ends share alignment, else bytes. + * $0 already holds the original dest for the return value. + */ +$backward: + addq $16, $18, $16 # E : dest end (one past last byte) + addq $17, $18, $17 # E : src end + xor $16, $17, $1 # E : ends share alignment? + and $1, 7, $1 # E : + bne $1, $back_bytes # U : no -> byte copy + +$back_head: + and $16, 7, $1 # E : dest end 0mod8 yet? + beq $1, $back_quads # U : + lda $16, -1($16) # E : dest-- + lda $17, -1($17) # E : src-- + ldbu $1, 0($17) # L : + stb $1, 0($16) # L : + subq $18, 1, $18 # E : count-- + beq $18, $nomoredata # U : + br $31, $back_head # U : + +$back_quads: + subq $18, 8, $2 # E : (count - 8) + blt $2, $back_bytes # U : < 8 left -> bytes +$back_q: + lda $16, -8($16) # E : dest -= 8 + lda $17, -8($17) # E : src -= 8 + ldq $1, 0($17) # L : + stq $1, 0($16) # L : + mov $2, $18 # E : commit count -= 8 + subq $18, 8, $2 # E : (count - 8) + bge $2, $back_q # U : + +$back_bytes: + beq $18, $nomoredata # U : +$back_b: + lda $16, -1($16) # E : + lda $17, -1($17) # E : + ldbu $1, 0($17) # L : + stb $1, 0($16) # L : + subq $18, 1, $18 # E : + bne $18, $back_b # U : + ret $31, ($26), 1 # L0 : + +END(memmove) +libc_hidden_builtin_def (memmove) From patchwork Wed Aug 12 01:19:48 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141157 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C6BF14B99F54 for ; Wed, 12 Aug 2026 01:25:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org C6BF14B99F54 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=Su4gqcNK X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb134.google.com (mail-yx1-xb134.google.com [IPv6:2607:f8b0:4864:20::b134]) by sourceware.org (Postfix) with ESMTPS id 461B14BA2E03 for ; Wed, 12 Aug 2026 01:20:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 461B14BA2E03 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 461B14BA2E03 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b134 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497623; cv=none; b=V4lDt6PvgNwzmXDNxNWk85iisNbwltMG2wAtvHfhfKdBSWjeoKJaJM1k+ObIa+tM5k5j835R24hEKGBxTmViYinFlY9MC73BYYkmGehputDp8xLO+sO/A2z1zT/M6MWpFmEaAEiEYdnCarEYwytecmB2AO1fDBzkLz53tb9NGJk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497623; c=relaxed/simple; bh=URnqzF/ETJ67Ky9NPEA7RcHBl1xpSZCXAw7ulbnkIhA=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=L7JX1iMEqgfDEBlJ5rhX5tZRozuLiCXsnwk00MVkezt/aupj3HVd7CFWzW7RVNo/llRuu6PgzdIPmbWayUCHBTfPZW7P9tf0KWcoIE/D7bnJlhYr0j+tHZTBAQkZ3aOFXZKlzkvqalhKw10MbLmSlt6aVBxinOxMFA02wx/0DE4= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=Su4gqcNK DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 461B14BA2E03 Received: by mail-yx1-xb134.google.com with SMTP id 956f58d0204a3-6680ae2031fso539750d50.3 for ; Tue, 11 Aug 2026 18:20:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497622; x=1787102422; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=HteST/pVr5MvYnnhNEqW7VI5cjtl5Df5lTQfJQ+De4Y=; b=Su4gqcNKfvz+rvtp5NlH5IgSWrCzW2k/SRsXxKstk6Ar/si8JGtDS/2DlhbxFJRLpK ncUc/g4Flv5v/U9PN+zdgZlTPg6+3u5yBWXwXzubBNwnp2CaNDrZ4s8hBNaonT1fJecg Ttzd/xtIEQnm2RNPpCv4kGGx38+QYMl6S9uS21MqSMaNW8jqHpvDnibcGQiDdBy+q31w 0YZi13UB3T/WAemHinhLMN6udhGBXmn793llOrjuPJ4i9sVMo/FjNHhCoE80fFdUeVdT 3VOt+3+GfDqrzriR/9k4fck0FtGDh/nxK3rcaZTZIEnVNH4TY7lPB8NOO2S+3wVDrwZj kzVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497622; x=1787102422; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=HteST/pVr5MvYnnhNEqW7VI5cjtl5Df5lTQfJQ+De4Y=; b=L7aoJ4V0XcsxazALrXWgq8ygL9ZGKv4JPwUt8t7YMM9ptT0TOcQUtE6xVpD5I0odsY dK2Yda/uGd3H1qDt/4x4ZQKRF780O68gdA0TaSQGgPPGLY3JtDEpzkYgLhfQrS9tUARE DTTma5p1G333lNtyetXXpu9AENTuPW9rYim9QcXCMFdJEY4oQsbsULx5IuGLThGfm/ec p5G+GmS41VfmAHnNvUNY4mbWFerlAp3AgcewzGj5A95g2PImZ+9GDXd4cLRs0DTXGt5E tXr4K9x8PniHiRj2i+1qkeENxlh5Ou8uKHFcoSXyZlgcUi7QzSv6WrwnmxoODXff3oV1 OHrg== X-Gm-Message-State: AOJu0YxZi5bzychuwmeyJiqT4vQ3/DaNFnj38upicB7oxnq8FshL01sm nWMKuQf/1hwttRrCSpW609TE0kkeDOVqTPjhEqU6P5YDfOjHG96kdqvHcV4K7e0SSEw= X-Gm-Gg: AR+sD13Qxrl6kDWu2au+BVDbWA5K9Qf2vkoyx6tVvpbcXWdJFObXj757LrnoN5oaGtO oxpxb76IPF2FK9Y14FPjrY3Xgg7nzQ5z1QIYI5Ym9pQOZyqoSAaCt0P/fm4kGK3cdO5UtWcfDOL WZklaqCweOIKTFVyEknjI1StK+l/x0+GpQ8SLYFBuaEZYCpciYAiixp9XDxdAbcAea80wLd/ceT kr3QUXelKajyJntYexaR56TD3OcEJnV3Q/rVbs5ynwgc72W1q1so0iIgBB+VfqSWHTmWz9k/1Jr qXY7EU5vtvAsYqn5tHSB9Fcczdfih4VGr5jJso/WZxti3jvMyatkWqXQEDpcAZaRMfS996P50Tv eATYuqj2KSy63c89PuaAEHFKmKFDZluT8jqpP/ulzuidiNSbWjXLWm308q7nQNvtL82nBaMiMrR eU4HDajO5lrNqLpwAnsRi5hGPVRfWVn40kqB5tBBPq+vF7r7uSAbeo7tJKPuQ= X-Received: by 2002:a53:d088:0:b0:668:9e51:5239 with SMTP id 956f58d0204a3-66b346fb2famr531017d50.44.1786497622621; Tue, 11 Aug 2026 18:20:22 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66b1ef5f935sm2087164d50.1.2026.08.11.18.20.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:21 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 11/20] alpha: add assembly mempcpy sharing the ev6 memcpy Date: Tue, 11 Aug 2026 21:19:48 -0400 Message-ID: <169c689ace6b5b034c5517a34d1b68dd517d867a.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Alpha had no assembly mempcpy and used the generic C. mempcpy differs from memcpy only in its return value (dest + n rather than dest), so build it from the ev6 memcpy with USE_AS_MEMPCPY, following the established convention used by other architectures. memcpy.S gains two small, behaviour-preserving changes so it can be reused: the entry computes the return value through USE_AS_MEMPCPY, and the misaligned path now derives its working dest pointer and alignment from the (still original) $16 instead of $0, freeing $0 to carry dest + n for mempcpy. The plain memcpy path is unchanged ($0 == $16 on entry). --- sysdeps/alpha/alphaev6/memcpy.S | 13 ++++++++++--- sysdeps/alpha/alphaev6/mempcpy.S | 27 +++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 3 deletions(-) create mode 100644 sysdeps/alpha/alphaev6/mempcpy.S diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S index d5b72d4961..d11138420d 100644 --- ./sysdeps/alpha/alphaev6/memcpy.S +++ ./sysdeps/alpha/alphaev6/memcpy.S @@ -40,7 +40,11 @@ ENTRY(memcpy) .prologue 0 - mov $16, $0 # E : copy dest to return +#ifdef USE_AS_MEMPCPY + addq $16, $18, $0 # E : mempcpy returns dest + n +#else + mov $16, $0 # E : memcpy returns dest +#endif ble $18, $nomoredata # U : done with the copy? xor $16, $17, $1 # E : are source and dest alignments the same? and $1, 7, $1 # E : are they the same mod 8? @@ -289,8 +293,9 @@ $tail_bytes: nop # E : $misaligned: - mov $0, $4 # E : dest temp - and $0, 7, $1 # E : dest alignment mod8 + mov $16, $4 # E : dest temp (orig dest still in $16 here; + # $0 may hold dest+n for mempcpy) + and $16, 7, $1 # E : dest alignment mod8 beq $1, $dest_0mod8 # U : life doesn't totally suck nop @@ -367,4 +372,6 @@ $nomoredata: nop # E : END(memcpy) +#ifndef USE_AS_MEMPCPY libc_hidden_builtin_def (memcpy) +#endif diff --git ./sysdeps/alpha/alphaev6/mempcpy.S ./sysdeps/alpha/alphaev6/mempcpy.S new file mode 100644 index 0000000000..95385e4111 --- /dev/null +++ ./sysdeps/alpha/alphaev6/mempcpy.S @@ -0,0 +1,27 @@ +/* Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +/* mempcpy is the ev6 memcpy with USE_AS_MEMPCPY, which only changes the + return value (dest + n instead of dest) and the exported symbol. */ + +#define USE_AS_MEMPCPY +#define memcpy __mempcpy +#include + +libc_hidden_def (__mempcpy) +weak_alias (__mempcpy, mempcpy) +libc_hidden_builtin_def (mempcpy) From patchwork Wed Aug 12 01:19:49 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141158 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3D3364B99F43 for ; Wed, 12 Aug 2026 01:25:39 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 3D3364B99F43 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=EAwqERar X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb134.google.com (mail-yx1-xb134.google.com [IPv6:2607:f8b0:4864:20::b134]) by sourceware.org (Postfix) with ESMTPS id DB5B64BB1C1A for ; Wed, 12 Aug 2026 01:20:26 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org DB5B64BB1C1A Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org DB5B64BB1C1A Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b134 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497627; cv=none; b=q6sVY/AVpf9USMQfXKv36vonbdR8sCqBe0vNyXl//vD/T2YCNM0A2T6/f4AbO0dhvAk6GVTHs4YcdnTj3e6gRS3fv0XNNS2vGhiP0hw0/TWqQHBxdb1wKttEzON/tTlxTIWsw998FHxP5kYLH2zsUDyp0yuWPM2fCokXvRlkgA0= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497627; c=relaxed/simple; bh=+cjbLSKXMgv4Y22YqVTkbSawGErmKOgQMFTpIZE2Dns=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=sINMdHZuA26YlW5mPeABg2XE+3bh7Zg/MuAemdC6/wig86lY6kVpOgeb7PfGnC5i//k133au33ajT2y0ILZK8sprF/QooMIc/6clfzu1w598xO9wb1xurqHZaHNA9ek5xz8f/2nBdy5ue9U3YTYBN2C9VsFwe2FI6MsrBkmCJAA= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=EAwqERar DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org DB5B64BB1C1A Received: by mail-yx1-xb134.google.com with SMTP id 956f58d0204a3-66ac4d40da4so738191d50.2 for ; Tue, 11 Aug 2026 18:20:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497626; x=1787102426; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=SeB19ywXTn/OYry/IG0hPPMdec0V0jr8ZkQpbJzbq3w=; b=EAwqERareCBTYbeMua9ZrhZCD8i/fiCFE5OH2dVZGdy8D0agv3m7+qay0Jd+kNhw9s VgFXHdPmoxdC06fQu1iYDPeLCgC9lYqW8n81/l/MNh5ZKu5uUAZ60A1CAGXIP6N1A2zg asPG00LLiazcuqbNkA2mZ9VvGE3vAYlIdmepqYLLL4r6dF/Ny0PowIKO3uzAXPAGj89y /YD6b52EDEXKmVXa0D4tuMlaaqw2Keuf8N4JJ4r3xkQs+4HVt47bNBrrn/FpeVBdWWzP 6ffS+JNMGB4txKGfGuimj67/YyF3yM+iGeEt4uMYMtGigTIZynn/93WSonSOFTshDNL3 PjHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497626; x=1787102426; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=SeB19ywXTn/OYry/IG0hPPMdec0V0jr8ZkQpbJzbq3w=; b=L9UWLwIsogiaDsMIXYvF95NIpfNIUkEjip+gncQyavGLxbiaDXsUCawvIkT+PPzqh5 JeW2b6a/P/UD/Ku1TEnWIhVP3hNqWIhJpblK7QFSF7laDZ+Dlyd/loU2nGNnB7q9/ToR rv4PZzMo+OigzbkzauqjHrUPfJIeEL5HakENlFTIy+6A8V053DRY03BEGDD0xPXyjd6w CQa2hGoIx3Q/iEX+B6xIMXvvbnh7Dq2PzIBKTpyhHi0r44jJZ8cwpr7eA4C/sDjll7ZS MMSeKM+1JoER0I9+gPtuhD1SeXCwyWiYWD3cxN0RJoxAGy7xekvpQBUcSmXsR0MQF1mC IeUQ== X-Gm-Message-State: AOJu0Yy1ka2G65SCfmQehZrSWHVdkY3AgeUFdZQ1TZDjcBlr52+udd6s yUrpsTIuSFo7mfYlnSY2jXdUDNLXe3ZZD/mUjmAYW/Ax3pEkvCdxBizEtQj7rvP8jhQ= X-Gm-Gg: AR+sD11DgVez8ZC1sSSvdjaENKDrQd4BTY7MLhLSn+6apBSiP7tOilb1AC6hScCtUwf jOY5Idic6c7Tx3ALkiH/pTQSOeqPYYj2+5GJHmC5b7+49t6v0pXzH4enia/n3f0qt9DtydkVeqP tfxSvHV14lIfjhKX2UGVO6Vc5HGs5KpKGOO3lAg+ic4lF1XOPKYdO6IFkpyF5G7ayzviYm53y2B nq39lW8dXP9GUGix+SgM0xIGr4axgGdT4io3hYlVL3vDvTpG0ygYO+u4xwMFWvTl2GeBpaiFdNM G8XzALLrL9+cVyVhbr/96gEvDKyezplm8k4n0Vg7+MPDaChu3JIW6FOsH8+z6Cgi52yI7zYY0Ds YrIUP3uEHRWqJelEmGd3jE/+vhBen//tqa/swnl8SkBYNRWFYSMTyjvbMBQ1oYUkgcGrbr1FoEQ aPQfcuMoH4jU7I9d1gL67zqGEZIJjOy44hI9/XmTQIpR4KTDFUQEujhcaKcVk= X-Received: by 2002:a05:690e:4394:b0:667:9f79:35cb with SMTP id 956f58d0204a3-66b343447d7mr445048d50.53.1786497626066; Tue, 11 Aug 2026 18:20:26 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66b1ef94d59sm2081972d50.6.2026.08.11.18.20.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:24 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 12/20] alpha: finish the ev6 memcpy and memmove tails without a byte loop Date: Tue, 11 Aug 2026 21:19:49 -0400 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SCC_10_SHORT_WORD_LINES, SCC_5_SHORT_WORD_LINES, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Once the quadword loop runs out there are 1 to 7 bytes left, and they were copied one at a time by a loop costing about eight issue slots a byte. For copies whose length is not a multiple of eight that dominates: on an EV68CB a 15-byte copy cost 40.7 cycles against 22.1 for a 16-byte one, so asking for one byte less than a whole number of quadwords made the call 85% dearer. Every path into $tail_quads leaves the source and destination both 0mod8, and the quadword loop preserves it, so the remainder can be done as at most one longword, one word and one byte with no unaligned access -- 0mod8 is 0mod4, and after the longword 4mod8 is still 2-aligned. That replaces up to seven loop trips with three tests and at most three load/store pairs. Measured on an EV68CB, hot, cycles per call: size before after 15 40.7 21.1 1.93x 23 41.9 24.1 1.74x 31 47.3 28.8 1.64x 7 34.7 25.6 1.36x 63 54.4 40.2 1.35x 95 57.3 42.3 1.36x 127 82.5 64.9 1.27x The counters confirm this is simply instruction count: at 15 bytes the call retires 69.2 instructions before and 43.1 after, at an IPC just over two with no Bcache misses and no replay traps. Short copies are issue-bound, unlike the long ones this file spends most of its comments on. Lengths leaving exactly one trailing byte pay about 1.5 cycles -- two extra instructions -- because the three tests cost more than a single trip of the old loop. A length of 9 goes from 18.3 to 19.8 cycles. That is the whole downside and it is worth it for a factor approaching two elsewhere. This is deliberately not memset's trailing masked store, which reads and rewrites the destination quadword: that touches bytes past the end of the destination, and memcpy has never written memory the caller did not ask for. Verified on an EV68CB against the C memcpy and memmove for every length from 0 to 600 at 72 destination alignments, and for memmove across source- destination gaps from 8 bytes to 1KB, with 512-byte guard regions checked for stray writes. --- sysdeps/alpha/alphaev6/memcpy.S | 45 +++++++++++++++++++++++++++----- sysdeps/alpha/alphaev6/memmove.S | 44 ++++++++++++++++++++++++++----- 2 files changed, 76 insertions(+), 13 deletions(-) diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S index d11138420d..ee251e8b8f 100644 --- ./sysdeps/alpha/alphaev6/memcpy.S +++ ./sysdeps/alpha/alphaev6/memcpy.S @@ -274,16 +274,47 @@ $less_than_8: nop # E : nop # E : - /* Trailing bytes */ + /* + * 1..7 trailing bytes, with source and destination both 0mod8 -- every + * path into $tail_quads guarantees that, and the quad loop above keeps + * it. Do them as at most one longword, one word and one byte rather + * than looping a byte at a time: the loop spends about eight issue + * slots per byte, which on an EV68CB made a 15-byte copy 85% dearer + * than a 16-byte one. Each step leaves the pointers aligned for the + * next -- 0mod8 is 0mod4, and 4mod8 is still 2-aligned -- so none of + * these accesses is unaligned. + * + * Deliberately not a single wider store masked down to size, the way + * memset finishes: that reads and rewrites bytes beyond the end of the + * destination, and memcpy has never touched memory the caller did not + * ask it to write. + */ $tail_bytes: - subq $18, 1, $18 # E : count-- - ldbu $1, 0($17) # L : fetch a byte - addq $17, 1, $17 # E : src++ + and $18, 4, $1 # E : longword to do? + and $18, 2, $3 # E : word to do? + beq $1, $tail_word # U : nop # E : - stb $1, 0($16) # L : store a byte - addq $16, 1, $16 # E : dest++ - bgt $18, $tail_bytes # U : more to be done? + ldl $2, 0($17) # L : 4 bytes + stl $2, 0($16) # L : + addq $17, 4, $17 # E : src += 4 + addq $16, 4, $16 # E : dest += 4 + +$tail_word: + and $18, 1, $1 # E : odd byte to do? + beq $3, $tail_byte # U : + ldwu $2, 0($17) # L : 2 bytes + stw $2, 0($16) # L : + + addq $17, 2, $17 # E : src += 2 + addq $16, 2, $16 # E : dest += 2 + nop # E : + nop # E : + +$tail_byte: + beq $1, $nomoredata # U : + ldbu $2, 0($17) # L : last byte + stb $2, 0($16) # L : nop # E : /* branching to exit takes 3 extra cycles, so replicate exit here */ diff --git ./sysdeps/alpha/alphaev6/memmove.S ./sysdeps/alpha/alphaev6/memmove.S index 7c0cca1391..d587a90dd9 100644 --- ./sysdeps/alpha/alphaev6/memmove.S +++ ./sysdeps/alpha/alphaev6/memmove.S @@ -258,15 +258,47 @@ $less_than_8: nop # E : nop # E : + /* + * 1..7 trailing bytes, with source and destination both 0mod8 -- every + * path into $tail_quads guarantees that, and the quad loop above keeps + * it. Do them as at most one longword, one word and one byte rather + * than looping a byte at a time: the loop spends about eight issue + * slots per byte, which on an EV68CB made a 15-byte copy 85% dearer + * than a 16-byte one. Each step leaves the pointers aligned for the + * next -- 0mod8 is 0mod4, and 4mod8 is still 2-aligned -- so none of + * these accesses is unaligned. + * + * Deliberately not a single wider store masked down to size, the way + * memset finishes: that reads and rewrites bytes beyond the end of the + * destination, and memcpy has never touched memory the caller did not + * ask it to write. + */ $tail_bytes: - subq $18, 1, $18 # E : count-- - ldbu $1, 0($17) # L : fetch a byte - addq $17, 1, $17 # E : src++ + and $18, 4, $1 # E : longword to do? + and $18, 2, $3 # E : word to do? + beq $1, $tail_word # U : nop # E : - stb $1, 0($16) # L : store a byte - addq $16, 1, $16 # E : dest++ - bgt $18, $tail_bytes # U : more to be done? + ldl $2, 0($17) # L : 4 bytes + stl $2, 0($16) # L : + addq $17, 4, $17 # E : src += 4 + addq $16, 4, $16 # E : dest += 4 + +$tail_word: + and $18, 1, $1 # E : odd byte to do? + beq $3, $tail_byte # U : + ldwu $2, 0($17) # L : 2 bytes + stw $2, 0($16) # L : + + addq $17, 2, $17 # E : src += 2 + addq $16, 2, $16 # E : dest += 2 + nop # E : + nop # E : + +$tail_byte: + beq $1, $nomoredata # U : + ldbu $2, 0($17) # L : last byte + stb $2, 0($16) # L : nop # E : ret $31, ($26), 1 # L0 : From patchwork Wed Aug 12 01:19:50 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141156 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 20CF84BAE7E4 for ; Wed, 12 Aug 2026 01:25:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 20CF84BAE7E4 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=b1mQaeoV X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb12b.google.com (mail-yx1-xb12b.google.com [IPv6:2607:f8b0:4864:20::b12b]) by sourceware.org (Postfix) with ESMTPS id 6FC054B99F58 for ; Wed, 12 Aug 2026 01:20:28 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6FC054B99F58 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 6FC054B99F58 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b12b ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497628; cv=none; b=MtKnZ4S2OggDbK2781Nz4H6VypZ2tFKtAjrHMkJLwtG2C0M6GlCs5is0aleZflYHs6jEwVwYmV9p+NdbD/P/uP7/HiAtWvj2/pnJ4qrHa9ctVD6231Phgvu6t14lTWJwIqEpSXJZSAwKKGE5XljRoLRBiquNHbjJm0AWCMXeTBs= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497628; c=relaxed/simple; bh=P6q2H2QtF70QboKyshOk1NZEfoqrdA3agKyE+whs2GY=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=ZuHI4pwR++T3EiWbiqRPrqUJOdwQt9DommDKQi6DKubU4ZNUIYaa8wskdCjKfQjcScCKxU5oxEWblFbJxnh7QRyggEEq7JDwvbu4YVlJctwL/q1Jp/faiUDwdVCxq1m3iobS3TaLIaAGLjJiY5Agk9Hz2XmOLiI1bDBQeiuOzrs= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=b1mQaeoV DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 6FC054B99F58 Received: by mail-yx1-xb12b.google.com with SMTP id 956f58d0204a3-66843304cbaso535100d50.2 for ; Tue, 11 Aug 2026 18:20:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497628; x=1787102428; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=tO8kuzrLF1vu0TivU0X7qAKV3jyN+P/53M2Vx7cHFBk=; b=b1mQaeoVIa8Vh9QJkgAriXqy2+QPYhpd83k4cDYha2hBYmd/kt+VYzloomhWeLj07c RgaghJk7yIz8jouhEytYgN4BMbkVM172FVocUzY5hwE3Eur+jPDI7W6UHsNR/nOEaXUN qXXMYzgwLmX8fYnsTtERHzzu3SuoXuvVt8HDiIpcTjEtBPaF0kIebF4Qiu7lOU0xY7TN V2/J4oPBIIaAzB8CAUhkm78BPERz2aXscNQd7GpSlFeK9onsLcj7Fe/2xLH6RgZ8W4Po /NdlpSXJrIr6vaFhvWx5hb7DdZlk1CXsDU/2OlF+vI4mi9qQTppjD8qatCkquIM4lGv3 tBPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497628; x=1787102428; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=tO8kuzrLF1vu0TivU0X7qAKV3jyN+P/53M2Vx7cHFBk=; b=kuP+jLzuW/KTXQFKVWWD85y7cvtLWTIoPsrHkuqAvbNcQMgPuFTXesGjlvYlzdthZZ 1gqjZZG2kU/6Anuj2F7lRkC0wt8bbElc5ualiM0goeToFDVe84eIfIAFQpx6ynBSSkk6 rdeYrPI7X1G+YmWma0SNYaXOLQ+oNJeDvZj1DtjF97tvOeaXxDQuTjGnTh24Dlao0Z3e X239HhhS6oJT1PgdFm4/GzUsW5jOo4sbr+mwnp2Ohn6ET+hctGTKpKzgbnn9MVPSPlxH X3nszShTRl6xYM1K622n6ouKAW40oTAC1dTaR2U1q/t6zIVwF6Ms3RnIxjkIkX83BGhS qxBg== X-Gm-Message-State: AOJu0Ywep5sQaToncbu4PwEFrJz5rqKvXEhSFswHI43e2TxGIB1v99Uq 9t+bq/KoQjE88tpjAZQIWX2/l9Xx1nGzEjpV8XyAL3mfShY0M1yYpXtKKXkOADGryHU= X-Gm-Gg: AR+sD12rxw8UglHK+tzSGhuhOJgoINUfHkFjx3rIvx+E+yZKJmkb1Pi8QHvDS8P9QBB WD2aYFSgo5wmc1Ai4FSp8ttHENLV1VBaK3P411IlTZstJTbrcf+HYUF8mwVUaUTe8S5zVpkBBvu 9zGYvGUUGv07JSpAszPDcPLABPCBjhcvZ2FQvomKOvjZjCiiBTeSG+lqeyDiVIuSCZV2DT1WXnn n+qQEb1BPRRajQ1rUnhwP86OYk16EaqdGNDISf8f09D9fJTnRrbBPuOkTHDLrw4edFdOQK6PovA C1nEdlg1BicVRZtUOWaZ1s6ZEaN0BgvHPTa4wGrOJjUiiy3IItJfkoOaHa+uG9my3Se0M6cMuEs EmmdJpTWVOdcrCxt90HAze7UWdJHmKUQlofNNbD1ZGLPF7nN7R/9u9M7SWdgZ3Il+qhrfapi23F NpTsZNnYWIILLHN7UGh7joTnT2r+V0PriuCXr6Te2t4Pr4rBKKRGBESMsvL24= X-Received: by 2002:a53:d058:0:20b0:668:1414:e9bd with SMTP id 956f58d0204a3-66b3434fed0mr579533d50.38.1786497627617; Tue, 11 Aug 2026 18:20:27 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66b1f2c1245sm2078816d50.21.2026.08.11.18.20.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:26 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 13/20] alpha: copy four quads a trip below the ev6 unrolled loop Date: Tue, 11 Aug 2026 21:19:50 -0400 Message-ID: <52ad635ef30caef7328b9c6cf81638260ee9e804.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Copies shorter than 128 bytes never reach the unrolled loop, and the remainder of the ones that do is up to 63 bytes; both were handled a quadword at a time by a loop spending eight issue slots per eight bytes. That is the wrong shape for the work. Short copies miss nothing, replay nothing and retire better than two instructions a cycle, so unlike the long copies this file spends most of its comments on they are issue-bound, and what they want is fewer instructions per byte. Take four quads a trip while at least 32 bytes remain, then fall into the existing one-quad loop. Measured on an EV68CB, hot, cycles per call: size before after 112 64.7 36.2 1.79x 120 65.9 37.7 1.75x 127 69.4 42.2 1.64x 160 80.0 52.8 1.52x 96 40.3 31.7 1.27x 64 32.2 27.1 1.19x 128 52.8 47.8 1.11x The loop body sits out of line, past every return path, and that placement is not incidental. With it inline just above $move_a_quad, copies of 1KB and 2KB lost about 5% -- 228 cycles becoming 237 at 1KB -- even though a copy that size never executes a single one of these instructions. Growing the straight-line distance between the unrolled loop and the tail is enough to do that on its own. Out of line, 1KB gains 4% and 2KB is level, and the only price is a few percent of the gain between 32 and 96 bytes. Copies below 32 bytes pay one cycle for the test that skips the new loop -- about 5% at 8 to 24 bytes, where the call is only 17 to 22 cycles to begin with. The instruction count at the head is unchanged, since the test takes the place of two nops; the cost is the branch itself. Note that GCC expands small constant-size copies inline, so the lengths that lose here are the ones least likely to reach this code at all. Sizes from 256 to 512 still measure 2-3% down for the same layout reason, which no placement tried made go away entirely. Verified on an EV68CB against the C memcpy and memmove for every length from 0 to 600 at 72 destination alignments, and for memmove across source- destination gaps from 8 bytes to 1KB, with 512-byte guard regions checked for stray writes. --- sysdeps/alpha/alphaev6/memcpy.S | 45 ++++++++++++++++++++++++++++++-- sysdeps/alpha/alphaev6/memmove.S | 45 ++++++++++++++++++++++++++++++-- 2 files changed, 86 insertions(+), 4 deletions(-) diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S index ee251e8b8f..d86c00b9be 100644 --- ./sysdeps/alpha/alphaev6/memcpy.S +++ ./sysdeps/alpha/alphaev6/memcpy.S @@ -253,8 +253,8 @@ $no_unroll: .align 4 subq $18, 8, $18 # E : At least a quad left? blt $18, $less_than_8 # U : Nope - nop # E : - nop # E : + subq $18, 24, $1 # E : four whole quads left? + bge $1, $move_four_quads # U : yes -- out of line, below $move_a_quad: ldq $1, 0($17) # L : fetch 8 @@ -402,6 +402,47 @@ $nomoredata: nop # E : nop # E : + + /* + * Four quads a trip, for the 32..127 byte copies that never reach the + * unrolled loop and for the remainder of the ones that do. Short + * copies are issue-bound -- they miss nothing and replay nothing, and + * retire better than two instructions a cycle -- so what they want is + * simply fewer instructions per byte than the one-quad loop above. + * + * This lives out of line, past every return path, on purpose. Placed + * inline just above $move_a_quad it costs long copies about 5% at 1KB + * and 2KB, even though a copy that size never executes a single one of + * these instructions, purely by pushing the unrolled loop and the tail + * further apart. Moving it down here recovers that and gives up only + * a few percent of the gain between 32 and 96 bytes. + * + * $18 counts bytes remaining less eight, so a full trip needs $18 >= 24. + * Falling out can leave $18 negative and $move_a_quad is a do-while, so + * the count is rechecked before going back to it. + */ + .align 4 +$move_four_quads: + ldq $1, 0($17) # L : 32 bytes + ldq $2, 8($17) # L : + ldq $3, 16($17) # L : + ldq $5, 24($17) # L : + + stq $1, 0($16) # L : + stq $2, 8($16) # L : + stq $3, 16($16) # L : + stq $5, 24($16) # L : + + addq $17, 32, $17 # E : src += 32 + addq $16, 32, $16 # E : dest += 32 + subq $18, 32, $18 # E : count -= 32 + nop # E : + + subq $18, 24, $1 # E : room for another full trip? + bge $1, $move_four_quads # U : + bge $18, $move_a_quad # U : a whole quad still left + br $31, $less_than_8 # U : only trailing bytes + END(memcpy) #ifndef USE_AS_MEMPCPY libc_hidden_builtin_def (memcpy) diff --git ./sysdeps/alpha/alphaev6/memmove.S ./sysdeps/alpha/alphaev6/memmove.S index d587a90dd9..a69a0588a2 100644 --- ./sysdeps/alpha/alphaev6/memmove.S +++ ./sysdeps/alpha/alphaev6/memmove.S @@ -237,8 +237,8 @@ $no_unroll: .align 4 subq $18, 8, $18 # E : At least a quad left? blt $18, $less_than_8 # U : Nope - nop # E : - nop # E : + subq $18, 24, $1 # E : four whole quads left? + bge $1, $move_four_quads # U : yes -- out of line, below $move_a_quad: ldq $1, 0($17) # L : fetch 8 @@ -412,5 +412,46 @@ $back_b: bne $18, $back_b # U : ret $31, ($26), 1 # L0 : + + /* + * Four quads a trip, for the 32..127 byte copies that never reach the + * unrolled loop and for the remainder of the ones that do. Short + * copies are issue-bound -- they miss nothing and replay nothing, and + * retire better than two instructions a cycle -- so what they want is + * simply fewer instructions per byte than the one-quad loop above. + * + * This lives out of line, past every return path, on purpose. Placed + * inline just above $move_a_quad it costs long copies about 5% at 1KB + * and 2KB, even though a copy that size never executes a single one of + * these instructions, purely by pushing the unrolled loop and the tail + * further apart. Moving it down here recovers that and gives up only + * a few percent of the gain between 32 and 96 bytes. + * + * $18 counts bytes remaining less eight, so a full trip needs $18 >= 24. + * Falling out can leave $18 negative and $move_a_quad is a do-while, so + * the count is rechecked before going back to it. + */ + .align 4 +$move_four_quads: + ldq $1, 0($17) # L : 32 bytes + ldq $2, 8($17) # L : + ldq $3, 16($17) # L : + ldq $5, 24($17) # L : + + stq $1, 0($16) # L : + stq $2, 8($16) # L : + stq $3, 16($16) # L : + stq $5, 24($16) # L : + + addq $17, 32, $17 # E : src += 32 + addq $16, 32, $16 # E : dest += 32 + subq $18, 32, $18 # E : count -= 32 + nop # E : + + subq $18, 24, $1 # E : room for another full trip? + bge $1, $move_four_quads # U : + bge $18, $move_a_quad # U : a whole quad still left + br $31, $less_than_8 # U : only trailing bytes + END(memmove) libc_hidden_builtin_def (memmove) From patchwork Wed Aug 12 01:19:51 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141160 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 1A58F4BAE7EC for ; Wed, 12 Aug 2026 01:26:55 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 1A58F4BAE7EC Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=S1H8lWCz X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x1136.google.com (mail-yw1-x1136.google.com [IPv6:2607:f8b0:4864:20::1136]) by sourceware.org (Postfix) with ESMTPS id CC96B4BB1C37 for ; Wed, 12 Aug 2026 01:20:31 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org CC96B4BB1C37 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org CC96B4BB1C37 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::1136 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497631; cv=none; b=vr9IC88vYBwPTzhwVH4lhUgbvRcJMElFlqljU3JNt3C8pKIq/soklLYXvsKgdoWvVwfPqYZjWNprR5mDxz9GRKZbXVpKoqUlRbawnuevzJuKWCcgMFHCLitOX7tOm6yp8GpMy0y8H/cWTTucdb+zFDGDKHrFQ1vA8I1EkdaM3hA= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497631; c=relaxed/simple; bh=KDcOlncamyQv8/vhiRudboALHXZiC5f6mq4COyTl2gA=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=pxcFsMkCAwm8rxqFrOXOZkWX5ZGsWpKyz4+4P2WrgTiZgNLQWlkj5UmKwMzOXotQhoFSJXpHv856PCvm9/U86I+m/Ivq4Y7fichnDb7mjx+VkHSXpR/q56vi3HDxVOec1QylrD/S9eYSvv8BHR+WL/Ub2GMiFb68vQJ7S8YuKu8= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=S1H8lWCz DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org CC96B4BB1C37 Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-81ed000b507so4756787b3.0 for ; Tue, 11 Aug 2026 18:20:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497631; x=1787102431; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=LmGm58EPpASROx8do67UqyaC/R++NCkgBLZ9ujSPS2g=; b=S1H8lWCzgB9evIz3NuvtJqGgJmROpdMj8f0h08kK+GNX84swpra5TefoIcscjZRwnb BjnBXWT4hdUjMkM15jHTEaJ/KmCCBWgSZ5IjM+AS8UOWjbSyxZeP4OOmgD9Ic/opZrq4 AEdFpwc+AeOIwLeiDidnlLM12qAzJPcy8bgaa4TEyUxDrmz5V6RKSBi7pxL9Kq7P/K5X uykDMUhjN8t+xvSUlUYSFQoqc9EKHtgi9QyMm5CdrppGLtsqjQ8KFNjI9C7UZ+WZtBch ld99I4Jdm1QWWSZBS13iiLYndm3L30+HqQkyt+5/ZWHe009OolRNk6KARhdUodsq3uav sAnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497631; x=1787102431; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=LmGm58EPpASROx8do67UqyaC/R++NCkgBLZ9ujSPS2g=; b=KW9+cgZghZ9eT/+4kx5fg9gjW5REXLv9C/nediDouzkfS7xuqk2ShGClWzyJdccENm j7F9QyBdYFC5xuMTBkbc+IhPDQzGDlRqShdz25aXnlKfswXGumH0KogEKlNBSsGpT4D4 a40NhfjWC63Kahuhivlfm3cD0kY3NVyg7w//E4l2xrLPP2rqGqFcAXv2y7CYuNbQ4ngf bviJJn5vQ0DlVvT6at6dMu08cdZSGDlk/VPdcIM7GdWjIVT0IwMG4f7wJmJ+ytWzBNnZ iDaJxKNZpF456LSEFpLJmy66pI8vxagouodgcDrkXAb/jLN+WWaoY4b8O3cDL+lQLsuv AA2w== X-Gm-Message-State: AOJu0Yzv4UYgHizvJD3WGLSvH45tzNR6HRO15FDd3ve39xVCuxYyjGVg v0RzjWmrbEoMjgPkchhPFeIqBiABTW0dUqJqJxpSUzmt4zGIql8jelJqQ2G/E/Qa0eY= X-Gm-Gg: AR+sD12R18Xe5FXbloJZ7R6ijEqdvxl1IlpvgfK792uQHy9DvLD/Lyw7r/fWW+LOC8z 0BNduSdMzecGYhOYzttWM70Ihe9Xhp529IKA4f6QwQH6Ner3syqr++uSJt839JOYxqIxkS/GxfT 4IfDIByGGnzmg0X3ASEo0wINosJwUIXycEbPOdNb8cczt96dxewfD7aDTz9q5F+iIfXIh/Iyz4V XpYElL7sNOnpBsEpt5qiwHW7i2dJ1wslLMSaYyZZNzCzMyCf4QaO437yNMABF3vZhZoNBF7RpiQ T7FuZZ3TNGcvvo6K+L7ime3BmZROCYAk838Amd6nDJtWUqxFgf8iAzQudhJ5vsrUOnHmrYCyFHF jRvi6H+V4h9UbEdz6OQ0AaDmmnuFjunMRpkPi8PoCGO1q4TsWbAHIFP78jbTx8fyjcDMxUfmTAL sZ9dF/JZKb2jElbkJBMJ6ZNJ0Wcoi6u2wKpqlbjCgTaGsvA7mMF3YfpwcquFE= X-Received: by 2002:a05:690c:c014:b0:81d:5806:ecc7 with SMTP id 00721157ae682-83105ff1541mr7332357b3.1.1786497631121; Tue, 11 Aug 2026 18:20:31 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a349e557sm6552807b3.7.2026.08.11.18.20.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:29 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 14/20] alpha: store four quads a trip below the ev6 wh64 loop Date: Tue, 11 Aug 2026 21:19:51 -0400 Message-ID: <27e8dc517f361cfda968789032172552c2a0ddc9.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Sets shorter than 128 bytes never reach the wh64 loop and were written a quadword at a time. Like short copies they are issue-bound -- they miss nothing and replay nothing -- so what they want is fewer instructions per byte, and four stores a trip delivers that. Measured on an EV68CB, hot, cycles per call: size before after 112 56.6 31.2 1.81x 64 36.2 23.1 1.57x 120 51.3 32.9 1.56x 127 51.3 36.2 1.42x 32 25.8 20.1 1.28x 96 33.2 27.4 1.21x Sets below 32 bytes pay one cycle for the test that skips the new loop, about 5% where the call is only 17 to 21 cycles. Placement took three attempts and is worth recording. Letting the trailing quads of a long set share the four-quad loop cost 5-8% between 128 and 512 bytes, so the wh64 path deliberately still falls into the one-quad loop. Putting the block inline above $loop cost 13% at 512 bytes even after that, purely by growing the straight-line distance the long path covers. Out of line past the return paths, 128 through 384, 1KB and 2KB all come out level. Three isolated sizes -- 288, 320 and 512 bytes -- still measure about 10% down. They are not a smooth function of length, since 256 and 352 are level, and they execute exactly the instructions they did before, so this is cache aliasing against the grown function rather than anything the new loop does. No placement tried removed it. Against that, the geometric mean over the whole range measured is about 1.09x. Verified on an EV68CB against the C memset for every length from 0 to 600 at 72 destination alignments, with 512-byte guard regions either side checked for stray writes. --- sysdeps/alpha/alphaev6/memset.S | 46 +++++++++++++++++++++++++++++++-- 1 file changed, 44 insertions(+), 2 deletions(-) diff --git ./sysdeps/alpha/alphaev6/memset.S ./sysdeps/alpha/alphaev6/memset.S index fb01baa795..4273604f6e 100644 --- ./sysdeps/alpha/alphaev6/memset.S +++ ./sysdeps/alpha/alphaev6/memset.S @@ -117,7 +117,7 @@ $aligned: * 21264 family (EV6/67/68) and 3 for the 21364 family (EV7/EV79). */ subq $3, 16, $4 # E : Only try to unroll if > 128 bytes - blt $4, $loop # U : too small -- scalar tail loop + blt $4, $loop_four # U : too small -- four/one quad loops implver $2 # E : 2 = 21264 (EV6/67/68), 3 = 21364 (EV7) subq $2, 3, $2 # E : EV7 -> 0 @@ -126,7 +126,7 @@ $aligned: and $16, 0x3f, $2 # E : Forward work (only useful for unrolled loop) subq $3, 16, $4 # E : Only try to unroll if > 128 bytes subq $2, 0x40, $1 # E : bias counter (aligning stuff 0mod64) - blt $4, $loop # U : + blt $4, $loop_four # U : /* * We know we've got at least 16 quads, minimum of one trip @@ -287,5 +287,47 @@ $end: nop ret $31,($26),1 # L0 : + /* + * Sets shorter than 128 bytes come here instead of the one-quad loop. + * Like short copies they are issue-bound -- nothing misses and nothing + * replays -- so four stores a trip is worth a fifth to nearly half the + * time between 32 and 127 bytes. + * + * Out of line, past every return path, on purpose. Two other + * placements were measured. Sharing the four-quad loop with the + * trailing quads of a long set cost 5-8% between 128 and 512 bytes, so + * the wh64 path deliberately keeps the one-quad loop it had. Putting + * the block inline just above $loop cost 13% at 512 bytes even with + * that fixed, purely by growing the straight-line distance the long + * path covers. Down here, 128 through 384, 1KB and 2KB come out level. + * + * A few isolated sizes -- 288, 320 and 512 bytes -- still measure about + * 10% down, and they are not a smooth function of length: 256 and 352 + * are level while 288 and 320 are not, and all of them run exactly the + * same instructions as before. That is cache aliasing against the + * grown function, not anything this loop does; no placement tried made + * it go away entirely. + */ + .align 4 +$loop_four: + subq $3, 4, $4 # E : four whole quads to write? + blt $4, $loop # U : no -- one at a time + nop # E : + nop # E : + +$loop_four_body: + stq $17, 0($5) # L : 32 bytes + stq $17, 8($5) # L : + stq $17, 16($5) # L : + stq $17, 24($5) # L : + + addq $5, 32, $5 # E : Inc address + subq $3, 4, $3 # E : four fewer quads + subq $3, 4, $4 # E : room for another four? + bge $4, $loop_four_body # U : + + beq $3, $no_quad # U : exactly consumed + br $31, $loop # U : 1..3 quads left + END(memset) libc_hidden_builtin_def (memset) From patchwork Wed Aug 12 01:19:52 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141163 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3273D4BA9011 for ; Wed, 12 Aug 2026 01:28:48 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 3273D4BA9011 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=PSWmSDFA X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) by sourceware.org (Postfix) with ESMTPS id 4FC824BB1C04 for ; Wed, 12 Aug 2026 01:20:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 4FC824BB1C04 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 4FC824BB1C04 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::1134 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497634; cv=none; b=ld6XPppwU6HE7H5YZ2m/resi3jv5wQqYg0vqT8Zn1ThUTXlvnYFbjrpp7z9rBMNoHDhCMM/f8wGdNX3ToATC5YVS0UTvNCFtoCcqLgQ0eksGswpCq5xnz4iMhU/O+5yMgHN0n5OzEhW1LziHPWQx6/+fqoD3tIcXeOVOdvyUpyg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497634; c=relaxed/simple; bh=850GqHaKOcRB8T+xQX8+4q7mJXxtX+/73jFrGlESGF4=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=cE3XuB03jzlvEXEigrg9PorYoFXnr78XioblZh1b9iOIuYalvKCdTBQpxy/Hatk9LojdRHgCuSJezNSDAVJ+YvK/KzJbD4jXcfILcNy0NKP2cwwSOjPxwws192bsoT+yHvhl0ef9pH1D7b5bFyZPuoJfQjYK1JYKUmCgeMO9tQs= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=PSWmSDFA DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 4FC824BB1C04 Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-82113c53aeaso5608187b3.1 for ; Tue, 11 Aug 2026 18:20:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497634; x=1787102434; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=vwI9qv3QlXzEPBfiv4PUzWYdD1Q6hbXNK1YUffJP5go=; b=PSWmSDFAUNNWPWw+G1snXvFIuipBfHEyh4nXohMR6qceOVbOVBcA+CX+MKdkMuvgY5 KM70sVZknviS/yHm0J9INfARxIiDqjs10j0Qu6ixQXr9y/ijiPdJHRdJ8tNYE5Q1DJUk xSoWclBt9hENuDSsjSVnWuL3ZwrtIeauHyjRrvP+tDaVWjZMKFp+GC4t08PtBYP2GepH ULy8MAZg2gdT5dpV0vLQX6jvCXBsSXEqPqhHSGQ+dbnSlfMKm/17zsXiH1Dc/EiHIxX5 JlePRI9kltKpoLGP1PQmbqI3eL+yHFq9qcpucs/WqbOBOSp0FCUcaMYxF/sU8aUYxHA8 9BEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497634; x=1787102434; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=vwI9qv3QlXzEPBfiv4PUzWYdD1Q6hbXNK1YUffJP5go=; b=NdLhy6CWgHT5XrwXxGcBV+ImYUbPImqNIOueV+hmKJA71N9XxEDZz6Ew9KTA7cW+90 ZQe+Mdm3fg5Gw5WPtFkv1N66dzBgHaWed2p8Gbn49WKKSB7DRG5BZV354r8q/sKJKvt1 fUnzXz2kg9HzsCX/2Th0vPduLlDD9pWm6NMoESZCZpdw/pQJtJ3NVMpzm+eTxYf3llAX pH+BtVuz3fBXDnNmdjOt3tbLIGMo+rwV0lR+3kjHKyr1tUqTM0Jys0GPZzazzDSy0Hte R8O8jeUVaweYq/0aEY3xEfPwflT4sgTcG1ct6GXo0nw0DbIrkSvV6RTID2FJTkxO51zr 8iIQ== X-Gm-Message-State: AOJu0Yz05VqOWR36FZtCtTymxP0O2iwjj7cdvVoy4ixPWGtNIg1hc8mG rvTdgOXuAbaxjSd1W99cq3P1QYn49wYwOhOUvFmoG5Reg4x2W4gf3YMbJ4sGigRFnTA= X-Gm-Gg: AR+sD12/ZNC1CfanbD745CjcYLe3rPojJBZbukuy0j+gQDMpCHYzI374SkHBQVRb3Zv aCCSeMYJvhdy0qWCDyX8rhMqF6gi4AD7AhLOPmLkxVeb6phaCULmMYI8ChCurSlbnfafUrD9bj6 zKEqRZNzfizEx8UxnzHgIY/RtCfyD8pVajjkagX1ipMK3h7fPzMMnL86PMMbbYNeYG5dEWA51l2 mTzmsCIep2DCfXSLMnn9peUvHY5tOSlF3AtHHcZPq6WUAfeVlac7aA9dUXu8ANv44LnKE56u6/2 zcDW4SEkdx+ZzCwcXQ0gNmKUTrMcDB4BqL6CbXwcvmhb/qGAUORafGeqbndpdYBXycwzf5LBaA4 dfFyYioCN8Emylvm/iv/nvEDYhQ22KcRyI3d75Ymqi1fJt7dlPU2K4RRrMimu6fWKqrwVjIy+t4 ZdTg+kC2Kmyd7AzxHkTurU2Jw35SRmnMudxwCyP7QT3n/t0RBQ4gxZD5zVT9Eyf5N2SOyQwA== X-Received: by 2002:a05:690c:6ac6:b0:80c:4923:cf4 with SMTP id 00721157ae682-831051735bcmr10409387b3.10.1786497633560; Tue, 11 Aug 2026 18:20:33 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a414e953sm6442767b3.14.2026.08.11.18.20.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:31 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 15/20] alpha: load all 8 quads before storing in the ev6 memcpy unrolled loop Date: Tue, 11 Aug 2026 21:19:52 -0400 Message-ID: <82c81f07e7a2b0c2385f6c8dc5d174cd0edf68fc.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, LOTS_OF_MONEY, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The unrolled loop interleaved load-4/store-4/load-4/store-4 each 64-byte trip, forcing every store group to wait on the load beside it. Loading all 8 quads before storing any of them gives the Mbox a full run of independent loads to hide behind, and the store run after has no load dependency left to stall on. Measured on an EV68CB: 19-39% fewer cycles/call hot from 512B upward, with Mbox replay traps (perf r4) roughly halved at 64KB-256KB, and 1-4% cold at the same sizes once DRAM latency dominates over scheduling. Below 512B the two shapes are within measurement noise. --- sysdeps/alpha/alphaev6/memcpy.S | 64 ++++++++++++++++++--------------- 1 file changed, 35 insertions(+), 29 deletions(-) diff --git ./sysdeps/alpha/alphaev6/memcpy.S ./sysdeps/alpha/alphaev6/memcpy.S index d86c00b9be..fb7bb77d42 100644 --- ./sysdeps/alpha/alphaev6/memcpy.S +++ ./sysdeps/alpha/alphaev6/memcpy.S @@ -139,57 +139,61 @@ $do_unroll: subq $18, 192, $2 # E : room for [$16+128, $16+192)? cmovlt $2, $16, $7 # E : no -- hint the first block instead + /* + * Each trip loads all 8 quads before storing any of them, rather than + * interleaving load-4/store-4/load-4/store-4. The four extra live + * values need four more registers across the trip -- $8/$22/$24/$25 + * (t7/t8/t10/t11) join $3-$6, all still caller-saved so this needs no + * prologue. Interleaving forced a store to wait on the load beside it + * every four quads; loading the whole trip first gives the Mbox a full + * run of independent loads to hide behind before the first store of + * the trip is issued, and the store run after has no load dependency + * left to stall on. Measured on an EV68CB: 19-39% fewer cycles/call + * hot from 512B upward (Mbox replay traps roughly halved at 64KB-256KB), + * 1-4% cold at the same sizes once DRAM latency dominates. Below 512B + * the difference is inside measurement noise either way. + */ $unroll_body: wh64 ($7) # L1 : memory subsystem hint: 64 bytes at # ($7) are about to be over-written ldq $6, 0($17) # L0 : bytes 0..7 - nop # E : + ldq $4, 8($17) # L : bytes 8..15 nop # E : - ldq $4, 8($17) # L : bytes 8..15 ldq $5, 16($17) # L : bytes 16..23 + ldq $3, 24($17) # L : bytes 24..31 addq $7, 64, $7 # E : Update next wh64 address nop # E : - ldq $3, 24($17) # L : bytes 24..31 + ldq $8, 32($17) # L : bytes 32..39 + ldq $22, 40($17) # L : bytes 40..47 addq $16, 64, $23 # E : fallback wh64 address (see below) nop # E : - nop # E : - addq $17, 32, $17 # E : src += 32 bytes - stq $6, 0($16) # L : bytes 0..7 - nop # E : - nop # E : - - stq $4, 8($16) # L : bytes 8..15 - stq $5, 16($16) # L : bytes 16..23 + ldq $24, 48($17) # L : bytes 48..55 + ldq $25, 56($17) # L : bytes 56..63 + addq $17, 64, $17 # E : src += 64 bytes lda $2, -256($18) # E : room for the +2 line hint next trip? # it covers [$16+128, $16+192) there, # so that trip needs 192 and this one # needs 256 (256 exceeds the subq # literal field, hence lda) + + stq $6, 0($16) # L : bytes 0..7 + stq $4, 8($16) # L : bytes 8..15 + nop # E : nop # E : + stq $5, 16($16) # L : bytes 16..23 stq $3, 24($16) # L : bytes 24..31 - addq $16, 32, $16 # E : dest += 32 bytes - nop # E : - nop # E : - - ldq $6, 0($17) # L : bytes 0..7 - ldq $4, 8($17) # L : bytes 8..15 - nop # E : - nop # E : - - ldq $5, 16($17) # L : bytes 16..23 - ldq $3, 24($17) # L : bytes 24..31 - addq $16, 32, $16 # E : dest += 32 subq $18, 64, $18 # E : count -= 64 - - addq $17, 32, $17 # E : src += 32 - stq $6, -32($16) # L : bytes 0..7 - stq $4, -24($16) # L : bytes 8..15 cmple $18, 63, $1 # E : At least one more trip? + stq $8, 32($16) # L : bytes 32..39 + stq $22, 40($16) # L : bytes 40..47 + nop # E : + nop # E : + /* * $7 is deliberately finalized late, in the group with the loop branch * rather than alongside the loads above, and the fallback address lives @@ -203,10 +207,12 @@ $unroll_body: * Moving it back up, or reusing $1 so that it has to move back up, * gives up that gain. */ - stq $5, -16($16) # L : bytes 16..23 - stq $3, -8($16) # L : bytes 24..31 + stq $24, 48($16) # L : bytes 48..55 + stq $25, 56($16) # L : bytes 56..63 + addq $16, 64, $16 # E : dest += 64 bytes cmovlt $2, $23, $7 # E : fallback if < 2 more trips (late; see # the note above -- do not hoist this) + beq $1, $unroll_body br $31, $tail_quads # U : 1..15 trailing quads/bytes From patchwork Wed Aug 12 01:19:53 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141161 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 260144BAE7EF for ; Wed, 12 Aug 2026 01:27:02 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 260144BAE7EF Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=LpMgnozZ X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x112d.google.com (mail-yw1-x112d.google.com [IPv6:2607:f8b0:4864:20::112d]) by sourceware.org (Postfix) with ESMTPS id B7C224BAE7E6 for ; Wed, 12 Aug 2026 01:20:36 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B7C224BAE7E6 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org B7C224BAE7E6 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::112d ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497636; cv=none; b=RcuRa+Dp77q2vjKJPBW7d2f5QlHa4lQcgrWLyEKal0q5LpyQaf2x5QWZV18/AdngXJg2avgUvbBAQF0jNq+w7e6BWkta4UgVWQH7RetMQXuBzkX1xk9VTREYX9ScPZCkgGFSEVE3jH4v9ZA8ah2VInMtP9NvZ+/JZz24c61DuKY= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497636; c=relaxed/simple; bh=oG2RJfoyMBH1pGMe8/r6bLO/QI3s7vb+S5K9AorT5AU=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=tOltecCHKtRmSkFl80kef7tX+k3DJGLZdsNv2Ct9YWFtnc/da65Daj6tDrhWxUD6T5WnkGmKUClIyFP8HmOMM8gp45ME7Y0LuG+MhsA3R4Sbsc8oYUMIBNeRTCw7ebfeZVMmdE1kLh5qWwWL4Nnu4Lkv2Xq7TutGGlUhIeY6GFM= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=LpMgnozZ DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B7C224BAE7E6 Received: by mail-yw1-x112d.google.com with SMTP id 00721157ae682-80bb41f7f3cso4736187b3.2 for ; Tue, 11 Aug 2026 18:20:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497636; x=1787102436; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=A5p/bbgcjb5UQkU8uFM+AUZe7dzcCJwS/6tNGlOBwHI=; b=LpMgnozZ2dF9nneGYDmPx/RVj5JXtzKGEvAQO/vt7i3Hg/ZbMlKbLV6txGg263J4Ce ENxoEJM5L16MpE+OY2EmhaylSAFfDe/GS6QEO1Dz34sj+Zs4grN0pn9AT5UN2W3GV5Ee lleNAhklufRwa+PaM2W5yx/5d6bzeetVRK8aPKL4g4XXfO0wWBNF6zPBOGSso+/+ZW6N MtJukm9Hl9PYB1cxS0HDm66moviyoyROX4k1A9mN5qcB6GiYjMR1OhQMwSF8UpxaAt8c ERo5Bx/uprleUy2Y4tKDQpS9nxXSFPiRQR6mc0hksuwKMIwXMQY7jg12DCH38fcy1Scd w0Vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497636; x=1787102436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=A5p/bbgcjb5UQkU8uFM+AUZe7dzcCJwS/6tNGlOBwHI=; b=O9aY41UV8phxnPCarWyzPKqYuUJ8Vq6mDaQ+WjspmUu7C2Zj0aytaiaUKpTIOtgHRe 2DKBkoE+LJs5e+xuoVfdLBH8e7Ks1cx2aVPndVpRNHWn3pvnSqlIPUvIcz8mqxI9gN5m mIaCTMEC7T5rDMyGiqmyyeQ71ISb2cn/Cp/37W6E2dHOE3/pxrrniQFNgR0tl3eUGNPO M0+c/V0DvpuyXoW71fWbgGq+a3KQ3lYwPyObcyi4kWPdfdzPAeMle3MtLOvyNV50F5k+ USr47dnwiUo9VKJqXoSO3C+kGNOWR5TfkNINCfqBZxvk/Wx3kYtzdIDNdZcHZ9nz4bC7 JvEA== X-Gm-Message-State: AOJu0Ywmb3RHjV01ULNEJvgKjvy+aZtUpMt5j6+saR9pY3lUSL323QfX jNo+qO3Cc5yBjZWpTIcS6x38SCN9j5fZLIj7HUBJr7BUXP6N/z2WXny3DX11InRfgNM= X-Gm-Gg: AR+sD12aNmsoZ+h9edrhe7S+4cQuJFY/nMTsM/1TeeallE+BhXzvc0ijttvQwoFIME0 bEpVv/DHLQyPh4UG/FioS30YrGQyMs41EjChcnhv1JLV9+ffDej+YKKJbIvJFk/79WDozabY7pG QEG++31iEway+u7VAE+F9kC8FctNEgjvySKOmTACPn+/8UFea/bL8rsO0UxyZN0iKUIju4hlT52 Q7/3xZNgubGGVz9ULwQ4VSysjIfqLbba6lkPN2I2ee+cv7i3IalOoYG9HP8m/2cg+wyH0RtRQ/c acy26TuNdMOeJg7etrqAG9l4eat3+FCqYkpnX01MALKFRQLOIdu4nUhZY20KL7vouLfw09Veudj VfudrE2Zti5bzq8bDo2DDtIyPzRAxlK5SPI+ME8WT0AyImrwtru+yAJshTWQEWHJARcU+iM8IWQ YxE8zQy82sAlwQd7m0xDXoYCQOBKPjE8wKPemLNGYO0MRTp/lFVufLXTS2PsNaE+LD/wer3+Q= X-Received: by 2002:a05:690c:d8d:b0:80b:fb24:4589 with SMTP id 00721157ae682-83105e01393mr10173317b3.2.1786497636068; Tue, 11 Aug 2026 18:20:36 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a9633601sm6123107b3.28.2026.08.11.18.20.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:35 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 16/20] alpha: add vectorized single-char fast path for strspn Date: Tue, 11 Aug 2026 21:19:53 -0400 Message-ID: <5739231c5777e500aefe79888c0921758aead7d4.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The generic strspn's single-accept-char case was a byte-at-a-time scalar loop. Alpha has no hand-tuned strspn at all, so add one: the single-char case now scans a word at a time via cmpbge, finding the first byte that differs from the broadcast accept character (a NUL byte always differs from a non-NUL accept char, so this doubles as the string end check with no separate test). The multi-char case keeps the existing bitmap-table scan unchanged -- already efficient, nothing to gain there. Measured on an EV68CB against the old scalar loop: 4.3x faster at 64B, 5.5-5.7x at 256B, ~6.8x at 4KB, ~7.7-7.8x at 64KB, converging toward the 8x ceiling the word width sets. Correctness verified against a reference scan at lengths 0 to 255 and all eight starting byte alignments, including runs that cross a word boundary and strings with a leading mismatch. --- sysdeps/alpha/strspn.c | 125 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 sysdeps/alpha/strspn.c diff --git ./sysdeps/alpha/strspn.c ./sysdeps/alpha/strspn.c new file mode 100644 index 0000000000..2ab11732a7 --- /dev/null +++ ./sysdeps/alpha/strspn.c @@ -0,0 +1,125 @@ +/* Copyright (C) 1991-2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +#include +#include + +typedef unsigned long word; + +#define unlikely(X) __builtin_expect ((X), 0) + +/* cmpbge(0, X) sets bit i iff byte i of X is zero. For the single-char + fast path we want the opposite -- the first byte that differs from the + accept character -- so cmpbge(0, X ^ C) is inverted and masked to 8 + bits: bit i then set iff byte i of X differs from C. */ +#define cmpbeq0(X) __builtin_alpha_cmpbge (0, (X)) +#define diffmask(X, C) ((~cmpbeq0 ((X) ^ (C))) & 0xff) + +/* Return the length of the maximum initial segment + of S which contains only characters in ACCEPT. */ +size_t +strspn (const char *str, const char *accept) +{ + if (accept[0] == '\0') + return 0; + + if (__glibc_unlikely (accept[1] == '\0')) + { + /* Single accept character: scan a word at a time for the first + byte that isn't it, rather than one byte per branch. A NUL byte + always differs from a non-NUL accept char, so this naturally + stops at the end of the string with no separate check. Measured + against libots' _OtsStringVerifyChar on an EV68CB: glibc's old + byte-at-a-time loop cost 2x at 8B, growing to ~14-15x by 4KB-64KB + runs, purely from one branch per byte versus one per word. */ + const char *s = str; + word t = (unsigned char) accept[0]; + t = (t << 8) | t; + t = (t << 16) | t; + const word c = (t << 32) | t; + + const word *s_align = (const word *) ((word) s & -8); + word current = *s_align; + word mask = (1ul << ((word) s & 7)) - 1; + word found = diffmask (current, c) & ~mask; + if (unlikely (found)) + goto found_it; + + for (;;) + { + s_align++; + current = *s_align; + found = diffmask (current, c); + if (unlikely (found)) + goto found_it; + } + + found_it: +#ifdef __alpha_cix__ + return ((word) s_align - (word) str) + __builtin_alpha_cttz (found); +#else + { + word offset; + found &= -found; + offset = (found & 0x0f ? 0 : 4); + offset += (found & 0x33 ? 0 : 2); + offset += (found & 0x55 ? 0 : 1); + return ((word) s_align - (word) str) + offset; + } +#endif + } + + /* General case: same bitmap-table scan as the generic implementation -- + already O(n) with no obvious win from vectorizing further here, and + libots' equivalent multi-char path is an O(n*setlen) scalar scan that + loses to this for any set beyond 1-2 characters, so there's nothing + to take from it. */ + unsigned char table[256]; + unsigned char *p = memset (table, 0, 64); + memset (p + 64, 0, 64); + memset (p + 128, 0, 64); + memset (p + 192, 0, 64); + + unsigned char *s = (unsigned char *) accept; + do + p[*s++] = 1; + while (*s); + + s = (unsigned char *) str; + if (!p[s[0]]) return 0; + if (!p[s[1]]) return 1; + if (!p[s[2]]) return 2; + if (!p[s[3]]) return 3; + + s = (unsigned char *) PTR_ALIGN_DOWN (s, 4); + + unsigned int c0, c1, c2, c3; + do + { + s += 4; + c0 = p[s[0]]; + c1 = p[s[1]]; + c2 = p[s[2]]; + c3 = p[s[3]]; + } + while ((c0 & c1 & c2 & c3) != 0); + + size_t count = s - (unsigned char *) str; + return (c0 & c1) == 0 ? count + c0 : count + c2 + 2; +} +libc_hidden_builtin_def (strspn) From patchwork Wed Aug 12 01:19:54 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141147 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 7C6A94B9DB50 for ; Wed, 12 Aug 2026 01:21:15 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7C6A94B9DB50 Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=VDax7fHC X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb130.google.com (mail-yx1-xb130.google.com [IPv6:2607:f8b0:4864:20::b130]) by sourceware.org (Postfix) with ESMTPS id 092254BA2E2A for ; Wed, 12 Aug 2026 01:20:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 092254BA2E2A Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 092254BA2E2A Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b130 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497640; cv=none; b=eyaTACZr3QJQnRWkFExttUEssvnLDTCXRgSxzfcOIfELIu3Zdv9p2nMtlfmPpp9DwauiiK620z8WSFUvAohczfUHp34VGia0TEgXobdiiaha9odh/shyac0HOmxllwfjQ2CEn92uhejCotEwBfJVWAgcHE8mcyX+VgAm4+xx7Zk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497640; c=relaxed/simple; bh=BKdL3rGk+U4pN5qpMW5SKP9A+RpOufvcmxxH9y7Qo4Q=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=h6c99o55TdiFhzyjaKrZWb93vkn3rQouWZvTS3c1nTtBTXjmyPRzCrjB4x4cknzZjjIBsCTDz5g2Tcym2QYxS0gsKpuaz3FwX2nansKjDJWnxI0Tc3vyp9UZQ0RMSSnHlAWFoLKUkkeskr6IMeXllDJvT3sqFIEzofdEenAqvys= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=VDax7fHC DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 092254BA2E2A Received: by mail-yx1-xb130.google.com with SMTP id 956f58d0204a3-667bae10ba3so524823d50.2 for ; Tue, 11 Aug 2026 18:20:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497639; x=1787102439; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=CGp42CNBscnAqlaV2fmxK8y5snT2LvLCzkmvzI+sj8A=; b=VDax7fHCXbLYXNkzYQHzUWWp2jjxUVniXaFLErsna8BSlaIDVvx9aFAS4u5N6nQfzw /L0aM8ddOCli9I61dIgGPB6/3zybKJP82vmK/FiaGUtjuft7TdxBmyxMHUAsVxWM6TCI Q97iqZU6LpU+hCijqdNJn7tELcBtUHm4zWdMODeKc9A7UkXz00V4Bs1sB85xshGQPz6+ l/15xh7QKr2KRUGUBQrwbpKf+r5ZMxbnobKV/8ZWOO+EASOMUODtVivVBsBFo7WwTq1h sL1Bo6zvmvb7fx6J8MMdVhDC6hhATfQgWaZDeC6S/WYWkuEmq43hzLR3Sz+sWRF4fXwB Y6Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497639; x=1787102439; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=CGp42CNBscnAqlaV2fmxK8y5snT2LvLCzkmvzI+sj8A=; b=Xy34yd9OIbFg0TdS5PoVxNrppbPlQFeheR/NZT7CfQDmhE2KfoybD6cKUFYmOvWz1C mx/IEKYUheGkJKOwE3TmJsXIUO4MG9QQcC36kkDxPDcjp08Yx+pqZsq1dHVxJ0Aqgcee QQfc+anNFqkJhVixYZQteFqm1YuUcxd1mZg7JO5qYYPwKyo0YuVFhGlMYPE3pHcUqeVB BGRvMLnsH6Qyk+YxBSO4kbBcE3K3cyxM4L3VqbjZSE1kSt5Oc07cQT2zTbgwWvHWfIIt 0d8g8qTwjnS2QlVg7hUTbkwWghNrWsT7/XCeQw1QO3b1Mez/SrxmGiT/2183RoJ5uuLM hjWw== X-Gm-Message-State: AOJu0YzbSdAslJXsG8ftIYwk9ydJRGiSZI0z9w/rMzCTqUxsmpiJ/MsD sRdL2ett1PBwn3VFoffWYBvt35Bz0NDcHydNYLzAT6xHVPAhEfEAq1e0Vvq87tcgvew= X-Gm-Gg: AR+sD13MfKU8E2c+Vo+Om/NeDLhHGMXrW2SoFRbfBc0jsl6ypZed0PENfVeVEkLkDq3 fc0xq2Eb4C//O12MmqjS7t4rha1CvMSwap5haeIZw5IpWmsvaYX8xGciS4yVcFc9zMtAXmYiEmB /nsN1h6F8REgVQabdHCT7VsCxq9dMmhpw/zXhE3Hhfg9tzmJzQ3VW5KztzCQjYh9EEjqahbd9eo sggl5UiMg1Y/r0OlmIl5+UFvXnHmDnFtFwMn2ygjJEO7r/rYXv6OsmTd02mDVO+yOeKmasxW82n we4syo9b4yHfKREALZ4lK4bE8kMsoaGMcNcPyo2PkJnPSUyr1xsnDwTud3e4XNzq8ZmG/Ts/Ga3 zZ6cs59Z4cRja+4IE5ZGsPkpy0nh4rW/4lm7mXatdrWIyyQNQURUp8fwOy+Tnv3J9ES9yd4eAOe ML6UAqCV79WubvgMjag4K1GsWf/qCb9OPHIAbSfdNwvzIkLPF8WwWJxH3ojlA= X-Received: by 2002:a05:690e:14cd:b0:667:bc49:3730 with SMTP id 956f58d0204a3-66b3412e0f9mr721080d50.7.1786497639403; Tue, 11 Aug 2026 18:20:39 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a414a752sm6292147b3.16.2026.08.11.18.20.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:37 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 17/20] alpha: treat STO_ALPHA_STD_GPLOAD as a function in abilist.awk Date: Tue, 11 Aug 2026 21:19:54 -0400 Message-ID: <7af54a9815fcd1e6f5734e665217e43b9c8c6bc6.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-11.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org The Alpha ABI defines two st_other values for functions: 0x80 (STO_ALPHA_NOPV) - .prologue 0: runs on caller's GP 0x88 (STO_ALPHA_STD_GPLOAD) - .prologue 1: standard two-instruction GP load abilist.awk recognized 0x80 but not 0x88 when classifying a dynamic symbol of object type as a function. Symbols like strncat that are weak aliases of .prologue 1 assembly functions appear as STT_NOTYPE with st_other=0x88 in the dynamic symbol table and were incorrectly classified as data. Add 0x88 to the check so both GP-setup variants are treated as functions. --- scripts/abilist.awk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git ./scripts/abilist.awk ./scripts/abilist.awk index 7ea1edf8c0..39f63d0803 100644 --- ./scripts/abilist.awk +++ ./scripts/abilist.awk @@ -70,8 +70,8 @@ $2 == "g" || $2 == "w" && (NF == 6 || NF == 7 || NF == 8) { type = "O"; seen_opd = 1; } - else if (type == "D" && NF == 8 && $7 == "0x80") { - # Alpha functions avoiding plt entry in users + else if (type == "D" && NF == 8 && ($7 == "0x80" || $7 == "0x88")) { + # Alpha functions: 0x80 = STO_ALPHA_NOPV, 0x88 = STO_ALPHA_STD_GPLOAD type = "F"; size = ""; seen_opd = -1; From patchwork Wed Aug 12 01:19:55 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141159 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 216634B99F5F for ; Wed, 12 Aug 2026 01:26:50 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 216634B99F5F Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=M33vp4Ub X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb12a.google.com (mail-yx1-xb12a.google.com [IPv6:2607:f8b0:4864:20::b12a]) by sourceware.org (Postfix) with ESMTPS id A65B64BAE7E4 for ; Wed, 12 Aug 2026 01:20:42 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A65B64BAE7E4 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org A65B64BAE7E4 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b12a ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497642; cv=none; b=jhWXwHIVWLAAno6q2ejGt2+IXDkHZYgjjqcnu7tSex/lA55YS8WXFRtaQhJ6fygWm87pyXtIDPX2ipy9ig1w44Ni3C2ymAc9gv9dMloRgwRsjnTzWcoKBmwEoUEmeCMFhU48PUkJViz1UseRWUEHAfnzNHqfJhEYKAQr518Ie38= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497642; c=relaxed/simple; bh=miwtazb2/LyhGOFnpq6DKBoLcO/s4EEbLphblSvXQoU=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=PIOvNQ8si8Gmjl8gn4ir2rKt3kPInQ/LNaKVzaanqDu2KHlruBpu2DkZgiV9rlnoHAtNc1w2K3gAuvQTlv+PxAZEtTFVbKYw8+0YNxTRMGV1uILiG7yxRhQTI6BKJYZ/ZKUXJpPaGXb9O8blwfTczQ10s2ZMpiYB+rxSr+RA60c= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=M33vp4Ub DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A65B64BAE7E4 Received: by mail-yx1-xb12a.google.com with SMTP id 956f58d0204a3-66899fe9077so825196d50.1 for ; Tue, 11 Aug 2026 18:20:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497642; x=1787102442; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=7mqABDO/g3Kw31UWtKCcwM7SRBA25GTNc4coKuHKKpc=; b=M33vp4Ub85qZC9dfPp9CtVnMZuJXww8csk9ugqgoHbvPvxKg4KtH7m1jFFehCdfexF wdbnjrjwMWP59gdIZyAhvPHzwVnLuNToCu4dfx1RD5FmRAYk4MWMpGDIYB2MD7PjAlpL 8bV8cA8wc7Xg8SNBS9jidC9I8macZPdEBOYn+yyotEx7291L2voR3v38W5wySA5I7MLW 7N09qjt3gXeB34FQyonxLY4v8v0QVXhLhcX3nlxmlq+GwS8yQTBxj/aRYg8c9zaOFSVS xgG8XxPcTB1IxqpxmEZB3cTLR7WyftmSRWeZHGutjFzLAwWbZrRdGCCe4DXJcLBH2cGp 6hNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497642; x=1787102442; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=7mqABDO/g3Kw31UWtKCcwM7SRBA25GTNc4coKuHKKpc=; b=QJ7HyBaI1NDD5M2pgUdq7oW+WUSop/QhfB9qnRqQLmJDBHTp0yn9Eu6TjPsWGL4C0i 7XhWGrd4KiHXCzoHTsA8CfStYYIVoinTLWUjaPSSHwIMLzAA0/3qFZUj9eWi6b3SCUIR nZwSigNMsfPeXBZWn+XxiuzT3S5Uj8HosS5nVQrnIWTlUmomhDj70+nqybl3ZdDl636/ w/6bPD5u7IuSQBe+ADYIUirDBR36BOoLSw1dpnyb72D7MfUEzkXf05cQUJV5epCWwkZg 7XqUZdL3nhCh9vML9WnufRoc9kGP2dFs38bWxXAcrf8nabEsvUFHmrHhGWPC8RFVDYnO genQ== X-Gm-Message-State: AOJu0YzfgluzlhxsBmOHvGJf7txjtN+LMh6vuzoFotrwr0mwmCsuwUDq 6WEa/rt2FbQxEx8DVIzJSLKlvPYZVyNFEggFfkXF92kyFwWKJN+COZ+URP1kK8LYesE= X-Gm-Gg: AR+sD10ljl5zJ+oN69Auaesu0EGgAY+YxtNuvT7P9MvicAuqC329iAKPhFBkqiof7OC xZru2ugWaIERVOL3S6QGFP24RiLqiDQCvEBS4QZG9uoOR+EO/HZD2woz2J9h1+iSUKVFTSoWK+Q V9zIBSqz3evlfz5yjxxVSM3iqgFaypO2w93HqEHP9DpDT91bJnKQGtbJ7AtYKRJPSJ7kXWs4A/9 qcpNLgWbnuWBY1e4Qu0ARg+DiBVnSen3JNqBQ648dGawhZcQ6a9ONFUsE1llUvDagpUtpIPCmMY /kX6Pk1ZrGbOiCDxamRVOzNNZmTiMuNmGC830bgIMP9QeqhtE1mNCfa4NCl28aLpyDaeyIN8qe8 ZEljrf6oAiOQRJbUxJZT8f8AHQmv9E7Kl7GMoKXQUmy3Xu39FbZXRHSrW4fahG/ZQKyQ80zWli9 HNOPXT7KejFRRTQ0tVdtM0RvAOxWJhKpIWl7yCEIb9vKWXFiWJ2dxdzId+d7s= X-Received: by 2002:a05:690e:190d:b0:668:99f0:92f0 with SMTP id 956f58d0204a3-66b3432e787mr769494d50.48.1786497641821; Tue, 11 Aug 2026 18:20:41 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a953c118sm6350437b3.32.2026.08.11.18.20.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:41 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 18/20] alpha: add IFUNC support to the dynamic linker Date: Tue, 11 Aug 2026 21:19:55 -0400 Message-ID: <46ed5941dbb09cf7b8dc3b0b67c68b904a94052f.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-10.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, KAM_STOCKGEN, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Define R_ALPHA_IRELATIVE (42) and teach the dynamic linker to resolve STT_GNU_IFUNC symbols with it. elf_machine_rela runs the resolver for R_ALPHA_IRELATIVE and stores what it returns, and ELF_MACHINE_IRELATIVE turns on the IRELATIVE pass in elf/do-rel.h. elf_machine_lazy_rel resolves an IFUNC PLT entry eagerly rather than deferring it to the trampoline, which has no way to tell an IFUNC apart from an ordinary lazy binding. A symbolic reference to an IFUNC needs the same treatment: for R_ALPHA_GLOB_DAT, R_ALPHA_JMP_SLOT and R_ALPHA_REFQUAD the address the lookup returns is the resolver, not the implementation it picks, so the resolver has to run there too, as the other targets do. Without it every call through such a symbol enters the resolver and takes the address it returns as the result of the call. With multi-arch enabled that is most of the string functions -- strlen returning an address, memcmp a nonzero constant for equal buffers, math/test-fenv crashing -- and quietly, since the values look like plausible integers. dl-irel.h provides elf_ifunc_invoke and elf_irela for a static binary, where the C library applies the IRELATIVE relocations itself at startup. --- elf/elf.h | 1 + sysdeps/alpha/dl-irel.h | 51 ++++++++++++++++++++++++++++++++++++++ sysdeps/alpha/dl-machine.h | 29 ++++++++++++++++++++++ 3 files changed, 81 insertions(+) create mode 100644 sysdeps/alpha/dl-irel.h diff --git ./elf/elf.h ./elf/elf.h index b482fcfb64..f93215d8e6 100644 --- ./elf/elf.h +++ ./elf/elf.h @@ -2544,6 +2544,7 @@ enum #define R_ALPHA_TPRELHI 39 #define R_ALPHA_TPRELLO 40 #define R_ALPHA_TPREL16 41 +#define R_ALPHA_IRELATIVE 42 /* Keep this the last entry. */ #define R_ALPHA_NUM 46 diff --git ./sysdeps/alpha/dl-irel.h ./sysdeps/alpha/dl-irel.h new file mode 100644 index 0000000000..bbceaf846c --- /dev/null +++ ./sysdeps/alpha/dl-irel.h @@ -0,0 +1,51 @@ +/* Machine-dependent ELF indirect relocation inline functions. + Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _DL_IREL_H +#define _DL_IREL_H + +#include +#include + +#define ELF_MACHINE_IRELA 1 + +static inline ElfW(Addr) +__attribute ((always_inline)) +elf_ifunc_invoke (ElfW(Addr) addr) +{ + return ((ElfW(Addr) (*) (void)) (addr)) (); +} + +static inline void +__attribute ((always_inline)) +elf_irela (const ElfW(Rela) *reloc) +{ + ElfW(Addr) *const reloc_addr = (void *) reloc->r_offset; + const unsigned long int r_type = ELFW(R_TYPE) (reloc->r_info); + + if (__glibc_likely (r_type == R_ALPHA_IRELATIVE)) + { + ElfW(Addr) value = elf_ifunc_invoke (reloc->r_addend); + *reloc_addr = value; + } + else + __libc_fatal ("Unexpected reloc type in static binary.\n"); +} + +#endif /* dl-irel.h */ diff --git ./sysdeps/alpha/dl-machine.h ./sysdeps/alpha/dl-machine.h index 7c500ea850..c357484cf1 100644 --- ./sysdeps/alpha/dl-machine.h +++ ./sysdeps/alpha/dl-machine.h @@ -203,6 +203,9 @@ _dl_start_user: \n\ /* A reloc type used for ld.so cmdline arg lookups to reject PLT entries. */ #define ELF_MACHINE_JMP_SLOT R_ALPHA_JMP_SLOT +/* The IRELATIVE relocation for this architecture. */ +#define ELF_MACHINE_IRELATIVE R_ALPHA_IRELATIVE + /* We define an initialization functions. This is called very early in * _dl_sysdep_start. */ #define DL_PLATFORM_INIT dl_platform_init () @@ -367,6 +370,16 @@ elf_machine_rela (struct link_map *map, struct r_scope_elem *scope[], sym_value += SYMBOL_ADDRESS (sym_map, sym, true); } +#ifndef RTLD_BOOTSTRAP + /* For an IFUNC symbol the address above is the resolver's; the + address to use is what calling it returns. */ + if (sym != NULL + && __glibc_unlikely (ELF64_ST_TYPE (sym->st_info) == STT_GNU_IFUNC) + && __glibc_likely (sym->st_shndx != SHN_UNDEF) + && __glibc_likely (!skip_ifunc)) + sym_value = ((Elf64_Addr (*) (void)) sym_value) (); +#endif + if (r_type == R_ALPHA_GLOB_DAT) *reloc_addr = sym_value; else if (r_type == R_ALPHA_JMP_SLOT) @@ -410,6 +423,15 @@ elf_machine_rela (struct link_map *map, struct r_scope_elem *scope[], } # endif } +#ifndef RTLD_BOOTSTRAP + else if (r_type == R_ALPHA_IRELATIVE) + { + Elf64_Addr value = map->l_addr + reloc->r_addend; + if (__glibc_likely (!skip_ifunc)) + value = ((Elf64_Addr (*) (void)) value) (); + *reloc_addr = value; + } +#endif else _dl_reloc_bad_type (map, r_type, 0); } @@ -451,6 +473,13 @@ elf_machine_lazy_rel (struct link_map *map, struct r_scope_elem *scope[], to the .plt. */ *reloc_addr += l_addr; } + else if (r_type == R_ALPHA_IRELATIVE) + { + Elf64_Addr value = l_addr + reloc->r_addend; + if (__glibc_likely (!skip_ifunc)) + value = ((Elf64_Addr (*) (void)) value) (); + *reloc_addr = value; + } else if (r_type == R_ALPHA_NONE) return; else From patchwork Wed Aug 12 01:19:56 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141162 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B0F9C4BA798D for ; Wed, 12 Aug 2026 01:28:22 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B0F9C4BA798D Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=Arqgw6zu X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yx1-xb134.google.com (mail-yx1-xb134.google.com [IPv6:2607:f8b0:4864:20::b134]) by sourceware.org (Postfix) with ESMTPS id 491BE4BA2E2A for ; Wed, 12 Aug 2026 01:20:44 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 491BE4BA2E2A Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 491BE4BA2E2A Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::b134 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497644; cv=none; b=fL30luiovX9Y7VpGcTim665AyHtUVcuIv7sCFsc6wLEbFYZUY2IES/JW26CEnhIPEhcGHohTW8YUiVvkHCZjqhm0aN2owVipbvMdvzzv0Z+ytKxni7yj8ezbSLqA4aiN9QXxKN8ni7oPI8KWavEtBLRMjW3hQgtQMohnlXqdUvk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497644; c=relaxed/simple; bh=lsMlfnIz7rCt2m6lNn5LmewxS2eVXdxU+qP9ysAhqfI=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=n8X2FgH6PMMCNE+/LNUQJ3ehZzoGdJI106N10OvR5DsKJZoAWd6ih87ie5ggyUicf2ErjzSpniDU0urXxwl0AGJAtOv7Ub/YFh1zywW85h0XK8au6Qj6g+sO73ndz5S7cY4JS+YYtIS7TJ3MrX29/HQS8HTAFHg8+ES+BaoBKAo= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=Arqgw6zu DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 491BE4BA2E2A Received: by mail-yx1-xb134.google.com with SMTP id 956f58d0204a3-6688dfaed31so551106d50.3 for ; Tue, 11 Aug 2026 18:20:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497643; x=1787102443; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qjKk4wmdEkYof68c6nS/8gpiFRkYRKWc0cihd7vTC0Q=; b=Arqgw6zuNpqSXiFyXJaFRp5o9rLneKrphrpdCyBrEnMlNJMPoq7jSNycJCIh2B9RQ3 RJSGZ+Bvin5anYkDZq78FcMRT0yTb9hL3YgEOrN/utm8tv8EmNoNFPiKlNfA58xLjpFm nJOcpzqb+rq+aZ82x5H/2FWyC1C/R5jS9jetjyDDjS7AVBB3PI66kA3V++F7MudTcHPH c0FMkKwYfpKT4/oW8roqTZTQ80pt3x4cS+/t87343COO8pX0XHPkv0c7CP4AUTyepUG5 wRWZS/bJG/iarR5hiKGq5Ia95BlfA0iqTz+YK0HvYZqsyBgypRPe2MjTsmp0cnMPcPOo Qj9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497643; x=1787102443; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=qjKk4wmdEkYof68c6nS/8gpiFRkYRKWc0cihd7vTC0Q=; b=OeJd9+QKamOYzLdpv6eb+UpkrwcHTxU1UZ5MA69NxEd1zDhctrl5r6nB3HFng+kLwW VP/SYF8R4jZSKWu5Rtx1gRExrliFuLu6pHqcKiq7MITlZlYFRd5+Owls2Bpz0an+865F 88qezl6At3huBZJkIYLKxcMQvPT4rWzjC1ci19LaIlNfaEK5324umbA6DUL1o8Sjv30r i3k7QdjUDZw57a5liK0ZA+uQSfA4B68OkY2NtW0JuVje4tsz0vowAQS7fcf0VseB2sWx Ae4twO9u7/+4tY9p6mA71FxsVa/xgdn2AxzhKD9MsuSmm/dZtHGqNnZTlBdSEogLb7TH Avvw== X-Gm-Message-State: AOJu0YzLADX+/yNO6dmUDAcbpf9wwhzDNb7wa7JsPpIm/vww7D94ib1i Y4M+DEsD8DZ7JeLBU4GuA7tTnTH1zApMJ2ciMqTN9hekMh1INL55Cn8aeMuTp/Sv7e0= X-Gm-Gg: AR+sD13fpkKmrVM9jWQ+Vm+MIxEKWj9IJf3pDtcT8DDWivHFSXD5N15GAfeIyuiTQd9 1IYo1rta2QFve6UCbld25Xdc+cuX8UQ5gUsGBYnCNRCdnrt25fpZCyiMenRi3HtzDpXoWDOezvt pVtFGn/IVfmjE8a0s/qPyt3eYbc4Rqy1ZWD2MZuLu2fO34FV3pmSLtn+86e+NVOWn0vH6eXTxP9 ZVr1ZVnQBFvKkp8ytGSqrFA7rKRXhrCo7izVjVVFDXqVulHyUEb8ct90fSR+i/nMRPHbTZqK38r oPMvtJ1HzFGPYtK4P/Y2pD7ymYeNhgs0VynonDKPnrlkwkoA9Fp7KUsmL/oBd9rFWXxiKS0eEnb x25t/4z3XFU4fNOZgZz9vii54jjSYkd55ORH2LXTGAtTI8xYlU7gZ5qrT3TfgSo3yGfSmAxrfG9 yfQuIwzIjuHLVoi4L+TjgVvyMoEu1317ByHjVUh5xVKliNy7nlZRV/ceY0M2c= X-Received: by 2002:a05:690e:d07:b0:669:8524:e733 with SMTP id 956f58d0204a3-66b34309b10mr793933d50.29.1786497643469; Tue, 11 Aug 2026 18:20:43 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-66b1f27d02dsm2073229d50.17.2026.08.11.18.20.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:42 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 19/20] alpha: add multiarch infrastructure for IFUNC dispatch Date: Tue, 11 Aug 2026 21:19:56 -0400 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-11.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Add the framework for runtime CPU-based dispatch on alpha. init-arch.h wraps the implver and amask instructions, which resolvers use directly rather than going through AT_HWCAP: both are unprivileged, so a resolver can ask the processor what it is without help from the kernel. IS_EV6_OR_LATER tests implver for the 21264 family, IS_EV7_OR_LATER for the 21364, and HAS_CIX asks amask for the count extension the 21264A introduced. ifunc-impl-list.c enumerates the implementations for the string tests, which run each one in turn rather than only whichever the resolver picks. The dynamic linker cannot dispatch through an IFUNC before it has relocated itself, so dl-symbol-redir-ifunc.h names an implementation directly for the functions it needs while bootstrapping. Only the static dynamic linker needs the redirection; the shared one has its own copies of these functions and the multiarch variants are not linked into it. configure inserts /multiarch immediately before for each term of the Implies chain, so a multiarch directory shadows only the assembly in its own directory. sysdeps/alpha/multiarch therefore sits after sysdeps/alpha/alphaev6 and sysdeps/alpha/alphaev67 in sysnames, and an --host=alphaev6 or later build picks the CPU assembly over the dispatchers, leaving IFUNC dispatch inert on exactly the configurations that have the tuned implementations. Give the two levels that carry assembly an Implies-only multiarch directory chaining down to sysdeps/alpha/multiarch, as powerpc does for its per-CPU directories, which places the dispatchers ahead of the assembly. A --disable-multi-arch build is unaffected: no multiarch directory enters sysnames and the assembly is used directly. --- sysdeps/alpha/alphaev6/multiarch/Implies | 1 + sysdeps/alpha/alphaev67/multiarch/Implies | 1 + sysdeps/alpha/multiarch/Makefile | 20 ++++++ .../alpha/multiarch/dl-symbol-redir-ifunc.h | 25 +++++++ sysdeps/alpha/multiarch/ifunc-impl-list.c | 64 +++++++++++++++++ sysdeps/alpha/multiarch/init-arch.h | 69 +++++++++++++++++++ 6 files changed, 180 insertions(+) create mode 100644 sysdeps/alpha/alphaev6/multiarch/Implies create mode 100644 sysdeps/alpha/alphaev67/multiarch/Implies create mode 100644 sysdeps/alpha/multiarch/Makefile create mode 100644 sysdeps/alpha/multiarch/dl-symbol-redir-ifunc.h create mode 100644 sysdeps/alpha/multiarch/ifunc-impl-list.c create mode 100644 sysdeps/alpha/multiarch/init-arch.h diff --git ./sysdeps/alpha/alphaev6/multiarch/Implies ./sysdeps/alpha/alphaev6/multiarch/Implies new file mode 100644 index 0000000000..c07181098c --- /dev/null +++ ./sysdeps/alpha/alphaev6/multiarch/Implies @@ -0,0 +1 @@ +alpha/multiarch diff --git ./sysdeps/alpha/alphaev67/multiarch/Implies ./sysdeps/alpha/alphaev67/multiarch/Implies new file mode 100644 index 0000000000..fb91f5988c --- /dev/null +++ ./sysdeps/alpha/alphaev67/multiarch/Implies @@ -0,0 +1 @@ +alpha/alphaev6/multiarch diff --git ./sysdeps/alpha/multiarch/Makefile ./sysdeps/alpha/multiarch/Makefile new file mode 100644 index 0000000000..87b08ec50d --- /dev/null +++ ./sysdeps/alpha/multiarch/Makefile @@ -0,0 +1,20 @@ +ifeq ($(subdir),string) +sysdep_routines += \ + memset_ev5 \ + memset_ev6 \ + rawmemchr_ev5 \ + rawmemchr_ev67 \ + stpcpy_ev5 \ + stpcpy_ev67 \ + stpncpy_ev5 \ + stpncpy_ev67 \ + strcat_ev5 \ + strcat_ev67 \ + strchr_ev5 \ + strchr_ev67 \ + strlen_ev5 \ + strlen_ev67 \ + strncat_ev5 \ + strncat_ev67 \ +# sysdep_routines +endif diff --git ./sysdeps/alpha/multiarch/dl-symbol-redir-ifunc.h ./sysdeps/alpha/multiarch/dl-symbol-redir-ifunc.h new file mode 100644 index 0000000000..9b7f1e258e --- /dev/null +++ ./sysdeps/alpha/multiarch/dl-symbol-redir-ifunc.h @@ -0,0 +1,25 @@ +/* Symbol redirection for loader/static initialization code. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _DL_IFUNC_GENERIC_H +#define _DL_IFUNC_GENERIC_H + +asm ("memset = __memset_ev5"); +asm ("strlen = __strlen_ev5"); + +#endif diff --git ./sysdeps/alpha/multiarch/ifunc-impl-list.c ./sysdeps/alpha/multiarch/ifunc-impl-list.c new file mode 100644 index 0000000000..cede63d975 --- /dev/null +++ ./sysdeps/alpha/multiarch/ifunc-impl-list.c @@ -0,0 +1,64 @@ +/* Enumerate available IFUNC implementations of a function. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include +#include + +size_t +__libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, + size_t max) +{ + size_t i = max; + + IFUNC_IMPL (i, name, memset, + IFUNC_IMPL_ADD (array, i, memset, IS_EV6_OR_LATER, __memset_ev6) + IFUNC_IMPL_ADD (array, i, memset, 1, __memset_ev5)); + + IFUNC_IMPL (i, name, rawmemchr, + IFUNC_IMPL_ADD (array, i, rawmemchr, HAS_CIX, ___rawmemchr_ev67) + IFUNC_IMPL_ADD (array, i, rawmemchr, 1, ___rawmemchr_ev5)); + + IFUNC_IMPL (i, name, stpcpy, + IFUNC_IMPL_ADD (array, i, stpcpy, HAS_CIX, ___stpcpy_ev67) + IFUNC_IMPL_ADD (array, i, stpcpy, 1, ___stpcpy_ev5)); + + IFUNC_IMPL (i, name, stpncpy, + IFUNC_IMPL_ADD (array, i, stpncpy, HAS_CIX, ___stpncpy_ev67) + IFUNC_IMPL_ADD (array, i, stpncpy, 1, ___stpncpy_ev5)); + + IFUNC_IMPL (i, name, strcat, + IFUNC_IMPL_ADD (array, i, strcat, HAS_CIX, __strcat_ev67) + IFUNC_IMPL_ADD (array, i, strcat, 1, __strcat_ev5)); + + IFUNC_IMPL (i, name, strchr, + IFUNC_IMPL_ADD (array, i, strchr, HAS_CIX, __strchr_ev67) + IFUNC_IMPL_ADD (array, i, strchr, 1, __strchr_ev5)); + + IFUNC_IMPL (i, name, strlen, + IFUNC_IMPL_ADD (array, i, strlen, HAS_CIX, __strlen_ev67) + IFUNC_IMPL_ADD (array, i, strlen, 1, __strlen_ev5)); + + IFUNC_IMPL (i, name, strncat, + IFUNC_IMPL_ADD (array, i, strncat, HAS_CIX, __strncat_ev67) + IFUNC_IMPL_ADD (array, i, strncat, 1, __strncat_ev5)); + + return 0; +} diff --git ./sysdeps/alpha/multiarch/init-arch.h ./sysdeps/alpha/multiarch/init-arch.h new file mode 100644 index 0000000000..03d5612e98 --- /dev/null +++ ./sysdeps/alpha/multiarch/init-arch.h @@ -0,0 +1,69 @@ +/* Define IFUNC resolver CPU detection for Alpha. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#ifndef _ALPHA_INIT_ARCH_H +#define _ALPHA_INIT_ARCH_H + +/* implver returns: + 0 = EV4 (21064) + 1 = EV5 (21164) + 2 = EV6 (21264) family (EV6, EV67, EV68) + 3 = EV7 (21364) */ + +static inline unsigned long +__attribute__ ((always_inline)) +alpha_implver (void) +{ + unsigned long implver; + __asm__ ("implver %0" : "=r" (implver)); + return implver; +} + +/* amask tests for optional extensions. Returns zero bits for + supported features. Known bits: + 0x01 = BWX (byte/word extensions, EV56+) + 0x02 = FIX (sqrt/ftoi extensions, EV6+) + 0x04 = CIX (count extensions, EV67+) + 0x08 = MVI (multimedia extensions, EV6+) */ + +static inline unsigned long +__attribute__ ((always_inline)) +alpha_amask (unsigned long mask) +{ + unsigned long result; + __asm__ ("amask %1, %0" : "=r" (result) : "rI" (mask)); + return result; +} + +#define ALPHA_IMPL_EV4 0 +#define ALPHA_IMPL_EV5 1 +#define ALPHA_IMPL_EV6 2 +#define ALPHA_IMPL_EV7 3 + +#define ALPHA_AMASK_BWX 0x01 +#define ALPHA_AMASK_FIX 0x02 +#define ALPHA_AMASK_CIX 0x04 +#define ALPHA_AMASK_MVI 0x08 + +#define IS_EV6_OR_LATER (alpha_implver () >= ALPHA_IMPL_EV6) +#define IS_EV7_OR_LATER (alpha_implver () >= ALPHA_IMPL_EV7) +#define HAS_CIX (!(alpha_amask (ALPHA_AMASK_CIX))) + +#define INIT_ARCH() + +#endif /* _ALPHA_INIT_ARCH_H */ From patchwork Wed Aug 12 01:19:57 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Turner X-Patchwork-Id: 141152 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id BA0E84BA902C for ; Wed, 12 Aug 2026 01:23:21 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BA0E84BA902C Authentication-Results: sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=sJvWCkqT X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-yw1-x1134.google.com (mail-yw1-x1134.google.com [IPv6:2607:f8b0:4864:20::1134]) by sourceware.org (Postfix) with ESMTPS id 5B3FA4B99F58 for ; Wed, 12 Aug 2026 01:20:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 5B3FA4B99F58 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 5B3FA4B99F58 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::1134 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497648; cv=none; b=b9mxEpvHW99blaU5dNj0yzFRHs9BmgE49cdRLhOoLyQf2LoeXCS3GbIMlbdsopUBM21lsrvDyXSXgGYKnNG9Se39AWKdjZMD4V0erJiB0r5KRSuxwyBwxhDUrtTEdTfseB06Tw0VvBZVpXggMqR6AFC/9iTbfLOnp/Q8g5nnFME= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1786497648; c=relaxed/simple; bh=jOJEqOEwnM4WySgyowcbUENwUFDV115Gl1kyAVoqROE=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=FVNXAw0FXhTOEqJYky3XiqxLf7NYIattfqh+Cl7bOd35iWopjQzBGYPEGjBcCC6N9upESa75fVYLMWw7D6UzdY265DGmT2k7UEPKVdUGINPK0AZD26m3kOZmC5RJuTX6KoJZEO01RBWyqEzhYOVW3KHu5s54KKDpwBhqh/8Ugeg= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20251104 header.b=sJvWCkqT DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 5B3FA4B99F58 Received: by mail-yw1-x1134.google.com with SMTP id 00721157ae682-8143daf89c7so5297227b3.1 for ; Tue, 11 Aug 2026 18:20:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1786497648; x=1787102448; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=GQR2v8CSITm3EVu7XRxi8eAnUiXzjPExKPfVw1esjBk=; b=sJvWCkqTtqKnQokPWQKJBooooqWGsR6ofy9l/gHfyuAieX15qgZ/dVHMhfb3OHDGkc w5lZ1KC/aqEsbR6gcX/Sd30vprCRGdJasLJfl/+A0PGqKbi8NaRxK+99mvTRd0feKIQr 1AZ56IqdA55WSnCJDmlynmfrhpdrHCSxA+HGuW3VfLg2wtptJqFTlGZ7m9gUuibb1072 5m16l5ZlIg7FX9fba7qjmdTE6ZLXXpR7zphEiUTskQ8JUMo4sd3EQ52O7hF72XJ6QS89 8hJKG4hxO9Yp+JHMLTLCEDLmv6Icnnnf4xzdphsEopBiNnUBw//ArElvU/WbtNXSjK08 M5Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786497648; x=1787102448; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=GQR2v8CSITm3EVu7XRxi8eAnUiXzjPExKPfVw1esjBk=; b=a2Sd63aXyaJkIyNcSZQCfQJRfOjqkkROQFs6CtzL24ml4LxeaRAMJiE9bHbvztkuDO AD9nVNi5Q9NojXDpxAjdByCaFDNa58FZuSpkf710+iKVi/FXhAqAfZCqPWV+Ndi/MJWS QHug4Payua6rkQfTnKCpH4e+fxMh5pWbtgiH9ND9di8Tv/kGMzsgzXOfjCKrt9TBwBS6 5rb5vqjNSXlZ7x6eyTqVCUUsQVbZpL7/HysNwdwCoO5bEBqrwQ/Iq+1zeSFoA5oXufSS enACJTsF+gNyugaUKTm8wKVzbglbNtSvvZw9HNxts0WlvcPDeoBOMQIqiY1wzFpe3kXK XiLA== X-Gm-Message-State: AOJu0Yw1gfY2p7O28/ZAqiKWMU/cbme9hES8euyOuc4dSHdkkzUNDgqs 4omi09EBjig+I9RU0319vGAXdtF0FTSHijmvAFRV1ugxPuRd0PREqA5GAC1ipjqVFa0= X-Gm-Gg: AR+sD11e5UCDBtgqCs0TTMxgxe647E3zWDjS1CnvOItg9ZjsHolXQPXimJd0JTjCar+ LpPyKtH5/laps83TR5D2h+OrefLPj08dZvXXHS5PjlpSNbXrZwL4X2Baw6w4gv2hBjXPeZBdEyK NcaSTFlt1t3NrY7weJNq+57iX654QankJ/guLNpiMZcmFfikKRHDnJCugoFH7hBW5Rpuus2lZfk lO7xmbVboBoYdvCoaBIekGblYlP74VANMj3tICX37OLcz0YqTBy4yVfstRPJ/y7ZfRXLm8T9zYE u2tJs6YDNoG/ObDN3e4J4BGz4wjaP4sC7uFT/MZgJSN0KYYAf/DyNj6OwRQALi2EbbrSPbYp5Ar JFNRfaV5YVbFtE8rkxB9Tefmq0k5/BfKEBhVTc5yk5aUQg4PlXQnQAcsigbVCCzmEAKckSX+WQ8 NFy7MOmr9UzP3eUhgHe4IzVo8C79yB3Hrueoa2VnY3BpmJfyrXl/3xypAmJ/Q= X-Received: by 2002:a05:690c:4512:b0:81d:e7d2:a46d with SMTP id 00721157ae682-83105e02b9fmr11367327b3.5.1786497646376; Tue, 11 Aug 2026 18:20:46 -0700 (PDT) Received: from localhost ([2600:1702:7a90:6f9f:482:39fe:7891:c084]) by smtp.gmail.com with ESMTPSA id 00721157ae682-830a41420d4sm6644857b3.12.2026.08.11.18.20.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Aug 2026 18:20:45 -0700 (PDT) From: Matt Turner To: libc-alpha@sourceware.org Cc: Magnus Lindholm , Michael Cree Subject: [PATCH 20/20] alpha: add multiarch IFUNC dispatchers Date: Tue, 11 Aug 2026 21:19:57 -0400 Message-ID: <005b561d639ed74c60d570e71ca06aa2a90a1232.1786497497.git.mattst88@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-11.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org Dispatch at runtime between the generic implementations and the EV6/EV67-optimized ones for twelve functions: memset: generic (EV4/EV5) vs EV6 assembly (dispatch on implver) memcpy: generic (C) vs EV6 assembly (dispatch on implver) memcmp: generic (C) vs EV6 assembly (dispatch on implver) memmove: generic (C) vs EV6 assembly (dispatch on implver) mempcpy: generic (C) vs EV6 assembly (dispatch on implver) strlen: generic vs EV67 CIX assembly (dispatch on amask CIX) stpcpy: generic vs EV67 CIX assembly stpncpy: generic vs EV67 CIX assembly strcat: generic vs EV67 CIX assembly strchr: generic vs EV67 CIX assembly rawmemchr: generic vs EV67 CIX assembly strncat: generic vs EV67 CIX assembly Each function gets a resolver in a .c file and two assembly wrappers (_ev5/_ev6 or _ev5/_ev67) that include the original source with renamed entry points. The mem* functions have no generic alpha assembly, so they use C generic wrappers for EV4/EV5; the EV6 mempcpy wrapper includes alphaev6/memcpy.S with USE_AS_MEMPCPY. The internal aliases are defined in the dispatchers, against the redirected type, so that calls from inside libc go through the IFUNC too. Defining them in the _ev5 variants instead, as the assembly they include does, would point __GI_strlen and the rest at the ev5 implementation and every caller inside libc would get that one whatever the CPU. string.h needs care in these files. It defines __mempcpy and __stpcpy as macros calling the corresponding builtin and, in the static library, redirects those builtins back to the __ names, which collides with the redirection the multiarch files set up; they define __NO_STRING_INLINES and NO_MEMPCPY_STPCPY_REDIRECT first, as the x86_64 and s390 versions do. mempcpy_generic.c names its function by setting MEMPCPY rather than by defining __mempcpy, which string/mempcpy.c undefines. __memcmpeq is aliased to the IFUNC alongside memcmp and bcmp. string/memcmp.c defines it as an alias of memcmp, but memcmp_generic.c neutralizes strong_alias so that the generic implementation does not bring the public names with it, and without the alias here the symbol would disappear from libc: an ABI break, since it is in the abilist, and a link failure for the tests that use it. With --enable-multi-arch a single binary selects the implementation the processor wants at load time. memcmp_ev6.S turns off strong_alias and libc_hidden_def as well as weak_alias, because the assembly it includes now defines the __memcmpeq alias and memcmp.c defines it for the dispatcher; without this both land in libc_pic.a and the link fails with a multiple definition of __memcmpeq. --- sysdeps/alpha/multiarch/Makefile | 8 +++ .../alpha/multiarch/dl-symbol-redir-ifunc.h | 12 ++++ sysdeps/alpha/multiarch/ifunc-impl-list.c | 16 +++++ sysdeps/alpha/multiarch/memcmp.c | 60 ++++++++++++++++++ sysdeps/alpha/multiarch/memcmp_ev6.S | 41 ++++++++++++ sysdeps/alpha/multiarch/memcmp_generic.c | 35 +++++++++++ sysdeps/alpha/multiarch/memcpy.c | 49 +++++++++++++++ sysdeps/alpha/multiarch/memcpy_ev6.S | 32 ++++++++++ sysdeps/alpha/multiarch/memcpy_generic.c | 26 ++++++++ sysdeps/alpha/multiarch/memmove.c | 49 +++++++++++++++ sysdeps/alpha/multiarch/memmove_ev6.S | 32 ++++++++++ sysdeps/alpha/multiarch/memmove_generic.c | 26 ++++++++ sysdeps/alpha/multiarch/mempcpy.c | 61 ++++++++++++++++++ sysdeps/alpha/multiarch/mempcpy_ev6.S | 39 ++++++++++++ sysdeps/alpha/multiarch/mempcpy_generic.c | 39 ++++++++++++ sysdeps/alpha/multiarch/memset.c | 49 +++++++++++++++ sysdeps/alpha/multiarch/memset_ev5.S | 33 ++++++++++ sysdeps/alpha/multiarch/memset_ev6.S | 32 ++++++++++ sysdeps/alpha/multiarch/rawmemchr.c | 55 ++++++++++++++++ sysdeps/alpha/multiarch/rawmemchr_ev5.S | 36 +++++++++++ sysdeps/alpha/multiarch/rawmemchr_ev67.S | 35 +++++++++++ sysdeps/alpha/multiarch/stpcpy.c | 62 +++++++++++++++++++ sysdeps/alpha/multiarch/stpcpy_ev5.S | 39 ++++++++++++ sysdeps/alpha/multiarch/stpcpy_ev67.S | 38 ++++++++++++ sysdeps/alpha/multiarch/stpncpy.c | 55 ++++++++++++++++ sysdeps/alpha/multiarch/stpncpy_ev5.S | 36 +++++++++++ sysdeps/alpha/multiarch/stpncpy_ev67.S | 35 +++++++++++ sysdeps/alpha/multiarch/strcat.c | 49 +++++++++++++++ sysdeps/alpha/multiarch/strcat_ev5.S | 33 ++++++++++ sysdeps/alpha/multiarch/strcat_ev67.S | 32 ++++++++++ sysdeps/alpha/multiarch/strchr.c | 50 +++++++++++++++ sysdeps/alpha/multiarch/strchr_ev5.S | 36 +++++++++++ sysdeps/alpha/multiarch/strchr_ev67.S | 35 +++++++++++ sysdeps/alpha/multiarch/strlen.c | 49 +++++++++++++++ sysdeps/alpha/multiarch/strlen_ev5.S | 33 ++++++++++ sysdeps/alpha/multiarch/strlen_ev67.S | 32 ++++++++++ sysdeps/alpha/multiarch/strncat.c | 49 +++++++++++++++ sysdeps/alpha/multiarch/strncat_ev5.S | 30 +++++++++ sysdeps/alpha/multiarch/strncat_ev67.S | 29 +++++++++ 39 files changed, 1487 insertions(+) create mode 100644 sysdeps/alpha/multiarch/memcmp.c create mode 100644 sysdeps/alpha/multiarch/memcmp_ev6.S create mode 100644 sysdeps/alpha/multiarch/memcmp_generic.c create mode 100644 sysdeps/alpha/multiarch/memcpy.c create mode 100644 sysdeps/alpha/multiarch/memcpy_ev6.S create mode 100644 sysdeps/alpha/multiarch/memcpy_generic.c create mode 100644 sysdeps/alpha/multiarch/memmove.c create mode 100644 sysdeps/alpha/multiarch/memmove_ev6.S create mode 100644 sysdeps/alpha/multiarch/memmove_generic.c create mode 100644 sysdeps/alpha/multiarch/mempcpy.c create mode 100644 sysdeps/alpha/multiarch/mempcpy_ev6.S create mode 100644 sysdeps/alpha/multiarch/mempcpy_generic.c create mode 100644 sysdeps/alpha/multiarch/memset.c create mode 100644 sysdeps/alpha/multiarch/memset_ev5.S create mode 100644 sysdeps/alpha/multiarch/memset_ev6.S create mode 100644 sysdeps/alpha/multiarch/rawmemchr.c create mode 100644 sysdeps/alpha/multiarch/rawmemchr_ev5.S create mode 100644 sysdeps/alpha/multiarch/rawmemchr_ev67.S create mode 100644 sysdeps/alpha/multiarch/stpcpy.c create mode 100644 sysdeps/alpha/multiarch/stpcpy_ev5.S create mode 100644 sysdeps/alpha/multiarch/stpcpy_ev67.S create mode 100644 sysdeps/alpha/multiarch/stpncpy.c create mode 100644 sysdeps/alpha/multiarch/stpncpy_ev5.S create mode 100644 sysdeps/alpha/multiarch/stpncpy_ev67.S create mode 100644 sysdeps/alpha/multiarch/strcat.c create mode 100644 sysdeps/alpha/multiarch/strcat_ev5.S create mode 100644 sysdeps/alpha/multiarch/strcat_ev67.S create mode 100644 sysdeps/alpha/multiarch/strchr.c create mode 100644 sysdeps/alpha/multiarch/strchr_ev5.S create mode 100644 sysdeps/alpha/multiarch/strchr_ev67.S create mode 100644 sysdeps/alpha/multiarch/strlen.c create mode 100644 sysdeps/alpha/multiarch/strlen_ev5.S create mode 100644 sysdeps/alpha/multiarch/strlen_ev67.S create mode 100644 sysdeps/alpha/multiarch/strncat.c create mode 100644 sysdeps/alpha/multiarch/strncat_ev5.S create mode 100644 sysdeps/alpha/multiarch/strncat_ev67.S diff --git ./sysdeps/alpha/multiarch/Makefile ./sysdeps/alpha/multiarch/Makefile index 87b08ec50d..09acc02716 100644 --- ./sysdeps/alpha/multiarch/Makefile +++ ./sysdeps/alpha/multiarch/Makefile @@ -1,5 +1,13 @@ ifeq ($(subdir),string) sysdep_routines += \ + memcmp_ev6 \ + memcmp_generic \ + memcpy_ev6 \ + memcpy_generic \ + memmove_ev6 \ + memmove_generic \ + mempcpy_ev6 \ + mempcpy_generic \ memset_ev5 \ memset_ev6 \ rawmemchr_ev5 \ diff --git ./sysdeps/alpha/multiarch/dl-symbol-redir-ifunc.h ./sysdeps/alpha/multiarch/dl-symbol-redir-ifunc.h index 9b7f1e258e..f15675748b 100644 --- ./sysdeps/alpha/multiarch/dl-symbol-redir-ifunc.h +++ ./sysdeps/alpha/multiarch/dl-symbol-redir-ifunc.h @@ -19,7 +19,19 @@ #ifndef _DL_IFUNC_GENERIC_H #define _DL_IFUNC_GENERIC_H +/* Only the static dynamic linker needs these. The shared one already has + its own copies of these functions, and redirecting them here would + refer to the multiarch variants, which are not linked into it. */ +#ifndef SHARED + +asm ("memcmp = __memcmp_generic"); +asm ("memcpy = __memcpy_generic"); +asm ("memmove = __memmove_generic"); +asm ("mempcpy = ___mempcpy_generic"); +asm ("__mempcpy = ___mempcpy_generic"); asm ("memset = __memset_ev5"); asm ("strlen = __strlen_ev5"); +#endif /* !SHARED */ + #endif diff --git ./sysdeps/alpha/multiarch/ifunc-impl-list.c ./sysdeps/alpha/multiarch/ifunc-impl-list.c index cede63d975..2df1eaf413 100644 --- ./sysdeps/alpha/multiarch/ifunc-impl-list.c +++ ./sysdeps/alpha/multiarch/ifunc-impl-list.c @@ -28,6 +28,22 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, { size_t i = max; + IFUNC_IMPL (i, name, memcmp, + IFUNC_IMPL_ADD (array, i, memcmp, IS_EV6_OR_LATER, __memcmp_ev6) + IFUNC_IMPL_ADD (array, i, memcmp, 1, __memcmp_generic)); + + IFUNC_IMPL (i, name, memcpy, + IFUNC_IMPL_ADD (array, i, memcpy, IS_EV6_OR_LATER, __memcpy_ev6) + IFUNC_IMPL_ADD (array, i, memcpy, 1, __memcpy_generic)); + + IFUNC_IMPL (i, name, memmove, + IFUNC_IMPL_ADD (array, i, memmove, IS_EV6_OR_LATER, __memmove_ev6) + IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_generic)); + + IFUNC_IMPL (i, name, mempcpy, + IFUNC_IMPL_ADD (array, i, mempcpy, IS_EV6_OR_LATER, ___mempcpy_ev6) + IFUNC_IMPL_ADD (array, i, mempcpy, 1, ___mempcpy_generic)); + IFUNC_IMPL (i, name, memset, IFUNC_IMPL_ADD (array, i, memset, IS_EV6_OR_LATER, __memset_ev6) IFUNC_IMPL_ADD (array, i, memset, 1, __memset_ev5)); diff --git ./sysdeps/alpha/multiarch/memcmp.c ./sysdeps/alpha/multiarch/memcmp.c new file mode 100644 index 0000000000..458b61eef9 --- /dev/null +++ ./sysdeps/alpha/multiarch/memcmp.c @@ -0,0 +1,60 @@ +/* Multiple versions of memcmp. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef memcmp +# define memcmp __redirect_memcmp +# undef __memcmpeq +# define __memcmpeq __redirect___memcmpeq +# include +# include + +extern __typeof (__redirect_memcmp) __libc_memcmp; + +extern __typeof (__redirect_memcmp) __memcmp_generic attribute_hidden; +extern __typeof (__redirect_memcmp) __memcmp_ev6 attribute_hidden; + +static inline __typeof (__redirect_memcmp) * +select_memcmp_ifunc (void) +{ + if (IS_EV6_OR_LATER) + return __memcmp_ev6; + + return __memcmp_generic; +} + +libc_ifunc (__libc_memcmp, select_memcmp_ifunc ()); + +# undef memcmp +# undef __memcmpeq +strong_alias (__libc_memcmp, memcmp); +weak_alias (memcmp, bcmp); +/* string/memcmp.c aliases __memcmpeq to memcmp, but memcmp_generic.c turns + strong_alias off, so nothing would define it here. */ +strong_alias (__libc_memcmp, __memcmpeq); +# ifdef SHARED +__hidden_ver1 (__memcmpeq, __GI___memcmpeq, __redirect___memcmpeq) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memcmp); +# endif +/* Internal callers use __GI_memcmp, which libc_hidden_builtin_def would + normally define; the name is redirected here, so define it by hand. */ +# ifdef SHARED +__hidden_ver1 (__libc_memcmp, __GI_memcmp, __redirect_memcmp) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memcmp); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/memcmp_ev6.S ./sysdeps/alpha/multiarch/memcmp_ev6.S new file mode 100644 index 0000000000..23ca99fd7f --- /dev/null +++ ./sysdeps/alpha/multiarch/memcmp_ev6.S @@ -0,0 +1,41 @@ +/* memcmp for Alpha, EV6 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define memcmp __memcmp_ev6 + +# undef weak_alias +# define weak_alias(a, b) + +/* memcmp.c defines the bcmp and __memcmpeq aliases for the dispatcher. */ +# undef strong_alias +# define strong_alias(a, b) + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +# undef libc_hidden_def +# define libc_hidden_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (memcmp, __memcmp_ev6) +#endif diff --git ./sysdeps/alpha/multiarch/memcmp_generic.c ./sysdeps/alpha/multiarch/memcmp_generic.c new file mode 100644 index 0000000000..65a17313f3 --- /dev/null +++ ./sysdeps/alpha/multiarch/memcmp_generic.c @@ -0,0 +1,35 @@ +/* memcmp for Alpha, generic C version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# define MEMCMP __memcmp_generic + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) + +# undef weak_alias +# define weak_alias(a, b) + +# undef strong_alias +# define strong_alias(a, b) + +# undef libc_hidden_def +# define libc_hidden_def(name) +#endif + +#include diff --git ./sysdeps/alpha/multiarch/memcpy.c ./sysdeps/alpha/multiarch/memcpy.c new file mode 100644 index 0000000000..2592706ede --- /dev/null +++ ./sysdeps/alpha/multiarch/memcpy.c @@ -0,0 +1,49 @@ +/* Multiple versions of memcpy. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef memcpy +# define memcpy __redirect_memcpy +# include +# include + +extern __typeof (__redirect_memcpy) __libc_memcpy; + +extern __typeof (__redirect_memcpy) __memcpy_generic attribute_hidden; +extern __typeof (__redirect_memcpy) __memcpy_ev6 attribute_hidden; + +static inline __typeof (__redirect_memcpy) * +select_memcpy_ifunc (void) +{ + if (IS_EV6_OR_LATER) + return __memcpy_ev6; + + return __memcpy_generic; +} + +libc_ifunc (__libc_memcpy, select_memcpy_ifunc ()); + +# undef memcpy +strong_alias (__libc_memcpy, memcpy); +/* Internal callers use __GI_memcpy, which libc_hidden_builtin_def would + normally define; the name is redirected here, so define it by hand. */ +# ifdef SHARED +__hidden_ver1 (__libc_memcpy, __GI_memcpy, __redirect_memcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memcpy); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/memcpy_ev6.S ./sysdeps/alpha/multiarch/memcpy_ev6.S new file mode 100644 index 0000000000..0f254e402c --- /dev/null +++ ./sysdeps/alpha/multiarch/memcpy_ev6.S @@ -0,0 +1,32 @@ +/* memcpy for Alpha, EV6 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define memcpy __memcpy_ev6 + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (memcpy, __memcpy_ev6) +#endif diff --git ./sysdeps/alpha/multiarch/memcpy_generic.c ./sysdeps/alpha/multiarch/memcpy_generic.c new file mode 100644 index 0000000000..1c5e3a96ee --- /dev/null +++ ./sysdeps/alpha/multiarch/memcpy_generic.c @@ -0,0 +1,26 @@ +/* memcpy for Alpha, generic C version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# define MEMCPY __memcpy_generic + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include diff --git ./sysdeps/alpha/multiarch/memmove.c ./sysdeps/alpha/multiarch/memmove.c new file mode 100644 index 0000000000..10b9f38b5e --- /dev/null +++ ./sysdeps/alpha/multiarch/memmove.c @@ -0,0 +1,49 @@ +/* Multiple versions of memmove. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef memmove +# define memmove __redirect_memmove +# include +# include + +extern __typeof (__redirect_memmove) __libc_memmove; + +extern __typeof (__redirect_memmove) __memmove_generic attribute_hidden; +extern __typeof (__redirect_memmove) __memmove_ev6 attribute_hidden; + +static inline __typeof (__redirect_memmove) * +select_memmove_ifunc (void) +{ + if (IS_EV6_OR_LATER) + return __memmove_ev6; + + return __memmove_generic; +} + +libc_ifunc (__libc_memmove, select_memmove_ifunc ()); + +# undef memmove +strong_alias (__libc_memmove, memmove); +/* Internal callers use __GI_memmove, which libc_hidden_builtin_def would + normally define; the name is redirected here, so define it by hand. */ +# ifdef SHARED +__hidden_ver1 (__libc_memmove, __GI_memmove, __redirect_memmove) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memmove); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/memmove_ev6.S ./sysdeps/alpha/multiarch/memmove_ev6.S new file mode 100644 index 0000000000..2130f8fb24 --- /dev/null +++ ./sysdeps/alpha/multiarch/memmove_ev6.S @@ -0,0 +1,32 @@ +/* memmove for Alpha, EV6 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define memmove __memmove_ev6 + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (memmove, __memmove_ev6) +#endif diff --git ./sysdeps/alpha/multiarch/memmove_generic.c ./sysdeps/alpha/multiarch/memmove_generic.c new file mode 100644 index 0000000000..cf5c962868 --- /dev/null +++ ./sysdeps/alpha/multiarch/memmove_generic.c @@ -0,0 +1,26 @@ +/* memmove for Alpha, generic C version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# define MEMMOVE __memmove_generic + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include diff --git ./sysdeps/alpha/multiarch/mempcpy.c ./sysdeps/alpha/multiarch/mempcpy.c new file mode 100644 index 0000000000..a68b08730b --- /dev/null +++ ./sysdeps/alpha/multiarch/mempcpy.c @@ -0,0 +1,61 @@ +/* Multiple versions of mempcpy. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef mempcpy +# define mempcpy __redirect_mempcpy +# undef __mempcpy +# define __mempcpy __redirect___mempcpy +/* string.h defines __mempcpy as a macro calling __builtin_mempcpy, and + redirects __builtin_mempcpy back to __mempcpy in the static library; + both get in the way of redirecting the name here. */ +# define __NO_STRING_INLINES +# define NO_MEMPCPY_STPCPY_REDIRECT +# include +# include + +extern __typeof (__redirect___mempcpy) __libc___mempcpy; + +extern __typeof (__redirect___mempcpy) ___mempcpy_generic attribute_hidden; +extern __typeof (__redirect___mempcpy) ___mempcpy_ev6 attribute_hidden; + +static inline __typeof (__redirect___mempcpy) * +select_mempcpy_ifunc (void) +{ + if (IS_EV6_OR_LATER) + return ___mempcpy_ev6; + + return ___mempcpy_generic; +} + +libc_ifunc (__libc___mempcpy, select_mempcpy_ifunc ()); + +# undef mempcpy +# undef __mempcpy +strong_alias (__libc___mempcpy, __mempcpy); +weak_alias (__mempcpy, mempcpy); +/* string.h declared the hidden prototype under the redirected name, so + libc_hidden_def would alias to a __GI_ symbol that is never defined. + Define it here against the redirected type instead. */ +# ifdef SHARED +__hidden_ver1 (__mempcpy, __GI___mempcpy, __redirect___mempcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (mempcpy); +__hidden_ver1 (__mempcpy, __GI_mempcpy, __redirect_mempcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (mempcpy); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/mempcpy_ev6.S ./sysdeps/alpha/multiarch/mempcpy_ev6.S new file mode 100644 index 0000000000..6e58e9caad --- /dev/null +++ ./sysdeps/alpha/multiarch/mempcpy_ev6.S @@ -0,0 +1,39 @@ +/* mempcpy for Alpha, EV6 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define USE_AS_MEMPCPY +# define memcpy ___mempcpy_ev6 + +# undef libc_hidden_def +# define libc_hidden_def(name) + +# undef weak_alias +# define weak_alias(a, b) + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (__mempcpy, ___mempcpy_ev6) +#endif diff --git ./sysdeps/alpha/multiarch/mempcpy_generic.c ./sysdeps/alpha/multiarch/mempcpy_generic.c new file mode 100644 index 0000000000..de34752af8 --- /dev/null +++ ./sysdeps/alpha/multiarch/mempcpy_generic.c @@ -0,0 +1,39 @@ +/* mempcpy for Alpha, generic C version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +/* Name the function through MEMPCPY rather than by defining __mempcpy: + string/mempcpy.c includes string.h, which defines __mempcpy as a macro + of its own, and then undefines it. */ +# define MEMPCPY ___mempcpy_generic + +# undef libc_hidden_def +# define libc_hidden_def(name) + +# undef weak_alias +# define weak_alias(a, b) + +/* The aliases below name __mempcpy, which this file does not define. */ +# undef static_weak_alias +# define static_weak_alias(a, b) + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include diff --git ./sysdeps/alpha/multiarch/memset.c ./sysdeps/alpha/multiarch/memset.c new file mode 100644 index 0000000000..7dc4c3f52f --- /dev/null +++ ./sysdeps/alpha/multiarch/memset.c @@ -0,0 +1,49 @@ +/* Multiple versions of memset. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef memset +# define memset __redirect_memset +# include +# include + +extern __typeof (__redirect_memset) __libc_memset; + +extern __typeof (__redirect_memset) __memset_ev5 attribute_hidden; +extern __typeof (__redirect_memset) __memset_ev6 attribute_hidden; + +static inline __typeof (__redirect_memset) * +select_memset_ifunc (void) +{ + if (IS_EV6_OR_LATER) + return __memset_ev6; + + return __memset_ev5; +} + +libc_ifunc (__libc_memset, select_memset_ifunc ()); + +# undef memset +strong_alias (__libc_memset, memset); +/* Internal callers reach this through __GI_memset. Alias it to the IFUNC + so that they get the implementation chosen for this CPU too. */ +# ifdef SHARED +__hidden_ver1 (__libc_memset, __GI_memset, __redirect_memset) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memset); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/memset_ev5.S ./sysdeps/alpha/multiarch/memset_ev5.S new file mode 100644 index 0000000000..d6844d086a --- /dev/null +++ ./sysdeps/alpha/multiarch/memset_ev5.S @@ -0,0 +1,33 @@ +/* memset for Alpha, EV4/EV5 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define memset __memset_ev5 + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) + +#endif + +#include + +#if IS_IN (rtld) +strong_alias (memset, __memset_ev5) +#endif diff --git ./sysdeps/alpha/multiarch/memset_ev6.S ./sysdeps/alpha/multiarch/memset_ev6.S new file mode 100644 index 0000000000..62550ea2cc --- /dev/null +++ ./sysdeps/alpha/multiarch/memset_ev6.S @@ -0,0 +1,32 @@ +/* memset for Alpha, EV6 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define memset __memset_ev6 + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (memset, __memset_ev6) +#endif diff --git ./sysdeps/alpha/multiarch/rawmemchr.c ./sysdeps/alpha/multiarch/rawmemchr.c new file mode 100644 index 0000000000..0235d5c7c3 --- /dev/null +++ ./sysdeps/alpha/multiarch/rawmemchr.c @@ -0,0 +1,55 @@ +/* Multiple versions of rawmemchr. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef rawmemchr +# define rawmemchr __redirect_rawmemchr +# undef __rawmemchr +# define __rawmemchr __redirect___rawmemchr +# include +# include + +extern __typeof (__redirect___rawmemchr) __libc___rawmemchr; + +extern __typeof (__redirect___rawmemchr) ___rawmemchr_ev5 attribute_hidden; +extern __typeof (__redirect___rawmemchr) ___rawmemchr_ev67 attribute_hidden; + +static inline __typeof (__redirect___rawmemchr) * +select_rawmemchr_ifunc (void) +{ + if (HAS_CIX) + return ___rawmemchr_ev67; + + return ___rawmemchr_ev5; +} + +libc_ifunc (__libc___rawmemchr, select_rawmemchr_ifunc ()); + +# undef rawmemchr +# undef __rawmemchr +strong_alias (__libc___rawmemchr, __rawmemchr); +weak_alias (__rawmemchr, rawmemchr); +/* Not libc_hidden_def: string.h declared the hidden prototype under the + redirected name, so that would alias to a __GI_ symbol nothing defines. + Alias it to the IFUNC so internal callers also get the implementation + chosen for this CPU. */ +# ifdef SHARED +__hidden_ver1 (__rawmemchr, __GI___rawmemchr, __redirect___rawmemchr) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (rawmemchr); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/rawmemchr_ev5.S ./sysdeps/alpha/multiarch/rawmemchr_ev5.S new file mode 100644 index 0000000000..8bbc83a525 --- /dev/null +++ ./sysdeps/alpha/multiarch/rawmemchr_ev5.S @@ -0,0 +1,36 @@ +/* rawmemchr for Alpha, generic version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define __rawmemchr ___rawmemchr_ev5 + +# undef weak_alias +# define weak_alias(a, b) + +# undef libc_hidden_def +# define libc_hidden_def(name) + +#endif + +#include + +#if IS_IN (rtld) +strong_alias (__rawmemchr, ___rawmemchr_ev5) +#endif diff --git ./sysdeps/alpha/multiarch/rawmemchr_ev67.S ./sysdeps/alpha/multiarch/rawmemchr_ev67.S new file mode 100644 index 0000000000..eb73d77712 --- /dev/null +++ ./sysdeps/alpha/multiarch/rawmemchr_ev67.S @@ -0,0 +1,35 @@ +/* rawmemchr for Alpha, EV67 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define __rawmemchr ___rawmemchr_ev67 + +# undef weak_alias +# define weak_alias(a, b) + +# undef libc_hidden_def +# define libc_hidden_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (__rawmemchr, ___rawmemchr_ev67) +#endif diff --git ./sysdeps/alpha/multiarch/stpcpy.c ./sysdeps/alpha/multiarch/stpcpy.c new file mode 100644 index 0000000000..de4c3ee38b --- /dev/null +++ ./sysdeps/alpha/multiarch/stpcpy.c @@ -0,0 +1,62 @@ +/* Multiple versions of stpcpy. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef stpcpy +# define stpcpy __redirect_stpcpy +# undef __stpcpy +# define __stpcpy __redirect___stpcpy +/* string.h defines __stpcpy as a macro calling __builtin_stpcpy, and + redirects __builtin_stpcpy back to __stpcpy in the static library; + both get in the way of redirecting the name here. */ +# define __NO_STRING_INLINES +# define NO_MEMPCPY_STPCPY_REDIRECT +# include +# include + +extern __typeof (__redirect___stpcpy) __libc___stpcpy; + +extern __typeof (__redirect___stpcpy) ___stpcpy_ev5 attribute_hidden; +extern __typeof (__redirect___stpcpy) ___stpcpy_ev67 attribute_hidden; + +static inline __typeof (__redirect___stpcpy) * +select_stpcpy_ifunc (void) +{ + if (HAS_CIX) + return ___stpcpy_ev67; + + return ___stpcpy_ev5; +} + +libc_ifunc (__libc___stpcpy, select_stpcpy_ifunc ()); + +# undef stpcpy +# undef __stpcpy +strong_alias (__libc___stpcpy, __stpcpy); +weak_alias (__stpcpy, stpcpy); +/* Not libc_hidden_def: string.h declared the hidden prototype under the + redirected name, so that would alias to a __GI_ symbol nothing defines. + Alias both internal names to the IFUNC so internal callers also get the + implementation chosen for this CPU. */ +# ifdef SHARED +__hidden_ver1 (__stpcpy, __GI___stpcpy, __redirect___stpcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (stpcpy); +__hidden_ver1 (__stpcpy, __GI_stpcpy, __redirect_stpcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (stpcpy); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/stpcpy_ev5.S ./sysdeps/alpha/multiarch/stpcpy_ev5.S new file mode 100644 index 0000000000..9525d46a8d --- /dev/null +++ ./sysdeps/alpha/multiarch/stpcpy_ev5.S @@ -0,0 +1,39 @@ +/* stpcpy for Alpha, generic version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define __stpcpy ___stpcpy_ev5 + +# undef weak_alias +# define weak_alias(a, b) + +# undef libc_hidden_def +# define libc_hidden_def(name) + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) + +#endif + +#include + +#if IS_IN (rtld) +strong_alias (__stpcpy, ___stpcpy_ev5) +#endif diff --git ./sysdeps/alpha/multiarch/stpcpy_ev67.S ./sysdeps/alpha/multiarch/stpcpy_ev67.S new file mode 100644 index 0000000000..fd145d3264 --- /dev/null +++ ./sysdeps/alpha/multiarch/stpcpy_ev67.S @@ -0,0 +1,38 @@ +/* stpcpy for Alpha, EV67 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define __stpcpy ___stpcpy_ev67 + +# undef weak_alias +# define weak_alias(a, b) + +# undef libc_hidden_def +# define libc_hidden_def(name) + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (__stpcpy, ___stpcpy_ev67) +#endif diff --git ./sysdeps/alpha/multiarch/stpncpy.c ./sysdeps/alpha/multiarch/stpncpy.c new file mode 100644 index 0000000000..e6d58a4754 --- /dev/null +++ ./sysdeps/alpha/multiarch/stpncpy.c @@ -0,0 +1,55 @@ +/* Multiple versions of stpncpy. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef stpncpy +# define stpncpy __redirect_stpncpy +# undef __stpncpy +# define __stpncpy __redirect___stpncpy +# include +# include + +extern __typeof (__redirect___stpncpy) __libc___stpncpy; + +extern __typeof (__redirect___stpncpy) ___stpncpy_ev5 attribute_hidden; +extern __typeof (__redirect___stpncpy) ___stpncpy_ev67 attribute_hidden; + +static inline __typeof (__redirect___stpncpy) * +select_stpncpy_ifunc (void) +{ + if (HAS_CIX) + return ___stpncpy_ev67; + + return ___stpncpy_ev5; +} + +libc_ifunc (__libc___stpncpy, select_stpncpy_ifunc ()); + +# undef stpncpy +# undef __stpncpy +strong_alias (__libc___stpncpy, __stpncpy); +weak_alias (__stpncpy, stpncpy); +/* Not libc_hidden_def: string.h declared the hidden prototype under the + redirected name, so that would alias to a __GI_ symbol nothing defines. + Alias it to the IFUNC so internal callers also get the implementation + chosen for this CPU. */ +# ifdef SHARED +__hidden_ver1 (__stpncpy, __GI___stpncpy, __redirect___stpncpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (stpncpy); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/stpncpy_ev5.S ./sysdeps/alpha/multiarch/stpncpy_ev5.S new file mode 100644 index 0000000000..e9995a517e --- /dev/null +++ ./sysdeps/alpha/multiarch/stpncpy_ev5.S @@ -0,0 +1,36 @@ +/* stpncpy for Alpha, generic version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define __stpncpy ___stpncpy_ev5 + +# undef weak_alias +# define weak_alias(a, b) + +# undef libc_hidden_def +# define libc_hidden_def(name) + +#endif + +#include + +#if IS_IN (rtld) +strong_alias (__stpncpy, ___stpncpy_ev5) +#endif diff --git ./sysdeps/alpha/multiarch/stpncpy_ev67.S ./sysdeps/alpha/multiarch/stpncpy_ev67.S new file mode 100644 index 0000000000..bb5ff45d6a --- /dev/null +++ ./sysdeps/alpha/multiarch/stpncpy_ev67.S @@ -0,0 +1,35 @@ +/* stpncpy for Alpha, EV67 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define __stpncpy ___stpncpy_ev67 + +# undef weak_alias +# define weak_alias(a, b) + +# undef libc_hidden_def +# define libc_hidden_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (__stpncpy, ___stpncpy_ev67) +#endif diff --git ./sysdeps/alpha/multiarch/strcat.c ./sysdeps/alpha/multiarch/strcat.c new file mode 100644 index 0000000000..bbba7bfa33 --- /dev/null +++ ./sysdeps/alpha/multiarch/strcat.c @@ -0,0 +1,49 @@ +/* Multiple versions of strcat. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef strcat +# define strcat __redirect_strcat +# include +# include + +extern __typeof (__redirect_strcat) __libc_strcat; + +extern __typeof (__redirect_strcat) __strcat_ev5 attribute_hidden; +extern __typeof (__redirect_strcat) __strcat_ev67 attribute_hidden; + +static inline __typeof (__redirect_strcat) * +select_strcat_ifunc (void) +{ + if (HAS_CIX) + return __strcat_ev67; + + return __strcat_ev5; +} + +libc_ifunc (__libc_strcat, select_strcat_ifunc ()); + +# undef strcat +strong_alias (__libc_strcat, strcat); +/* Internal callers reach this through __GI_strcat. Alias it to the IFUNC + so that they get the implementation chosen for this CPU too. */ +# ifdef SHARED +__hidden_ver1 (__libc_strcat, __GI_strcat, __redirect_strcat) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (strcat); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/strcat_ev5.S ./sysdeps/alpha/multiarch/strcat_ev5.S new file mode 100644 index 0000000000..5b0c421381 --- /dev/null +++ ./sysdeps/alpha/multiarch/strcat_ev5.S @@ -0,0 +1,33 @@ +/* strcat for Alpha, generic version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define strcat __strcat_ev5 + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) + +#endif + +#include + +#if IS_IN (rtld) +strong_alias (strcat, __strcat_ev5) +#endif diff --git ./sysdeps/alpha/multiarch/strcat_ev67.S ./sysdeps/alpha/multiarch/strcat_ev67.S new file mode 100644 index 0000000000..fbb4296336 --- /dev/null +++ ./sysdeps/alpha/multiarch/strcat_ev67.S @@ -0,0 +1,32 @@ +/* strcat for Alpha, EV67 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define strcat __strcat_ev67 + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (strcat, __strcat_ev67) +#endif diff --git ./sysdeps/alpha/multiarch/strchr.c ./sysdeps/alpha/multiarch/strchr.c new file mode 100644 index 0000000000..57ca43ab81 --- /dev/null +++ ./sysdeps/alpha/multiarch/strchr.c @@ -0,0 +1,50 @@ +/* Multiple versions of strchr. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef strchr +# define strchr __redirect_strchr +# include +# include + +extern __typeof (__redirect_strchr) __libc_strchr; + +extern __typeof (__redirect_strchr) __strchr_ev5 attribute_hidden; +extern __typeof (__redirect_strchr) __strchr_ev67 attribute_hidden; + +static inline __typeof (__redirect_strchr) * +select_strchr_ifunc (void) +{ + if (HAS_CIX) + return __strchr_ev67; + + return __strchr_ev5; +} + +libc_ifunc (__libc_strchr, select_strchr_ifunc ()); + +# undef strchr +strong_alias (__libc_strchr, strchr); +weak_alias (strchr, index); +/* Internal callers reach this through __GI_strchr. Alias it to the IFUNC + so that they get the implementation chosen for this CPU too. */ +# ifdef SHARED +__hidden_ver1 (__libc_strchr, __GI_strchr, __redirect_strchr) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (strchr); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/strchr_ev5.S ./sysdeps/alpha/multiarch/strchr_ev5.S new file mode 100644 index 0000000000..2bed0a475f --- /dev/null +++ ./sysdeps/alpha/multiarch/strchr_ev5.S @@ -0,0 +1,36 @@ +/* strchr for Alpha, generic version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define strchr __strchr_ev5 + +# undef weak_alias +# define weak_alias(a, b) + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) + +#endif + +#include + +#if IS_IN (rtld) +strong_alias (strchr, __strchr_ev5) +#endif diff --git ./sysdeps/alpha/multiarch/strchr_ev67.S ./sysdeps/alpha/multiarch/strchr_ev67.S new file mode 100644 index 0000000000..22dcd12ef2 --- /dev/null +++ ./sysdeps/alpha/multiarch/strchr_ev67.S @@ -0,0 +1,35 @@ +/* strchr for Alpha, EV67 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define strchr __strchr_ev67 + +# undef weak_alias +# define weak_alias(a, b) + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (strchr, __strchr_ev67) +#endif diff --git ./sysdeps/alpha/multiarch/strlen.c ./sysdeps/alpha/multiarch/strlen.c new file mode 100644 index 0000000000..7c9bda3e67 --- /dev/null +++ ./sysdeps/alpha/multiarch/strlen.c @@ -0,0 +1,49 @@ +/* Multiple versions of strlen. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef strlen +# define strlen __redirect_strlen +# include +# include + +extern __typeof (__redirect_strlen) __libc_strlen; + +extern __typeof (__redirect_strlen) __strlen_ev5 attribute_hidden; +extern __typeof (__redirect_strlen) __strlen_ev67 attribute_hidden; + +static inline __typeof (__redirect_strlen) * +select_strlen_ifunc (void) +{ + if (HAS_CIX) + return __strlen_ev67; + + return __strlen_ev5; +} + +libc_ifunc (__libc_strlen, select_strlen_ifunc ()); + +# undef strlen +strong_alias (__libc_strlen, strlen); +/* Internal callers reach this through __GI_strlen. Alias it to the IFUNC + so that they get the implementation chosen for this CPU too. */ +# ifdef SHARED +__hidden_ver1 (__libc_strlen, __GI_strlen, __redirect_strlen) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (strlen); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/strlen_ev5.S ./sysdeps/alpha/multiarch/strlen_ev5.S new file mode 100644 index 0000000000..8894b77486 --- /dev/null +++ ./sysdeps/alpha/multiarch/strlen_ev5.S @@ -0,0 +1,33 @@ +/* strlen for Alpha, generic version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define strlen __strlen_ev5 + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) + +#endif + +#include + +#if IS_IN (rtld) +strong_alias (strlen, __strlen_ev5) +#endif diff --git ./sysdeps/alpha/multiarch/strlen_ev67.S ./sysdeps/alpha/multiarch/strlen_ev67.S new file mode 100644 index 0000000000..1de4775802 --- /dev/null +++ ./sysdeps/alpha/multiarch/strlen_ev67.S @@ -0,0 +1,32 @@ +/* strlen for Alpha, EV67 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define strlen __strlen_ev67 + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +#endif + +#include + +#if IS_IN (rtld) +strong_alias (strlen, __strlen_ev67) +#endif diff --git ./sysdeps/alpha/multiarch/strncat.c ./sysdeps/alpha/multiarch/strncat.c new file mode 100644 index 0000000000..b844518d1a --- /dev/null +++ ./sysdeps/alpha/multiarch/strncat.c @@ -0,0 +1,49 @@ +/* Multiple versions of strncat. Alpha version. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +# undef strncat +# define strncat __redirect_strncat +# include +# include + +extern __typeof (__redirect_strncat) __libc_strncat; + +extern __typeof (__redirect_strncat) __strncat_ev5 attribute_hidden; +extern __typeof (__redirect_strncat) __strncat_ev67 attribute_hidden; + +static inline __typeof (__redirect_strncat) * +select_strncat_ifunc (void) +{ + if (HAS_CIX) + return __strncat_ev67; + + return __strncat_ev5; +} + +libc_ifunc (__libc_strncat, select_strncat_ifunc ()); + +# undef strncat +strong_alias (__libc_strncat, strncat); +/* Internal callers reach this through __GI_strncat. Alias it to the IFUNC + so that they get the implementation chosen for this CPU too. */ +# ifdef SHARED +__hidden_ver1 (__libc_strncat, __GI_strncat, __redirect_strncat) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (strncat); +# endif +#endif diff --git ./sysdeps/alpha/multiarch/strncat_ev5.S ./sysdeps/alpha/multiarch/strncat_ev5.S new file mode 100644 index 0000000000..591ac9173d --- /dev/null +++ ./sysdeps/alpha/multiarch/strncat_ev5.S @@ -0,0 +1,30 @@ +/* strncat for Alpha, generic version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define strncat __strncat_ev5 + +#endif + +#include + +#if IS_IN (rtld) +strong_alias (strncat, __strncat_ev5) +#endif diff --git ./sysdeps/alpha/multiarch/strncat_ev67.S ./sysdeps/alpha/multiarch/strncat_ev67.S new file mode 100644 index 0000000000..3b878d3618 --- /dev/null +++ ./sysdeps/alpha/multiarch/strncat_ev67.S @@ -0,0 +1,29 @@ +/* strncat for Alpha, EV67 version for multiarch dispatch. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) +# define strncat __strncat_ev67 +#endif + +#include + +#if IS_IN (rtld) +strong_alias (strncat, __strncat_ev67) +#endif