From patchwork Mon Mar 23 02:25:34 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: daichengrong X-Patchwork-Id: 132170 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from vm01.sourceware.org (localhost [127.0.0.1]) by sourceware.org (Postfix) with ESMTP id 3C4734BB3B90 for ; Mon, 23 Mar 2026 02:26:25 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 3C4734BB3B90 X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from cstnet.cn (smtp25.cstnet.cn [159.226.251.25]) by sourceware.org (Postfix) with ESMTPS id 285BF4B9DB6E for ; Mon, 23 Mar 2026 02:25:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 285BF4B9DB6E Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=iscas.ac.cn Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=iscas.ac.cn ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 285BF4B9DB6E Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=159.226.251.25 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1774232747; cv=none; b=l/606DmslUCvQvzALFMt8p10G4rgQODU93QDSyIINNnFRq2H58z5A0Nvz5340EcC4/pc6M194T8mBuT+oCuirQ1XcTW6Ri8DKt358m9erf1wwTiSitzjDDsn89IZmypslDJ46mHcaKTd7PY/KTV4V+0kHwxjbOW8kpo70WlFZ2g= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1774232747; c=relaxed/simple; bh=QSFf21oCFk/5/Wo7aI/GGKgRBFuVGepKLyogMsgNPb0=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=rJH4VP3R1AYnjNsRxwt8WkELgxSh2FOVj2ucwPJloh8OPHKXLt5pVVQ2b7ZRjfpXaYCKElt5lMjewSIpCvRk2hGNnoZFjjqVkOcPEB+BnGkKHthnlp5FBKpIMIPbLILp1+7PAAEGHKAw8pnAwJ1gEIDLUlk4k51UwjDk5mEmleg= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 285BF4B9DB6E Received: from localhost.localdomain (unknown [121.237.245.153]) by APP-05 (Coremail) with SMTP id zQCowAD3iAyjpMBpQ50vCw--.41217S2; Mon, 23 Mar 2026 10:25:40 +0800 (CST) From: daichengrong To: libc-alpha@sourceware.org Cc: Jeffrey Law , Adhemerval Zanella Netto , daichengrong Subject: [PATCH] riscv: add RVV-optimized memchr Date: Mon, 23 Mar 2026 10:25:34 +0800 Message-Id: <20260323022534.10684-1-daichengrong@iscas.ac.cn> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-CM-TRANSID: zQCowAD3iAyjpMBpQ50vCw--.41217S2 X-Coremail-Antispam: 1UD129KBjvJXoW3tw4rKF4DAw45GrW3JryxZrb_yoWDtF48pF Z5u3WUKFWkGws7GrWSgF1Yq3W5Ar95GFn8KryY9w4UJ3yUAw4xXFs2vw15GF93JrWF93yF vFn8WFyUurs7AaDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUkq14x267AKxVWUJVW8JwAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2ocxC64kIII0Yj41l84x0c7CEw4AK67xGY2AK02 1l84ACjcxK6xIIjxv20xvE14v26F1j6w1UM28EF7xvwVC0I7IYx2IY6xkF7I0E14v26F4j 6r4UJwA2z4x0Y4vEx4A2jsIE14v26F4UJVW0owA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_Cr 1j6rxdM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7xfMcIj 6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVW8JVWxJwAm72CE4IkC6x0Yz7v_Jr 0_Gr1lF7xvr2IYc2Ij64vIr41lF7I21c0EjII2zVCS5cI20VAGYxC7MxkIecxEwVAFwVW8 JwCF04k20xvY0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r 1j6r18MI8I3I0E7480Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_JF0_Jw1lIxkGc2Ij 64vIr41lIxAIcVC0I7IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Jr 0_Gr1lIxAIcVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF 0xvEx4A2jsIEc7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7VUjfcTDUUUUU== X-Originating-IP: [121.237.245.153] X-CM-SenderInfo: pgdluxxhqj201qj6x2xfdvhtffof0/ X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_STATUS, KAM_SHORT, RCVD_IN_DNSWL_BLOCKED, RCVD_IN_VALIDITY_RPBL_BLOCKED, RCVD_IN_VALIDITY_SAFE_BLOCKED, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~patchwork=sourceware.org@sourceware.org This patch introduces an RVV (RISC-V Vector Extension) based implementation of memchr, leveraging vector instructions to improve data-level parallelism. The implementation has been tested on SG2044 (RVV 128-bit) and Banana Pi F3 (RVV 256-bit) platforms. Benchmark results show no observable regression on small inputs, while providing significant performance improvements on larger inputs. If this approach proves effective, additional optimized mem/str routines that have already been implemented and validated on existing platforms, and demonstrate performance gains, will be submitted in follow-up patches. Signed-off-by: daichengrong --- sysdeps/riscv/multiarch/memchr-generic.c | 24 ++++ sysdeps/riscv/multiarch/memchr-vector.S | 26 ++++ sysdeps/riscv/rvv/memchr.S | 129 ++++++++++++++++++ .../unix/sysv/linux/riscv/multiarch/Makefile | 3 + .../linux/riscv/multiarch/ifunc-impl-list.c | 6 + .../unix/sysv/linux/riscv/multiarch/memchr.c | 57 ++++++++ 6 files changed, 245 insertions(+) create mode 100644 sysdeps/riscv/multiarch/memchr-generic.c create mode 100644 sysdeps/riscv/multiarch/memchr-vector.S create mode 100644 sysdeps/riscv/rvv/memchr.S create mode 100644 sysdeps/unix/sysv/linux/riscv/multiarch/memchr.c diff --git a/sysdeps/riscv/multiarch/memchr-generic.c b/sysdeps/riscv/multiarch/memchr-generic.c new file mode 100644 index 0000000000..a95e16ee51 --- /dev/null +++ b/sysdeps/riscv/multiarch/memchr-generic.c @@ -0,0 +1,24 @@ +/* Re-include the default memchr implementation. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN(libc) +# define MEMCHR __memchr_generic +# include +#endif diff --git a/sysdeps/riscv/multiarch/memchr-vector.S b/sysdeps/riscv/multiarch/memchr-vector.S new file mode 100644 index 0000000000..951f256245 --- /dev/null +++ b/sysdeps/riscv/multiarch/memchr-vector.S @@ -0,0 +1,26 @@ +/* Re-include the RISC-V RVV based memchr implementation. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN(libc) +# define MEMCHR __memchr_vector +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) +# undef weak_alias +# define weak_alias(name, alias) +# include +#endif diff --git a/sysdeps/riscv/rvv/memchr.S b/sysdeps/riscv/rvv/memchr.S new file mode 100644 index 0000000000..8dbd2ec28a --- /dev/null +++ b/sysdeps/riscv/rvv/memchr.S @@ -0,0 +1,129 @@ +/* RISC-V RVV based memchr. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include + +#ifndef MEMCHR +# define MEMCHR __memchr +#endif + +#define srcin a0 +#define chrin a1 +#define cntin a2 +#define result a0 + +#define src a0 +#define cntrem a2 +#define cntrem_4 a5 + +#define vdata v0 +#define vmask v16 +#define first_index a4 + +#define tmp t0 +#define vset_vl t6 + +#define src_2 a3 +#define src_3 a6 +#define src_4 a7 + +#define vdata_1 v0 +#define vdata_2 v4 +#define vdata_3 v8 +#define vdata_4 v12 + +#define vmask_1 v16 +#define vmask_2 v20 +#define vmask_3 v24 +#define vmask_4 v28 + +#define first_index_1 a4 +#define first_index_2 t2 +#define first_index_3 t3 +#define first_index_4 t4 + +ENTRY (MEMCHR) +.option push +.option arch, +v +.option arch, +zba + beqz cntin, L(ret) + and tmp, cntin, 0x3F + beqz tmp, L(loop_pre) + vsetvli vset_vl, tmp, e8, m8, ta, ma + vle8.v vdata, (srcin) + vmseq.vx vmask, vdata, chrin + vfirst.m first_index, vmask + bgez first_index, L(found) + + beq vset_vl, cntin, L(ret) + + add src, srcin, vset_vl + sub cntrem, cntin, vset_vl + +L(loop_pre): + srli cntrem_4, cntrem, 2 +L(loop): + vsetvli vset_vl, cntrem_4, e8, m4, ta, ma + + vle8.v vdata_1, (src) + add src_2, src, vset_vl + vle8.v vdata_2, (src_2) + add src_3, src_2, vset_vl + vle8.v vdata_3, (src_3) + add src_4, src_3, vset_vl + vle8.v vdata_4, (src_4) + + vmseq.vx vmask_1, vdata_1, chrin + vmseq.vx vmask_2, vdata_2, chrin + vmseq.vx vmask_3, vdata_3, chrin + vmseq.vx vmask_4, vdata_4, chrin + + vfirst.m first_index, vmask_1 + vfirst.m first_index_2, vmask_2 + vfirst.m first_index_3, vmask_3 + vfirst.m first_index_4, vmask_4 + + bgez first_index_1, L(found1) + bgez first_index_2, L(found2) + bgez first_index_3, L(found3) + bgez first_index_4, L(found4) + + add src, src_4, vset_vl + sub cntrem_4, cntrem_4, vset_vl + bnez cntrem_4, L(loop) +L(ret): + li result, 0 + ret +L(found4): + add result, src_4, first_index_4 + ret +L(found3): + add result, src_3, first_index_3 + ret +L(found2): + add result, src_2, first_index_2 + ret +L(found1): +L(found): + add result, src, first_index + ret +.option pop +END (MEMCHR) +weak_alias (MEMCHR, memchr) +libc_hidden_builtin_def (memchr) diff --git a/sysdeps/unix/sysv/linux/riscv/multiarch/Makefile b/sysdeps/unix/sysv/linux/riscv/multiarch/Makefile index a865090a53..41ff6f5de3 100644 --- a/sysdeps/unix/sysv/linux/riscv/multiarch/Makefile +++ b/sysdeps/unix/sysv/linux/riscv/multiarch/Makefile @@ -1,6 +1,9 @@ ifeq ($(subdir),string) sysdep_routines += \ memcpy \ + memchr \ + memchr-generic \ + memchr-vector \ memcpy-generic \ memcpy_noalignment \ memset \ diff --git a/sysdeps/unix/sysv/linux/riscv/multiarch/ifunc-impl-list.c b/sysdeps/unix/sysv/linux/riscv/multiarch/ifunc-impl-list.c index a3b5731411..3d107b2a97 100644 --- a/sysdeps/unix/sysv/linux/riscv/multiarch/ifunc-impl-list.c +++ b/sysdeps/unix/sysv/linux/riscv/multiarch/ifunc-impl-list.c @@ -43,6 +43,12 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, rvv_enabled = true; } + IFUNC_IMPL (i, name, memchr, + IFUNC_IMPL_ADD (array, i, memchr, rvv_enabled, + __memchr_vector) + IFUNC_IMPL_ADD (array, i, memchr, 1, __memchr_generic)) + + IFUNC_IMPL (i, name, memcpy, IFUNC_IMPL_ADD (array, i, memcpy, fast_unaligned, __memcpy_noalignment) diff --git a/sysdeps/unix/sysv/linux/riscv/multiarch/memchr.c b/sysdeps/unix/sysv/linux/riscv/multiarch/memchr.c new file mode 100644 index 0000000000..0844a33760 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/multiarch/memchr.c @@ -0,0 +1,57 @@ +/* Multiple versions of memchr. + All versions must be listed in ifunc-impl-list.c. + Copyright (C) 2026 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +/* Redefine memchr so that the compiler won't complain about the type + mismatch with the IFUNC selector in strong_alias, below. */ +# undef memchr +# define memchr __redirect_memchr +# include +# include +# include +# include +# include + +extern __typeof (__redirect_memchr) __libc_memchr; + +extern __typeof (__redirect_memchr) __memchr_generic attribute_hidden; +extern __typeof (__redirect_memchr) __memchr_vector attribute_hidden; + +static inline __typeof (__redirect_memchr) * +select_memchr_ifunc (uint64_t dl_hwcap, __riscv_hwprobe_t hwprobe_func) +{ + unsigned long long int v; + if (__riscv_hwprobe_one (hwprobe_func, RISCV_HWPROBE_KEY_IMA_EXT_0, &v) == 0 + && (v & RISCV_HWPROBE_IMA_V) == RISCV_HWPROBE_IMA_V) + return __memchr_vector; + return __memchr_generic; +} + +riscv_libc_ifunc (__libc_memchr, select_memchr_ifunc); + +# undef memchr +strong_alias (__libc_memchr, memchr); +strong_alias (memchr, __memchr) +# ifdef SHARED +__hidden_ver1 (memchr, __GI_memchr, __redirect_memchr) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memchr); +# endif +#else +# include +#endif