From patchwork Mon Feb 6 19:48:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 64368 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 46EBB38555A2 for ; Mon, 6 Feb 2023 19:48:55 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) by sourceware.org (Postfix) with ESMTPS id 025D0385840C for ; Mon, 6 Feb 2023 19:48:39 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 025D0385840C Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pg1-x529.google.com with SMTP id u75so2356172pgc.10 for ; Mon, 06 Feb 2023 11:48:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=onGJF2WkgUzITfJXKQnpdBxYZJ7jWiUI6+AyD8+t9RU=; b=Y+xqdnApz32NAlTUBB61EydsuQVBuqnF0PyUTnePxzDNUb7DWLh/q/IhFZT+UBt9cJ IMunz8/M5UGYtgdJ2NDi3ptXEHrg+FcbdVN5/vsajJN8526xG5rZWm1GbdaG6MV4GKC4 XpV50SRX+enJIcrNxsmEfhdD1NbecGrpKfAj2i2cDHBL4FvyP3XIXNKHw/S8cYprMbci SSyU7LKToVO/QKQJTr3JnX8MZLiwrUIvU/zBuXzOK6fwzLHYrhgBE/fE7ZuaEuOTm5fY luJgbL9w7xifKlWi/vmRgbKVYT6dkkeyDFqZnk803c0X6/I8pG1B1npVXDfvzqv+eCVl mEpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=onGJF2WkgUzITfJXKQnpdBxYZJ7jWiUI6+AyD8+t9RU=; b=5eyn2oFcFuW+ZS97NULUUUt9Q0HWYTdd6y3/+hSFhTgX4Hej8nJg71Astdm1CSXbA0 E/Mw2g5d6DFOyL5+MWZRxaMy0skEONosJcq7yZQaRJRKF1pF81lfvu3XmVDwfvIJj5Iu WX66R+nXSyLkimeYx53VrhsXiseVt+qVauQYwGXVJfFlyfOxexGwrH3bHRA6ifgBhNFt VsoAE0UZe0trYae/tAUgSOl/tiq2oGfb20WpzPGNqzIk2iuV7qaMWhwqkgVcZP8ZLc6g begMJ972t7Y53AePA47zsPAGW1afdP5hJKLcJIKkYCzXxLCmO4d2YRQRWiyzfO8wb5AO 0g8g== X-Gm-Message-State: AO0yUKXHnJsNku6ZvwiSp/NC9SeyQdKPCEZv/C3Z7LUCthq++90LQphY qO34FPuK+7iYO75/ebxfGj3tAZD5SqU/EBmm X-Google-Smtp-Source: AK7set/7w+u3Hbb3mxooIaXMToTOjrpI69/gLTOHpDdqfVx9GFQDMzOEW7Ky5u63fNqJC9MX/Ae5vA== X-Received: by 2002:aa7:9aef:0:b0:592:61a5:40 with SMTP id y15-20020aa79aef000000b0059261a50040mr631918pfp.16.1675712917638; Mon, 06 Feb 2023 11:48:37 -0800 (PST) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id g20-20020aa796b4000000b0058bb0fb6295sm7512548pfk.26.2023.02.06.11.48.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Feb 2023 11:48:37 -0800 (PST) From: Evan Green To: libc-alpha@sourceware.org Cc: slewis@rivosinc.com, vineetg@rivosinc.com, palmer@rivosinc.com, Evan Green Subject: [PATCH 1/2] riscv: Add Linux hwprobe syscall support Date: Mon, 6 Feb 2023 11:48:18 -0800 Message-Id: <20230206194819.1679472-2-evan@rivosinc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230206194819.1679472-1-evan@rivosinc.com> References: <20230206194819.1679472-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Add awareness and a thin wrapper function around a new Linux system call that allows callers to get architecture and microarchitecture information about the CPUs from the kernel. This can be used to do things like dynamically choose a memcpy implementation. Signed-off-by: Evan Green --- sysdeps/unix/sysv/linux/riscv/Makefile | 4 +-- sysdeps/unix/sysv/linux/riscv/Versions | 3 ++ sysdeps/unix/sysv/linux/riscv/hwprobe.c | 30 ++++++++++++++++ .../unix/sysv/linux/riscv/rv32/arch-syscall.h | 1 + .../unix/sysv/linux/riscv/rv32/libc.abilist | 1 + .../unix/sysv/linux/riscv/rv64/arch-syscall.h | 1 + .../unix/sysv/linux/riscv/rv64/libc.abilist | 1 + sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h | 34 +++++++++++++++++++ sysdeps/unix/sysv/linux/syscall-names.list | 1 + 9 files changed, 74 insertions(+), 2 deletions(-) create mode 100644 sysdeps/unix/sysv/linux/riscv/hwprobe.c create mode 100644 sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h diff --git a/sysdeps/unix/sysv/linux/riscv/Makefile b/sysdeps/unix/sysv/linux/riscv/Makefile index 4b6eacb32f..45cc29e40d 100644 --- a/sysdeps/unix/sysv/linux/riscv/Makefile +++ b/sysdeps/unix/sysv/linux/riscv/Makefile @@ -1,6 +1,6 @@ ifeq ($(subdir),misc) -sysdep_headers += sys/cachectl.h -sysdep_routines += flush-icache +sysdep_headers += sys/cachectl.h sys/hwprobe.h +sysdep_routines += flush-icache hwprobe endif ifeq ($(subdir),stdlib) diff --git a/sysdeps/unix/sysv/linux/riscv/Versions b/sysdeps/unix/sysv/linux/riscv/Versions index 5625d2a0b8..891ae05730 100644 --- a/sysdeps/unix/sysv/linux/riscv/Versions +++ b/sysdeps/unix/sysv/linux/riscv/Versions @@ -8,4 +8,7 @@ libc { GLIBC_2.27 { __riscv_flush_icache; } + GLIBC_2.37 { + __riscv_hwprobe; + } } diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe.c b/sysdeps/unix/sysv/linux/riscv/hwprobe.c new file mode 100644 index 0000000000..ef6dccb9db --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe.c @@ -0,0 +1,30 @@ +/* RISC-V hardware feature probing support on Linux + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +int +__riscv_hwprobe (struct riscv_hwprobe *pairs, long pair_count, + long cpu_count, unsigned long *cpus, unsigned long flags) +{ + return INLINE_SYSCALL (riscv_hwprobe, 5, pairs, pair_count, + cpu_count, cpus, flags); +} diff --git a/sysdeps/unix/sysv/linux/riscv/rv32/arch-syscall.h b/sysdeps/unix/sysv/linux/riscv/rv32/arch-syscall.h index 202520ee25..2416e041c8 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv32/arch-syscall.h +++ b/sysdeps/unix/sysv/linux/riscv/rv32/arch-syscall.h @@ -198,6 +198,7 @@ #define __NR_request_key 218 #define __NR_restart_syscall 128 #define __NR_riscv_flush_icache 259 +#define __NR_riscv_hwprobe 258 #define __NR_rseq 293 #define __NR_rt_sigaction 134 #define __NR_rt_sigpending 136 diff --git a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist index ff90d1bff2..f4c391d3be 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist @@ -2396,3 +2396,4 @@ GLIBC_2.36 pidfd_open F GLIBC_2.36 pidfd_send_signal F GLIBC_2.36 process_madvise F GLIBC_2.36 process_mrelease F +GLIBC_2.37 __riscv_hwprobe F diff --git a/sysdeps/unix/sysv/linux/riscv/rv64/arch-syscall.h b/sysdeps/unix/sysv/linux/riscv/rv64/arch-syscall.h index 4e65f337d4..a32bc82f60 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv64/arch-syscall.h +++ b/sysdeps/unix/sysv/linux/riscv/rv64/arch-syscall.h @@ -205,6 +205,7 @@ #define __NR_request_key 218 #define __NR_restart_syscall 128 #define __NR_riscv_flush_icache 259 +#define __NR_riscv_hwprobe 258 #define __NR_rseq 293 #define __NR_rt_sigaction 134 #define __NR_rt_sigpending 136 diff --git a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist index f1017f6ec5..0f57bbe9e1 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist @@ -2596,3 +2596,4 @@ GLIBC_2.36 pidfd_open F GLIBC_2.36 pidfd_send_signal F GLIBC_2.36 process_madvise F GLIBC_2.36 process_mrelease F +GLIBC_2.37 __riscv_hwprobe F diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h new file mode 100644 index 0000000000..da8cdc90bf --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h @@ -0,0 +1,34 @@ +/* RISC-V architecture probe interface + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#ifndef _SYS_HWPROBE_H +#define _SYS_HWPROBE_H 1 + +#include +#include + +__BEGIN_DECLS + +int +__riscv_hwprobe (struct riscv_hwprobe *pairs, long pair_count, + long cpu_count, unsigned long *cpus, unsigned long flags); + +__END_DECLS + +#endif /* sys/hwprobe.h */ diff --git a/sysdeps/unix/sysv/linux/syscall-names.list b/sysdeps/unix/sysv/linux/syscall-names.list index 822498d3e3..4f4a62e91c 100644 --- a/sysdeps/unix/sysv/linux/syscall-names.list +++ b/sysdeps/unix/sysv/linux/syscall-names.list @@ -477,6 +477,7 @@ renameat2 request_key restart_syscall riscv_flush_icache +riscv_hwprobe rmdir rseq rt_sigaction From patchwork Mon Feb 6 19:48:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 64369 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 01F313857C55 for ; Mon, 6 Feb 2023 19:48:58 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by sourceware.org (Postfix) with ESMTPS id 464543858C39 for ; Mon, 6 Feb 2023 19:48:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 464543858C39 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pf1-x42f.google.com with SMTP id 203so9203888pfx.6 for ; Mon, 06 Feb 2023 11:48:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AZDFBl10Fm3/ZS/xA2p8N32SDHmlDprhbjcu1w3J2n0=; b=p98SFxBcfKPXOhjqY2xe8IskL7yJTt5esMAvp6a8FXXM9h4ssWmec0u4at7Ds5wetm W4pZ+vdxY9e3cD4S3j3jfWEWR1UjTCBfKOzFT3oaIrkbHsYcydQsOLLPWohKw/NAKwlo ZNNYbuCY2/M/mWQwWI98j56Sht4XNaRosPBRi2BQzbzHMUhsOjiAMxJk1EEXCu6jGpsP x1zcobYz03PuA9eTUk/TpmBLV2jKKpz9H7kvJApSKn5jEWP50hF+2o44dGgzZHcqhp7W 25+TUKYvaCfow4xGDO/Glh/EGeq0gCXPHJ8FdsIjYdirN4pnAhRybgcXn0X5Z4HFscm5 1IYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AZDFBl10Fm3/ZS/xA2p8N32SDHmlDprhbjcu1w3J2n0=; b=0b1cH1XloAEKaotIpeyUrctTGMrWl6CkmH85rmWPBKpY36WPlYakb0TL6H+sxUx2NY IWlXkjeaD4YV9BoAoYOptO3vfOAhjhNO3C6pFL5KI+Tfe+Uhox68UbRVIgzDQxGKfWVm u6iSmUMRjR5gZqHJRI2opBL8qNYxTc0MMXMwshRMVSQjqqnyO9e6xGoF4DqvOZnJrCS3 /TO/+HxdCkY7iH1e3/whANyF89gtPIxVolhM39COC7Jg4wEtTd2T5O1/0NvTivpAe/hx evXBqb7WU4SMKp6lmEZpMY9xYTv7OuucwLnDZiI+oyimIWfNJvUhSZ307vehf+dzy3/W cNxA== X-Gm-Message-State: AO0yUKWEKlRWLG72JnaNet5CaXC5r2aXc+OqGWAkht76oBHFHFwrz25s Mcr3yoP8kcb/loaXdwW44wBIqZqnMLAsqhBR X-Google-Smtp-Source: AK7set8eEWOajULOb+BCYsqljZyHuB7jUfxCCm7Pr0o+9O8OVXDQg9/dxox5IAFsEQWyOSLHgVB6aQ== X-Received: by 2002:aa7:97b3:0:b0:597:c44d:ce21 with SMTP id d19-20020aa797b3000000b00597c44dce21mr707067pfq.10.1675712918781; Mon, 06 Feb 2023 11:48:38 -0800 (PST) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id g20-20020aa796b4000000b0058bb0fb6295sm7512548pfk.26.2023.02.06.11.48.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Feb 2023 11:48:38 -0800 (PST) From: Evan Green To: libc-alpha@sourceware.org Cc: slewis@rivosinc.com, vineetg@rivosinc.com, palmer@rivosinc.com, Evan Green Subject: [PATCH 2/2] riscv: Add and use alignment-ignorant memcpy Date: Mon, 6 Feb 2023 11:48:19 -0800 Message-Id: <20230206194819.1679472-3-evan@rivosinc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230206194819.1679472-1-evan@rivosinc.com> References: <20230206194819.1679472-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" For CPU implementations that can perform unaligned accesses with little or no performance penalty, create a memcpy implementation that does not bother aligning buffers. It will use a block of integer registers, a single integer register, and fall back to bytewise copy for the remainder. Signed-off-by: Evan Green --- --- sysdeps/riscv/memcopy.h | 28 +++++ sysdeps/riscv/memcpy.c | 65 +++++++++++ sysdeps/riscv/memcpy_noalignment.S | 103 ++++++++++++++++++ sysdeps/unix/sysv/linux/riscv/Makefile | 4 + .../unix/sysv/linux/riscv/memcpy-generic.c | 24 ++++ 5 files changed, 224 insertions(+) create mode 100644 sysdeps/riscv/memcopy.h create mode 100644 sysdeps/riscv/memcpy.c create mode 100644 sysdeps/riscv/memcpy_noalignment.S create mode 100644 sysdeps/unix/sysv/linux/riscv/memcpy-generic.c diff --git a/sysdeps/riscv/memcopy.h b/sysdeps/riscv/memcopy.h new file mode 100644 index 0000000000..21f6081b5f --- /dev/null +++ b/sysdeps/riscv/memcopy.h @@ -0,0 +1,28 @@ +/* memcopy.h -- definitions for memory copy functions. RISC-V version. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +/* + * Redefine the generic memcpy implementation to __memcpy_generic, so + * the memcpy ifunc can select between generic and special versions. + * In rtld, don't bother with all the ifunciness. + */ +#if IS_IN (libc) +#define MEMCPY __memcpy_generic +#endif diff --git a/sysdeps/riscv/memcpy.c b/sysdeps/riscv/memcpy.c new file mode 100644 index 0000000000..1ba25ef976 --- /dev/null +++ b/sysdeps/riscv/memcpy.c @@ -0,0 +1,65 @@ +/* Multiple versions of memcpy. + All versions must be listed in ifunc-impl-list.c. + Copyright (C) 2017-2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +/* Redefine memcpy so that the compiler won't complain about the type + mismatch with the IFUNC selector in strong_alias, below. */ +# undef memcpy +# define memcpy __redirect_memcpy +# include +#include +#include + +#define INIT_ARCH() + +extern __typeof (__redirect_memcpy) __libc_memcpy; + +extern __typeof (__redirect_memcpy) __memcpy_generic attribute_hidden; +extern __typeof (__redirect_memcpy) __memcpy_noalignment attribute_hidden; + +static inline __typeof (__redirect_memcpy) * +select_memcpy_ifunc (void) +{ + INIT_ARCH (); + + struct riscv_hwprobe pair; + + pair.key = RISCV_HWPROBE_KEY_CPUPERF_0; + if (__riscv_hwprobe(&pair, 1, 0, NULL, 0) != 0) + return __memcpy_generic; + + if ((pair.key > 0) && + (pair.value & RISCV_HWPROBE_MISALIGNED_FAST) == + RISCV_HWPROBE_MISALIGNED_FAST) + return __memcpy_noalignment; + + return __memcpy_generic; +} + +libc_ifunc (__libc_memcpy, select_memcpy_ifunc ()); + +# undef memcpy +strong_alias (__libc_memcpy, memcpy); +# ifdef SHARED +__hidden_ver1 (memcpy, __GI_memcpy, __redirect_memcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memcpy); +# endif + +#endif + diff --git a/sysdeps/riscv/memcpy_noalignment.S b/sysdeps/riscv/memcpy_noalignment.S new file mode 100644 index 0000000000..fe1d9213c4 --- /dev/null +++ b/sysdeps/riscv/memcpy_noalignment.S @@ -0,0 +1,103 @@ +/* memcpy for RISC-V, ignoring buffer alignment + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +#include + +/* void *memcpy(void *, const void *, size_t) */ +ENTRY (__memcpy_noalignment) + move t6, a0 /* Preserve return value */ + + /* Round down to the nearest "page" size */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + add a3, a1, a4 +1: + /* Copy "pages" (chunks of 16 registers) */ + REG_L a4, 0(a1) + REG_L a5, SZREG(a1) + REG_L a6, 2*SZREG(a1) + REG_L a7, 3*SZREG(a1) + REG_L t0, 4*SZREG(a1) + REG_L t1, 5*SZREG(a1) + REG_L t2, 6*SZREG(a1) + REG_L t3, 7*SZREG(a1) + REG_L t4, 8*SZREG(a1) + REG_L t5, 9*SZREG(a1) + REG_S a4, 0(t6) + REG_S a5, SZREG(t6) + REG_S a6, 2*SZREG(t6) + REG_S a7, 3*SZREG(t6) + REG_S t0, 4*SZREG(t6) + REG_S t1, 5*SZREG(t6) + REG_S t2, 6*SZREG(t6) + REG_S t3, 7*SZREG(t6) + REG_S t4, 8*SZREG(t6) + REG_S t5, 9*SZREG(t6) + REG_L a4, 10*SZREG(a1) + REG_L a5, 11*SZREG(a1) + REG_L a6, 12*SZREG(a1) + REG_L a7, 13*SZREG(a1) + REG_L t0, 14*SZREG(a1) + REG_L t1, 15*SZREG(a1) + addi a1, a1, 16*SZREG + REG_S a4, 10*SZREG(t6) + REG_S a5, 11*SZREG(t6) + REG_S a6, 12*SZREG(t6) + REG_S a7, 13*SZREG(t6) + REG_S t0, 14*SZREG(t6) + REG_S t1, 15*SZREG(t6) + addi t6, t6, 16*SZREG + bltu a1, a3, 1b + andi a2, a2, (16*SZREG)-1 /* Update count */ + +2: + /* Remainder is smaller than a page, compute native word count */ + beqz a2, 6f + andi a5, a2, ~(SZREG-1) + andi a2, a2, (SZREG-1) + add a3, a1, a5 + /* Jump directly to byte copy if no words. */ + beqz a5, 4f + +3: + /* Use single native register copy */ + REG_L a4, 0(a1) + addi a1, a1, SZREG + REG_S a4, 0(t6) + addi t6, t6, SZREG + bltu a1, a3, 3b + + /* Jump directly out if no more bytes */ + beqz a2, 6f + +4: + /* Copy the last few individual bytes */ + add a3, a1, a2 +5: + lb a4, 0(a1) + addi a1, a1, 1 + sb a4, 0(t6) + addi t6, t6, 1 + bltu a1, a3, 5b +6: + ret + +END (__memcpy_noalignment) + +hidden_def (__memcpy_noalignment) diff --git a/sysdeps/unix/sysv/linux/riscv/Makefile b/sysdeps/unix/sysv/linux/riscv/Makefile index 45cc29e40d..aa9ea443d6 100644 --- a/sysdeps/unix/sysv/linux/riscv/Makefile +++ b/sysdeps/unix/sysv/linux/riscv/Makefile @@ -7,6 +7,10 @@ ifeq ($(subdir),stdlib) gen-as-const-headers += ucontext_i.sym endif +ifeq ($(subdir),string) +sysdep_routines += memcpy memcpy-generic memcpy_noalignment +endif + abi-variants := ilp32 ilp32d lp64 lp64d ifeq (,$(filter $(default-abi),$(abi-variants))) diff --git a/sysdeps/unix/sysv/linux/riscv/memcpy-generic.c b/sysdeps/unix/sysv/linux/riscv/memcpy-generic.c new file mode 100644 index 0000000000..0abe03f7f5 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/memcpy-generic.c @@ -0,0 +1,24 @@ +/* Re-include the default memcpy implementation. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +extern __typeof (memcpy) __memcpy_generic; +hidden_proto(__memcpy_generic) + +#include