From patchwork Wed Jul 12 19:36:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 72569 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 74249385AFA7 for ; Wed, 12 Jul 2023 19:36:56 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by sourceware.org (Postfix) with ESMTPS id 781113858C74 for ; Wed, 12 Jul 2023 19:36:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 781113858C74 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-66c729f5618so6577368b3a.1 for ; Wed, 12 Jul 2023 12:36:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689190599; x=1691782599; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H5VDLJjzLpIaHbaRaSZ6f+nS1sQornTPs7yu/rkxFY4=; b=XUdHeDnKqk7JUrVze9YIJmNl/Yvek+lfBeM3E+bDKrORazCzEU/a8uL1dCFXo4a8E+ cm1qmbccikGy0zuxOAcdKi5jYSNtiososhzbC4eFoKMTo9wB6xO5NzcvWDhIYgr9RAAw Jd5i/91GVfHq5G7T3ZPhs5z1IFIi+AeQ5Bv9bZOZsIChRDutD2kG1oqWy7qg7oN3QdrH udW7JgAlqsYs2b+pquzFWyylZAawyDi9I7SfIcWus/B8MWytBTzIat3H5JxDyc/HJD67 f+x6nv92n1rht3xcniFHNXxkdufIdETJJFl/2lcQvuDfB/9Emw5Kfj0re/YyXa2sS65Z Ti0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689190599; x=1691782599; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H5VDLJjzLpIaHbaRaSZ6f+nS1sQornTPs7yu/rkxFY4=; b=jHwXjtW9nC4eAS891B87Y5rLgnFJLavdGEUdto1NuVTPPzlTiSnPCVdXefDile3Wtn 7ufmkJ3lPiAXbCBC4GoIX0z9w8L7v7B6m0j+7s1ocLmaVlbXsDVgUctZajCBAGmzVNsz eHG6FLkaS2VI2VH8b0dmVSDQvHLHPfMeLHkI43TalWkHohjcYXKiLlVUPmqqh2c74NjI fQafHWpKVTu7InPDZdIVSeuqEdcl+Ry7UC9bwcSBLuDJVVN1QB6uFeTWHcX+jNopqCT+ FwoF6InLJtZs8A98YgDeGp3kd9Pao2XyDwpnqE3iJY+kpXJW39c4gajK6+q82WvBOtKk D/RQ== X-Gm-Message-State: ABy/qLZGpg0KM+tMgU2Ev5lIqN5oB99VqDynRDSS7URG4XoKwXq/TP5j janC57hQvFeHd7pZVCSbh3C9fbh2xrVzB3Xrlb4= X-Google-Smtp-Source: APBJJlEljVByg1zLoXvpMny3S0TQqrNU79/6HqTs44WoRN4bI53ty1uOso+CqAFLqHKalRUx1Cq/yQ== X-Received: by 2002:a05:6a00:985:b0:682:4e4c:48bc with SMTP id u5-20020a056a00098500b006824e4c48bcmr25700788pfg.21.1689190599212; Wed, 12 Jul 2023 12:36:39 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id h20-20020aa786d4000000b0065dd1e7c2c1sm3983141pfo.63.2023.07.12.12.36.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jul 2023 12:36:38 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, slewis@rivosinc.com, palmer@rivosinc.com, Florian Weimer , Evan Green Subject: [PATCH v5 1/4] riscv: Add Linux hwprobe syscall support Date: Wed, 12 Jul 2023 12:36:25 -0700 Message-Id: <20230712193629.2880253-2-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230712193629.2880253-1-evan@rivosinc.com> References: <20230712193629.2880253-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Add awareness and a thin wrapper function around a new Linux system call that allows callers to get architecture and microarchitecture information about the CPUs from the kernel. This can be used to do things like dynamically choose a memcpy implementation. Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- (no changes since v4) Changes in v4: - Remove __USE_GNU (Florian) - __nonnull, __wur, __THROW, and __fortified_attr_access decorations (Florian) - change long to long int (Florian) - Fix comment formatting (Florian) - Update backup kernel header content copy. - Fix function declaration formatting (Florian) - Changed export versions to 2.38 Changes in v3: - Update argument types to match v4 kernel interface Changes in v2: - hwprobe.h: Use __has_include and duplicate Linux content to make compilation work when Linux headers are absent (Adhemerval) - hwprobe.h: Put declaration under __USE_GNU (Adhemerval) - Use INLINE_SYSCALL_CALL (Adhemerval) - Update versions - Update UNALIGNED_MASK to match kernel v3 series. sysdeps/unix/sysv/linux/riscv/Makefile | 4 +- sysdeps/unix/sysv/linux/riscv/Versions | 3 + sysdeps/unix/sysv/linux/riscv/hwprobe.c | 30 ++++++++ .../unix/sysv/linux/riscv/rv32/libc.abilist | 1 + .../unix/sysv/linux/riscv/rv64/libc.abilist | 1 + sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h | 72 +++++++++++++++++++ 6 files changed, 109 insertions(+), 2 deletions(-) create mode 100644 sysdeps/unix/sysv/linux/riscv/hwprobe.c create mode 100644 sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h diff --git a/sysdeps/unix/sysv/linux/riscv/Makefile b/sysdeps/unix/sysv/linux/riscv/Makefile index 4b6eacb32f..45cc29e40d 100644 --- a/sysdeps/unix/sysv/linux/riscv/Makefile +++ b/sysdeps/unix/sysv/linux/riscv/Makefile @@ -1,6 +1,6 @@ ifeq ($(subdir),misc) -sysdep_headers += sys/cachectl.h -sysdep_routines += flush-icache +sysdep_headers += sys/cachectl.h sys/hwprobe.h +sysdep_routines += flush-icache hwprobe endif ifeq ($(subdir),stdlib) diff --git a/sysdeps/unix/sysv/linux/riscv/Versions b/sysdeps/unix/sysv/linux/riscv/Versions index 5625d2a0b8..0c4016382d 100644 --- a/sysdeps/unix/sysv/linux/riscv/Versions +++ b/sysdeps/unix/sysv/linux/riscv/Versions @@ -8,4 +8,7 @@ libc { GLIBC_2.27 { __riscv_flush_icache; } + GLIBC_2.38 { + __riscv_hwprobe; + } } diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe.c b/sysdeps/unix/sysv/linux/riscv/hwprobe.c new file mode 100644 index 0000000000..a8a14d29a5 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe.c @@ -0,0 +1,30 @@ +/* RISC-V hardware feature probing support on Linux + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +int __riscv_hwprobe (struct riscv_hwprobe *pairs, size_t pair_count, + size_t cpu_count, unsigned long int *cpus, + unsigned int flags) +{ + return INLINE_SYSCALL_CALL (riscv_hwprobe, pairs, pair_count, + cpu_count, cpus, flags); +} diff --git a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist index b9740a1afc..8fab4a606f 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist @@ -2436,3 +2436,4 @@ GLIBC_2.38 strlcat F GLIBC_2.38 strlcpy F GLIBC_2.38 wcslcat F GLIBC_2.38 wcslcpy F +GLIBC_2.38 __riscv_hwprobe F diff --git a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist index e3b4656aa2..1ebb91deed 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist @@ -2636,3 +2636,4 @@ GLIBC_2.38 strlcat F GLIBC_2.38 strlcpy F GLIBC_2.38 wcslcat F GLIBC_2.38 wcslcpy F +GLIBC_2.38 __riscv_hwprobe F diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h new file mode 100644 index 0000000000..b27af5cb07 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h @@ -0,0 +1,72 @@ +/* RISC-V architecture probe interface + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#ifndef _SYS_HWPROBE_H +#define _SYS_HWPROBE_H 1 + +#include +#include +#ifdef __has_include +# if __has_include () +# include +# endif +#endif + +/* Define a (probably stale) version of the interface if the Linux headers + aren't present. */ +#ifndef RISCV_HWPROBE_KEY_MVENDORID +struct riscv_hwprobe { + signed long long int key; + unsigned long long int value; +}; + +#define RISCV_HWPROBE_KEY_MVENDORID 0 +#define RISCV_HWPROBE_KEY_MARCHID 1 +#define RISCV_HWPROBE_KEY_MIMPID 2 +#define RISCV_HWPROBE_KEY_BASE_BEHAVIOR 3 +#define RISCV_HWPROBE_BASE_BEHAVIOR_IMA (1 << 0) +#define RISCV_HWPROBE_KEY_IMA_EXT_0 4 +#define RISCV_HWPROBE_IMA_FD (1 << 0) +#define RISCV_HWPROBE_IMA_C (1 << 1) +#define RISCV_HWPROBE_IMA_V (1 << 2) +#define RISCV_HWPROBE_EXT_ZBA (1 << 3) +#define RISCV_HWPROBE_EXT_ZBB (1 << 4) +#define RISCV_HWPROBE_EXT_ZBS (1 << 5) +#define RISCV_HWPROBE_KEY_CPUPERF_0 5 +#define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0) +#define RISCV_HWPROBE_MISALIGNED_EMULATED (1 << 0) +#define RISCV_HWPROBE_MISALIGNED_SLOW (2 << 0) +#define RISCV_HWPROBE_MISALIGNED_FAST (3 << 0) +#define RISCV_HWPROBE_MISALIGNED_UNSUPPORTED (4 << 0) +#define RISCV_HWPROBE_MISALIGNED_MASK (7 << 0) + +#endif /* RISCV_HWPROBE_KEY_MVENDORID */ + +__BEGIN_DECLS + +extern int __riscv_hwprobe (struct riscv_hwprobe *pairs, size_t pair_count, + size_t cpu_count, unsigned long int *cpus, + unsigned int flags) + __THROW __nonnull ((1)) __wur + __fortified_attr_access (__read_write__, 1, 2) + __fortified_attr_access (__read_only__, 4, 3); + +__END_DECLS + +#endif /* sys/hwprobe.h */ From patchwork Wed Jul 12 19:36:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 72570 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 015EF3858C62 for ; Wed, 12 Jul 2023 19:37:00 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by sourceware.org (Postfix) with ESMTPS id B9AAE3857711 for ; Wed, 12 Jul 2023 19:36:43 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B9AAE3857711 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-666e6ecb52dso4478497b3a.2 for ; Wed, 12 Jul 2023 12:36:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689190602; x=1691782602; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TzZCzP51tI2Kmf6IzjQzegq47jrOLoeCA9W09QZCblE=; b=hGE42BMznrFr0jjUBT7kj1myRNF753bjgTRvQ0Y+RJltPKNorMUyUqXucNrILLPNi6 FbvmkJrge8DsZhasc20I9U01XDqFsZH4VnA0icLq8+oMqabtek9MSMUdj4m3S8aD7M+g BDYPAZz+fBCmgLYyo2ueG/H/fLsWd8+GeE/wKy7oKr5EErpgLfi4PwK9MYJzGPPgP6or JJtO1SlTBwGpwaxqoF5FKcVSblPFUeeZKdOR9m1rr/Rxmklv84xwmpjYCnoy6LqVxAGL cfoEbZzea4Ta7ycvq+mmMP9EdtulEtfwXo+ocuAYAslOV7dHbImVf2J/qLILbB79/3fQ /Sxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689190602; x=1691782602; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TzZCzP51tI2Kmf6IzjQzegq47jrOLoeCA9W09QZCblE=; b=Y3liVdsS+H125hFahj1EgxcyNSzhJVTHtc4KIGPk/QFH4aVhLJwz8d1wLoBg/7AAnM +kMjB47z92IuwsYlU0c1JkA2ugRsHjgOvvDwKLnmjDpZXiyLvsIkNdCYaOVbBk3Tb2OX cSnMA0a/txkxIGVh84GVF6krQwEkVshBkxIs5TXUPIREiircrv+MwBEBvYgMH8p1ypnp oObyAyO7CUpmOJxg1hPLPC/0ugdLi+dVfRgtvpiIIjX3cgBUbciONG07e9Q8SPF3bMwD ASyLWhEEdTzu7ug1RTMCjNu4sPCt/2FOKygrJO9aaPEjI9t1AKRiEtbGpUkmmuDWF6qA DqXQ== X-Gm-Message-State: ABy/qLYKMVdyq31qtAfN5YdHRdMtgWxwhlJiPI9KU9/hOn+e/d/VAZjC y5xUMuM2aDpM/+4C5kh1IGu3vJX8fuk8BNlqMsY= X-Google-Smtp-Source: APBJJlFqBKQ0nZR8UoCzO4pUt9AyJwh4Zv0FfX4kXCor1t1p7df2QHYh1otTnHRakK0yZerH3537kg== X-Received: by 2002:aa7:8891:0:b0:681:3ed2:b493 with SMTP id z17-20020aa78891000000b006813ed2b493mr17662625pfe.26.1689190602457; Wed, 12 Jul 2023 12:36:42 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id h20-20020aa786d4000000b0065dd1e7c2c1sm3983141pfo.63.2023.07.12.12.36.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jul 2023 12:36:42 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, slewis@rivosinc.com, palmer@rivosinc.com, Florian Weimer , Evan Green Subject: [PATCH v5 2/4] riscv: Add hwprobe vdso call support Date: Wed, 12 Jul 2023 12:36:26 -0700 Message-Id: <20230712193629.2880253-3-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230712193629.2880253-1-evan@rivosinc.com> References: <20230712193629.2880253-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" The new riscv_hwprobe syscall also comes with a vDSO for faster answers to your most common questions. Call in today to speak with a kernel representative near you! Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- (no changes since v3) Changes in v3: - Add the "return" to the vsyscall - Fix up vdso arg types to match kernel v4 version - Remove ifdef around INLINE_VSYSCALL (Adhemerval) Changes in v2: - Add vDSO interface sysdeps/unix/sysv/linux/dl-vdso-setup.c | 10 ++++++++++ sysdeps/unix/sysv/linux/dl-vdso-setup.h | 3 +++ sysdeps/unix/sysv/linux/riscv/hwprobe.c | 5 +++-- sysdeps/unix/sysv/linux/riscv/sysdep.h | 1 + 4 files changed, 17 insertions(+), 2 deletions(-) diff --git a/sysdeps/unix/sysv/linux/dl-vdso-setup.c b/sysdeps/unix/sysv/linux/dl-vdso-setup.c index 97eaaeac37..ed8b1ef426 100644 --- a/sysdeps/unix/sysv/linux/dl-vdso-setup.c +++ b/sysdeps/unix/sysv/linux/dl-vdso-setup.c @@ -71,6 +71,16 @@ PROCINFO_CLASS int (*_dl_vdso_clock_getres_time64) (clockid_t, # ifdef HAVE_GET_TBFREQ PROCINFO_CLASS uint64_t (*_dl_vdso_get_tbfreq)(void) RELRO; # endif + +/* RISC-V specific ones. */ +# ifdef HAVE_RISCV_HWPROBE +PROCINFO_CLASS int (*_dl_vdso_riscv_hwprobe)(void *, + size_t, + size_t, + unsigned long *, + unsigned int) RELRO; +# endif + #endif #undef RELRO diff --git a/sysdeps/unix/sysv/linux/dl-vdso-setup.h b/sysdeps/unix/sysv/linux/dl-vdso-setup.h index 867072b897..39eafd5316 100644 --- a/sysdeps/unix/sysv/linux/dl-vdso-setup.h +++ b/sysdeps/unix/sysv/linux/dl-vdso-setup.h @@ -47,6 +47,9 @@ setup_vdso_pointers (void) #ifdef HAVE_GET_TBFREQ GLRO(dl_vdso_get_tbfreq) = dl_vdso_vsym (HAVE_GET_TBFREQ); #endif +#ifdef HAVE_RISCV_HWPROBE + GLRO(dl_vdso_riscv_hwprobe) = dl_vdso_vsym (HAVE_RISCV_HWPROBE); +#endif } #endif diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe.c b/sysdeps/unix/sysv/linux/riscv/hwprobe.c index a8a14d29a5..14f7136998 100644 --- a/sysdeps/unix/sysv/linux/riscv/hwprobe.c +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe.c @@ -20,11 +20,12 @@ #include #include #include +#include int __riscv_hwprobe (struct riscv_hwprobe *pairs, size_t pair_count, size_t cpu_count, unsigned long int *cpus, unsigned int flags) { - return INLINE_SYSCALL_CALL (riscv_hwprobe, pairs, pair_count, - cpu_count, cpus, flags); + /* The vDSO may be able to provide the answer without a syscall. */ + return INLINE_VSYSCALL(riscv_hwprobe, 5, pairs, pair_count, cpu_count, cpus, flags); } diff --git a/sysdeps/unix/sysv/linux/riscv/sysdep.h b/sysdeps/unix/sysv/linux/riscv/sysdep.h index 5583b96d23..ee015dfeb6 100644 --- a/sysdeps/unix/sysv/linux/riscv/sysdep.h +++ b/sysdeps/unix/sysv/linux/riscv/sysdep.h @@ -156,6 +156,7 @@ /* List of system calls which are supported as vsyscalls (for RV32 and RV64). */ # define HAVE_GETCPU_VSYSCALL "__vdso_getcpu" +# define HAVE_RISCV_HWPROBE "__vdso_riscv_hwprobe" # undef HAVE_INTERNAL_BRK_ADDR_SYMBOL # define HAVE_INTERNAL_BRK_ADDR_SYMBOL 1 From patchwork Wed Jul 12 19:36:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 72572 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 760BA3856DC0 for ; Wed, 12 Jul 2023 19:37:23 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by sourceware.org (Postfix) with ESMTPS id 38B4D3857716 for ; Wed, 12 Jul 2023 19:36:46 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 38B4D3857716 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-666eec46206so6568037b3a.3 for ; Wed, 12 Jul 2023 12:36:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689190605; x=1691782605; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2v+4hVuFPCcPSWwcGERx7u1WLPoLmXsH49B9YcmvQio=; b=F0zXw5ibNJsza9YjiSVY0c4JIsu35hOH5a4BOgn/Xt3QaYN1LNV0r21MNwJdSK7Toy e1LbA5jp/kBtqUE/GWs93xCHTELnm9WnzPj9H3eFs7Y+ctpMnWQ8k78KHlDKvkrVjgAP 8N5zyYtWPe6WBOa83gWg96NprV7lIwddC6Q5Nqzsk0xrSbkqtryMlaU7XEOTW2kO465Q YkXfLkKE9b0y6CDavUs6wxQr/Vp2nx2uGf/JW2AHjg6/uKKkWS4bWZsbWR0hb14pjZ48 0B9h27QVfVqdjyulRzn66+Apsyc+j6ceTNo+Smm9tlK/g3h6ex6o1ofXGXEiia4/AnGK nnLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689190605; x=1691782605; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2v+4hVuFPCcPSWwcGERx7u1WLPoLmXsH49B9YcmvQio=; b=OsqYMo3MjO8zdQfq6wpJCTWIHZl2uYAC5n3aenzY2Cekceo+qQlU1OyqxCNfDzsyBQ Cs2gq8rvgnVH4tJu+CGfx/QNVWYtIq/SlNgwQoXAubZ+XlkPMgTQu4XOLmaSfBdBNyCs d5qlafHyz7M1B8wJLcgkme5bYDS3TuR6n+HPUWHDoIRlIhAYRNqZ6ZJBu6IEeJQU/Nnb wH2F58EOqf1es0Ueu/z69kONLtVkwvEX9OTHUXvhy+kNsGEeOdS7KHs5bb4NNg/tpUEt d/nFoqKnkVGLgP63TiXtitlayxgmqXY2UuVRFpmjpVSjdwiyHWQ6GDvYMW3bOSTEZzBc 5vFg== X-Gm-Message-State: ABy/qLYiAd+DtE0+FA23B80+upGiC3TLFaIBX7iMgUSAjHW2s6nOssyo uPM/F0ActsUVL+w/wygi7ORmemStfsaCMKXvP34= X-Google-Smtp-Source: APBJJlGRwdI2TTPZ6PlC47TFKOYIt7UX+MdDjfiSWoKCc7BKJXOTgpmTqpVzOlYsD60iUa5JQ9eUsg== X-Received: by 2002:a05:6a20:543:b0:12f:dc31:a71e with SMTP id 3-20020a056a20054300b0012fdc31a71emr15605416pzq.56.1689190604957; Wed, 12 Jul 2023 12:36:44 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id h20-20020aa786d4000000b0065dd1e7c2c1sm3983141pfo.63.2023.07.12.12.36.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jul 2023 12:36:44 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, slewis@rivosinc.com, palmer@rivosinc.com, Florian Weimer , Evan Green Subject: [PATCH v5 3/4] riscv: Add ifunc-compatible hwprobe function Date: Wed, 12 Jul 2023 12:36:27 -0700 Message-Id: <20230712193629.2880253-4-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230712193629.2880253-1-evan@rivosinc.com> References: <20230712193629.2880253-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" ifunc selector functions are called very early, and are not able to call external functions that require relocation. This presents a problem for applications and libraries outside of glibc that may want to use __riscv_hwprobe() in their ifunc selectors. Add a new function, __riscv_hwprobe_early(), that is compiled into libc_nonshared.a and is safe to use in application and library ifunc selectors. The _early() variant checks a weak alias of __riscv_hwprobe, __riscv_hwprobe_weak, to see if the symbol is available, and simply calls that if so. However if it's called before the symbol is available, it makes the system call directly. The early call has some caveats, primarily that it does not take advantage of the cached vDSO data, and does not set errno (another external symbol) upon failing. The original dynamic __riscv_hwprobe() function is still available for users that don't need the overhead of handling early init cases. Signed-off-by: Evan Green --- Changes in v5: - Introduced __riscv_hwprobe_early() sysdeps/unix/sysv/linux/riscv/Makefile | 3 +- sysdeps/unix/sysv/linux/riscv/Versions | 1 + sysdeps/unix/sysv/linux/riscv/hwprobe.c | 2 ++ .../unix/sysv/linux/riscv/hwprobe_static.c | 36 +++++++++++++++++++ .../unix/sysv/linux/riscv/rv32/libc.abilist | 1 + .../unix/sysv/linux/riscv/rv64/libc.abilist | 1 + sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h | 21 +++++++++++ 7 files changed, 64 insertions(+), 1 deletion(-) create mode 100644 sysdeps/unix/sysv/linux/riscv/hwprobe_static.c diff --git a/sysdeps/unix/sysv/linux/riscv/Makefile b/sysdeps/unix/sysv/linux/riscv/Makefile index 45cc29e40d..fd8fd48ade 100644 --- a/sysdeps/unix/sysv/linux/riscv/Makefile +++ b/sysdeps/unix/sysv/linux/riscv/Makefile @@ -1,6 +1,7 @@ ifeq ($(subdir),misc) sysdep_headers += sys/cachectl.h sys/hwprobe.h -sysdep_routines += flush-icache hwprobe +sysdep_routines += flush-icache hwprobe hwprobe_static +static-only-routines += hwprobe_static endif ifeq ($(subdir),stdlib) diff --git a/sysdeps/unix/sysv/linux/riscv/Versions b/sysdeps/unix/sysv/linux/riscv/Versions index 0c4016382d..571d4c2046 100644 --- a/sysdeps/unix/sysv/linux/riscv/Versions +++ b/sysdeps/unix/sysv/linux/riscv/Versions @@ -10,5 +10,6 @@ libc { } GLIBC_2.38 { __riscv_hwprobe; + __riscv_hwprobe_weak; } } diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe.c b/sysdeps/unix/sysv/linux/riscv/hwprobe.c index 14f7136998..a8bd63fca4 100644 --- a/sysdeps/unix/sysv/linux/riscv/hwprobe.c +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe.c @@ -29,3 +29,5 @@ int __riscv_hwprobe (struct riscv_hwprobe *pairs, size_t pair_count, /* The vDSO may be able to provide the answer without a syscall. */ return INLINE_VSYSCALL(riscv_hwprobe, 5, pairs, pair_count, cpu_count, cpus, flags); } + +weak_alias(__riscv_hwprobe, __riscv_hwprobe_weak) diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe_static.c b/sysdeps/unix/sysv/linux/riscv/hwprobe_static.c new file mode 100644 index 0000000000..b1c0f4089f --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe_static.c @@ -0,0 +1,36 @@ +/* RISC-V hardware feature probing support on Linux, ifunc-compatible + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include + +attribute_hidden +int __riscv_hwprobe_early (struct riscv_hwprobe *pairs, size_t pair_count, + size_t cpu_count, unsigned long int *cpus, + unsigned int flags) +{ + if (!__riscv_hwprobe_weak) { + /* Use INTERNAL_SYSCALL since errno is not available. */ + return INTERNAL_SYSCALL(riscv_hwprobe, 5, pairs, pair_count, cpu_count, cpus, flags); + } + + return __riscv_hwprobe_weak(pairs, pair_count, cpu_count, cpus, flags); +} diff --git a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist index 8fab4a606f..942e397be0 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist @@ -2437,3 +2437,4 @@ GLIBC_2.38 strlcpy F GLIBC_2.38 wcslcat F GLIBC_2.38 wcslcpy F GLIBC_2.38 __riscv_hwprobe F +GLIBC_2.38 __riscv_hwprobe_weak F diff --git a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist index 1ebb91deed..bc140fa7df 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist @@ -2637,3 +2637,4 @@ GLIBC_2.38 strlcpy F GLIBC_2.38 wcslcat F GLIBC_2.38 wcslcpy F GLIBC_2.38 __riscv_hwprobe F +GLIBC_2.38 __riscv_hwprobe_weak F diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h index b27af5cb07..d739371806 100644 --- a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h @@ -67,6 +67,27 @@ extern int __riscv_hwprobe (struct riscv_hwprobe *pairs, size_t pair_count, __fortified_attr_access (__read_write__, 1, 2) __fortified_attr_access (__read_only__, 4, 3); +extern int __riscv_hwprobe_weak (struct riscv_hwprobe *pairs, size_t pair_count, + size_t cpu_count, unsigned long int *cpus, + unsigned int flags) + __THROW __nonnull ((1)) __wur + __fortified_attr_access (__read_write__, 1, 2) + __fortified_attr_access (__read_only__, 4, 3); + +# ifdef weak_extern +weak_extern (__riscv_hwprobe_weak) +# else +# pragma weak __riscv_hwprobe_weak +# endif + +extern int __riscv_hwprobe_early (struct riscv_hwprobe *pairs, size_t pair_count, + size_t cpu_count, unsigned long int *cpus, + unsigned int flags) + __THROW __nonnull ((1)) __wur + __fortified_attr_access (__read_write__, 1, 2) + __fortified_attr_access (__read_only__, 4, 3) + __attribute__ ((__visibility__ ("hidden"))); + __END_DECLS #endif /* sys/hwprobe.h */ From patchwork Wed Jul 12 19:36:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 72571 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E3C39385E00A for ; Wed, 12 Jul 2023 19:37:17 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by sourceware.org (Postfix) with ESMTPS id AD20F3857728 for ; Wed, 12 Jul 2023 19:36:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org AD20F3857728 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pf1-x42a.google.com with SMTP id d2e1a72fcca58-66872d4a141so5156852b3a.1 for ; Wed, 12 Jul 2023 12:36:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689190607; x=1691782607; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HFlRt+dWurSeFBCGa596ITmT9C+F7+kxw23cSLyHAnw=; b=ei37n7/P9y0I8zJZp19ZqWXHMqOWuIqG1/er2jE/4moCNUlDcQS6l6Dg/fdVgHWzR6 IIE2w5GCxk5961nC11Ed9j+UxHMMCVqbiMpQN2XENlAwL5MS/9HHwWeUhEPsE7yaP2d+ r6iNpukWPQPIqC26D9TtcToJuwEGGJEMjiUVauqmVPpuHqpb5Bde1+2dYZ9WQYi7Xeuo YNWpCsxNI8VqyoLYVgG3oFeVruRvYe+UAOamGP6i6cnq7qqaw1Dz2qLjZnJNBGjAf31o 04szyeXUMqL3j2i+ufnEIeSBMpI45QoCqJO1ks0bceOFPZFJnG7908ccTH2e3NF5cljF P35Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689190607; x=1691782607; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HFlRt+dWurSeFBCGa596ITmT9C+F7+kxw23cSLyHAnw=; b=MYB1Mhj03P2dPckmRY8P4tiyvM5BK8uZHLNITgtAxl9etxZFkWyyPIw+iSm8z34rWB ppCOGmdtdZywVoDAHr8OxL7gXQf0zSEGm2TdVG0TcbtyZqcZ9JisTV+76Ezx6XogZwIG 7R11IOj8dGMfyaV2UoCmn/waTKPZ0dM+4+h6CRaLoBju2D9syJ0/yXUCGvWECF5iSabu qG9CO6S3+9g0YKfZhZLq4k4BxhjjpfStG7n8RKfnz3x9Zo7CYGm4mhmXtLtNKLy3EPYi +VepHGxcJj3beMmF1hy6ciMLODa3ShSP0HMUFwYE46irmZjYsxJO2C9OKcuzKpg56Mpv tI2g== X-Gm-Message-State: ABy/qLYoQxO1CZm0FPtMboLtjljsAhIeMXXkb7UFyOtYPjtgkWSwI3Hd +LLhFafBvkzHrbL4Re4SnKI69rDqiE4SoTIyKYc= X-Google-Smtp-Source: APBJJlG8augF4qAVOJ3bFs8EMJPWHh6KtcFMsk0zEKgsVAeoIKRy4rlkko8HubgyyohodWL9d3IzcQ== X-Received: by 2002:a05:6a00:1816:b0:666:ecf4:ed6d with SMTP id y22-20020a056a00181600b00666ecf4ed6dmr18901887pfa.18.1689190607185; Wed, 12 Jul 2023 12:36:47 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id h20-20020aa786d4000000b0065dd1e7c2c1sm3983141pfo.63.2023.07.12.12.36.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jul 2023 12:36:46 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, slewis@rivosinc.com, palmer@rivosinc.com, Florian Weimer , Evan Green Subject: [PATCH v5 4/4] riscv: Add and use alignment-ignorant memcpy Date: Wed, 12 Jul 2023 12:36:28 -0700 Message-Id: <20230712193629.2880253-5-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230712193629.2880253-1-evan@rivosinc.com> References: <20230712193629.2880253-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" For CPU implementations that can perform unaligned accesses with little or no performance penalty, create a memcpy implementation that does not bother aligning buffers. It will use a block of integer registers, a single integer register, and fall back to bytewise copy for the remainder. Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- Changes in v5: - Do unaligned word access for final trailing bytes (Richard) Changes in v4: - Fixed comment style (Florian) Changes in v3: - Word align dest for large memcpy()s. - Add tags - Remove spurious blank line from sysdeps/riscv/memcpy.c Changes in v2: - Used _MASK instead of _FAST value itself. --- sysdeps/riscv/memcopy.h | 26 ++++ sysdeps/riscv/memcpy.c | 64 +++++++++ sysdeps/riscv/memcpy_noalignment.S | 134 ++++++++++++++++++ sysdeps/unix/sysv/linux/riscv/Makefile | 4 + .../unix/sysv/linux/riscv/memcpy-generic.c | 24 ++++ 5 files changed, 252 insertions(+) create mode 100644 sysdeps/riscv/memcopy.h create mode 100644 sysdeps/riscv/memcpy.c create mode 100644 sysdeps/riscv/memcpy_noalignment.S create mode 100644 sysdeps/unix/sysv/linux/riscv/memcpy-generic.c diff --git a/sysdeps/riscv/memcopy.h b/sysdeps/riscv/memcopy.h new file mode 100644 index 0000000000..2b685c8aa0 --- /dev/null +++ b/sysdeps/riscv/memcopy.h @@ -0,0 +1,26 @@ +/* memcopy.h -- definitions for memory copy functions. RISC-V version. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +/* Redefine the generic memcpy implementation to __memcpy_generic, so + the memcpy ifunc can select between generic and special versions. + In rtld, don't bother with all the ifunciness. */ +#if IS_IN (libc) +#define MEMCPY __memcpy_generic +#endif diff --git a/sysdeps/riscv/memcpy.c b/sysdeps/riscv/memcpy.c new file mode 100644 index 0000000000..fdb8dc3208 --- /dev/null +++ b/sysdeps/riscv/memcpy.c @@ -0,0 +1,64 @@ +/* Multiple versions of memcpy. + All versions must be listed in ifunc-impl-list.c. + Copyright (C) 2017-2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +/* Redefine memcpy so that the compiler won't complain about the type + mismatch with the IFUNC selector in strong_alias, below. */ +# undef memcpy +# define memcpy __redirect_memcpy +# include +#include +#include + +#define INIT_ARCH() + +extern __typeof (__redirect_memcpy) __libc_memcpy; + +extern __typeof (__redirect_memcpy) __memcpy_generic attribute_hidden; +extern __typeof (__redirect_memcpy) __memcpy_noalignment attribute_hidden; + +static inline __typeof (__redirect_memcpy) * +select_memcpy_ifunc (void) +{ + INIT_ARCH (); + + struct riscv_hwprobe pair; + + pair.key = RISCV_HWPROBE_KEY_CPUPERF_0; + if (__riscv_hwprobe(&pair, 1, 0, NULL, 0) != 0) + return __memcpy_generic; + + if ((pair.key > 0) && + (pair.value & RISCV_HWPROBE_MISALIGNED_MASK) == + RISCV_HWPROBE_MISALIGNED_FAST) + return __memcpy_noalignment; + + return __memcpy_generic; +} + +libc_ifunc (__libc_memcpy, select_memcpy_ifunc ()); + +# undef memcpy +strong_alias (__libc_memcpy, memcpy); +# ifdef SHARED +__hidden_ver1 (memcpy, __GI_memcpy, __redirect_memcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memcpy); +# endif + +#endif diff --git a/sysdeps/riscv/memcpy_noalignment.S b/sysdeps/riscv/memcpy_noalignment.S new file mode 100644 index 0000000000..907b858b18 --- /dev/null +++ b/sysdeps/riscv/memcpy_noalignment.S @@ -0,0 +1,134 @@ +/* memcpy for RISC-V, ignoring buffer alignment + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +#include + +/* void *memcpy(void *, const void *, size_t) */ +ENTRY (__memcpy_noalignment) + move t6, a0 /* Preserve return value */ + + /* Bail if 0 */ + beqz a2, 6f + + /* Jump to byte copy if size < SZREG */ + li a4, SZREG + bltu a2, a4, 5f + + /* Round down to the nearest "page" size */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + add a3, a1, a4 + + /* Copy the first word to get dest word aligned */ + andi a5, t6, SZREG-1 + beqz a5, 1f + REG_L a6, (a1) + REG_S a6, (t6) + + /* Align dst up to a word, move src and size as well. */ + addi t6, t6, SZREG-1 + andi t6, t6, ~(SZREG-1) + sub a5, t6, a0 + add a1, a1, a5 + sub a2, a2, a5 + + /* Recompute page count */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + +1: + /* Copy "pages" (chunks of 16 registers) */ + REG_L a4, 0(a1) + REG_L a5, SZREG(a1) + REG_L a6, 2*SZREG(a1) + REG_L a7, 3*SZREG(a1) + REG_L t0, 4*SZREG(a1) + REG_L t1, 5*SZREG(a1) + REG_L t2, 6*SZREG(a1) + REG_L t3, 7*SZREG(a1) + REG_L t4, 8*SZREG(a1) + REG_L t5, 9*SZREG(a1) + REG_S a4, 0(t6) + REG_S a5, SZREG(t6) + REG_S a6, 2*SZREG(t6) + REG_S a7, 3*SZREG(t6) + REG_S t0, 4*SZREG(t6) + REG_S t1, 5*SZREG(t6) + REG_S t2, 6*SZREG(t6) + REG_S t3, 7*SZREG(t6) + REG_S t4, 8*SZREG(t6) + REG_S t5, 9*SZREG(t6) + REG_L a4, 10*SZREG(a1) + REG_L a5, 11*SZREG(a1) + REG_L a6, 12*SZREG(a1) + REG_L a7, 13*SZREG(a1) + REG_L t0, 14*SZREG(a1) + REG_L t1, 15*SZREG(a1) + addi a1, a1, 16*SZREG + REG_S a4, 10*SZREG(t6) + REG_S a5, 11*SZREG(t6) + REG_S a6, 12*SZREG(t6) + REG_S a7, 13*SZREG(t6) + REG_S t0, 14*SZREG(t6) + REG_S t1, 15*SZREG(t6) + addi t6, t6, 16*SZREG + bltu a1, a3, 1b + andi a2, a2, (16*SZREG)-1 /* Update count */ + +2: + /* Remainder is smaller than a page, compute native word count */ + beqz a2, 6f + andi a5, a2, ~(SZREG-1) + andi a2, a2, (SZREG-1) + add a3, a1, a5 + /* Jump directly to last word if no words. */ + beqz a5, 4f + +3: + /* Use single native register copy */ + REG_L a4, 0(a1) + addi a1, a1, SZREG + REG_S a4, 0(t6) + addi t6, t6, SZREG + bltu a1, a3, 3b + + /* Jump directly out if no more bytes */ + beqz a2, 6f + +4: + /* Copy the last word unaligned */ + add a3, a1, a2 + add a4, t6, a2 + REG_L a4, -SZREG(a4) + REG_S a4, -SZREG(a3) + ret + +5: + /* Copy bytes when the total copy is . */ + +#include + +extern __typeof (memcpy) __memcpy_generic; +hidden_proto(__memcpy_generic) + +#include