From patchwork Fri Apr 7 23:07:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 67539 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id ADF723856DC0 for ; Fri, 7 Apr 2023 23:07:37 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by sourceware.org (Postfix) with ESMTPS id 1D20B3858D32 for ; Fri, 7 Apr 2023 23:07:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1D20B3858D32 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x633.google.com with SMTP id ji2so1521249plb.0 for ; Fri, 07 Apr 2023 16:07:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20210112.gappssmtp.com; s=20210112; t=1680908841; x=1683500841; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qV6ObTkPeZQv5lrZcGdaOqFQVSPJRHj3Ff7a977zjVQ=; b=VnMBe3e2iFzYot4uW+xQiU+Akvgfa8uitmEee2/nrvEzqTYr7XoxZHv+oHSx1Ic+FV SQbDm3mcf7uoAiM2AKiGl0TL/poRv94RZidXZ19z5y1/qcj/bJ2PQxsZ0BAG7HQMyiXu sp73eUEatLpikkpzbdsetMxWNz5K8N7/t/TErgAQ5F2HioMdNXA4LXE1BMJ6h2uLiyqb 5QMoEgGigCA6+Fgp5FStdAAF9S0dpy1RSDF2tQKetvM1UzdVX8F0lIbVK825M0ztsLJ5 HgFjJA6S2JQM05Y6BP2PEzSQSb+hCjebJLOWxcxW7sWsFieaRFAz2i4NqATJjlRAlnQg nIPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680908841; x=1683500841; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qV6ObTkPeZQv5lrZcGdaOqFQVSPJRHj3Ff7a977zjVQ=; b=aFwvqrEA65H5MIQjgNIbEU2Sq5dMimxuuJKfOWPxfs57ilzK2G/+SnKVTuvP06svka RZ+y1yVLz0Td1SugGQmFJX7wU/Lna79aRr+ZaOIZzqkKre0F51zNwD3OCxCNjCAO0FS6 d7jL7DW9l6CZV2ugW4R+62nPxLl7zE41dFopZL7ne0LTE2Gie4efug+sDjMyVv+fz7IW r5bvzzUz+YeyBQXI1XpTNid7VE0+0a7Dw935+C9SrUWdnqI0TD+/42y3EQrE0dbrIebu FkMxBXXTHnyQ4Wg7Kcp9F6jRMEKGNcLNbtXr1b/ZwNRc2WgjcGzhnDAEkVmsXLkOMCM9 8gLg== X-Gm-Message-State: AAQBX9ccWmpBtJAz8GCBJF08dvVWnBevyL3XzoWHEfP9rY/bk5WbjcBv dGz997sZEw3JNzfeaiSLdfOuOciMft42Fv17VxM= X-Google-Smtp-Source: AKy350Z8of36dRlL7n5DXauw3AvBdugL7PD+Zoc2f8hEmt03bO7wDL/s5EAAwK8I+HFwiYeEmnSF3Q== X-Received: by 2002:a17:90b:4ad1:b0:23d:9ae:a49a with SMTP id mh17-20020a17090b4ad100b0023d09aea49amr4483352pjb.30.1680908840690; Fri, 07 Apr 2023 16:07:20 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id ei7-20020a17090ae54700b00240ab3c5f66sm3224691pjb.29.2023.04.07.16.07.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Apr 2023 16:07:20 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, palmer@rivosinc.com, slewis@rivosinc.com, Evan Green Subject: [PATCH v3 1/3] riscv: Add Linux hwprobe syscall support Date: Fri, 7 Apr 2023 16:07:08 -0700 Message-Id: <20230407230711.2621614-2-evan@rivosinc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230407230711.2621614-1-evan@rivosinc.com> References: <20230407230711.2621614-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Add awareness and a thin wrapper function around a new Linux system call that allows callers to get architecture and microarchitecture information about the CPUs from the kernel. This can be used to do things like dynamically choose a memcpy implementation. Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- Changes in v3: - Update argument types to match v4 kernel interface Changes in v2: - hwprobe.h: Use __has_include and duplicate Linux content to make compilation work when Linux headers are absent (Adhemerval) - hwprobe.h: Put declaration under __USE_GNU (Adhemerval) - Use INLINE_SYSCALL_CALL (Adhemerval) - Update versions - Update UNALIGNED_MASK to match kernel v3 series. sysdeps/unix/sysv/linux/riscv/Makefile | 4 +- sysdeps/unix/sysv/linux/riscv/Versions | 3 + sysdeps/unix/sysv/linux/riscv/hwprobe.c | 30 ++++++++ .../unix/sysv/linux/riscv/rv32/arch-syscall.h | 1 + .../unix/sysv/linux/riscv/rv32/libc.abilist | 1 + .../unix/sysv/linux/riscv/rv64/arch-syscall.h | 1 + .../unix/sysv/linux/riscv/rv64/libc.abilist | 1 + sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h | 68 +++++++++++++++++++ sysdeps/unix/sysv/linux/syscall-names.list | 1 + 9 files changed, 108 insertions(+), 2 deletions(-) create mode 100644 sysdeps/unix/sysv/linux/riscv/hwprobe.c create mode 100644 sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h diff --git a/sysdeps/unix/sysv/linux/riscv/Makefile b/sysdeps/unix/sysv/linux/riscv/Makefile index 4b6eacb32f..45cc29e40d 100644 --- a/sysdeps/unix/sysv/linux/riscv/Makefile +++ b/sysdeps/unix/sysv/linux/riscv/Makefile @@ -1,6 +1,6 @@ ifeq ($(subdir),misc) -sysdep_headers += sys/cachectl.h -sysdep_routines += flush-icache +sysdep_headers += sys/cachectl.h sys/hwprobe.h +sysdep_routines += flush-icache hwprobe endif ifeq ($(subdir),stdlib) diff --git a/sysdeps/unix/sysv/linux/riscv/Versions b/sysdeps/unix/sysv/linux/riscv/Versions index 5625d2a0b8..8717b62a4a 100644 --- a/sysdeps/unix/sysv/linux/riscv/Versions +++ b/sysdeps/unix/sysv/linux/riscv/Versions @@ -8,4 +8,7 @@ libc { GLIBC_2.27 { __riscv_flush_icache; } + GLIBC_2.39 { + __riscv_hwprobe; + } } diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe.c b/sysdeps/unix/sysv/linux/riscv/hwprobe.c new file mode 100644 index 0000000000..e9023e2fba --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe.c @@ -0,0 +1,30 @@ +/* RISC-V hardware feature probing support on Linux + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +int +__riscv_hwprobe (struct riscv_hwprobe *pairs, size_t pair_count, + size_t cpu_count, unsigned long *cpus, unsigned int flags) +{ + return INLINE_SYSCALL_CALL (riscv_hwprobe, pairs, pair_count, + cpu_count, cpus, flags); +} diff --git a/sysdeps/unix/sysv/linux/riscv/rv32/arch-syscall.h b/sysdeps/unix/sysv/linux/riscv/rv32/arch-syscall.h index 202520ee25..2416e041c8 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv32/arch-syscall.h +++ b/sysdeps/unix/sysv/linux/riscv/rv32/arch-syscall.h @@ -198,6 +198,7 @@ #define __NR_request_key 218 #define __NR_restart_syscall 128 #define __NR_riscv_flush_icache 259 +#define __NR_riscv_hwprobe 258 #define __NR_rseq 293 #define __NR_rt_sigaction 134 #define __NR_rt_sigpending 136 diff --git a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist index b716f5c763..1548587eb1 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist @@ -2428,3 +2428,4 @@ GLIBC_2.38 __isoc23_wcstoull F GLIBC_2.38 __isoc23_wcstoull_l F GLIBC_2.38 __isoc23_wcstoumax F GLIBC_2.38 __isoc23_wscanf F +GLIBC_2.39 __riscv_hwprobe F diff --git a/sysdeps/unix/sysv/linux/riscv/rv64/arch-syscall.h b/sysdeps/unix/sysv/linux/riscv/rv64/arch-syscall.h index 4e65f337d4..a32bc82f60 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv64/arch-syscall.h +++ b/sysdeps/unix/sysv/linux/riscv/rv64/arch-syscall.h @@ -205,6 +205,7 @@ #define __NR_request_key 218 #define __NR_restart_syscall 128 #define __NR_riscv_flush_icache 259 +#define __NR_riscv_hwprobe 258 #define __NR_rseq 293 #define __NR_rt_sigaction 134 #define __NR_rt_sigpending 136 diff --git a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist index 774e777b65..49c0577b61 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist @@ -2628,3 +2628,4 @@ GLIBC_2.38 __isoc23_wcstoull F GLIBC_2.38 __isoc23_wcstoull_l F GLIBC_2.38 __isoc23_wcstoumax F GLIBC_2.38 __isoc23_wscanf F +GLIBC_2.39 __riscv_hwprobe F diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h new file mode 100644 index 0000000000..49e27ee855 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h @@ -0,0 +1,68 @@ +/* RISC-V architecture probe interface + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#ifndef _SYS_HWPROBE_H +#define _SYS_HWPROBE_H 1 + +#include +#include +#ifdef __has_include +# if __has_include () +# include +# endif +#endif + +/* + * Define a (probably stale) version of the interface if the Linux headers + * aren't present. + */ +#ifndef RISCV_HWPROBE_KEY_MVENDORID +struct riscv_hwprobe { + signed long long key; + unsigned long long value; +}; + +#define RISCV_HWPROBE_KEY_MVENDORID 0 +#define RISCV_HWPROBE_KEY_MARCHID 1 +#define RISCV_HWPROBE_KEY_MIMPID 2 +#define RISCV_HWPROBE_KEY_BASE_BEHAVIOR 3 +#define RISCV_HWPROBE_BASE_BEHAVIOR_IMA (1 << 0) +#define RISCV_HWPROBE_KEY_IMA_EXT_0 4 +#define RISCV_HWPROBE_IMA_FD (1 << 0) +#define RISCV_HWPROBE_IMA_C (1 << 1) +#define RISCV_HWPROBE_KEY_CPUPERF_0 5 +#define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0) +#define RISCV_HWPROBE_MISALIGNED_EMULATED (1 << 0) +#define RISCV_HWPROBE_MISALIGNED_SLOW (2 << 0) +#define RISCV_HWPROBE_MISALIGNED_FAST (3 << 0) +#define RISCV_HWPROBE_MISALIGNED_MASK (7 << 0) + +#endif // RISCV_HWPROBE_KEY_MVENDORID + +__BEGIN_DECLS + +#ifdef __USE_GNU +int +__riscv_hwprobe (struct riscv_hwprobe *pairs, size_t pair_count, + size_t cpu_count, unsigned long *cpus, unsigned int flags); +#endif + +__END_DECLS + +#endif /* sys/hwprobe.h */ diff --git a/sysdeps/unix/sysv/linux/syscall-names.list b/sysdeps/unix/sysv/linux/syscall-names.list index 5d27b5279c..bbc0e01d3a 100644 --- a/sysdeps/unix/sysv/linux/syscall-names.list +++ b/sysdeps/unix/sysv/linux/syscall-names.list @@ -477,6 +477,7 @@ renameat2 request_key restart_syscall riscv_flush_icache +riscv_hwprobe rmdir rseq rt_sigaction From patchwork Fri Apr 7 23:07:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 67541 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0351B3855584 for ; Fri, 7 Apr 2023 23:08:01 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by sourceware.org (Postfix) with ESMTPS id 13D783858002 for ; Fri, 7 Apr 2023 23:07:25 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 13D783858002 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pj1-x102a.google.com with SMTP id qe8-20020a17090b4f8800b0023f07253a2cso44393307pjb.3 for ; Fri, 07 Apr 2023 16:07:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20210112.gappssmtp.com; s=20210112; t=1680908843; x=1683500843; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XMG14HTGNa6uAe7w+Dfmw/9dDeb1ZA19ComYuXgl3iY=; b=F3LjIBenjGDuSCgDJmBo7pn6GaUYs9EQkiX4yU5SpfbHBNIw2K6t32ynD8sXIEGipo WXh8KU5f8laGvFrnWjwQyvVb5cWCnuXgNl1th37QEim1rfmstwPxphFzjoerfvGUM3CS irF5YW72TjiPqykxwTiYjFhDySh7W97q5a3pqkDPAnNTDEUc597ibrfyRimwNOMhCXCl 3vqLspcUSclwawR4Knqmb/65TrZjegee0DYmnM14gY+0jPIUBXPTv45yCqKSWpB+shPL LKucUaQQTze87/U65wPeEQVX4O4BCZPzAoay5PJcbyUyua6VSLrEz+09cfclikKibjtK uIAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680908843; x=1683500843; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XMG14HTGNa6uAe7w+Dfmw/9dDeb1ZA19ComYuXgl3iY=; b=lhNuhMU6Yv0GGWcdbbRcNypAWzT7vh3t3A1VlURfuK3POHYbM3/tRAbxz9A+H/5aNQ M1WCWEbxZ762jO33ZkuNHbxquUYoVr4SKJMGEAEODg/vQZwqYXqNl9gjV2m6E4DxWKCT vt/MCK0ruW7yR9pOWMVv/NrUyOytgxEHH3pFIL3ql4Xjdz0OPt4TsyLxbiFewRZFqy4T UmkUjwggd6PAdTwfrSJvRb8rcL8r2op/PL8fQZJ6Pr3GKGJm+rkJ0jFjSRU4rmCneOWy +hZpzmE9X1hVXUAhcsY5Y8Q+gIPl/CvB3gINLWR1TffayUQ+ltvPIlH7xNPZH8K3H4gE gs+Q== X-Gm-Message-State: AAQBX9efFOzyVmKTVFXAOC9igymJMevTFMojzd0VRCyIRcC9q8YgVRDg wlBpAon9i/SY2d5BoYh+fj6NYp2tkkG6l2BA7b4= X-Google-Smtp-Source: AKy350bgYZ9YwPpLRk3rZJIB+dQx46041gnD56rT2aGNIGSF3KGG9r0P5jtCzIiUoHWO3mKDqOlDkw== X-Received: by 2002:a17:90a:e7c3:b0:233:f786:35ca with SMTP id kb3-20020a17090ae7c300b00233f78635camr418372pjb.35.1680908843694; Fri, 07 Apr 2023 16:07:23 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id ei7-20020a17090ae54700b00240ab3c5f66sm3224691pjb.29.2023.04.07.16.07.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Apr 2023 16:07:23 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, palmer@rivosinc.com, slewis@rivosinc.com, Evan Green Subject: [PATCH v3 2/3] riscv: Add hwprobe vdso call support Date: Fri, 7 Apr 2023 16:07:09 -0700 Message-Id: <20230407230711.2621614-3-evan@rivosinc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230407230711.2621614-1-evan@rivosinc.com> References: <20230407230711.2621614-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" The new riscv_hwprobe syscall also comes with a vDSO for faster answers to your most common questions. Call in today to speak with a kernel representative near you! Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- Changes in v3: - Add the "return" to the vsyscall - Fix up vdso arg types to match kernel v4 version - Remove ifdef around INLINE_VSYSCALL (Adhemerval) Changes in v2: - Add vDSO interface sysdeps/unix/sysv/linux/dl-vdso-setup.c | 10 ++++++++++ sysdeps/unix/sysv/linux/dl-vdso-setup.h | 3 +++ sysdeps/unix/sysv/linux/riscv/hwprobe.c | 5 +++-- sysdeps/unix/sysv/linux/riscv/sysdep.h | 1 + 4 files changed, 17 insertions(+), 2 deletions(-) diff --git a/sysdeps/unix/sysv/linux/dl-vdso-setup.c b/sysdeps/unix/sysv/linux/dl-vdso-setup.c index 68fa8de641..3fc558afdd 100644 --- a/sysdeps/unix/sysv/linux/dl-vdso-setup.c +++ b/sysdeps/unix/sysv/linux/dl-vdso-setup.c @@ -71,6 +71,16 @@ PROCINFO_CLASS int (*_dl_vdso_clock_getres_time64) (clockid_t, # ifdef HAVE_GET_TBFREQ PROCINFO_CLASS uint64_t (*_dl_vdso_get_tbfreq)(void) RELRO; # endif + +/* RISC-V specific ones. */ +# ifdef HAVE_RISCV_HWPROBE +PROCINFO_CLASS int (*_dl_vdso_riscv_hwprobe)(void *, + size_t, + size_t, + unsigned long *, + unsigned int) RELRO; +# endif + #endif #undef RELRO diff --git a/sysdeps/unix/sysv/linux/dl-vdso-setup.h b/sysdeps/unix/sysv/linux/dl-vdso-setup.h index 867072b897..39eafd5316 100644 --- a/sysdeps/unix/sysv/linux/dl-vdso-setup.h +++ b/sysdeps/unix/sysv/linux/dl-vdso-setup.h @@ -47,6 +47,9 @@ setup_vdso_pointers (void) #ifdef HAVE_GET_TBFREQ GLRO(dl_vdso_get_tbfreq) = dl_vdso_vsym (HAVE_GET_TBFREQ); #endif +#ifdef HAVE_RISCV_HWPROBE + GLRO(dl_vdso_riscv_hwprobe) = dl_vdso_vsym (HAVE_RISCV_HWPROBE); +#endif } #endif diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe.c b/sysdeps/unix/sysv/linux/riscv/hwprobe.c index e9023e2fba..6bf28f574b 100644 --- a/sysdeps/unix/sysv/linux/riscv/hwprobe.c +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe.c @@ -20,11 +20,12 @@ #include #include #include +#include int __riscv_hwprobe (struct riscv_hwprobe *pairs, size_t pair_count, size_t cpu_count, unsigned long *cpus, unsigned int flags) { - return INLINE_SYSCALL_CALL (riscv_hwprobe, pairs, pair_count, - cpu_count, cpus, flags); + /* The vDSO may be able to provide the answer without a syscall. */ + return INLINE_VSYSCALL(riscv_hwprobe, 5, pairs, pair_count, cpu_count, cpus, flags); } diff --git a/sysdeps/unix/sysv/linux/riscv/sysdep.h b/sysdeps/unix/sysv/linux/riscv/sysdep.h index 4af5fe5dbc..ba17aaaff2 100644 --- a/sysdeps/unix/sysv/linux/riscv/sysdep.h +++ b/sysdeps/unix/sysv/linux/riscv/sysdep.h @@ -155,6 +155,7 @@ /* List of system calls which are supported as vsyscalls (for RV32 and RV64). */ # define HAVE_GETCPU_VSYSCALL "__vdso_getcpu" +# define HAVE_RISCV_HWPROBE "__vdso_riscv_hwprobe" # undef HAVE_INTERNAL_BRK_ADDR_SYMBOL # define HAVE_INTERNAL_BRK_ADDR_SYMBOL 1 From patchwork Fri Apr 7 23:07:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 67540 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 1146D385783F for ; Fri, 7 Apr 2023 23:07:51 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by sourceware.org (Postfix) with ESMTPS id 9E7E63858C1F for ; Fri, 7 Apr 2023 23:07:28 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 9E7E63858C1F Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x62a.google.com with SMTP id kx12so2223644plb.12 for ; Fri, 07 Apr 2023 16:07:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20210112.gappssmtp.com; s=20210112; t=1680908847; x=1683500847; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YkaGDQE+tPHHgll1cZde3EZiWrEIMjVRjFYMfVGQSEg=; b=LAOuux/I61btKejMDMRE2yCa8y0XX47M9Az9mTRmj/pQT55Ew/0jHMW95y8poHAI5R 6DCA4Hh/lraVGzvP6/yR8uQp0AX0IMQTHJCVNen7pKkCdj1kHSejIFizDhtG4Ab/hWap KfpMjBd6PQS8pTrtMBuTLiNBlbvYJan+5tsSmxHpKOYl6STMwYy9eMYVWSs2qq21dnvY hiWMfBgSkIJDEyiLyALR7WaPMj5+HWme1YaAJgprmwqJoBR2g+9SkTYEstvJVlCNVTnv MyF0WxKJRNPzHVDNQNxOiCy8PvsdTsYgTGHHGsdUkrJcU90F9n9eBtaVG4MZh72JDvC5 SDzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680908847; x=1683500847; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YkaGDQE+tPHHgll1cZde3EZiWrEIMjVRjFYMfVGQSEg=; b=l/Q2HpJC4jCyCStBHN88Jt1LsiIV8YjQp5vq8sxCA0+1aDjddLfvr12Of3tnqa+e7x S1q+LIu+GUP9xqdqdIPCv1fafnhyw/Ts0ZfxPZUgVFJg5ycHflPN/zoGSvqNyKCb6y8K TSOtXscA9xATgmxTEN3jNkztF8bel9hSdM5kcn7yJGAruYga5bcZzPho5oVdGmj6mNE1 nbklLZkoVWdbjxXcY+x4ZJ/KEBlBy8SdRqs1psmLI0vJM2AoVrjq7yHP06VwyDE8GxYc NF1tkiVVzKLwt3JN1yvuE80RtqgzyDcCxIq8cOAM0M9qvdZdvOoTkIEiTgvmuqLm9v1x Xd+g== X-Gm-Message-State: AAQBX9fpxcZNzA09p7iP/cUcwWaJiRLwzFzQIINmR/1yAMFWIvijJg0C c3HOJnkYp9hcjZpzbvzYRq/bWaKEm6FAxbowZxg= X-Google-Smtp-Source: AKy350ZBA4WGPEMIaHfMPNnRf9X/aKqKVAN/0Nsh3ZO+q+5DgWr7+9f2/mRanYHG0PK3yXCq9x0H1Q== X-Received: by 2002:a17:90b:4c89:b0:23d:1e5f:eea6 with SMTP id my9-20020a17090b4c8900b0023d1e5feea6mr4140293pjb.24.1680908847149; Fri, 07 Apr 2023 16:07:27 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id ei7-20020a17090ae54700b00240ab3c5f66sm3224691pjb.29.2023.04.07.16.07.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Apr 2023 16:07:26 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, palmer@rivosinc.com, slewis@rivosinc.com, Evan Green Subject: [PATCH v3 3/3] riscv: Add and use alignment-ignorant memcpy Date: Fri, 7 Apr 2023 16:07:10 -0700 Message-Id: <20230407230711.2621614-4-evan@rivosinc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230407230711.2621614-1-evan@rivosinc.com> References: <20230407230711.2621614-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" For CPU implementations that can perform unaligned accesses with little or no performance penalty, create a memcpy implementation that does not bother aligning buffers. It will use a block of integer registers, a single integer register, and fall back to bytewise copy for the remainder. Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- Changes in v3: - Word align dest for large memcpy()s. - Add tags - Remove spurious blank line from sysdeps/riscv/memcpy.c Changes in v2: - Used _MASK instead of _FAST value itself. --- sysdeps/riscv/memcopy.h | 28 ++++ sysdeps/riscv/memcpy.c | 64 +++++++++ sysdeps/riscv/memcpy_noalignment.S | 121 ++++++++++++++++++ sysdeps/unix/sysv/linux/riscv/Makefile | 4 + .../unix/sysv/linux/riscv/memcpy-generic.c | 24 ++++ 5 files changed, 241 insertions(+) create mode 100644 sysdeps/riscv/memcopy.h create mode 100644 sysdeps/riscv/memcpy.c create mode 100644 sysdeps/riscv/memcpy_noalignment.S create mode 100644 sysdeps/unix/sysv/linux/riscv/memcpy-generic.c diff --git a/sysdeps/riscv/memcopy.h b/sysdeps/riscv/memcopy.h new file mode 100644 index 0000000000..21f6081b5f --- /dev/null +++ b/sysdeps/riscv/memcopy.h @@ -0,0 +1,28 @@ +/* memcopy.h -- definitions for memory copy functions. RISC-V version. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +/* + * Redefine the generic memcpy implementation to __memcpy_generic, so + * the memcpy ifunc can select between generic and special versions. + * In rtld, don't bother with all the ifunciness. + */ +#if IS_IN (libc) +#define MEMCPY __memcpy_generic +#endif diff --git a/sysdeps/riscv/memcpy.c b/sysdeps/riscv/memcpy.c new file mode 100644 index 0000000000..fdb8dc3208 --- /dev/null +++ b/sysdeps/riscv/memcpy.c @@ -0,0 +1,64 @@ +/* Multiple versions of memcpy. + All versions must be listed in ifunc-impl-list.c. + Copyright (C) 2017-2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +/* Redefine memcpy so that the compiler won't complain about the type + mismatch with the IFUNC selector in strong_alias, below. */ +# undef memcpy +# define memcpy __redirect_memcpy +# include +#include +#include + +#define INIT_ARCH() + +extern __typeof (__redirect_memcpy) __libc_memcpy; + +extern __typeof (__redirect_memcpy) __memcpy_generic attribute_hidden; +extern __typeof (__redirect_memcpy) __memcpy_noalignment attribute_hidden; + +static inline __typeof (__redirect_memcpy) * +select_memcpy_ifunc (void) +{ + INIT_ARCH (); + + struct riscv_hwprobe pair; + + pair.key = RISCV_HWPROBE_KEY_CPUPERF_0; + if (__riscv_hwprobe(&pair, 1, 0, NULL, 0) != 0) + return __memcpy_generic; + + if ((pair.key > 0) && + (pair.value & RISCV_HWPROBE_MISALIGNED_MASK) == + RISCV_HWPROBE_MISALIGNED_FAST) + return __memcpy_noalignment; + + return __memcpy_generic; +} + +libc_ifunc (__libc_memcpy, select_memcpy_ifunc ()); + +# undef memcpy +strong_alias (__libc_memcpy, memcpy); +# ifdef SHARED +__hidden_ver1 (memcpy, __GI_memcpy, __redirect_memcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memcpy); +# endif + +#endif diff --git a/sysdeps/riscv/memcpy_noalignment.S b/sysdeps/riscv/memcpy_noalignment.S new file mode 100644 index 0000000000..80f5e09ebb --- /dev/null +++ b/sysdeps/riscv/memcpy_noalignment.S @@ -0,0 +1,121 @@ +/* memcpy for RISC-V, ignoring buffer alignment + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +#include + +/* void *memcpy(void *, const void *, size_t) */ +ENTRY (__memcpy_noalignment) + move t6, a0 /* Preserve return value */ + + /* Round down to the nearest "page" size */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + add a3, a1, a4 + + /* Copy the first word to get dest word aligned */ + andi a5, t6, SZREG-1 + beqz a5, 1f + REG_L a6, (a1) + REG_S a6, (t6) + + /* Align dst up to a word, move src and size as well. */ + addi t6, t6, SZREG-1 + andi t6, t6, ~(SZREG-1) + sub a5, t6, a0 + add a1, a1, a5 + sub a2, a2, a5 + + /* Recompute page count */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + +1: + /* Copy "pages" (chunks of 16 registers) */ + REG_L a4, 0(a1) + REG_L a5, SZREG(a1) + REG_L a6, 2*SZREG(a1) + REG_L a7, 3*SZREG(a1) + REG_L t0, 4*SZREG(a1) + REG_L t1, 5*SZREG(a1) + REG_L t2, 6*SZREG(a1) + REG_L t3, 7*SZREG(a1) + REG_L t4, 8*SZREG(a1) + REG_L t5, 9*SZREG(a1) + REG_S a4, 0(t6) + REG_S a5, SZREG(t6) + REG_S a6, 2*SZREG(t6) + REG_S a7, 3*SZREG(t6) + REG_S t0, 4*SZREG(t6) + REG_S t1, 5*SZREG(t6) + REG_S t2, 6*SZREG(t6) + REG_S t3, 7*SZREG(t6) + REG_S t4, 8*SZREG(t6) + REG_S t5, 9*SZREG(t6) + REG_L a4, 10*SZREG(a1) + REG_L a5, 11*SZREG(a1) + REG_L a6, 12*SZREG(a1) + REG_L a7, 13*SZREG(a1) + REG_L t0, 14*SZREG(a1) + REG_L t1, 15*SZREG(a1) + addi a1, a1, 16*SZREG + REG_S a4, 10*SZREG(t6) + REG_S a5, 11*SZREG(t6) + REG_S a6, 12*SZREG(t6) + REG_S a7, 13*SZREG(t6) + REG_S t0, 14*SZREG(t6) + REG_S t1, 15*SZREG(t6) + addi t6, t6, 16*SZREG + bltu a1, a3, 1b + andi a2, a2, (16*SZREG)-1 /* Update count */ + +2: + /* Remainder is smaller than a page, compute native word count */ + beqz a2, 6f + andi a5, a2, ~(SZREG-1) + andi a2, a2, (SZREG-1) + add a3, a1, a5 + /* Jump directly to byte copy if no words. */ + beqz a5, 4f + +3: + /* Use single native register copy */ + REG_L a4, 0(a1) + addi a1, a1, SZREG + REG_S a4, 0(t6) + addi t6, t6, SZREG + bltu a1, a3, 3b + + /* Jump directly out if no more bytes */ + beqz a2, 6f + +4: + /* Copy the last few individual bytes */ + add a3, a1, a2 +5: + lb a4, 0(a1) + addi a1, a1, 1 + sb a4, 0(t6) + addi t6, t6, 1 + bltu a1, a3, 5b +6: + ret + +END (__memcpy_noalignment) + +hidden_def (__memcpy_noalignment) diff --git a/sysdeps/unix/sysv/linux/riscv/Makefile b/sysdeps/unix/sysv/linux/riscv/Makefile index 45cc29e40d..aa9ea443d6 100644 --- a/sysdeps/unix/sysv/linux/riscv/Makefile +++ b/sysdeps/unix/sysv/linux/riscv/Makefile @@ -7,6 +7,10 @@ ifeq ($(subdir),stdlib) gen-as-const-headers += ucontext_i.sym endif +ifeq ($(subdir),string) +sysdep_routines += memcpy memcpy-generic memcpy_noalignment +endif + abi-variants := ilp32 ilp32d lp64 lp64d ifeq (,$(filter $(default-abi),$(abi-variants))) diff --git a/sysdeps/unix/sysv/linux/riscv/memcpy-generic.c b/sysdeps/unix/sysv/linux/riscv/memcpy-generic.c new file mode 100644 index 0000000000..0abe03f7f5 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/memcpy-generic.c @@ -0,0 +1,24 @@ +/* Re-include the default memcpy implementation. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +extern __typeof (memcpy) __memcpy_generic; +hidden_proto(__memcpy_generic) + +#include