From patchwork Wed May 13 23:08:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Blanchard X-Patchwork-Id: 39243 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 877D2388B03B; Wed, 13 May 2020 23:08:42 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 877D2388B03B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1589411322; bh=eZc0v3gcYQgR1lnFUwmRDF9qovGU8RoO3FNxiaEdjys=; h=Date:To:Subject:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=IEqXdVuWDabayPl26Lsg881XbVapF3sR54Iabqe+SRC/W+ZfANuXzlm45Zne6k3pg fLGvaHLPY8miuhgq+btKFdAzjM7L8DV1Ctsx0PBLj/X/nnikOVOR39mMife8vEiYVD Tt/f1N9/vo+ThUAQo1J4sYYpJjFISWN7w31ksCFE= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from ozlabs.org (ozlabs.org [IPv6:2401:3900:2:1::2]) by sourceware.org (Postfix) with ESMTPS id 6C200385DC00 for ; Wed, 13 May 2020 23:08:39 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 6C200385DC00 Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 49Mr2D5pSCz9sSd; Thu, 14 May 2020 09:08:36 +1000 (AEST) Date: Thu, 14 May 2020 09:08:35 +1000 To: libc-alpha@sourceware.org Subject: [PATCH 2/2] powerpc: Optimized stpcpy for POWER9 Message-ID: <20200514090835.1e1cdc49@kryten.localdomain> In-Reply-To: <20200514090026.6889aa77@kryten.localdomain> References: <20200514090026.6889aa77@kryten.localdomain> X-Mailer: Mutt/1.8.0 (2017-02-23) MIME-Version: 1.0 X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, JMQ_SPF_NEUTRAL, KAM_NUMSUBJECT, KAM_SHORT, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Anton Blanchard via Libc-alpha From: Anton Blanchard Reply-To: Anton Blanchard Cc: Raphael M Zinsly Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" Add stpcpy support to the POWER9 strcpy. This is up to 40% faster on small strings and up to 90% faster on long relatively unaligned strings, compared to the POWER8 version. A few examples: __stpcpy_power9 __stpcpy_power8 Length 20, alignments in bytes 4/ 4: 2.58246 4.8788 Length 1024, alignments in bytes 1/ 6: 24.8186 47.8528 --- sysdeps/powerpc/powerpc64/le/power9/stpcpy.S | 24 ++++++ sysdeps/powerpc/powerpc64/le/power9/strcpy.S | 73 +++++++++++++++---- sysdeps/powerpc/powerpc64/multiarch/Makefile | 2 +- .../powerpc64/multiarch/ifunc-impl-list.c | 4 + .../powerpc64/multiarch/stpcpy-power9.S | 24 ++++++ sysdeps/powerpc/powerpc64/multiarch/stpcpy.c | 17 +++-- 6 files changed, 123 insertions(+), 21 deletions(-) create mode 100644 sysdeps/powerpc/powerpc64/le/power9/stpcpy.S create mode 100644 sysdeps/powerpc/powerpc64/multiarch/stpcpy-power9.S diff --git a/sysdeps/powerpc/powerpc64/le/power9/stpcpy.S b/sysdeps/powerpc/powerpc64/le/power9/stpcpy.S new file mode 100644 index 0000000000..44425cb1e8 --- /dev/null +++ b/sysdeps/powerpc/powerpc64/le/power9/stpcpy.S @@ -0,0 +1,24 @@ +/* Optimized stpcpy implementation for PowerPC64/POWER9. + Copyright (C) 2015-2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#define USE_AS_STPCPY +#include + +weak_alias (__stpcpy, stpcpy) +libc_hidden_def (__stpcpy) +libc_hidden_builtin_def (stpcpy) diff --git a/sysdeps/powerpc/powerpc64/le/power9/strcpy.S b/sysdeps/powerpc/powerpc64/le/power9/strcpy.S index 5749228054..ce8f503291 100644 --- a/sysdeps/powerpc/powerpc64/le/power9/strcpy.S +++ b/sysdeps/powerpc/powerpc64/le/power9/strcpy.S @@ -18,19 +18,35 @@ #include -#ifndef STRCPY -# define STRCPY strcpy -#endif +#ifdef USE_AS_STPCPY +# ifndef STPCPY +# define FUNC_NAME __stpcpy +# else +# define FUNC_NAME STPCPY +# endif +#else +# ifndef STRCPY +# define FUNC_NAME strcpy +# else +# define FUNC_NAME STRCPY +# endif +#endif /* !USE_AS_STPCPY */ /* Implements the function char * [r3] strcpy (char *dest [r3], const char *src [r4]) + or + + char * [r3] stpcpy (char *dest [r3], const char *src [r4]) + + if USE_AS_STPCPY is defined. + The implementation can load bytes past a null terminator, but only up to the next 16B boundary, so it never crosses a page. */ .machine power9 -ENTRY_TOCLESS (STRCPY, 4) +ENTRY_TOCLESS (FUNC_NAME, 4) CALL_MCOUNT 2 /* NULL string optimisation */ @@ -53,8 +69,8 @@ ENTRY_TOCLESS (STRCPY, 4) vperm v0,v18,v0,v1 vcmpequb v6,v0,v18 /* 0xff if byte is NULL, 0x00 otherwise */ - vctzlsbb r8,v6 /* Number of trailing zeroes */ - addi r8,r8,1 /* Add null terminator */ + vctzlsbb r7,v6 /* Number of trailing zeroes */ + addi r8,r7,1 /* Add null terminator */ /* r8 = bytes including null r9 = bytes to get source 16B aligned @@ -68,6 +84,11 @@ ENTRY_TOCLESS (STRCPY, 4) sldi r10,r8,56 /* stxvl wants size in top 8 bits */ stxvl 32+v0,r11,r10 /* Partial store */ +#ifdef USE_AS_STPCPY + /* stpcpy returns the dest address plus the size not counting the + final '\0'. */ + add r3,r11,r7 +#endif blr L(no_null): @@ -106,28 +127,43 @@ L(loop): L(tail1): vctzlsbb r8,v6 - addi r8,r8,1 - sldi r9,r8,56 /* stxvl wants size in top 8 bits */ + addi r9,r8,1 + sldi r9,r9,56 /* stxvl wants size in top 8 bits */ stxvl 32+v0,r11,r9 +#ifdef USE_AS_STPCPY + /* stpcpy returns the dest address plus the size not counting the + final '\0'. */ + add r3,r11,r8 +#endif blr L(tail2): stxv 32+v0,0(r11) vctzlsbb r8,v6 /* Number of trailing zeroes */ - addi r8,r8,1 /* Add null terminator */ - sldi r10,r8,56 /* stxvl wants size in top 8 bits */ + addi r9,r8,1 /* Add null terminator */ + sldi r10,r9,56 /* stxvl wants size in top 8 bits */ addi r11,r11,16 stxvl 32+v1,r11,r10 /* Partial store */ +#ifdef USE_AS_STPCPY + /* stpcpy returns the dest address plus the size not counting the + final '\0'. */ + add r3,r11,r8 +#endif blr L(tail3): stxv 32+v0,0(r11) stxv 32+v1,16(r11) vctzlsbb r8,v6 /* Number of trailing zeroes */ - addi r8,r8,1 /* Add null terminator */ - sldi r10,r8,56 /* stxvl wants size in top 8 bits */ + addi r9,r8,1 /* Add null terminator */ + sldi r10,r9,56 /* stxvl wants size in top 8 bits */ addi r11,r11,32 stxvl 32+v2,r11,r10 /* Partial store */ +#ifdef USE_AS_STPCPY + /* stpcpy returns the dest address plus the size not counting the + final '\0'. */ + add r3,r11,r8 +#endif blr L(tail4): @@ -135,10 +171,17 @@ L(tail4): stxv 32+v1,16(r11) stxv 32+v2,32(r11) vctzlsbb r8,v6 /* Number of trailing zeroes */ - addi r8,r8,1 /* Add null terminator */ - sldi r10,r8,56 /* stxvl wants size in top 8 bits */ + addi r9,r8,1 /* Add null terminator */ + sldi r10,r9,56 /* stxvl wants size in top 8 bits */ addi r11,r11,48 stxvl 32+v3,r11,r10 /* Partial store */ +#ifdef USE_AS_STPCPY + /* stpcpy returns the dest address plus the size not counting the + final '\0'. */ + add r3,r11,r8 +#endif blr -END (STRCPY) +END (FUNC_NAME) +#ifndef USE_AS_STPCPY libc_hidden_builtin_def (strcpy) +#endif diff --git a/sysdeps/powerpc/powerpc64/multiarch/Makefile b/sysdeps/powerpc/powerpc64/multiarch/Makefile index db11345053..61a6901f7d 100644 --- a/sysdeps/powerpc/powerpc64/multiarch/Makefile +++ b/sysdeps/powerpc/powerpc64/multiarch/Makefile @@ -32,7 +32,7 @@ sysdep_routines += memcpy-power8-cached memcpy-power7 memcpy-a2 memcpy-power6 \ strncase-power8 ifneq (,$(filter %le,$(config-machine))) -sysdep_routines += strcmp-power9 strncmp-power9 strcpy-power9 +sysdep_routines += strcmp-power9 strncmp-power9 strcpy-power9 stpcpy-power9 endif CFLAGS-strncase-power7.c += -mcpu=power7 -funroll-loops CFLAGS-strncase_l-power7.c += -mcpu=power7 -funroll-loops diff --git a/sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c b/sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c index ad11ede20e..8021d8d8fa 100644 --- a/sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c +++ b/sysdeps/powerpc/powerpc64/multiarch/ifunc-impl-list.c @@ -98,6 +98,10 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, /* Support sysdeps/powerpc/powerpc64/multiarch/stpcpy.c. */ IFUNC_IMPL (i, name, stpcpy, +#ifdef __LITTLE_ENDIAN__ + IFUNC_IMPL_ADD (array, i, strncmp, hwcap2 & PPC_FEATURE2_ARCH_3_00, + __stpcpy_power9) +#endif IFUNC_IMPL_ADD (array, i, stpcpy, hwcap2 & PPC_FEATURE2_ARCH_2_07, __stpcpy_power8) IFUNC_IMPL_ADD (array, i, stpcpy, hwcap & PPC_FEATURE_HAS_VSX, diff --git a/sysdeps/powerpc/powerpc64/multiarch/stpcpy-power9.S b/sysdeps/powerpc/powerpc64/multiarch/stpcpy-power9.S new file mode 100644 index 0000000000..a728d49fd2 --- /dev/null +++ b/sysdeps/powerpc/powerpc64/multiarch/stpcpy-power9.S @@ -0,0 +1,24 @@ +/* Optimized stpcpy implementation for POWER9/PPC64. + Copyright (C) 2015-2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#define STPCPY __stpcpy_power9 + +#undef libc_hidden_builtin_def +#define libc_hidden_builtin_def(name) + +#include diff --git a/sysdeps/powerpc/powerpc64/multiarch/stpcpy.c b/sysdeps/powerpc/powerpc64/multiarch/stpcpy.c index 0a63c8f51a..68e1e648c3 100644 --- a/sysdeps/powerpc/powerpc64/multiarch/stpcpy.c +++ b/sysdeps/powerpc/powerpc64/multiarch/stpcpy.c @@ -26,13 +26,20 @@ extern __typeof (__stpcpy) __stpcpy_ppc attribute_hidden; extern __typeof (__stpcpy) __stpcpy_power7 attribute_hidden; extern __typeof (__stpcpy) __stpcpy_power8 attribute_hidden; +# ifdef __LITTLE_ENDIAN__ +extern __typeof (__stpcpy) __stpcpy_power9 attribute_hidden; +# endif libc_ifunc_hidden (__stpcpy, __stpcpy, - (hwcap2 & PPC_FEATURE2_ARCH_2_07) - ? __stpcpy_power8 - : (hwcap & PPC_FEATURE_HAS_VSX) - ? __stpcpy_power7 - : __stpcpy_ppc); +# ifdef __LITTLE_ENDIAN__ + (hwcap2 & PPC_FEATURE2_ARCH_3_00) + ? __stpcpy_power9 : +# endif + (hwcap2 & PPC_FEATURE2_ARCH_2_07) + ? __stpcpy_power8 + : (hwcap & PPC_FEATURE_HAS_VSX) + ? __stpcpy_power7 + : __stpcpy_ppc); weak_alias (__stpcpy, stpcpy) libc_hidden_def (__stpcpy)