From patchwork Wed Dec 7 08:52:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noah Goldstein X-Patchwork-Id: 61646 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8B7FC398243F for ; Wed, 7 Dec 2022 08:59:32 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 8B7FC398243F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1670403572; bh=a2QmjC7OtpfDb1a2oPH9CmhaPMWmiAbC7sQwhmMBFO0=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=aC1atYZ895bfYYfujVCqHU0nVVWh1J3HIrqELaBpUY06jzlMBMxLUH7QJsKi/BhY2 HvvnNK/3057S0nNKMN74SKe5bztow0C7V74aUimvOZy4nccZ3lriOtlka7YGLG/y76 bbL2Pum+Ew1DFnE8JiFGQPFhYc3sYQkrAlIpYClA= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by sourceware.org (Postfix) with ESMTPS id 431E43949096 for ; Wed, 7 Dec 2022 08:53:46 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 431E43949096 Received: by mail-ed1-x532.google.com with SMTP id l11so23957697edb.4 for ; Wed, 07 Dec 2022 00:53:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a2QmjC7OtpfDb1a2oPH9CmhaPMWmiAbC7sQwhmMBFO0=; b=RoE0SJ31jvQakvyuFg0caHAiCH+ow9gxzlGERFYhJi0t5nmpdzFN45iLhtR9MCs3JR g3QfFKdnE9wkzv7Ked1MxhTelXJ6fhVgM0+2ZPvK6Qvmjomji39CWOdcHp661BSI/++b g7FIWgqxV1CdOzCV+0XxYJBP7wjTgHB74O72Y7zpKx02cc1wfs72pVSNfKf0hN6lA/BI N9iXfhbjHbVfE4nh+FLpB9lm628LIIIR3pAUq472Nl8wOxcA6OfwkedpXYryQ8yXF59X F6VJjT9taJWtcQqbYtCJ9S2UUHSyCMn/Ydduho5c8qMw6yzAjXuFMIIkW6JXtMoLBE42 RtoA== X-Gm-Message-State: ANoB5pn81EkDrbSfs3obeMk43mJMXc4507wJJwXjlCaxDCzLJYs+QEUz y9kWM5EyHdU9AEMKFZP7nqIRIwuL3e4= X-Google-Smtp-Source: AA0mqf45BAufZ3GyH2UVtj6nBIN7nHRn9eo8o2NjxkhNb0voEHrVj+4S0XWFb95C1ALWY288LKSe/Q== X-Received: by 2002:a05:6402:5517:b0:461:c563:defa with SMTP id fi23-20020a056402551700b00461c563defamr79641125edb.72.1670403225480; Wed, 07 Dec 2022 00:53:45 -0800 (PST) Received: from noahgold-desk.lan (2603-8080-1301-76c6-feb7-1b9b-f2dd-08f7.res6.spectrum.com. [2603:8080:1301:76c6:feb7:1b9b:f2dd:8f7]) by smtp.gmail.com with ESMTPSA id k17-20020aa7c051000000b0046bd3b366f9sm1931767edo.32.2022.12.07.00.53.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 00:53:45 -0800 (PST) To: libc-alpha@sourceware.org Cc: goldstein.w.n@gmail.com, hjl.tools@gmail.com, andrey.kolesov@intel.com, carlos@systemhalted.org Subject: [PATCH v1 25/27] x86/fpu: Optimize svml_s_logf4_core_sse4.S Date: Wed, 7 Dec 2022 00:52:34 -0800 Message-Id: <20221207085236.1424424-25-goldstein.w.n@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221207085236.1424424-1-goldstein.w.n@gmail.com> References: <20221207085236.1424424-1-goldstein.w.n@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Noah Goldstein via Libc-alpha From: Noah Goldstein Reply-To: Noah Goldstein Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" 1. Improve special values case which ends up covering ~half of all float bit patterns. 2. Cleanup some missed optimizations in instruction selection / unnecissary repeated rodata references. 3. Remove unused rodata. 4. Use common data definitions where possible. As well instead of using the shared `__svml_slogf_data` just define the data locally. This is because; 1) Its not really ideal for the sse4/avx2 to reuse the avx512 tables as it pollute the cache which unnecessarily large blocks. 2) Really only one of the versions is ever expected to be used by a given process so there isn't any constructive caching between them. And 3) there is not enough data shared to make up for the first two reasons. Code Size Change: -286 Bytes (235 - 521) Input New Time / Old Time 0F (0x00000000) -> 0.5750 0F (0x0000ffff, Denorm) -> 0.9206 .1F (0x3dcccccd) -> 0.9389 5F (0x40a00000) -> 0.9361 2315255808F (0x4f0a0000) -> 0.9306 -NaN (0xffffffff) -> 0.5226 --- .../fpu/multiarch/svml_s_logf4_core_sse4.S | 327 +++++++++--------- 1 file changed, 156 insertions(+), 171 deletions(-) diff --git a/sysdeps/x86_64/fpu/multiarch/svml_s_logf4_core_sse4.S b/sysdeps/x86_64/fpu/multiarch/svml_s_logf4_core_sse4.S index 20ad054eac..42d09db8df 100644 --- a/sysdeps/x86_64/fpu/multiarch/svml_s_logf4_core_sse4.S +++ b/sysdeps/x86_64/fpu/multiarch/svml_s_logf4_core_sse4.S @@ -16,179 +16,164 @@ License along with the GNU C Library; if not, see . */ + + +#define LOCAL_DATA_NAME __svml_slog_data_internal +#include "svml_s_common_sse4_rodata_offsets.h" + +#define _sPoly_7 0 +#define _sPoly_6 16 +#define _sPoly_5 32 +#define _sPoly_4 48 +#define _sPoly_3 64 +#define _sPoly_2 80 + #include -#include "svml_s_logf_data.h" .section .text.sse4, "ax", @progbits ENTRY (_ZGVbN4v_logf_sse4) -/* - ALGORITHM DESCRIPTION: - - log(x) = exponent_x*log(2) + log(mantissa_x), if mantissa_x<4/3 - log(x) = (exponent_x+1)*log(2) + log(0.5*mantissa_x), if mantissa_x>4/3 - - R = mantissa_x - 1, if mantissa_x<4/3 - R = 0.5*mantissa_x - 1, if mantissa_x>4/3 - |R|< 1/3 - - log(1+R) is approximated as a polynomial: degree 9 for 1-ulp, - degree 7 for 4-ulp, degree 3 for half-precision. */ - - pushq %rbp - cfi_adjust_cfa_offset (8) - cfi_rel_offset (%rbp, 0) - movq %rsp, %rbp - cfi_def_cfa_register (%rbp) - andq $-64, %rsp - subq $320, %rsp - -/* reduction: compute r,n */ - movaps %xmm0, %xmm2 - -/* check for working range, - set special argument mask (denormals/zero/Inf/NaN) */ - movq __svml_slog_data@GOTPCREL(%rip), %rax - movdqu _iHiDelta(%rax), %xmm1 - movdqu _iLoRange(%rax), %xmm4 - paddd %xmm0, %xmm1 - movdqu _iBrkValue(%rax), %xmm3 - pcmpgtd %xmm1, %xmm4 - movdqu _iOffExpoMask(%rax), %xmm1 - psubd %xmm3, %xmm2 - pand %xmm2, %xmm1 - -/* exponent_x (mantissa_x<4/3) or exponent_x+1 (mantissa_x>4/3) */ - psrad $23, %xmm2 - paddd %xmm3, %xmm1 - movups _sPoly_7(%rax), %xmm5 - -/* mantissa_x (mantissa_x<4/3), or 0.5*mantissa_x (mantissa_x>4/3) */ - cvtdq2ps %xmm2, %xmm6 - -/* reduced argument R */ - subps _sOne(%rax), %xmm1 - movmskps %xmm4, %ecx - -/* final reconstruction: - add exponent_value*log2 to polynomial result */ - mulps _sLn2(%rax), %xmm6 - -/* polynomial evaluation starts here */ - mulps %xmm1, %xmm5 - addps _sPoly_6(%rax), %xmm5 - mulps %xmm1, %xmm5 - addps _sPoly_5(%rax), %xmm5 - mulps %xmm1, %xmm5 - addps _sPoly_4(%rax), %xmm5 - mulps %xmm1, %xmm5 - addps _sPoly_3(%rax), %xmm5 - mulps %xmm1, %xmm5 - addps _sPoly_2(%rax), %xmm5 - mulps %xmm1, %xmm5 - addps _sPoly_1(%rax), %xmm5 - mulps %xmm1, %xmm5 - -/* polynomial evaluation end */ - mulps %xmm1, %xmm5 - addps %xmm5, %xmm1 - addps %xmm6, %xmm1 - testl %ecx, %ecx - jne .LBL_1_3 - -.LBL_1_2: - cfi_remember_state - movdqa %xmm1, %xmm0 - movq %rbp, %rsp - cfi_def_cfa_register (%rsp) - popq %rbp - cfi_adjust_cfa_offset (-8) - cfi_restore (%rbp) - ret - -.LBL_1_3: - cfi_restore_state - movups %xmm0, 192(%rsp) - movups %xmm1, 256(%rsp) - je .LBL_1_2 - - xorb %dl, %dl - xorl %eax, %eax - movups %xmm8, 112(%rsp) - movups %xmm9, 96(%rsp) - movups %xmm10, 80(%rsp) - movups %xmm11, 64(%rsp) - movups %xmm12, 48(%rsp) - movups %xmm13, 32(%rsp) - movups %xmm14, 16(%rsp) - movups %xmm15, (%rsp) - movq %rsi, 136(%rsp) - movq %rdi, 128(%rsp) - movq %r12, 168(%rsp) - cfi_offset_rel_rsp (12, 168) - movb %dl, %r12b - movq %r13, 160(%rsp) - cfi_offset_rel_rsp (13, 160) - movl %ecx, %r13d - movq %r14, 152(%rsp) - cfi_offset_rel_rsp (14, 152) - movl %eax, %r14d - movq %r15, 144(%rsp) - cfi_offset_rel_rsp (15, 144) - cfi_remember_state - -.LBL_1_6: - btl %r14d, %r13d - jc .LBL_1_12 - -.LBL_1_7: - lea 1(%r14), %esi - btl %esi, %r13d - jc .LBL_1_10 - -.LBL_1_8: - incb %r12b - addl $2, %r14d - cmpb $16, %r12b - jb .LBL_1_6 - - movups 112(%rsp), %xmm8 - movups 96(%rsp), %xmm9 - movups 80(%rsp), %xmm10 - movups 64(%rsp), %xmm11 - movups 48(%rsp), %xmm12 - movups 32(%rsp), %xmm13 - movups 16(%rsp), %xmm14 - movups (%rsp), %xmm15 - movq 136(%rsp), %rsi - movq 128(%rsp), %rdi - movq 168(%rsp), %r12 - cfi_restore (%r12) - movq 160(%rsp), %r13 - cfi_restore (%r13) - movq 152(%rsp), %r14 - cfi_restore (%r14) - movq 144(%rsp), %r15 - cfi_restore (%r15) - movups 256(%rsp), %xmm1 - jmp .LBL_1_2 - -.LBL_1_10: - cfi_restore_state - movzbl %r12b, %r15d - movss 196(%rsp,%r15,8), %xmm0 - - call JUMPTARGET(logf) - - movss %xmm0, 260(%rsp,%r15,8) - jmp .LBL_1_8 - -.LBL_1_12: - movzbl %r12b, %r15d - movss 192(%rsp,%r15,8), %xmm0 - - call JUMPTARGET(logf) - - movss %xmm0, 256(%rsp,%r15,8) - jmp .LBL_1_7 - + movdqu COMMON_DATA(_ILoRange)(%rip), %xmm1 + /* ALGORITHM DESCRIPTION: + if mantissa_x<4/3 + log(x) = exponent_x*log(2) + log(mantissa_x) + if mantissa_x>4/3 + log(x) = (exponent_x+1)*log(2) + log(0.5*mantissa_x) + + R = mantissa_x - 1, if mantissa_x<4/3 + R = 0.5*mantissa_x - 1, if mantissa_x>4/3 + |R|< 1/3 + + log(1+R) is approximated as a polynomial: degree 9 for + 1-ulp, degree 7 for 4-ulp, degree 3 for half-precision. */ + + /* check for working range, set special argument mask + (denormals/zero/Inf/NaN). */ + movdqu COMMON_DATA(_NotiOffExpoMask)(%rip), %xmm2 + movaps %xmm0, %xmm3 + psubd %xmm2, %xmm3 + pcmpgtd %xmm3, %xmm1 + movmskps %xmm1, %eax + + movdqu COMMON_DATA(_IBrkValue)(%rip), %xmm1 + movaps %xmm0, %xmm3 + psubd %xmm1, %xmm0 + pandn %xmm0, %xmm2 + paddd %xmm1, %xmm2 + /* reduced argument R. */ + subps COMMON_DATA(_OneF)(%rip), %xmm2 + /* exponent_x (mantissa_x<4/3), + or exponent_x+1 (mantissa_x>4/3). */ + psrad $0x17, %xmm0 + /* mantissa_x (mantissa_x<4/3), + or 0.5 mantissa_x (mantissa_x>4/3). */ + cvtdq2ps %xmm0, %xmm0 + /* final reconstruction: add exponent_value * log2 to polynomial + result. */ + mulps COMMON_DATA(_Ln2)(%rip), %xmm0 + movups LOCAL_DATA(_sPoly_7)(%rip), %xmm1 + /* polynomial evaluation starts here. */ + mulps %xmm2, %xmm1 + addps LOCAL_DATA(_sPoly_6)(%rip), %xmm1 + mulps %xmm2, %xmm1 + addps LOCAL_DATA(_sPoly_5)(%rip), %xmm1 + mulps %xmm2, %xmm1 + addps LOCAL_DATA(_sPoly_4)(%rip), %xmm1 + mulps %xmm2, %xmm1 + addps LOCAL_DATA(_sPoly_3)(%rip), %xmm1 + mulps %xmm2, %xmm1 + addps LOCAL_DATA(_sPoly_2)(%rip), %xmm1 + mulps %xmm2, %xmm1 + addps COMMON_DATA(_Neg5F)(%rip), %xmm1 + mulps %xmm2, %xmm1 + /* polynomial evaluation end. */ + mulps %xmm2, %xmm1 + addps %xmm1, %xmm2 + addps %xmm2, %xmm0 + testl %eax, %eax + jne L(SPECIAL_VALUES_BRANCH) + ret + + /* Cold case. edx has 1s where there was a special value that + more so than speed here. */ +L(SPECIAL_VALUES_BRANCH): + /* Stack coming in 16-byte aligned. Set 8-byte misaligned so on + call entry will be 16-byte aligned. */ + subq $0x38, %rsp + movups %xmm0, 24(%rsp) + movups %xmm3, 40(%rsp) + + /* Use rbx/rbp for callee save registers as they get short + encoding for many instructions (as compared with r12/r13). */ + movq %rbx, (%rsp) + cfi_offset (rbx, -64) + movq %rbp, 8(%rsp) + cfi_offset (rbp, -56) + /* edx has 1s where there was a special value that needs to be + handled by a tanhf call. */ + movl %eax, %ebx +L(SPECIAL_VALUES_LOOP): + + /* use rbp as index for special value that is saved across calls + to tanhf. We technically don't need a callee save register + here as offset to rsp is always [0, 12] so we can restore + rsp by realigning to 64. Essentially the tradeoff is 1 extra + save/restore vs 2 extra instructions in the loop. */ + xorl %ebp, %ebp + bsfl %ebx, %ebp + + /* Scalar math fucntion call to process special input. */ + movss 40(%rsp, %rbp, 4), %xmm0 + call logf@PLT + + /* No good way to avoid the store-forwarding fault this will + cause on return. `lfence` avoids the SF fault but at greater + cost as it serialized stack/callee save restoration. */ + movss %xmm0, 24(%rsp, %rbp, 4) + + leal -1(%rbx), %eax + andl %eax, %ebx + jnz L(SPECIAL_VALUES_LOOP) + + /* All results have been written to 24(%rsp). */ + movups 24(%rsp), %xmm0 + movq (%rsp), %rbx + cfi_restore (rbx) + movq 8(%rsp), %rbp + cfi_restore (rbp) + addq $56, %rsp + cfi_def_cfa_offset (8) + ret END (_ZGVbN4v_logf_sse4) + + .section .rodata.sse4, "a" + .align 16 + + /* Data table for vector implementations of function logf. The + table may contain polynomial, reduction, lookup coefficients + and other coefficients obtained through different methods of + research and experimental work. */ + + .globl LOCAL_DATA_NAME +LOCAL_DATA_NAME: + /* Polynomial sPoly[] coefficients:. */ + /* -1.5177205204963684082031250e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_7, 0xbe1b6a22) + + /* 1.6964881122112274169921875e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_6, 0x3e2db86b) + + /* -1.6462457180023193359375000e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_5, 0xbe289358) + + /* 1.9822503626346588134765625e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_4, 0x3e4afb81) + + /* -2.5004664063453674316406250e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_3, 0xbe80061d) + + /* 3.3336564898490905761718750e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_2, 0x3eaaaee7) + + + .type LOCAL_DATA_NAME, @object + .size LOCAL_DATA_NAME, .-LOCAL_DATA_NAME