From patchwork Wed Dec 7 08:52:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noah Goldstein X-Patchwork-Id: 61648 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A0A2C395B417 for ; Wed, 7 Dec 2022 09:00:19 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A0A2C395B417 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1670403619; bh=vpUFb9WV1mGpyr4fdxrDcL/jMZ5y8ufWmm6RthjBwMc=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=xAJ6OFsp3Uq6VqEm8CI9NpZ/HSbN0CrIND/EoEeyrBQJBLLdzCvkQOhtYI7s7UP/K /7t0/vpfWA2BWr/CwxwrkMfAko8zMqxpB0XciZwqe3A7wzKJilXlRlinw+pb4Y6plW vN2GNwRJY72CtyP2g7hBEoO1AJLNiStsqfYbyQSE= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) by sourceware.org (Postfix) with ESMTPS id ED068395A430 for ; Wed, 7 Dec 2022 08:53:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org ED068395A430 Received: by mail-ed1-x532.google.com with SMTP id m19so23902206edj.8 for ; Wed, 07 Dec 2022 00:53:48 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vpUFb9WV1mGpyr4fdxrDcL/jMZ5y8ufWmm6RthjBwMc=; b=BNbtbiB8sAHLHMl51CE36HkY4xtMj9mftfNWM6cP+Ka1sSaOGNisnlOYRdAzkAV1jP u1Cp6dk8NipmzrfE5T7ZxkkIM4f2q6u+ohOw03OPiwHQCAgbRQg7b2P3v1XPljXQCi+r jvQ7Kc39rPM/1L5yjDYw7G+4md912YTDpLq+pxvXAHbPyxrt1630aWlh9UG0S5zJFXQ+ yOLswQsMB12Sajs8OQ47wO8FtNAg8aMCW4RwM/W82opIe5ZkUGjka8k/v/huKb9eyxJb vTsmzavC10RSGiOzOotUREg/2DMY86OcYuhCu3Jd4NG0UjCOwVCA9WJkfWe2wIhODY4L 6ySA== X-Gm-Message-State: ANoB5pkxmpN94O/j9VJ2V8I4TEUg7547ZeelHhIZ+nT+BPQWVC67v5BI eXBtaiPo6VbrWPxK50t/niY8vWVoDDs= X-Google-Smtp-Source: AA0mqf4Zc1BCEMJxi7C7GspLWZjwf7v0KKJb6cBwANoZIqntDZX3LF38Ife56TzBDEPIUBlR/SC4Ng== X-Received: by 2002:aa7:c415:0:b0:46c:4b56:8c06 with SMTP id j21-20020aa7c415000000b0046c4b568c06mr17122405edq.230.1670403227264; Wed, 07 Dec 2022 00:53:47 -0800 (PST) Received: from noahgold-desk.lan (2603-8080-1301-76c6-feb7-1b9b-f2dd-08f7.res6.spectrum.com. [2603:8080:1301:76c6:feb7:1b9b:f2dd:8f7]) by smtp.gmail.com with ESMTPSA id k17-20020aa7c051000000b0046bd3b366f9sm1931767edo.32.2022.12.07.00.53.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 00:53:46 -0800 (PST) To: libc-alpha@sourceware.org Cc: goldstein.w.n@gmail.com, hjl.tools@gmail.com, andrey.kolesov@intel.com, carlos@systemhalted.org Subject: [PATCH v1 26/27] x86/fpu: Optimize svml_s_logf8_core_avx2.S Date: Wed, 7 Dec 2022 00:52:35 -0800 Message-Id: <20221207085236.1424424-26-goldstein.w.n@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221207085236.1424424-1-goldstein.w.n@gmail.com> References: <20221207085236.1424424-1-goldstein.w.n@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Noah Goldstein via Libc-alpha From: Noah Goldstein Reply-To: Noah Goldstein Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" 1. Improve special values case which ends up covering ~half of all float bit patterns. 2. Cleanup some missed optimizations in instruction selection / unnecissary repeated rodata references. 3. Remove unused rodata. 4. Use common data definitions where possible. As well instead of using the shared `__svml_slogf_data` just define the data locally. This is because; 1) Its not really ideal for the sse4/avx2 to reuse the avx512 tables as it pollute the cache which unnecessarily large blocks. 2) Really only one of the versions is ever expected to be used by a given process so there isn't any constructive caching between them. And 3) there is not enough data shared to make up for the first two reasons. Code Size Change: -314 Bytes (230 - 544) Input New Time / Old Time 0F (0x00000000) -> 0.6246 0F (0x0000ffff, Denorm) -> 0.9525 .1F (0x3dcccccd) -> 0.9021 5F (0x40a00000) -> 0.9113 2315255808F (0x4f0a0000) -> 0.8293 -NaN (0xffffffff) -> 0.5745 --- .../fpu/multiarch/svml_s_logf8_core_avx2.S | 342 +++++++++--------- 1 file changed, 181 insertions(+), 161 deletions(-) diff --git a/sysdeps/x86_64/fpu/multiarch/svml_s_logf8_core_avx2.S b/sysdeps/x86_64/fpu/multiarch/svml_s_logf8_core_avx2.S index 616312c695..35f63b7879 100644 --- a/sysdeps/x86_64/fpu/multiarch/svml_s_logf8_core_avx2.S +++ b/sysdeps/x86_64/fpu/multiarch/svml_s_logf8_core_avx2.S @@ -16,169 +16,189 @@ License along with the GNU C Library; if not, see . */ +#define LOCAL_DATA_NAME __svml_slog_data_internal +#include "svml_s_common_avx2_rodata_offsets.h" + +/* Offsets for data table __svml_slog_data_internal. */ +#define _sPoly_7 0 +#define _sPoly_6 32 +#define _sPoly_5 64 +#define _sPoly_4 96 +#define _sPoly_3 128 +#define _sPoly_2 160 + #include -#include "svml_s_logf_data.h" .section .text.avx2, "ax", @progbits ENTRY(_ZGVdN8v_logf_avx2) -/* - ALGORITHM DESCRIPTION: - - log(x) = exponent_x*log(2) + log(mantissa_x), if mantissa_x<4/3 - log(x) = (exponent_x+1)*log(2) + log(0.5*mantissa_x), if mantissa_x>4/3 - - R = mantissa_x - 1, if mantissa_x<4/3 - R = 0.5*mantissa_x - 1, if mantissa_x>4/3 - |R|< 1/3 - - log(1+R) is approximated as a polynomial: degree 9 for 1-ulp, - degree 7 for 4-ulp, degree 3 for half-precision. */ - - pushq %rbp - cfi_adjust_cfa_offset (8) - cfi_rel_offset (%rbp, 0) - movq %rsp, %rbp - cfi_def_cfa_register (%rbp) - andq $-64, %rsp - subq $448, %rsp - movq __svml_slog_data@GOTPCREL(%rip), %rax - vmovaps %ymm0, %ymm2 - vmovups _iBrkValue(%rax), %ymm6 - vmovups _iLoRange(%rax), %ymm1 -/* check for working range, - set special argument mask (denormals/zero/Inf/NaN) */ - vpaddd _iHiDelta(%rax), %ymm2, %ymm7 - -/* reduction: compute r,n */ - vpsubd %ymm6, %ymm2, %ymm4 - -/* exponent_x (mantissa_x<4/3) or exponent_x+1 (mantissa_x>4/3) */ - vpsrad $23, %ymm4, %ymm3 - vpand _iOffExpoMask(%rax), %ymm4, %ymm5 - vmovups _sPoly_7(%rax), %ymm4 - vcvtdq2ps %ymm3, %ymm0 - -/* mantissa_x (mantissa_x<4/3), or 0.5*mantissa_x (mantissa_x>4/3) */ - vpaddd %ymm6, %ymm5, %ymm3 - -/* reduced argument R */ - vsubps _sOne(%rax), %ymm3, %ymm5 - -/* polynomial evaluation starts here */ - vfmadd213ps _sPoly_6(%rax), %ymm5, %ymm4 - vfmadd213ps _sPoly_5(%rax), %ymm5, %ymm4 - vfmadd213ps _sPoly_4(%rax), %ymm5, %ymm4 - vfmadd213ps _sPoly_3(%rax), %ymm5, %ymm4 - vfmadd213ps _sPoly_2(%rax), %ymm5, %ymm4 - vfmadd213ps _sPoly_1(%rax), %ymm5, %ymm4 - vmulps %ymm5, %ymm4, %ymm6 - -/* polynomial evaluation end */ - vfmadd213ps %ymm5, %ymm5, %ymm6 - vpcmpgtd %ymm7, %ymm1, %ymm1 - vmovmskps %ymm1, %ecx - -/* final reconstruction: - add exponent_value*log2 to polynomial result */ - vfmadd132ps _sLn2(%rax), %ymm6, %ymm0 - testl %ecx, %ecx - jne .LBL_1_3 - -.LBL_1_2: - cfi_remember_state - movq %rbp, %rsp - cfi_def_cfa_register (%rsp) - popq %rbp - cfi_adjust_cfa_offset (-8) - cfi_restore (%rbp) - ret - -.LBL_1_3: - cfi_restore_state - vmovups %ymm2, 320(%rsp) - vmovups %ymm0, 384(%rsp) - je .LBL_1_2 - - xorb %dl, %dl - xorl %eax, %eax - vmovups %ymm8, 224(%rsp) - vmovups %ymm9, 192(%rsp) - vmovups %ymm10, 160(%rsp) - vmovups %ymm11, 128(%rsp) - vmovups %ymm12, 96(%rsp) - vmovups %ymm13, 64(%rsp) - vmovups %ymm14, 32(%rsp) - vmovups %ymm15, (%rsp) - movq %rsi, 264(%rsp) - movq %rdi, 256(%rsp) - movq %r12, 296(%rsp) - cfi_offset_rel_rsp (12, 296) - movb %dl, %r12b - movq %r13, 288(%rsp) - cfi_offset_rel_rsp (13, 288) - movl %ecx, %r13d - movq %r14, 280(%rsp) - cfi_offset_rel_rsp (14, 280) - movl %eax, %r14d - movq %r15, 272(%rsp) - cfi_offset_rel_rsp (15, 272) - cfi_remember_state - -.LBL_1_6: - btl %r14d, %r13d - jc .LBL_1_12 - -.LBL_1_7: - lea 1(%r14), %esi - btl %esi, %r13d - jc .LBL_1_10 - -.LBL_1_8: - incb %r12b - addl $2, %r14d - cmpb $16, %r12b - jb .LBL_1_6 - - vmovups 224(%rsp), %ymm8 - vmovups 192(%rsp), %ymm9 - vmovups 160(%rsp), %ymm10 - vmovups 128(%rsp), %ymm11 - vmovups 96(%rsp), %ymm12 - vmovups 64(%rsp), %ymm13 - vmovups 32(%rsp), %ymm14 - vmovups (%rsp), %ymm15 - vmovups 384(%rsp), %ymm0 - movq 264(%rsp), %rsi - movq 256(%rsp), %rdi - movq 296(%rsp), %r12 - cfi_restore (%r12) - movq 288(%rsp), %r13 - cfi_restore (%r13) - movq 280(%rsp), %r14 - cfi_restore (%r14) - movq 272(%rsp), %r15 - cfi_restore (%r15) - jmp .LBL_1_2 - -.LBL_1_10: - cfi_restore_state - movzbl %r12b, %r15d - vmovss 324(%rsp,%r15,8), %xmm0 - vzeroupper - - call JUMPTARGET(logf) - - vmovss %xmm0, 388(%rsp,%r15,8) - jmp .LBL_1_8 - -.LBL_1_12: - movzbl %r12b, %r15d - vmovss 320(%rsp,%r15,8), %xmm0 - vzeroupper - - call JUMPTARGET(logf) - - vmovss %xmm0, 384(%rsp,%r15,8) - jmp .LBL_1_7 - + /* ALGORITHM DESCRIPTION: + if mantissa_x<4/3 + log(x) = exponent_x*log(2) + log(mantissa_x) + if mantissa_x>4/3 + log(x) = (exponent_x+1)*log(2) + log(0.5*mantissa_x) + + R = mantissa_x - 1, if mantissa_x<4/3 + R = 0.5*mantissa_x - 1, if mantissa_x>4/3 + |R|< 1/3 + + log(1+R) is approximated as a polynomial: degree 9 for + 1-ulp, degree 7 for 4-ulp, degree 3 for half-precision. */ + vmovups COMMON_DATA(_IBrkValue)(%rip), %ymm6 + + + vmovups COMMON_DATA(_NotiOffExpoMask)(%rip), %ymm7 + + /* reduction: compute r,n. */ + vpsubd %ymm6, %ymm0, %ymm4 + + /* exponent_x (mantissa_x<4/3) or + exponent_x+1 (mantissa_x>4/3). */ + vpsrad $23, %ymm4, %ymm3 + vpandn %ymm4, %ymm7, %ymm5 + vmovups LOCAL_DATA(_sPoly_7)(%rip), %ymm4 + vcvtdq2ps %ymm3, %ymm2 + + /* mantissa_x (mantissa_x<4/3), + or 0.5*mantissa_x (mantissa_x>4/3). */ + vpaddd %ymm6, %ymm5, %ymm3 + + /* reduced argument R. */ + vsubps COMMON_DATA(_OneF)(%rip), %ymm3, %ymm5 + + /* polynomial evaluation starts here. */ + vfmadd213ps LOCAL_DATA(_sPoly_6)(%rip), %ymm5, %ymm4 + vfmadd213ps LOCAL_DATA(_sPoly_5)(%rip), %ymm5, %ymm4 + vfmadd213ps LOCAL_DATA(_sPoly_4)(%rip), %ymm5, %ymm4 + vfmadd213ps LOCAL_DATA(_sPoly_3)(%rip), %ymm5, %ymm4 + vfmadd213ps LOCAL_DATA(_sPoly_2)(%rip), %ymm5, %ymm4 + vfmadd213ps COMMON_DATA(_Neg5F)(%rip), %ymm5, %ymm4 + vmulps %ymm5, %ymm4, %ymm6 + + /* polynomial evaluation end. */ + vfmadd213ps %ymm5, %ymm5, %ymm6 + + vmovups COMMON_DATA(_ILoRange)(%rip), %ymm1 + /* check for working range, set special argument mask + (denormals/zero/Inf/NaN). */ + vpsubd %ymm7, %ymm0, %ymm7 + + + vpcmpgtd %ymm7, %ymm1, %ymm1 + vmovmskps %ymm1, %ecx + + /* final reconstruction: add exponent_value*log2 to polynomial + result. */ + vfmadd132ps COMMON_DATA(_Ln2)(%rip), %ymm6, %ymm2 + testl %ecx, %ecx + /* Branch to process special inputs. */ + jne L(SPECIAL_VALUES_BRANCH) + vmovaps %ymm2, %ymm0 + ret + + + /* Cold case. edx has 1s where there was a special value that + needs to be handled by a atanhf call. Optimize for code size + more so than speed here. */ +L(SPECIAL_VALUES_BRANCH): + /* Use r13 to save/restore the stack. This allows us to use rbp + as callee save register saving code size. */ + pushq %r13 + cfi_adjust_cfa_offset (8) + cfi_offset (r13, -16) + /* Need to callee save registers to preserve state across tanhf + calls. */ + pushq %rbx + cfi_adjust_cfa_offset (8) + cfi_offset (rbx, -24) + pushq %rbp + cfi_adjust_cfa_offset (8) + cfi_offset (rbp, -32) + movq %rsp, %r13 + cfi_def_cfa_register (r13) + + /* Align stack and make room for 2x ymm vectors. */ + andq $-32, %rsp + addq $-64, %rsp + + /* Save all already computed inputs. */ + vmovups %ymm2, (%rsp) + /* Save original input (ymm2 unchanged up to this point). */ + vmovups %ymm0, 32(%rsp) + + vzeroupper + + /* edx has 1s where there was a special value that needs to be + handled by a atanhf call. */ + movl %ecx, %ebx +L(SPECIAL_VALUES_LOOP): + + /* use rbp as index for special value that is saved across calls + to atanhf. We technically don't need a callee save register + here as offset to rsp is always [0, 28] so we can restore + rsp by realigning to 64. Essentially the tradeoff is 1 extra + save/restore vs 2 extra instructions in the loop. Realigning + also costs more code size. */ + xorl %ebp, %ebp + tzcntl %ebx, %ebp + + /* Scalar math fucntion call to process special input. */ + vmovss 32(%rsp, %rbp, 4), %xmm0 + call logf@PLT + + /* No good way to avoid the store-forwarding fault this will + cause on return. `lfence` avoids the SF fault but at greater + cost as it serialized stack/callee save restoration. */ + vmovss %xmm0, (%rsp, %rbp, 4) + + blsrl %ebx, %ebx + jnz L(SPECIAL_VALUES_LOOP) + + + + /* All results have been written to (%rsp). */ + vmovups (%rsp), %ymm0 + /* Restore rsp. */ + movq %r13, %rsp + cfi_def_cfa_register (rsp) + /* Restore callee save registers. */ + popq %rbp + cfi_adjust_cfa_offset (-8) + cfi_restore (rbp) + popq %rbx + cfi_adjust_cfa_offset (-8) + cfi_restore (rbp) + popq %r13 + cfi_adjust_cfa_offset (-8) + cfi_restore (r13) + ret END(_ZGVdN8v_logf_avx2) + + .section .rodata.avx2, "a" + .align 32 + + /* Data table for vector implementations of function logf. The + table may contain polynomial, reduction, lookup coefficients + and other coefficients obtained through different methods of + research and experimental work. */ +LOCAL_DATA_NAME: + /* Polynomial sPoly[] coefficients:. */ + /* -1.5177205204963684082031250e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_7, 0xbe1b6a22) + + /* 1.6964881122112274169921875e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_6, 0x3e2db86b) + + /* -1.6462457180023193359375000e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_5, 0xbe289358) + + /* 1.9822503626346588134765625e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_4, 0x3e4afb81) + + /* -2.5004664063453674316406250e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_3, 0xbe80061d) + + /* 3.3336564898490905761718750e-01. */ + DATA_VEC (LOCAL_DATA_NAME, _sPoly_2, 0x3eaaaee7) + + .type LOCAL_DATA_NAME, @object + .size LOCAL_DATA_NAME, .-LOCAL_DATA_NAME