From patchwork Wed Dec 7 08:52:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noah Goldstein X-Patchwork-Id: 61647 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 028B1398AC23 for ; Wed, 7 Dec 2022 08:59:48 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 028B1398AC23 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1670403588; bh=gXVHzU0Rm6liJzo/uCAV7Bn3JuAX9sBVhOSN/YgxeFA=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=Anz3BFMBx1kBYFfpptR8xLyPydjndEIWrV3KEY/87oVDghNeORyauFUPTSS/9DLQ4 XJ7FP2uwBFbxrC92zrcgzc1gC9ncTUUCeE84/qQSOrnQiOZN1lFSlXHegjLVk9f82r P55uaF+pEX6MTh85JG5YR5WLqZPfNJ3qXmHGNBxA= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) by sourceware.org (Postfix) with ESMTPS id C124B3957793 for ; Wed, 7 Dec 2022 08:53:25 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org C124B3957793 Received: by mail-ed1-x52e.google.com with SMTP id c66so17629505edf.5 for ; Wed, 07 Dec 2022 00:53:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gXVHzU0Rm6liJzo/uCAV7Bn3JuAX9sBVhOSN/YgxeFA=; b=svOwDeFw2YM7cW/JGs1YyDJGOO4688sas1wBR5wA13JlVNn4A1OSZEFS8LbEcIf0BE ihABJnB2sqWa4VjDykemqOLoNbKp8iRVu1zp+0A3IMZYBlc4duXBgV8VHM5AotP5ZCgx GB0/czaTuLfTj+s4Rhujc7gzPe4Kex1mgd3D1bssW19dgQsT/dM6Eo4ww41nlE3zBo8e 9oSbedo/mUTqgoLJBv9fzHjQFc7Fqv+MEAcmLjFCTwuLq7Mg9BBcqPFfseBUuZqZDQ25 hgs1+NWhWyVFtqoVdUx9wQUSds1+JarFzaNM8QgLLf1+WIX92h/24TgkhECQyq4ngrKm AIAA== X-Gm-Message-State: ANoB5pmyAImDJbUbAwkR46KYvcg/U0JzZi0iBKLMwbikufp2QX5LxVFV rlQWHbkjzfTomu2WNINRbpZ6ZK83DZg= X-Google-Smtp-Source: AA0mqf7isPv7O64k7pZ5Ydn38rIZXW7GeHzps+NFyCEZigH5Bm9sL4VhJAgiBg/LubUXSeZsVMXgCA== X-Received: by 2002:a05:6402:899:b0:46a:c6d3:eeb1 with SMTP id e25-20020a056402089900b0046ac6d3eeb1mr48982727edy.141.1670403204120; Wed, 07 Dec 2022 00:53:24 -0800 (PST) Received: from noahgold-desk.lan (2603-8080-1301-76c6-feb7-1b9b-f2dd-08f7.res6.spectrum.com. [2603:8080:1301:76c6:feb7:1b9b:f2dd:8f7]) by smtp.gmail.com with ESMTPSA id k17-20020aa7c051000000b0046bd3b366f9sm1931767edo.32.2022.12.07.00.53.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 00:53:23 -0800 (PST) To: libc-alpha@sourceware.org Cc: goldstein.w.n@gmail.com, hjl.tools@gmail.com, andrey.kolesov@intel.com, carlos@systemhalted.org Subject: [PATCH v1 24/27] x86/fpu: Optimize svml_s_logf16_core_avx512.S Date: Wed, 7 Dec 2022 00:52:33 -0800 Message-Id: <20221207085236.1424424-24-goldstein.w.n@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221207085236.1424424-1-goldstein.w.n@gmail.com> References: <20221207085236.1424424-1-goldstein.w.n@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Noah Goldstein via Libc-alpha From: Noah Goldstein Reply-To: Noah Goldstein Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" 1. Improve special values case which ends up covering ~half of all float bit patterns. 2. Cleanup some missed optimizations in instruction selection / unnecissary repeated rodata references. 3. Remove unused rodata. 4. Use common data definitions where possible. As well instead of using the shared `__svml_slogf_data` just define the data locally. This is because; 1) Its not really ideal for the sse4/avx2 to reuse the avx512 tables as it pollute the cache which unnecessarily large blocks. 2) Really only one of the versions is ever expected to be used by a given process so there isn't any constructive caching between them. And 3) there is not enough data shared to make up for the first two reasons. Lastly, the avx512-skl implementation no longer uses `vpternlogd` (the only AVX512-DQ instruction) so make the avx512-knl implementation just a alias to the SKL one. Code Size Change: -1285 Bytes (267 - 1552) (With removing KNL impl) Input New Time / Old Time 0F (0x00000000) -> 0.7324 0F (0x0000ffff, Denorm) -> 0.9708 .1F (0x3dcccccd) -> 0.9535 5F (0x40a00000) -> 0.9699 2315255808F (0x4f0a0000) -> 0.9909 -NaN (0xffffffff) -> 0.6798 --- .../fpu/multiarch/svml_s_logf16_core_avx512.S | 553 ++++++------------ 1 file changed, 173 insertions(+), 380 deletions(-) diff --git a/sysdeps/x86_64/fpu/multiarch/svml_s_logf16_core_avx512.S b/sysdeps/x86_64/fpu/multiarch/svml_s_logf16_core_avx512.S index eb830a6579..ebc959aa74 100644 --- a/sysdeps/x86_64/fpu/multiarch/svml_s_logf16_core_avx512.S +++ b/sysdeps/x86_64/fpu/multiarch/svml_s_logf16_core_avx512.S @@ -16,388 +16,181 @@ License along with the GNU C Library; if not, see . */ +#define LOCAL_DATA_NAME __svml_slog_data_internal +#define LOCAL_DATA_NAME_UNALIGNED __svml_slog_data_internal_unaligned +#include "svml_s_common_evex512_rodata_offsets.h" + +/* Offsets for data table __svml_slog_data_internal_unaligned. + 4 bytes each. */ +#define _iLoRange 0 + +/* Offsets for data table __svml_slog_data_internal + 64 bytes each. */ +#define _iBrkValue 0 +#define _sPoly_7 64 +#define _sPoly_6 128 +#define _sPoly_5 192 +#define _sPoly_4 256 +#define _sPoly_3 320 +#define _sPoly_2 384 +#define _sLn2 448 + #include -#include "svml_s_logf_data.h" -#include "svml_s_wrapper_impl.h" .section .text.evex512, "ax", @progbits -ENTRY (_ZGVeN16v_logf_knl) -/* - ALGORITHM DESCRIPTION: - - log(x) = exponent_x*log(2) + log(mantissa_x), if mantissa_x<4/3 - log(x) = (exponent_x+1)*log(2) + log(0.5*mantissa_x), if mantissa_x>4/3 - - R = mantissa_x - 1, if mantissa_x<4/3 - R = 0.5*mantissa_x - 1, if mantissa_x>4/3 - |R|< 1/3 - - log(1+R) is approximated as a polynomial: degree 9 for 1-ulp, - degree 7 for 4-ulp, degree 3 for half-precision. */ - - pushq %rbp - cfi_adjust_cfa_offset (8) - cfi_rel_offset (%rbp, 0) - movq %rsp, %rbp - cfi_def_cfa_register (%rbp) - andq $-64, %rsp - subq $1280, %rsp - movq __svml_slog_data@GOTPCREL(%rip), %rax - movl $-1, %ecx - -/* reduction: compute r,n */ - vpsubd _iBrkValue(%rax), %zmm0, %zmm2 - vmovups _sPoly_7(%rax), %zmm7 - vpandd _iOffExpoMask(%rax), %zmm2, %zmm3 - -/* exponent_x (mantissa_x<4/3) or exponent_x+1 (mantissa_x>4/3) */ - vpsrad $23, %zmm2, %zmm4 - -/* check for working range, - set special argument mask (denormals/zero/Inf/NaN) - */ - vpaddd _iHiDelta(%rax), %zmm0, %zmm1 - -/* mantissa_x (mantissa_x<4/3), or 0.5*mantissa_x (mantissa_x>4/3) */ - vpaddd _iBrkValue(%rax), %zmm3, %zmm6 - vpcmpd $1, _iLoRange(%rax), %zmm1, %k1 - vcvtdq2ps {rn-sae}, %zmm4, %zmm1 - -/* reduced argument R */ - vsubps _sOne(%rax), %zmm6, %zmm8 - vpbroadcastd %ecx, %zmm5{%k1}{z} - -/* polynomial evaluation starts here */ - vfmadd213ps _sPoly_6(%rax), %zmm8, %zmm7 - vptestmd %zmm5, %zmm5, %k0 - kmovw %k0, %ecx - vfmadd213ps _sPoly_5(%rax), %zmm8, %zmm7 - vfmadd213ps _sPoly_4(%rax), %zmm8, %zmm7 - vfmadd213ps _sPoly_3(%rax), %zmm8, %zmm7 - vfmadd213ps _sPoly_2(%rax), %zmm8, %zmm7 - vfmadd213ps _sPoly_1(%rax), %zmm8, %zmm7 - vmulps %zmm8, %zmm7, %zmm9 - -/* polynomial evaluation end */ - vfmadd213ps %zmm8, %zmm8, %zmm9 - -/* - final reconstruction: - add exponent_value*log2 to polynomial result - */ - vfmadd132ps _sLn2(%rax), %zmm9, %zmm1 - testl %ecx, %ecx - jne .LBL_1_3 - -.LBL_1_2: - cfi_remember_state - vmovaps %zmm1, %zmm0 - movq %rbp, %rsp - cfi_def_cfa_register (%rsp) - popq %rbp - cfi_adjust_cfa_offset (-8) - cfi_restore (%rbp) - ret - -.LBL_1_3: - cfi_restore_state - vmovups %zmm0, 1152(%rsp) - vmovups %zmm1, 1216(%rsp) - je .LBL_1_2 - - xorb %dl, %dl - kmovw %k4, 1048(%rsp) - xorl %eax, %eax - kmovw %k5, 1040(%rsp) - kmovw %k6, 1032(%rsp) - kmovw %k7, 1024(%rsp) - vmovups %zmm16, 960(%rsp) - vmovups %zmm17, 896(%rsp) - vmovups %zmm18, 832(%rsp) - vmovups %zmm19, 768(%rsp) - vmovups %zmm20, 704(%rsp) - vmovups %zmm21, 640(%rsp) - vmovups %zmm22, 576(%rsp) - vmovups %zmm23, 512(%rsp) - vmovups %zmm24, 448(%rsp) - vmovups %zmm25, 384(%rsp) - vmovups %zmm26, 320(%rsp) - vmovups %zmm27, 256(%rsp) - vmovups %zmm28, 192(%rsp) - vmovups %zmm29, 128(%rsp) - vmovups %zmm30, 64(%rsp) - vmovups %zmm31, (%rsp) - movq %rsi, 1064(%rsp) - movq %rdi, 1056(%rsp) - movq %r12, 1096(%rsp) - cfi_offset_rel_rsp (12, 1096) - movb %dl, %r12b - movq %r13, 1088(%rsp) - cfi_offset_rel_rsp (13, 1088) - movl %ecx, %r13d - movq %r14, 1080(%rsp) - cfi_offset_rel_rsp (14, 1080) - movl %eax, %r14d - movq %r15, 1072(%rsp) - cfi_offset_rel_rsp (15, 1072) - cfi_remember_state - -.LBL_1_6: - btl %r14d, %r13d - jc .LBL_1_12 - -.LBL_1_7: - lea 1(%r14), %esi - btl %esi, %r13d - jc .LBL_1_10 - -.LBL_1_8: - addb $1, %r12b - addl $2, %r14d - cmpb $16, %r12b - jb .LBL_1_6 - - kmovw 1048(%rsp), %k4 - movq 1064(%rsp), %rsi - kmovw 1040(%rsp), %k5 - movq 1056(%rsp), %rdi - kmovw 1032(%rsp), %k6 - movq 1096(%rsp), %r12 - cfi_restore (%r12) - movq 1088(%rsp), %r13 - cfi_restore (%r13) - kmovw 1024(%rsp), %k7 - vmovups 960(%rsp), %zmm16 - vmovups 896(%rsp), %zmm17 - vmovups 832(%rsp), %zmm18 - vmovups 768(%rsp), %zmm19 - vmovups 704(%rsp), %zmm20 - vmovups 640(%rsp), %zmm21 - vmovups 576(%rsp), %zmm22 - vmovups 512(%rsp), %zmm23 - vmovups 448(%rsp), %zmm24 - vmovups 384(%rsp), %zmm25 - vmovups 320(%rsp), %zmm26 - vmovups 256(%rsp), %zmm27 - vmovups 192(%rsp), %zmm28 - vmovups 128(%rsp), %zmm29 - vmovups 64(%rsp), %zmm30 - vmovups (%rsp), %zmm31 - movq 1080(%rsp), %r14 - cfi_restore (%r14) - movq 1072(%rsp), %r15 - cfi_restore (%r15) - vmovups 1216(%rsp), %zmm1 - jmp .LBL_1_2 - -.LBL_1_10: - cfi_restore_state - movzbl %r12b, %r15d - vmovss 1156(%rsp,%r15,8), %xmm0 - call JUMPTARGET(logf) - vmovss %xmm0, 1220(%rsp,%r15,8) - jmp .LBL_1_8 - -.LBL_1_12: - movzbl %r12b, %r15d - vmovss 1152(%rsp,%r15,8), %xmm0 - call JUMPTARGET(logf) - vmovss %xmm0, 1216(%rsp,%r15,8) - jmp .LBL_1_7 -END (_ZGVeN16v_logf_knl) - ENTRY (_ZGVeN16v_logf_skx) -/* - ALGORITHM DESCRIPTION: - - log(x) = exponent_x*log(2) + log(mantissa_x), if mantissa_x<4/3 - log(x) = (exponent_x+1)*log(2) + log(0.5*mantissa_x), if mantissa_x>4/3 - - R = mantissa_x - 1, if mantissa_x<4/3 - R = 0.5*mantissa_x - 1, if mantissa_x>4/3 - |R|< 1/3 - - log(1+R) is approximated as a polynomial: degree 9 for 1-ulp, - degree 7 for 4-ulp, degree 3 for half-precision. */ - - pushq %rbp - cfi_adjust_cfa_offset (8) - cfi_rel_offset (%rbp, 0) - movq %rsp, %rbp - cfi_def_cfa_register (%rbp) - andq $-64, %rsp - subq $1280, %rsp - movq __svml_slog_data@GOTPCREL(%rip), %rax - vpternlogd $0xff, %zmm6, %zmm6, %zmm6 - vmovups _iBrkValue(%rax), %zmm4 - vmovups _sPoly_7(%rax), %zmm8 - -/* - check for working range, - set special argument mask (denormals/zero/Inf/NaN) - */ - vpaddd _iHiDelta(%rax), %zmm0, %zmm1 - -/* reduction: compute r,n */ - vpsubd %zmm4, %zmm0, %zmm2 - vpcmpd $5, _iLoRange(%rax), %zmm1, %k1 - -/* exponent_x (mantissa_x<4/3) or exponent_x+1 (mantissa_x>4/3) */ - vpsrad $23, %zmm2, %zmm5 - vpandd _iOffExpoMask(%rax), %zmm2, %zmm3 - -/* mantissa_x (mantissa_x<4/3), or 0.5*mantissa_x (mantissa_x>4/3) */ - vpaddd %zmm4, %zmm3, %zmm7 - -/* reduced argument R */ - vsubps _sOne(%rax), %zmm7, %zmm9 - -/* polynomial evaluation starts here */ - vfmadd213ps _sPoly_6(%rax), %zmm9, %zmm8 - vfmadd213ps _sPoly_5(%rax), %zmm9, %zmm8 - vfmadd213ps _sPoly_4(%rax), %zmm9, %zmm8 - vfmadd213ps _sPoly_3(%rax), %zmm9, %zmm8 - vfmadd213ps _sPoly_2(%rax), %zmm9, %zmm8 - vfmadd213ps _sPoly_1(%rax), %zmm9, %zmm8 - vmulps %zmm9, %zmm8, %zmm10 - -/* polynomial evaluation end */ - vfmadd213ps %zmm9, %zmm9, %zmm10 - vpandnd %zmm1, %zmm1, %zmm6{%k1} - vptestmd %zmm6, %zmm6, %k0 - vcvtdq2ps {rn-sae}, %zmm5, %zmm1 - kmovw %k0, %ecx - -/* - final reconstruction: - add exponent_value*log2 to polynomial result - */ - vfmadd132ps _sLn2(%rax), %zmm10, %zmm1 - testl %ecx, %ecx - jne .LBL_2_3 - -.LBL_2_2: - cfi_remember_state - vmovaps %zmm1, %zmm0 - movq %rbp, %rsp - cfi_def_cfa_register (%rsp) - popq %rbp - cfi_adjust_cfa_offset (-8) - cfi_restore (%rbp) - ret - -.LBL_2_3: - cfi_restore_state - vmovups %zmm0, 1152(%rsp) - vmovups %zmm1, 1216(%rsp) - je .LBL_2_2 - - xorb %dl, %dl - xorl %eax, %eax - kmovw %k4, 1048(%rsp) - kmovw %k5, 1040(%rsp) - kmovw %k6, 1032(%rsp) - kmovw %k7, 1024(%rsp) - vmovups %zmm16, 960(%rsp) - vmovups %zmm17, 896(%rsp) - vmovups %zmm18, 832(%rsp) - vmovups %zmm19, 768(%rsp) - vmovups %zmm20, 704(%rsp) - vmovups %zmm21, 640(%rsp) - vmovups %zmm22, 576(%rsp) - vmovups %zmm23, 512(%rsp) - vmovups %zmm24, 448(%rsp) - vmovups %zmm25, 384(%rsp) - vmovups %zmm26, 320(%rsp) - vmovups %zmm27, 256(%rsp) - vmovups %zmm28, 192(%rsp) - vmovups %zmm29, 128(%rsp) - vmovups %zmm30, 64(%rsp) - vmovups %zmm31, (%rsp) - movq %rsi, 1064(%rsp) - movq %rdi, 1056(%rsp) - movq %r12, 1096(%rsp) - cfi_offset_rel_rsp (12, 1096) - movb %dl, %r12b - movq %r13, 1088(%rsp) - cfi_offset_rel_rsp (13, 1088) - movl %ecx, %r13d - movq %r14, 1080(%rsp) - cfi_offset_rel_rsp (14, 1080) - movl %eax, %r14d - movq %r15, 1072(%rsp) - cfi_offset_rel_rsp (15, 1072) - cfi_remember_state - -.LBL_2_6: - btl %r14d, %r13d - jc .LBL_2_12 - -.LBL_2_7: - lea 1(%r14), %esi - btl %esi, %r13d - jc .LBL_2_10 - -.LBL_2_8: - incb %r12b - addl $2, %r14d - cmpb $16, %r12b - jb .LBL_2_6 - - kmovw 1048(%rsp), %k4 - kmovw 1040(%rsp), %k5 - kmovw 1032(%rsp), %k6 - kmovw 1024(%rsp), %k7 - vmovups 960(%rsp), %zmm16 - vmovups 896(%rsp), %zmm17 - vmovups 832(%rsp), %zmm18 - vmovups 768(%rsp), %zmm19 - vmovups 704(%rsp), %zmm20 - vmovups 640(%rsp), %zmm21 - vmovups 576(%rsp), %zmm22 - vmovups 512(%rsp), %zmm23 - vmovups 448(%rsp), %zmm24 - vmovups 384(%rsp), %zmm25 - vmovups 320(%rsp), %zmm26 - vmovups 256(%rsp), %zmm27 - vmovups 192(%rsp), %zmm28 - vmovups 128(%rsp), %zmm29 - vmovups 64(%rsp), %zmm30 - vmovups (%rsp), %zmm31 - vmovups 1216(%rsp), %zmm1 - movq 1064(%rsp), %rsi - movq 1056(%rsp), %rdi - movq 1096(%rsp), %r12 - cfi_restore (%r12) - movq 1088(%rsp), %r13 - cfi_restore (%r13) - movq 1080(%rsp), %r14 - cfi_restore (%r14) - movq 1072(%rsp), %r15 - cfi_restore (%r15) - jmp .LBL_2_2 - -.LBL_2_10: - cfi_restore_state - movzbl %r12b, %r15d - vmovss 1156(%rsp,%r15,8), %xmm0 - vzeroupper - vmovss 1156(%rsp,%r15,8), %xmm0 - - call JUMPTARGET(logf) - - vmovss %xmm0, 1220(%rsp,%r15,8) - jmp .LBL_2_8 - -.LBL_2_12: - movzbl %r12b, %r15d - vmovss 1152(%rsp,%r15,8), %xmm0 - vzeroupper - vmovss 1152(%rsp,%r15,8), %xmm0 - - call JUMPTARGET(logf) - - vmovss %xmm0, 1216(%rsp,%r15,8) - jmp .LBL_2_7 - + /* ALGORITHM DESCRIPTION: + if mantissa_x<4/3 + log(x) = exponent_x*log(2) + log(mantissa_x) + if mantissa_x>4/3 + log(x) = (exponent_x+1)*log(2) + log(0.5*mantissa_x) + + R = mantissa_x - 1, if mantissa_x<4/3 + R = 0.5*mantissa_x - 1, if mantissa_x>4/3 + |R|< 1/3 + + log(1+R) is approximated as a polynomial: degree 9 for + 1-ulp, degree 7 for 4-ulp, degree 3 for half-precision. */ + vmovups LOCAL_DATA(_iBrkValue)(%rip), %zmm4 + vmovups LOCAL_DATA(_sPoly_7)(%rip), %zmm8 + + /* check for working range, set special argument mask + (denormals/zero/Inf/NaN). */ + vmovups COMMON_DATA(_NotiOffExpoMask)(%rip), %zmm3 + vpsubd %zmm3, %zmm0, %zmm1 + + /* reduction: compute r,n. */ + vpsubd %zmm4, %zmm0, %zmm2 + vpcmpd $2, LOCAL_DATA_UNALIGNED(_iLoRange)(%rip){1to16}, %zmm1, %k0 + + /* exponent_x (mantissa_x<4/3), + or exponent_x+1 (mantissa_x>4/3). */ + vpsrad $23, %zmm2, %zmm5 + vpandnd %zmm2, %zmm3, %zmm3 + + /* mantissa_x (mantissa_x<4/3), + or 0.5 mantissa_x (mantissa_x>4/3). */ + vpaddd %zmm4, %zmm3, %zmm7 + + /* reduced argument R. */ + vsubps COMMON_DATA(_OneF)(%rip), %zmm7, %zmm9 + + /* polynomial evaluation starts here. */ + vfmadd213ps LOCAL_DATA(_sPoly_6)(%rip), %zmm9, %zmm8 + vfmadd213ps LOCAL_DATA(_sPoly_5)(%rip), %zmm9, %zmm8 + vfmadd213ps LOCAL_DATA(_sPoly_4)(%rip), %zmm9, %zmm8 + vfmadd213ps LOCAL_DATA(_sPoly_3)(%rip), %zmm9, %zmm8 + vfmadd213ps LOCAL_DATA(_sPoly_2)(%rip), %zmm9, %zmm8 + vfmadd213ps COMMON_DATA(_Neg5F)(%rip), %zmm9, %zmm8 + vmulps %zmm9, %zmm8, %zmm10 + + /* polynomial evaluation end. */ + vfmadd213ps %zmm9, %zmm9, %zmm10 + vcvtdq2ps {rn-sae}, %zmm5, %zmm1 + kmovw %k0, %ecx + /* final reconstruction: add exponent_value + log2 to polynomial result. */ + vfmadd132ps LOCAL_DATA(_sLn2)(%rip), %zmm10, %zmm1 + /* Branch to process special inputs. */ + testl %ecx, %ecx + jne L(SPECIAL_VALUES_BRANCH) + + vmovaps %zmm1, %zmm0 + ret + + /* Cold case. edx has 1s where there was a special value that + needs to be handled by a tanf call. Optimize for code size + moreso than speed here. */ +L(SPECIAL_VALUES_BRANCH): + /* Use r13 to save/restore the stack. This allows us to use rbp + as callee save register saving code size. */ + pushq %r13 + cfi_def_cfa (rsp, 16) + /* Need to callee save registers to preserve state across tanf + calls. */ + pushq %rbx + cfi_def_cfa (rsp, 24) + pushq %rbp + cfi_def_cfa (rsp, 32) + movq %rsp, %r13 + cfi_def_cfa (r13, 32) + + /* Align stack and make room for 2x zmm vectors. */ + andq $-64, %rsp + addq $-128, %rsp + + /* Save origional input. */ + vmovaps %zmm0, 64(%rsp) + /* Save all already computed inputs. */ + vmovaps %zmm1, (%rsp) + + vzeroupper + + /* edx has 1s where there was a special value that needs to be + handled by a tanf call. */ + movl %ecx, %ebx +L(SPECIAL_VALUES_LOOP): + + /* use rbp as index for special value that is saved across calls + to tanf. We technically don't need a callee save register + here as offset to rsp is always [0, 56] so we can restore + rsp by realigning to 64. Essentially the tradeoff is 1 extra + save/restore vs 2 extra instructions in the loop. Realigning + also costs more code size. */ + xorl %ebp, %ebp + tzcntl %ebx, %ebp + + /* Scalar math fucntion call to process special input. */ + movss 64(%rsp, %rbp, 4), %xmm0 + call logf@PLT + + /* No good way to avoid the store-forwarding fault this will + cause on return. `lfence` avoids the SF fault but at greater + cost as it serialized stack/callee save restoration. */ + movss %xmm0, (%rsp, %rbp, 4) + + blsrl %ebx, %ebx + jnz L(SPECIAL_VALUES_LOOP) + + + /* All results have been written to 64(%rsp). */ + vmovaps (%rsp), %zmm0 + /* Restore rsp. */ + movq %r13, %rsp + cfi_def_cfa (rsp, 32) + /* Restore callee save registers. */ + popq %rbp + cfi_def_cfa (rsp, 24) + popq %rbx + cfi_def_cfa (rsp, 16) + popq %r13 + ret END (_ZGVeN16v_logf_skx) + +strong_alias (_ZGVeN16v_logf_skx, _ZGVeN16v_logf_knl) + + .section .rodata.evex512, "a" + .align 64 + + /* Data table for vector implementations of function logf. The + table may contain polynomial, reduction, lookup coefficients + and other coefficients obtained through different methods of + research and experimental work. */ +LOCAL_DATA_NAME_UNALIGNED: + float_block (LOCAL_DATA_NAME_UNALIGNED, _iLoRange, 0x00ffffff) + + .type LOCAL_DATA_NAME_UNALIGNED, @object + .size LOCAL_DATA_NAME_UNALIGNED, .-LOCAL_DATA_NAME_UNALIGNED + +LOCAL_DATA_NAME: + DATA_VEC (LOCAL_DATA_NAME, _iBrkValue, 0x3f2aaaab) + DATA_VEC (LOCAL_DATA_NAME, _sPoly_7, 0xbe1b6a22) + DATA_VEC (LOCAL_DATA_NAME, _sPoly_6, 0x3e2db86b) + DATA_VEC (LOCAL_DATA_NAME, _sPoly_5, 0xbe289358) + DATA_VEC (LOCAL_DATA_NAME, _sPoly_4, 0x3e4afb81) + DATA_VEC (LOCAL_DATA_NAME, _sPoly_3, 0xbe80061d) + DATA_VEC (LOCAL_DATA_NAME, _sPoly_2, 0x3eaaaee7) + DATA_VEC (LOCAL_DATA_NAME, _sLn2, 0x3f317218) + + .type LOCAL_DATA_NAME, @object + .size LOCAL_DATA_NAME, .-LOCAL_DATA_NAME