From patchwork Mon Jan 4 15:17:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 41629 X-Patchwork-Delegate: azanella@linux.vnet.ibm.com Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A41B5382E06C; Mon, 4 Jan 2021 15:17:12 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A41B5382E06C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1609773432; bh=uunEzMxsk0Cbd4igBb1a4d94b1Xm4tMnfynh/BraJ44=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=UabeM1d3+pIOeHKmK4x8J7M+y1xjifSeCeIq04mQ5Ct19yaEyoE0PYoXg2XHkeZA9 j4kj3GJb/RBJ+WMs1tuybKpmhg5uheVmWhKsNhzCrRdtPYsbcaK7iGRUHyc6991fgf TCd+0bi0v6BW8BiV8B0k9V8eyOfWE5Iq+GnWqUH0= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by sourceware.org (Postfix) with ESMTPS id 01AB63842422 for ; Mon, 4 Jan 2021 15:17:11 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 01AB63842422 Received: by mail-pj1-x102b.google.com with SMTP id f14so10259192pju.4 for ; Mon, 04 Jan 2021 07:17:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=uunEzMxsk0Cbd4igBb1a4d94b1Xm4tMnfynh/BraJ44=; b=se1GkfJj+Os9T02eUFgOmJhpf9gjkR7OERqPpnV2cEh6upizrSADE51VFKgiJn9BcE T8rMnVWcUMtcjzOdfsS9qJreRH0PwFhZ9iot9/7+EVQslhv22b0Spzkeiy0uNzdgd0H4 MrfO/eLs8Qnw176m7vn3uvfDHmIMg9/LCAyeyPjq4WXhAY181rsf6GuloyKHCniaM2CR EoHs/2VW5FuytsZwJvj51wIl8Q41DL3Vk/H/kCyOU8lOe8KwzucDGv5zF+cviF9HOoOn pX7VjlE/rdapaJnDJHNIJO6tzMAM8CgMVwdLDsuFERX1OAueGh3+imdlJAejg2+OWOP6 WqDQ== X-Gm-Message-State: AOAM533XMh5J6YTQswUlY7Jca5QVX61QlJsAVpzVZVc2XNeExgJ7QsRL TOaHl2NDpUDcj3/Lb4PX7KF41SaVyO0= X-Google-Smtp-Source: ABdhPJxhMYrLB+nwww74PKMZB3EpUU3DWMb5XutMz/0vt8ZRlz5mQszqRSwAr7ExGOlttUJ1v6zsuQ== X-Received: by 2002:a17:90a:5914:: with SMTP id k20mr30108313pji.199.1609773429882; Mon, 04 Jan 2021 07:17:09 -0800 (PST) Received: from gnu-cfl-2.localdomain (c-69-181-90-243.hsd1.ca.comcast.net. [69.181.90.243]) by smtp.gmail.com with ESMTPSA id q70sm22248223pja.39.2021.01.04.07.17.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Jan 2021 07:17:09 -0800 (PST) Received: from gnu-cfl-2.localdomain (localhost [IPv6:::1]) by gnu-cfl-2.localdomain (Postfix) with ESMTP id 5C7241A013B for ; Mon, 4 Jan 2021 07:17:08 -0800 (PST) To: libc-alpha@sourceware.org Subject: V2 [PATCH] x86-64: Avoid rep movsb with short distance [BZ #27130] Date: Mon, 4 Jan 2021 07:17:06 -0800 Message-Id: <20210104151706.2129490-1-hjl.tools@gmail.com> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 X-Spam-Status: No, score=-3039.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Libc-alpha" From: "H.J. Lu" Reply-To: "H.J. Lu" Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" When copying with "rep movsb", if the distance between source and destination is N*4GB + [1..63] with N >= 0, performance may be very slow. This patch updates memmove-vec-unaligned-erms.S for AVX and AVX512 versions with the distance in RCX: cmpl $63, %ecx // Don't use "rep movsb" if ECX <= 63 jbe L(Don't use rep movsb") Use "rep movsb" Benchtests data with bench-memcpy, bench-memcpy-large, bench-memcpy-random and bench-memcpy-walk on Skylake, Ice Lake and Tiger Lake show that its performance impact is within noise range as "rep movsb" is only used for data size >= 4KB. Changes from V1: 1. Check distance of N*4GB + [1..63] with N >= 0 instead of [1..63]. --- .../multiarch/memmove-vec-unaligned-erms.S | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S b/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S index 7d54095f04..0980c95378 100644 --- a/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S +++ b/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S @@ -56,6 +56,13 @@ # endif #endif +/* Avoid short distance rep movsb only with non-SSE vector. */ +#ifndef AVOID_SHORT_DISTANCE_REP_MOVSB +# define AVOID_SHORT_DISTANCE_REP_MOVSB (VEC_SIZE > 16) +#else +# define AVOID_SHORT_DISTANCE_REP_MOVSB 0 +#endif + #ifndef PREFETCH # define PREFETCH(addr) prefetcht0 addr #endif @@ -243,7 +250,21 @@ L(movsb): cmpq %r9, %rdi /* Avoid slow backward REP MOVSB. */ jb L(more_8x_vec_backward) +# if AVOID_SHORT_DISTANCE_REP_MOVSB + movq %rdi, %rcx + subq %rsi, %rcx + jmp 2f +# endif 1: +# if AVOID_SHORT_DISTANCE_REP_MOVSB + movq %rsi, %rcx + subq %rdi, %rcx +2: +/* Avoid "rep movsb" if RCX, the distance between source and destination, + is N*4GB + [1..63] with N >= 0. */ + cmpl $63, %ecx + jbe L(more_2x_vec) /* Avoid "rep movsb" if ECX <= 63. */ +# endif mov %RDX_LP, %RCX_LP rep movsb L(nop):