From patchwork Wed Jan 13 15:18:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: develop--- via Libc-alpha X-Patchwork-Id: 41708 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4CD03389683F; Wed, 13 Jan 2021 15:19:37 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 4CD03389683F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1610551177; bh=acYf4D5HuhnVIQfSled2kKjo++7/XS/pH0Y5ltfDReQ=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=qfbHUxLf8JlANwWUFPqtFo6TWVBPDNVgfg7W5HAzRkkjoK/Ygxi+g72ds2cm8hgS6 H67mUio64zDthqPvzc+hA2LZsTWnSSK3gOnOYFBG0hnVmWeOCXD9+/Tuv2uPfbte6a 5bRtj9X6EnngWx93B5Q+nrSbZa7pvEVeQ1XGPx1o= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2049.outbound.protection.outlook.com [40.107.244.49]) by sourceware.org (Postfix) with ESMTPS id 7FA2D3896831 for ; Wed, 13 Jan 2021 15:19:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 7FA2D3896831 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kx8crjLkApsQvd18ZyTFAL1JZlrzA2TmumiVUEwySxqqFbHfxSBiwPnKhoi35NQzBWFO2w83hNsLY/k1ESAIr1VGRUXweJmS3ATqjglvX1O0M7DoQJTki6SfhBNCMNFMgvhMktyAKIbOSXM0a09BHtx3KbOfJIjaGcpnt3qlpXOB56wbEC0S47jJKMQWl89cGrH5FS8gFZY72xK3l5ubUlth/djU39Kr/wooKvNUd9tbJQrlm3R2A76yB2J59kjYCKhRad4hNyUmmzvPXJ3tlooB5V29AIomZ4m4Rd070GV6/s/ZAVe3ef8lKwOAt1KmptauTLW7z6OT/cxSFsOX6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=acYf4D5HuhnVIQfSled2kKjo++7/XS/pH0Y5ltfDReQ=; b=Zuef+aYrzm53AfL4jPSy2ZjgGFRg0kPRqfO56awWDbNmDFIXHCnnjnlDnq7Q7Jf4YotCo3vbL2DSgYTrakpGUx2yPK30ByKEvXJIZg7aRFJGwIRxKZBExBcLIMwVm3cxdBD54JNiNUpwnawKe5UwQZw72nZl8n8rxie+4LibOWuSqCkCn1Qh5NhWU2Xq0M0sgexP5X9l1Z9oa31Te091/QW7qJ83HUChwQronziAxy6SGUBkbh5Tn7760rDeyKWwzlt7CWXvstojYLuCSEiwAn7c4RaLdqZ1GcXMuYk9czYEASr0s4nQzywXMCJNqfSHOfWkCnOWkioRZXDN0miRrQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none Received: from BN6PR1201MB0081.namprd12.prod.outlook.com (2603:10b6:405:4f::12) by BN8PR12MB2979.namprd12.prod.outlook.com (2603:10b6:408:66::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3742.7; Wed, 13 Jan 2021 15:19:31 +0000 Received: from BN6PR1201MB0081.namprd12.prod.outlook.com ([fe80::20fa:a1a2:9e7d:1370]) by BN6PR1201MB0081.namprd12.prod.outlook.com ([fe80::20fa:a1a2:9e7d:1370%11]) with mapi id 15.20.3742.012; Wed, 13 Jan 2021 15:19:31 +0000 To: hjl.tools@gmail.com Subject: [PATCH] x86: Adding an upper bound for Enhanced REP MOVSB. Date: Wed, 13 Jan 2021 20:48:52 +0530 Message-Id: <20210113151852.431112-1-sajan.karumanchi@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-Originating-IP: [165.204.156.251] X-ClientProxiedBy: MA1PR0101CA0019.INDPRD01.PROD.OUTLOOK.COM (2603:1096:a00:21::29) To BN6PR1201MB0081.namprd12.prod.outlook.com (2603:10b6:405:4f::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from amd.com (165.204.156.251) by MA1PR0101CA0019.INDPRD01.PROD.OUTLOOK.COM (2603:1096:a00:21::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3763.10 via Frontend Transport; Wed, 13 Jan 2021 15:19:29 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: b24a8960-de37-41a1-2001-08d8b7d6a087 X-MS-TrafficTypeDiagnostic: BN8PR12MB2979: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FhfHYo8xBfh4+CejgFnMbFtJvCgcTZ7hzEoMAqd/pQ4+DUi4Vk+OoIb3vFZzoYqJny7Ecpzo77twAV9DPaxcCaedeMrZs+6bTwZWLiSpun7rYFOB08ItOocWEMNPotDXRTn9FQPbYYTrsf+AIsrSGYS5EuH6QVjQHq/VnHUVjcnSp8V+hdRSu8zzTq0NnQ9Pg2aGrZaCUG+/47P5EiujhFctmU4nYs7lRStPxq1KRbOCAGCRc1pRKEncJh14ti+eXkgiINRZUORQCV1QMRsz0b7qSqoMGCOHochkroVtXzac/hIl0ChTcQKV84Zu6aFkq+t3AVdBun3ntY24xchINVOHI92U5weJWOwNA6QsGgBJym+OilsZCu3RfKz/JxosGj/tvw8YUjHOWor7YNpjeg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN6PR1201MB0081.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(366004)(346002)(39860400002)(376002)(136003)(7696005)(66476007)(66556008)(86362001)(6666004)(8936002)(66946007)(956004)(8676002)(2906002)(5660300002)(2616005)(83380400001)(54906003)(26005)(6916009)(186003)(316002)(16526019)(55016002)(1076003)(478600001)(52116002)(8886007)(4326008)(9686003)(36756003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: CAFXTyTREhDw9e9DYn0oeDrO34MfYaqDIfBKR0tTw2mHdOpiJdeA4G+VKyFFgaBt9+oH9gicZzGNq41dZFc7gcbeHxlZH9ByQWbtKMHcq518jSFGLAJxkv45lVKeFSX68v+fsKoYcWBN07U+aLMd37Bnw2uy98FDRQkZNGDyJ4FcuUjLNf/4gPnMOakSTBgNAS90TYX6sor+Bfd6Xqm07PfNodgSWKDG+FXCYcVg9L+nCDIUdIqWRpa6C4ZzgvaeCXeiKoPiw38Ot4w1M/SoFnKH88+g65xBcp1kLos07asPoWL8tYjV4kEf1PFFLLdANO6HQp4xs0KoT07Y6aYJW2Ky28gfmYFl/YEJqop21Ry0YoOl+YMzGD8NBWKoU2Kq7OXvXqPADnKFN1WeFnWFHw6i6HwBOuyACjjXM1jKMytUtBkB1BTmw18vHHOoXVdDgSY25GdAWZU6qWaLNfXUEdoW3DMhve3naf50I29x18QrUwkL+pdQ8deDwgO0aCzxO06PgBr2WtL41FsAVJ8L4+8r+8f49hJbUC2CtgqoobJqtSUJ20fcMrpyG/oA+bBQG+XTtJobmb/g9w1aJ5r3YbqiCT+ZrGEo5zSA+3P0ZZ701QVAAOiIXFM2fxAg0SUrtCoiqr7vABnOJy9/70dIFHd4uRxAjMOfrkpXuVAzsbFo05RtBXbPt5cAJwpI2yuUmhixmDVRJRfCVebiJ7pC715j570MNF1zM2xUM6uyRskaXL/U7xRCtFNKA8Sm8szOnNsmQwETZDbn1Ps8g3GaRzLgEy2SthkLD2AMzs317edTvikMixZaz7wVEiS3ItbSXQQgefX6Q2lGS6dLHzJ3kysbbkzM9gU9M5mL9UqAtoTXD3WZmbJDReZYQ1RuL8tNy4FfEXyeVmZnqEpfpbIYq0ynsyc1IUtalv4aC4WWu53SlaeQXglVbW+sNMGETaA3NHl41BzJ7c1Hfy5i2M5c/1xZotLi5TiMBL5GnQ/DdVs7VburJ6rqGCw54oC1jA+D X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-AuthSource: BN6PR1201MB0081.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Jan 2021 15:19:31.5780 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-Network-Message-Id: b24a8960-de37-41a1-2001-08d8b7d6a087 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3ZIe1K5eqGE6d+Rdc0y3Ez0rR2DaDB4PTR3BRZZKJWv2v3INuU7lhs+Xbf81eoSCxLPc2e8Fjt/yu+zewnhk4A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB2979 X-Spam-Status: No, score=-14.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MSGID_FROM_MTA_HEADER, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "sajan.karumanchi--- via Libc-alpha" From: develop--- via Libc-alpha Reply-To: sajan.karumanchi@amd.com Cc: Sajan Karumanchi , Premachandra Mallappa , libc-alpha@sourceware.org, fweimer@redhat.com Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" From: Sajan Karumanchi In the process of optimizing memcpy for AMD machines, we have found the vector move operations are outperforming enhanced REP MOVSB for data transfers above the L2 cache size on Zen3 architectures. To handle this use case, we are adding an upper bound parameter on enhanced REP MOVSB:'__x86_max_rep_movsb_threshold'. As per large-bench results, we are configuring this parameter to the L2 cache size for AMD machines and applicable from Zen3 architecture supporting the ERMS feature. For architectures other than AMD, it is the computed value of non-temporal threshold parameter. Reviewed-by: Premachandra Mallappa --- sysdeps/x86/cacheinfo.h | 14 ++++++++++++++ .../x86_64/multiarch/memmove-vec-unaligned-erms.S | 8 ++++++-- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/sysdeps/x86/cacheinfo.h b/sysdeps/x86/cacheinfo.h index 00d2d8a52a..f20b7fea4f 100644 --- a/sysdeps/x86/cacheinfo.h +++ b/sysdeps/x86/cacheinfo.h @@ -45,6 +45,9 @@ long int __x86_rep_movsb_threshold attribute_hidden = 2048; /* Threshold to use Enhanced REP STOSB. */ long int __x86_rep_stosb_threshold attribute_hidden = 2048; +/* Threshold to stop using Enhanced REP MOVSB. */ +long int __x86_max_rep_movsb_threshold attribute_hidden; + static void get_common_cache_info (long int *shared_ptr, unsigned int *threads_ptr, long int core) @@ -351,6 +354,11 @@ init_cacheinfo (void) /* Account for exclusive L2 and L3 caches. */ shared += core; } + /* ERMS feature is implemented from Zen3 architecture and it is + performing poorly for data above L2 cache size. Henceforth, adding + an upper bound threshold parameter to limit the usage of Enhanced + REP MOVSB operations and setting its value to L2 cache size. */ + __x86_max_rep_movsb_threshold = core; } } @@ -423,6 +431,12 @@ init_cacheinfo (void) else __x86_rep_movsb_threshold = rep_movsb_threshold; + /* Setting the upper bound of ERMS to the known default value of + non-temporal threshold for architectures other than AMD. */ + if (cpu_features->basic.kind != arch_kind_amd) + __x86_max_rep_movsb_threshold = __x86_shared_non_temporal_threshold; + + # if HAVE_TUNABLES __x86_rep_stosb_threshold = cpu_features->rep_stosb_threshold; # endif diff --git a/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S b/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S index 0980c95378..c7e75bfbda 100644 --- a/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S +++ b/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S @@ -30,7 +30,10 @@ load and aligned store. Load the last 4 * VEC and first VEC before the loop and store them after the loop to support overlapping addresses. - 6. If size >= __x86_shared_non_temporal_threshold and there is no + 6. On machines with ERMS feature, if size greater than equal or to + __x86_rep_movsb_threshold and less than + __x86_max_rep_movsb_threshold, then REP MOVSB will be used. + 7. If size >= __x86_shared_non_temporal_threshold and there is no overlap between destination and source, use non-temporal store instead of aligned store. */ @@ -240,7 +243,8 @@ L(return): ret L(movsb): - cmp __x86_shared_non_temporal_threshold(%rip), %RDX_LP + /* Avoid REP MOVSB for sizes above max threshold. */ + cmp __x86_max_rep_movsb_threshold(%rip), %RDX_LP jae L(more_8x_vec) cmpq %rsi, %rdi jb 1f