From patchwork Wed Nov 3 15:04:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "H.J. Lu" X-Patchwork-Id: 46998 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A5A273858C39 for ; Wed, 3 Nov 2021 15:05:38 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A5A273858C39 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1635951938; bh=OXNNYAthW6AU1ZCoYyUbCgfATKxA2iuEku5j6hwd0ZQ=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=yT5Ww0F5HP12uceBXGdJeuJ8HAlumQGZR1oKG6Gmx5CpzML+7pdHHE0geRSygE5FV 7HU3A+vz3winOxmsG7Yt7bKZ2M8wrMFbSkiEN2XPeThULQS2FskM7+4Lzjg9CN3TIF QmtC6iFiTRWFiPJ0ZY4HRVBCnzcJ04zxocoQCb2M= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by sourceware.org (Postfix) with ESMTPS id E7484385841B for ; Wed, 3 Nov 2021 15:04:17 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org E7484385841B Received: by mail-pg1-x536.google.com with SMTP id j9so2646310pgh.1 for ; Wed, 03 Nov 2021 08:04:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=OXNNYAthW6AU1ZCoYyUbCgfATKxA2iuEku5j6hwd0ZQ=; b=tou8YyqEdrZZDW1DodqUxb+iUb7fQ799IPV4ws6T/8z6zCQMsPYuXn3eiEO+9fDyDp cdcdjQQvhCeHF7MrCokCJNTsWjTxjZ8KmvrZDD+PifhWniriL/4cxB7Ir6dTfvo/yn60 /9CFbA8DL7UGkAY4nWh5YgVI5f0UbqUt/F9jYd7hQnxw9hLpsP0LyiHk+TvFh8P0iqA+ Syq0esivRJWyiHhRcWDu1j/kLisT7IsvaCEBrGV4k6xW0BQFxkO6GHPOGBcvWYzHtI5N o4Nk1vqubaBhawANdqE24aMhXlLBQAk4Ymn5t3ZNMvdH0K9sKCCnhNzgeQ9y8pr+Tp1p ikZQ== X-Gm-Message-State: AOAM530kRsKbVe1Vezs3dPYhkYxoyU7CibiKdA1/9tSlOlgbJoR80lpj K8x4Hv9tizFCJOPXIwpvGU0gecLxbqw= X-Google-Smtp-Source: ABdhPJzFj0/Z5M3wSuivvXtSxmR0kS2wZeEEQqwRMUcWnYdWyZI7suMc4k8xavpEEvOzq/TBF4tpww== X-Received: by 2002:a05:6a00:1786:b0:481:21d3:fd2 with SMTP id s6-20020a056a00178600b0048121d30fd2mr14350669pfg.81.1635951856901; Wed, 03 Nov 2021 08:04:16 -0700 (PDT) Received: from gnu-cfl-2.localdomain ([172.58.35.133]) by smtp.gmail.com with ESMTPSA id p9sm2631496pfn.7.2021.11.03.08.04.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Nov 2021 08:04:16 -0700 (PDT) Received: from gnu-cfl-2.lan (localhost [IPv6:::1]) by gnu-cfl-2.localdomain (Postfix) with ESMTP id 4FF3E1A08BE for ; Wed, 3 Nov 2021 08:04:15 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH] x86: Optimize atomic_compare_and_exchange_[val|bool]_acq [BZ #28537] Date: Wed, 3 Nov 2021 08:04:15 -0700 Message-Id: <20211103150415.1211388-1-hjl.tools@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Spam-Status: No, score=-3030.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: "H.J. Lu via Libc-alpha" From: "H.J. Lu" Reply-To: "H.J. Lu" Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" From the CPU's point of view, getting a cache line for writing is more expensive than reading. See Appendix A.2 Spinlock in: https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/xeon-lock-scaling-analysis-paper.pdf The full compare and swap will grab the cache line exclusive and cause excessive cache line bouncing. Check the current memory value first and return immediately if writing cache line may fail to reduce cache line bouncing on contended locks. This fixes BZ# 28537. --- sysdeps/x86/atomic-machine.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/sysdeps/x86/atomic-machine.h b/sysdeps/x86/atomic-machine.h index 2692d94a92..92c7cf58b7 100644 --- a/sysdeps/x86/atomic-machine.h +++ b/sysdeps/x86/atomic-machine.h @@ -73,9 +73,19 @@ typedef uintmax_t uatomic_max_t; #define ATOMIC_EXCHANGE_USES_CAS 0 #define atomic_compare_and_exchange_val_acq(mem, newval, oldval) \ - __sync_val_compare_and_swap (mem, oldval, newval) + ({ __typeof (*(mem)) oldmem = *(mem), ret; \ + ret = (oldmem == (oldval) \ + ? __sync_val_compare_and_swap (mem, oldval, newval) \ + : oldmem); \ + ret; }) #define atomic_compare_and_exchange_bool_acq(mem, newval, oldval) \ - (! __sync_bool_compare_and_swap (mem, oldval, newval)) + ({ __typeof (*(mem)) old = *(mem); \ + int ret; \ + if (old != (oldval)) \ + ret = 1; \ + else \ + ret = !__sync_bool_compare_and_swap (mem, oldval, newval); \ + ret; }) #define __arch_c_compare_and_exchange_val_8_acq(mem, newval, oldval) \