From patchwork Mon Aug 30 18:52:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella X-Patchwork-Id: 44813 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4B0D83858419 for ; Mon, 30 Aug 2021 18:52:51 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 4B0D83858419 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1630349571; bh=719KjI0cMF3TyoEm/c0dU5O+toMVWIKJp4sJtvDkixI=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=psoFdZvz3RmSloU/iWXLxU+JCM0FvbSY+Dj2UUQCKunVzTtpP+MdBZFo8ACzvTCGT FyOnVxGazC28IsEOui6XJJ+JpGKPqJ/uMDwLta2T7dk3ZzTP3C1tV8AvEF9xxZj7Fy Hw9RA89/wFyQAz3Tk7cnXUXBRmz50CfY0Pjx3aw4= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) by sourceware.org (Postfix) with ESMTPS id AE1373858C3B for ; Mon, 30 Aug 2021 18:52:20 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org AE1373858C3B Received: by mail-qk1-x733.google.com with SMTP id c10so16756460qko.11 for ; Mon, 30 Aug 2021 11:52:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=719KjI0cMF3TyoEm/c0dU5O+toMVWIKJp4sJtvDkixI=; b=QStC5iLcyRDliY7rClklpT5lunQG+M/GCjeGYaAeqy4AVo2sSrUPdVisTgWQuDAMWm hmsE1WKqHdkWrq/vELNYTQdR1lBCO1N0kIjdRyXwxPIj5ogVVwNhduNPcL7w374muTg9 jwmYbM0LxRiSculOnXsbFdsBqRDKKsEwdukbPW5NmTe+Dph6tLQ3YmnRYl4Qi+7Ix/ZQ zUkWb/tyP9Copd7srQVtGs58DYy5OgypHZvn5WpBpPJ4IlXW3t7moKaI6podHpR582JC 4+6rg+K9mpB3JC2CxjpsglpAxa0dgZtpw09EFkMmXLr4/ZhWRZlaMnboTjnwpdFwieD1 r4Wg== X-Gm-Message-State: AOAM533Stfktz8afC/Yy/l7REKJSdsnsnLuo1bwF1hIsJ0XCe5ZAY5U+ ocXdXilPv9oBp43X/yXbS86biRU4Cxi0Gw== X-Google-Smtp-Source: ABdhPJz62oD6ZxwD8Ipd0T6PNzbBgsFDdn67RpSKry91UUQQ4k47UGrqDJfRA7F71H1ER4GFA6QI0g== X-Received: by 2002:a05:620a:15e8:: with SMTP id p8mr23791178qkm.27.1630349540040; Mon, 30 Aug 2021 11:52:20 -0700 (PDT) Received: from birita.. ([2804:431:c7ca:1a68:7647:1f41:2147:1ed2]) by smtp.gmail.com with ESMTPSA id m187sm11752338qkd.131.2021.08.30.11.52.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Aug 2021 11:52:19 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH v4 0/7] malloc: Improve Huge Page support Date: Mon, 30 Aug 2021 15:52:08 -0300 Message-Id: <20210830185215.449572-1-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 X-Spam-Status: No, score=-5.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Guillaume Morin , Siddhesh Poyarekar Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Linux currently supports two ways to use Huge Pages: either by using specific flags directly with the syscall (MAP_HUGETLB for mmap(), or SHM_HUGETLB for shmget()), or by using Transparent Huge Pages (THP) where the kernel tries to move allocated anonymous pages to Huge Pages blocks transparently to application. Also, THP current support three different modes [1]: 'never', 'madvise', and 'always'. The 'never' is self-explanatory and 'always' will enable THP for all anonymous memory. However, 'madvise' is still the default for some systems and for such cases THP will be only used if the memory range is explicity advertise by the program through a madvise(MADV_HUGEPAGE) call. This patchset adds a two new tunables to improve malloc() support with Huge Page, 'glibc.malloc.hugetlb' with the supported values: - 0: default value, do not enable any huge page usage. - 1: instructs the system allocator to issue a madvise(MADV_HUGEPAGE) call after a mmap() for sizes larger than the default huge page size and on sbrk() calls to extend the program data segment. - 2 or larger: instructthe system allocator to round allocation to huge page sizes along with the required flags (MAP_HUGETLB for Linux). If the memory allocation fails, the default system page size is used instead. A positive value larger than 2 sets a specific huge page size and the value is checked against the supported one by the system. The 'glibc.malloc.hugetlb=2' aims to replace the 'morecore' removed callback from 2.34 for libhugetlbfs (where the library tries to leverage the huge pages usage instead to provide a system allocator). By implementing the support directly on the mmap() code patch there is no need to try emulate the morecore()/sbrk() semantic which simplifies the code and make memory shrink logic more straighforward. I did also a sniff test check with SPECcpu2017 intspeed on a Ryzen 9 5900X machine using gcc 10.3 to compare glibc.malloc.hugetlb=0 and glibc.malloc.hugetlb=2 (and THP set to 'madvise'). The improvement is about 7.5% for hugetlb=2 (10.7 vs 11.5). Changes from previous version: - Fixed the area shrink logic, where the pagesize was not updated. - Removed malloc/tst-free-errno* from hugetlb2 tests set, since it requires to know which page size was used by the malloc call. - Add huge page support on main arena. Adhemerval Zanella (7): malloc: Add madvise support for Transparent Huge Pages malloc: Add THP/madvise support for sbrk malloc: Move mmap logic to its own function malloc: Add Huge Page support for mmap() malloc: Add huge page support to arenas malloc: Move MORECORE fallback mmap to sysmalloc_mmap_fallback malloc: Enable huge page support on main arena NEWS | 11 +- Rules | 36 +++ elf/dl-tunables.list | 4 + elf/tst-rtld-list-tunables.exp | 1 + include/libc-pointer-arith.h | 10 + malloc/Makefile | 23 ++ malloc/arena.c | 133 +++++--- malloc/malloc-internal.h | 1 + malloc/malloc.c | 357 ++++++++++++++------- malloc/morecore.c | 2 - manual/tunables.texi | 16 + sysdeps/generic/Makefile | 8 + sysdeps/generic/malloc-hugepages.c | 38 +++ sysdeps/generic/malloc-hugepages.h | 44 +++ sysdeps/unix/sysv/linux/malloc-hugepages.c | 202 ++++++++++++ 15 files changed, 731 insertions(+), 155 deletions(-) create mode 100644 sysdeps/generic/malloc-hugepages.c create mode 100644 sysdeps/generic/malloc-hugepages.h create mode 100644 sysdeps/unix/sysv/linux/malloc-hugepages.c