From patchwork Tue Dec 14 18:57:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 48907 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id BFF42385AC3A for ; Tue, 14 Dec 2021 18:58:33 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BFF42385AC3A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1639508313; bh=3wh6I9LznYduNAKW1yPE1Kquhyh8jCmxQ96npuWMAdY=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=mgYC4es8O3CYCLu23iTHGhMTD8K4EzKJzSYFSAH0JlRlGeycn3RzXJCNVrabrjfyf LSiNUTH85z6LFD4tDp5ODGj8W7c4Ug6WSEUvbrpoiJHBtoTxneCHqalCFzyQ30nqGR EnLG5W9BYgl1Zz1T2CnOqPRAOEYQHDnMP34yTEiI= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e]) by sourceware.org (Postfix) with ESMTPS id 78CDA385AC36 for ; Tue, 14 Dec 2021 18:58:11 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 78CDA385AC36 Received: by mail-qt1-x82e.google.com with SMTP id n15so19434764qta.0 for ; Tue, 14 Dec 2021 10:58:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=3wh6I9LznYduNAKW1yPE1Kquhyh8jCmxQ96npuWMAdY=; b=O6DypOip9kQSi54vStb+MnvD2BS4dh9Fn2tc44FlakqD244jCGSNl7fSvyuv2WeTJH 0X0fMhRqhZsY+AowOqXpDxVDK0aUKF9Te1X/pQj7dysiYdBv2cay095VHsZ7nBPrK/+t MOljp1YjXEo7BJr9UA8pjxl1YnaCckieBtQRMsNGRU+mGbWp3X0qKApVjeXkHk+egXH2 Sc8LlHvHGH4eaX1XNb0WoGouVxsF8g/NvvmWc993iX1xxDDqk/0s7d5gHxtNhCMhQ8tD 9z2ZhR0viySBLV4UWp9mhKLDr/SjTy7yYAO1p0Ax9vsSfH/wPriN2gZlbervop3F/IpZ BX6g== X-Gm-Message-State: AOAM530oFPOM7kYFfWe9MjI8zQzc2l6v+MCaN8JdiskSnfhD4ECSHCvL TUQEysNM4usdUCyq0PhZmmczJtQQtXiiQQ== X-Google-Smtp-Source: ABdhPJzYIxdJEvoYpaYQGGwfyK2RqChCWWts2v6eJvW/TwDDTxUy4tx07upNHCOcrFYUaAAbu5+FSA== X-Received: by 2002:a05:622a:1053:: with SMTP id f19mr8058082qte.451.1639508290700; Tue, 14 Dec 2021 10:58:10 -0800 (PST) Received: from birita.. ([2804:431:c7ca:103f:1000:c46d:a2d6:9bed]) by smtp.gmail.com with ESMTPSA id j124sm422848qkd.98.2021.12.14.10.58.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Dec 2021 10:58:10 -0800 (PST) To: libc-alpha@sourceware.org, Siddhesh Poyarekar Subject: [PATCH v5 0/7] malloc: Improve Huge Page support Date: Tue, 14 Dec 2021 15:57:59 -0300 Message-Id: <20211214185806.4109231-1-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 X-Spam-Status: No, score=-3.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Guillaume Morin Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Linux currently supports two ways to use Huge Pages: either by using specific flags directly with the syscall (MAP_HUGETLB for mmap, or SHM_HUGETLB for shmget), or by using Transparent Huge Pages (THP) where the kernel will try to move allocated anonymous pages to Huge Pages blocks transparent to application. Also, THP current support three different modes [1]: 'never', 'madvise', and 'always'. The 'never' is self-explanatory and 'always' will enable THP for all anonymous memory. However, 'madvise' is still the default for some systems and for such cases THP will be only used if the memory range is explicity advertise by the program through a madvise(MADV_HUGEPAGE) call. This patchset adds a two new tunables to improve malloc() support with Huge Page, 'glibc.malloc.hugetlb' with the supported values: - 0: default value, do not enable any huge page usage. - 1: instruct the system allocator to issue a madvise(MADV_HUGEPAGE) call after a mmap() one for sizes larger than the default huge page size and on sbrk() calls to extend the program data segment. - 2: instruct the system allocator to round allocation to huge page sizes along with the required flags (MAP_HUGETLB for Linux). If the memory allocation fails, the default system page size is used instead. - >2: same as '2' but to use a specific huge page size. The value is checked against the supported one by the system. The 'glibc.malloc.hugetlb=2' aims to replace the 'morecore' removed callback from 2.34 for libhugetlbfs (where the library tries to leverage the huge pages usage instead to provide a system allocator). By implementing the support directly on the mmap() code patch there is no need to try emulate the morecore/sbrk semantic which simplifies the code and make memory shrink logic more straighforward. --- v5: Rebased on current master, fixed some typos and comment, and changed the return type of __malloc_default_thp_pagesize (to match the kernel). v4: Fixed the area shrink logic, removed malloc/tst-free-errno* from hugetlb2 tests set, added huge page support on main arena. v3: Only use one tunable, 'glibc.malloc.hugetlb', disabled madvise if THP is set to always, added support to arenas. v2: Renamed thp_pagesize to thp_madvise and make it a boolean state, added MAP_HUGETLB support for mmap, removed system specific hooks for THP huge page size in favor of Linux generic implementation, initial program segments need to be page aligned for the first madvise call. Adhemerval Zanella (7): malloc: Add madvise support for Transparent Huge Pages malloc: Add THP/madvise support for sbrk malloc: Move mmap logic to its own function malloc: Add Huge Page support for mmap() malloc: Add huge page support to arenas malloc: Move MORECORE fallback mmap to sysmalloc_mmap_fallback malloc: Enable huge page support on main arena NEWS | 7 + Rules | 36 +++ elf/dl-tunables.list | 4 + elf/tst-rtld-list-tunables.exp | 1 + include/libc-pointer-arith.h | 8 + malloc/Makefile | 23 ++ malloc/arena.c | 143 ++++++--- malloc/malloc-internal.h | 1 + malloc/malloc.c | 354 ++++++++++++++------- malloc/morecore.c | 4 - manual/tunables.texi | 17 + sysdeps/generic/Makefile | 8 + sysdeps/generic/malloc-hugepages.c | 39 +++ sysdeps/generic/malloc-hugepages.h | 44 +++ sysdeps/unix/sysv/linux/malloc-hugepages.c | 201 ++++++++++++ 15 files changed, 734 insertions(+), 156 deletions(-) create mode 100644 sysdeps/generic/malloc-hugepages.c create mode 100644 sysdeps/generic/malloc-hugepages.h create mode 100644 sysdeps/unix/sysv/linux/malloc-hugepages.c