From patchwork Fri Aug 13 21:04:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 44665 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B0787384601E for ; Fri, 13 Aug 2021 21:07:35 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B0787384601E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1628888855; bh=apMFwwmfIcsxjTxApyESd04PwTllj31dd8f8cdQa9/Q=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=CL9hiEg/BvCu7WyvF6Z/LlA9zpLk1h/r3hd6a3a7nmX3rjFrDfVVhOEZAPiqruNmO /+Sgz51cjV+Fa3gcHcQ820e45UZ6TLVo+5n1f9Z19wzHnwXhPEroiFBj1c24h111Jm r/VDXSAnbXcE8uT64JqlfmVOiwrdf5NcI2R7/SUE= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) by sourceware.org (Postfix) with ESMTPS id 07750385F034 for ; Fri, 13 Aug 2021 21:04:37 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 07750385F034 Received: by mail-pl1-x62c.google.com with SMTP id e15so13611449plh.8 for ; Fri, 13 Aug 2021 14:04:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=apMFwwmfIcsxjTxApyESd04PwTllj31dd8f8cdQa9/Q=; b=Rw4ixC2h40WRXmLBk7k9XcG9yDe8qgkwky+fFiJ31z1ERlM+fnWniylnuIw4V7Seyz wuf+B3M+6qEZmt1zW8DQKJTnyzfYJoRLmnu2Pzvag8kIkUOpRmep4+bG+UGn10+EPFLd 8ReQB8SjpFjY3ajrTmpVIE5iyzRmxJHm7c5As/KXr9uoo6AeEZ8SCo81wXgUH6pp4AkE UJKeO4UYb6R7zWxeRCsyNJnlv+aPItXaIYI4693UenxSy4oEBEjs6Bqgaoxaayvlp93a 9k9VfAdPnz5K8m2tOtza0FSpLcoEBUtweV/ApUpdPoAy4of7/PPIE/ukb8Dzq5zAIVsA EibA== X-Gm-Message-State: AOAM531Rcd02oEX9PnSTqLvY7D5P6IWfhGX3THtexgcMlyXJuHPM5bdz Qg7rkaXL0pcRGGVBXrjwu41btZaTQgle0Q== X-Google-Smtp-Source: ABdhPJy3Cm7FkceDhUFUwP98txHCKfer1zGtmp7/ibijBBr8+Tf5vjR4JQIMmweMN8erAZBe3YtqGA== X-Received: by 2002:a62:32c7:0:b029:3cd:fba0:3218 with SMTP id y190-20020a6232c70000b02903cdfba03218mr4220559pfy.52.1628888675818; Fri, 13 Aug 2021 14:04:35 -0700 (PDT) Received: from birita.. ([2804:431:c7cb:9dce:7cf8:4c28:b9e7:2767]) by smtp.gmail.com with ESMTPSA id v15sm3259462pff.105.2021.08.13.14.04.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Aug 2021 14:04:35 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH 1/3] malloc: Add madvise support for Transparent Huge Pages Date: Fri, 13 Aug 2021 18:04:27 -0300 Message-Id: <20210813210429.1147112-2-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210813210429.1147112-1-adhemerval.zanella@linaro.org> References: <20210813210429.1147112-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Siddhesh Poyarekar , Guillaume Morin Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Linux Transparent Huge Pages (THP) current support three different states: 'never', 'madvise', and 'always'. The 'never' is self-explanatory and 'always' will enable THP for all anonymous memory. However, 'madvise' is still the default for some system and for such case THP will be only used if the memory range is explicity advertise by the program through the madvise(MADV_HUGEPAGE) call. To enable it a new tunable is provided, 'glibc.malloc.thp_pagesize', where the user can either enable THP through madvise using the default huge page size by using a value of '1' or by specifying a different large page size if the system supports it (Linux current only support one page size for THP, even if the architecture supports multiple sizes). This patch issues the madvise(MADV_HUGEPAGE) call after a successful mmap() call at sysmalloc(). The default malloc_verify_thp_pagesize() does not enable it even if the tunable is set. Checked on x86_64-linux-gnu. --- NEWS | 5 +++- elf/dl-tunables.list | 5 ++++ elf/tst-rtld-list-tunables.exp | 1 + malloc/arena.c | 5 ++++ malloc/malloc-internal.h | 1 + malloc/malloc.c | 45 ++++++++++++++++++++++++++++++++++ manual/tunables.texi | 11 +++++++++ sysdeps/generic/malloc-thp.h | 32 ++++++++++++++++++++++++ 8 files changed, 104 insertions(+), 1 deletion(-) create mode 100644 sysdeps/generic/malloc-thp.h diff --git a/NEWS b/NEWS index 79c895e382..85b7933e4d 100644 --- a/NEWS +++ b/NEWS @@ -9,7 +9,10 @@ Version 2.35 Major new features: - [Add new features here] +* On Linux, a new tunable, glibc.malloc.thp_pagesize, can be used to + make malloc issue madvise plus MADV_HUGEPAGE on mmap and sbrk calls. + It force the use of Transparent Huge Pages when madvise global mode + is set and might improve performance depending of the workload. Deprecated and removed features, and other changes affecting compatibility: diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list index 8ddd4a2314..77d3662ffd 100644 --- a/elf/dl-tunables.list +++ b/elf/dl-tunables.list @@ -92,6 +92,11 @@ glibc { minval: 0 security_level: SXID_IGNORE } + thp_pagesize { + type: SIZE_T + minval: 0 + default: 0 + } } cpu { hwcap_mask { diff --git a/elf/tst-rtld-list-tunables.exp b/elf/tst-rtld-list-tunables.exp index 9f66c52885..532af4eabc 100644 --- a/elf/tst-rtld-list-tunables.exp +++ b/elf/tst-rtld-list-tunables.exp @@ -8,6 +8,7 @@ glibc.malloc.perturb: 0 (min: 0, max: 255) glibc.malloc.tcache_count: 0x0 (min: 0x0, max: 0x[f]+) glibc.malloc.tcache_max: 0x0 (min: 0x0, max: 0x[f]+) glibc.malloc.tcache_unsorted_limit: 0x0 (min: 0x0, max: 0x[f]+) +glibc.malloc.thp_pagesize: 0x0 (min: 0x0, max: 0x[f]+) glibc.malloc.top_pad: 0x0 (min: 0x0, max: 0x[f]+) glibc.malloc.trim_threshold: 0x0 (min: 0x0, max: 0x[f]+) glibc.rtld.nns: 0x4 (min: 0x1, max: 0x10) diff --git a/malloc/arena.c b/malloc/arena.c index 667484630e..7ec316a906 100644 --- a/malloc/arena.c +++ b/malloc/arena.c @@ -231,6 +231,7 @@ TUNABLE_CALLBACK_FNDECL (set_tcache_count, size_t) TUNABLE_CALLBACK_FNDECL (set_tcache_unsorted_limit, size_t) #endif TUNABLE_CALLBACK_FNDECL (set_mxfast, size_t) +TUNABLE_CALLBACK_FNDECL (set_thp_pagesize, size_t) #else /* Initialization routine. */ #include @@ -331,6 +332,7 @@ ptmalloc_init (void) TUNABLE_CALLBACK (set_tcache_unsorted_limit)); # endif TUNABLE_GET (mxfast, size_t, TUNABLE_CALLBACK (set_mxfast)); + TUNABLE_GET (thp_pagesize, size_t, TUNABLE_CALLBACK (set_thp_pagesize)); #else if (__glibc_likely (_environ != NULL)) { @@ -509,6 +511,9 @@ new_heap (size_t size, size_t top_pad) __munmap (p2, HEAP_MAX_SIZE); return 0; } + + sysmadvise_thp (p2, size); + h = (heap_info *) p2; h->size = size; h->mprotect_size = size; diff --git a/malloc/malloc-internal.h b/malloc/malloc-internal.h index 0c7b5a183c..2efef06f35 100644 --- a/malloc/malloc-internal.h +++ b/malloc/malloc-internal.h @@ -22,6 +22,7 @@ #include #include #include +#include /* Called in the parent process before a fork. */ void __malloc_fork_lock_parent (void) attribute_hidden; diff --git a/malloc/malloc.c b/malloc/malloc.c index e065785af7..52ea84a63d 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -1881,6 +1881,11 @@ struct malloc_par INTERNAL_SIZE_T arena_test; INTERNAL_SIZE_T arena_max; +#if HAVE_TUNABLES + /* Transparent Large Page support. */ + INTERNAL_SIZE_T thp_pagesize; +#endif + /* Memory map support */ int n_mmaps; int n_mmaps_max; @@ -2009,6 +2014,20 @@ free_perturb (char *p, size_t n) #include +/* ----------- Routines dealing with transparent huge pages ----------- */ + +static inline void +sysmadvise_thp (void *p, INTERNAL_SIZE_T size) +{ +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + /* Do not consider areas smaller than a huge page or if the tunable is + not active. */ + if (mp_.thp_pagesize == 0 || size < mp_.thp_pagesize) + return; + __madvise (p, size, MADV_HUGEPAGE); +#endif +} + /* ------------------- Support for multiple arenas -------------------- */ #include "arena.c" @@ -2446,6 +2465,8 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) if (mm != MAP_FAILED) { + sysmadvise_thp (mm, size); + /* The offset to the start of the mmapped region is stored in the prev_size field of the chunk. This allows us to adjust @@ -2607,6 +2628,8 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) if (size > 0) { brk = (char *) (MORECORE (size)); + if (brk != (char *) (MORECORE_FAILURE)) + sysmadvise_thp (brk, size); LIBC_PROBE (memory_sbrk_more, 2, brk, size); } @@ -2638,6 +2661,8 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) if (mbrk != MAP_FAILED) { + sysmadvise_thp (mbrk, size); + /* We do not need, and cannot use, another sbrk call to find end */ brk = mbrk; snd_brk = brk + size; @@ -2749,6 +2774,8 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) correction = 0; snd_brk = (char *) (MORECORE (0)); } + else + sysmadvise_thp (snd_brk, correction); } /* handle non-contiguous cases */ @@ -2989,6 +3016,8 @@ mremap_chunk (mchunkptr p, size_t new_size) if (cp == MAP_FAILED) return 0; + sysmadvise_thp (cp, new_size); + p = (mchunkptr) (cp + offset); assert (aligned_OK (chunk2mem (p))); @@ -5325,6 +5354,22 @@ do_set_mxfast (size_t value) return 0; } +#if HAVE_TUNABLES +static __always_inline int +do_set_thp_pagesize (size_t value) +{ + /* Only enable THP through madvise if the arch-specific return size is + larger than the default page size. */ + if (value > 0) + { + size_t thps = malloc_verify_thp_pagesize (value); + if (thps != GLRO(dl_pagesize)) + mp_.thp_pagesize = thps; + } + return 0; +} +#endif + int __libc_mallopt (int param_number, int value) { diff --git a/manual/tunables.texi b/manual/tunables.texi index 658547c613..3364e85ef5 100644 --- a/manual/tunables.texi +++ b/manual/tunables.texi @@ -270,6 +270,17 @@ pointer, so add 4 on 32-bit systems or 8 on 64-bit systems to the size passed to @code{malloc} for the largest bin size to enable. @end deftp +@deftp Tunable glibc.malloc.thp_pagesize +This tunable enables support for Transparent Huge Page through @code{madvise} +with @code{MADV_HUGEPAGE} on the allocated memory range after @code{malloc} +calls the system allocator. Each architecture defines set of possible values, +and the input value is rounded to the supported one. + +The default value of this tunable is 0, which disable its usage. The value +of 1 meants to use the default Huge Page size for the architecture, and +a value larger than 2 is rounded to the supported size. +@end deftp + @node Dynamic Linking Tunables @section Dynamic Linking Tunables @cindex dynamic linking tunables diff --git a/sysdeps/generic/malloc-thp.h b/sysdeps/generic/malloc-thp.h new file mode 100644 index 0000000000..d70ceb8e1e --- /dev/null +++ b/sysdeps/generic/malloc-thp.h @@ -0,0 +1,32 @@ +/* Transparent Huge Page support. Generic implementation. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; see the file COPYING.LIB. If + not, see . */ + +#ifndef _MALLOC_THP_H +#define _MALLOC_THP_H + +#include + +/* Return the prefered large page size for the request PAGESIZE. The + requested value of 1 means the default size for the architecture. */ +static inline size_t +malloc_verify_thp_pagesize (size_t pagesize) +{ + return GLRO(dl_pagesize); +} + +#endif /* _MALLOC_THP_H */ From patchwork Fri Aug 13 21:04:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 44663 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 365833851C14 for ; Fri, 13 Aug 2021 21:05:54 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 365833851C14 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1628888754; bh=fLi5CLlxkvnLcH7KfYOhUCZf4O5gpRbn9DcBfGpMQIQ=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=Bcq+FKl8byVAPmvubfGmUNHjJT7o9xhJ0P7nnwwF5xbXVU7NdhzVpZlF8uavpjprr BDLPrveEbi1zfU8j/5YqIoFUs95eFfiQ101SFFSpOtQK3/FCHhYTE9emV47JRz4w8z DqsUfMSnxAOp24LJ0+MzIgkYe56luIg3nvdt/4xw= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by sourceware.org (Postfix) with ESMTPS id A6897386184A for ; Fri, 13 Aug 2021 21:04:38 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org A6897386184A Received: by mail-pj1-x102b.google.com with SMTP id u13-20020a17090abb0db0290177e1d9b3f7so22524124pjr.1 for ; Fri, 13 Aug 2021 14:04:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fLi5CLlxkvnLcH7KfYOhUCZf4O5gpRbn9DcBfGpMQIQ=; b=Y8ttn0UVIuVyiIBfEZ0br48yVfAk9trTmOS5ZNs3MgD9cCESIQs06vo+sJ6KG1JvnG 0SiVqtMJpIG69dFD5xce8DpVOoY6kkae1LoFEZW5m5V/aU82IxEomZHtHlrQtsReFQ5H hu/4La6H9pM8bsld7xHhR3yk6HRsOFhPST+ggtdP3d3/+ARTRF73KQ+T3quVjVA5dlrz 2HYSjTDPBTpNzbCL8vvdVI6GS3M5VgsMXCV1osgR52MNzN2c7nFrxcz0GvN3FbYY+H5e r/q/ja8Hw3WCCJLmhVykKk+NSDgJS2aD73rF7R6JRkGz3Tv0hNq2GIvBuB6JiHWvM7/D F0Vw== X-Gm-Message-State: AOAM531rDf7lLT/EIO7mnvW7Cu8zK/dsht4g0OCsg8QFJKsszZ0B1fr3 qJXjMSrLSDp+Gh3ZvoZVLgg14Zy7Ee5KEA== X-Google-Smtp-Source: ABdhPJyKWmEFpSf3incQfYjNe9ol04wATnNePEKOClQojF6sZ6vkqJQjXI/3h0LPbbZ3agEeNiIOBQ== X-Received: by 2002:a17:902:c950:b029:12d:2ada:9ef7 with SMTP id i16-20020a170902c950b029012d2ada9ef7mr3525259pla.61.1628888677639; Fri, 13 Aug 2021 14:04:37 -0700 (PDT) Received: from birita.. ([2804:431:c7cb:9dce:7cf8:4c28:b9e7:2767]) by smtp.gmail.com with ESMTPSA id v15sm3259462pff.105.2021.08.13.14.04.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Aug 2021 14:04:37 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH 2/3] malloc: Add THP/madvise support for sbrk Date: Fri, 13 Aug 2021 18:04:28 -0300 Message-Id: <20210813210429.1147112-3-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210813210429.1147112-1-adhemerval.zanella@linaro.org> References: <20210813210429.1147112-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-13.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=unavailable autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Siddhesh Poyarekar , Guillaume Morin Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" For the main arena, the sbrk() might the preferable syscall instead of mmap(). And the granularity used when increasing the program segment is the default page size. To increase effectiveness with Transparent Huge Page with madvise, the large page size is use instead. This is enabled with the new tunable 'glibc.malloc.thp_pagesize'. Checked on x86_64-linux-gnu. --- malloc/malloc.c | 40 +++++++++++++++++++++++++++++++++++----- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index 52ea84a63d..7cd586c866 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -2028,6 +2028,38 @@ sysmadvise_thp (void *p, INTERNAL_SIZE_T size) #endif } +static inline long int +thp_brk_align_up (long int size) +{ + INTERNAL_SIZE_T r = size; +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + /* Defined in brk.c. */ + extern void *__curbrk; + if (mp_.thp_pagesize != 0) + { + uintptr_t top = ALIGN_UP ((uintptr_t)__curbrk + size, mp_.thp_pagesize); + r = top - (uintptr_t)__curbrk; + } + else +#endif + r = ALIGN_UP (size, GLRO(dl_pagesize)); + return r; +} + +static inline long +thp_brk_align_down (long int top) +{ + long r; +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + if (mp_.thp_pagesize != 0) + r = ALIGN_DOWN (top, mp_.thp_pagesize); + else +#endif + r = ALIGN_DOWN (top, GLRO(dl_pagesize)); + return r; +} + + /* ------------------- Support for multiple arenas -------------------- */ #include "arena.c" @@ -2610,14 +2642,14 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) size -= old_size; /* - Round to a multiple of page size. + Round to a multiple of page size or huge page size. If MORECORE is not contiguous, this ensures that we only call it with whole-page arguments. And if MORECORE is contiguous and this is not first time through, this preserves page-alignment of previous calls. Otherwise, we correct to page-align below. */ - size = ALIGN_UP (size, pagesize); + size = thp_brk_align_up (size); /* Don't try to call MORECORE if argument is so big as to appear @@ -2900,10 +2932,8 @@ systrim (size_t pad, mstate av) long released; /* Amount actually released */ char *current_brk; /* address returned by pre-check sbrk call */ char *new_brk; /* address returned by post-check sbrk call */ - size_t pagesize; long top_area; - pagesize = GLRO (dl_pagesize); top_size = chunksize (av->top); top_area = top_size - MINSIZE - 1; @@ -2911,7 +2941,7 @@ systrim (size_t pad, mstate av) return 0; /* Release in pagesize units and round down to the nearest page. */ - extra = ALIGN_DOWN(top_area - pad, pagesize); + extra = thp_brk_align_down (top_area - pad); if (extra == 0) return 0; From patchwork Fri Aug 13 21:04:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 44664 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C739E3857424 for ; Fri, 13 Aug 2021 21:06:49 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org C739E3857424 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1628888809; bh=P/TcJ6r/PgTtmm016ijdlKo/YH3/X1qB+XmBSMDngPw=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=JKhJ9YTwEU3tn2i7hpWPCEhEtveIfYUQkWG6Zx0oKteluFdI8z8+smFL9PRgdu0gT b3dz79aVRoM0eIr9HQ5zs/UTIrn8WMpS+ATrbGHNTGiLTp3FHD56B6RLKXXfFEZ6k5 HJBuy9xF4JaDGEqHIn6ilrZTSFVX7OMWiWejWNg8= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by sourceware.org (Postfix) with ESMTPS id AD29D3889C30 for ; Fri, 13 Aug 2021 21:04:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org AD29D3889C30 Received: by mail-pj1-x1029.google.com with SMTP id 28-20020a17090a031cb0290178dcd8a4d1so12057388pje.0 for ; Fri, 13 Aug 2021 14:04:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P/TcJ6r/PgTtmm016ijdlKo/YH3/X1qB+XmBSMDngPw=; b=DL2P8+yGdvyc9H/Db+7DeO/L0EOIfDFE/BmqJqhatVka5dBeIthQQqEwcA71ESWIGz D1FNz7NO5dir5TbixQn8K2Meqk4BAtcNDhBBbU5k/1+0W3OXl+4lmLk+EWW8eF+B3JB4 kbAlyRI2BbixTiqCv1cE60mXFLR444IaGgF7kLQuGhki3t2zjlcuegLIyFtfsQvGBhgl 20cy9hapr6AWwgfQLwJpMWNyrzEzJfmhEUF1snQLeVBuMjB4JQzT6B5KSwiOeuOU0/2j eKZB7pMNxOVPBUsQZ4iGAt5RB7UCJthoPz3WFsytbl8LfrXnayRvOMJ7lh3bEq01Vztd zhKw== X-Gm-Message-State: AOAM5317JvnFoyiVR3vdG6Laz8x5/vXPEfpz2UcettX7lho/QeyIctQt sLr96BvXHLcj9AxLTxTLjsjl2xatgeGD0w== X-Google-Smtp-Source: ABdhPJwky/dqL0sLXhp7bMIu0hx6OmtdYJpGunXP4t1cIF1IBHO+GG/UtmV80i5IeGw3f2lwiFqFVA== X-Received: by 2002:a17:903:244d:b029:12d:77ee:2dd1 with SMTP id l13-20020a170903244db029012d77ee2dd1mr3538079pls.72.1628888679503; Fri, 13 Aug 2021 14:04:39 -0700 (PDT) Received: from birita.. ([2804:431:c7cb:9dce:7cf8:4c28:b9e7:2767]) by smtp.gmail.com with ESMTPSA id v15sm3259462pff.105.2021.08.13.14.04.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Aug 2021 14:04:39 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH 3/3] malloc: Add arch-specific malloc_verify_thp_pagesize for Linux Date: Fri, 13 Aug 2021 18:04:29 -0300 Message-Id: <20210813210429.1147112-4-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210813210429.1147112-1-adhemerval.zanella@linaro.org> References: <20210813210429.1147112-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-13.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Siddhesh Poyarekar , Guillaume Morin Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" Not all architectures have Transparent Huge Page (THP) support enabled by default, so this patch only adds support for the one that have HAVE_ARCH_TRANSPARENT_HUGEPAGE defined. Also, Linux THP only support one Huge Page size, so malloc_verify_thp_pagesize() returns only the default value for the architecture. The x86, sparc, and riscv are straightforward since they only support one possible value. AArch64 and mips, which supports multiple pages sizes, can use a direct map to the Large Page support. PowerPC is the only architecture where its THP size depends not only on the configured page size, but also on which MMU is used. For this case the sysfs file is used instead. Checked on x86_64-linux-gnu, aarch64-linux-gnu, and powerpc64le-linux-gnu. --- sysdeps/unix/sysv/linux/aarch64/malloc-thp.h | 40 ++++++++++++++ sysdeps/unix/sysv/linux/mips/malloc-thp.h | 39 ++++++++++++++ sysdeps/unix/sysv/linux/powerpc/malloc-thp.h | 56 ++++++++++++++++++++ sysdeps/unix/sysv/linux/riscv/malloc-thp.h | 32 +++++++++++ sysdeps/unix/sysv/linux/s390/malloc-thp.h | 33 ++++++++++++ sysdeps/unix/sysv/linux/sparc/malloc-thp.h | 36 +++++++++++++ sysdeps/unix/sysv/linux/x86/malloc-thp.h | 32 +++++++++++ 7 files changed, 268 insertions(+) create mode 100644 sysdeps/unix/sysv/linux/aarch64/malloc-thp.h create mode 100644 sysdeps/unix/sysv/linux/mips/malloc-thp.h create mode 100644 sysdeps/unix/sysv/linux/powerpc/malloc-thp.h create mode 100644 sysdeps/unix/sysv/linux/riscv/malloc-thp.h create mode 100644 sysdeps/unix/sysv/linux/s390/malloc-thp.h create mode 100644 sysdeps/unix/sysv/linux/sparc/malloc-thp.h create mode 100644 sysdeps/unix/sysv/linux/x86/malloc-thp.h diff --git a/sysdeps/unix/sysv/linux/aarch64/malloc-thp.h b/sysdeps/unix/sysv/linux/aarch64/malloc-thp.h new file mode 100644 index 0000000000..e2e65446f2 --- /dev/null +++ b/sysdeps/unix/sysv/linux/aarch64/malloc-thp.h @@ -0,0 +1,40 @@ +/* Transparent Huge Page support. AArch64 implementation. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; see the file COPYING.LIB. If + not, see . */ + +#ifndef _MALLOC_THP_H +#define _MALLOC_THP_H + +#include +#include + +/* Return the prefered large page size for the request PAGESIZE. The + requested value of 1 means the default size for the architecutre. + Returning 0 disables Large Parse usage. */ +static inline size_t +malloc_verify_thp_pagesize (size_t pagesize) +{ + /* AArch64 THP size depends of the default page size: + 4k -> 2m + 16k -> 32m + 64k -> 512m */ + int page_shift = __builtin_ctzl (GLRO(dl_pagesize)); + printf ("%s: page_shift=%d\n", __func__, page_shift); + return 1UL << ((page_shift - 3) * 2 + 3); +} + +#endif /* _MALLOC_THP_H */ diff --git a/sysdeps/unix/sysv/linux/mips/malloc-thp.h b/sysdeps/unix/sysv/linux/mips/malloc-thp.h new file mode 100644 index 0000000000..d8cdbd026d --- /dev/null +++ b/sysdeps/unix/sysv/linux/mips/malloc-thp.h @@ -0,0 +1,39 @@ +/* Transparent Huge Page support. Generic implementation. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; see the file COPYING.LIB. If + not, see . */ + +#ifndef _MALLOC_THP_H +#define _MALLOC_THP_H + +#include + +/* Return the prefered large page size for the request PAGESIZE. The + requested value of 1 means the default size for the architecture. */ +static inline size_t +malloc_verify_thp_pagesize (size_t pagesize) +{ + /* MIPS THP size depends of the default page size: + 4k -> 2m + 8k -> 8m + 16k -> 32m + 32k -> 128m + 64k -> 512m */ + int page_shift = __builtin_ctzl (GLRO(dl_pagesize)); + return 1UL << (page_shift + (page_shift - 3)); +} + +#endif /* _MALLOC_THP_H */ diff --git a/sysdeps/unix/sysv/linux/powerpc/malloc-thp.h b/sysdeps/unix/sysv/linux/powerpc/malloc-thp.h new file mode 100644 index 0000000000..c3fcfa3386 --- /dev/null +++ b/sysdeps/unix/sysv/linux/powerpc/malloc-thp.h @@ -0,0 +1,56 @@ +/* Transparent Huge Page support. PowerPC implementation. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; see the file COPYING.LIB. If + not, see . */ + +#ifndef _MALLOC_THP_H +#define _MALLOC_THP_H + +#include +#include + +/* Return the prefered large page size for the request PAGESIZE. The + requested value of 1 means the default size for the architecture. */ +static inline size_t +malloc_verify_thp_pagesize (size_t pagesize) +{ + /* PowerPC THP size depends of the default page size and which MMU hardware + is used. So no easy way to statically map it, query the kernel + instead. */ + int fd = __open64_nocancel ( + "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY); + if (fd == -1) + return 0; + + size_t hps = 0; + + char str[INT_BUFSIZE_BOUND (size_t)]; + ssize_t r = __read_nocancel (fd, str, sizeof (str)); + if (r > 0) + for (ssize_t i = 0; i < r; i++) + { + if (str[i] == '\n') + break; + hps *= 10; + hps += str[i] - '0'; + } + + __close_nocancel (fd); + + return hps; +} + +#endif /* _MALLOC_THP_H */ diff --git a/sysdeps/unix/sysv/linux/riscv/malloc-thp.h b/sysdeps/unix/sysv/linux/riscv/malloc-thp.h new file mode 100644 index 0000000000..aa38ca6dd6 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/malloc-thp.h @@ -0,0 +1,32 @@ +/* Transparent Huge Page support. RISCV implementation. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; see the file COPYING.LIB. If + not, see . */ + +#ifndef _MALLOC_THP_H +#define _MALLOC_THP_H + +#include + +/* Return the prefered large page size for the request PAGESIZE. The + requested value of 1 means the default size for the architecture. */ +static inline size_t +malloc_verify_thp_pagesize (size_t pagesize) +{ + return 1UL << 21; +} + +#endif /* _MALLOC_THP_H */ diff --git a/sysdeps/unix/sysv/linux/s390/malloc-thp.h b/sysdeps/unix/sysv/linux/s390/malloc-thp.h new file mode 100644 index 0000000000..be6c26aa3b --- /dev/null +++ b/sysdeps/unix/sysv/linux/s390/malloc-thp.h @@ -0,0 +1,33 @@ +/* Transparent Huge Page support. Generic implementation. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; see the file COPYING.LIB. If + not, see . */ + +#ifndef _MALLOC_THP_H +#define _MALLOC_THP_H + +#include + +/* Return the prefered large page size for the request PAGESIZE. The + requested value of 1 means the default size for the architecture. */ +static inline size_t +malloc_verify_thp_pagesize (size_t pagesize) +{ + /* s390 uses 1M for THP. */ + return 1UL << 20; +} + +#endif /* _MALLOC_THP_H */ diff --git a/sysdeps/unix/sysv/linux/sparc/malloc-thp.h b/sysdeps/unix/sysv/linux/sparc/malloc-thp.h new file mode 100644 index 0000000000..83f6fdc114 --- /dev/null +++ b/sysdeps/unix/sysv/linux/sparc/malloc-thp.h @@ -0,0 +1,36 @@ +/* Transparent Huge Page support. SPARC implementation. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; see the file COPYING.LIB. If + not, see . */ + +#ifndef _MALLOC_THP_H +#define _MALLOC_THP_H + +#include + +/* Return the prefered large page size for the request PAGESIZE. The + requested value of 1 means the default size for the architecture. */ +static inline size_t +malloc_verify_thp_pagesize (size_t pagesize) +{ +#ifdef __arch64__ + return 1UL << 23; +#else + return 1UL << 18; +#endif +} + +#endif /* _MALLOC_THP_H */ diff --git a/sysdeps/unix/sysv/linux/x86/malloc-thp.h b/sysdeps/unix/sysv/linux/x86/malloc-thp.h new file mode 100644 index 0000000000..d94cb578c2 --- /dev/null +++ b/sysdeps/unix/sysv/linux/x86/malloc-thp.h @@ -0,0 +1,32 @@ +/* Transparent Huge Page support. Generic implementation. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; see the file COPYING.LIB. If + not, see . */ + +#ifndef _MALLOC_THP_H +#define _MALLOC_THP_H + +#include + +/* Return the prefered large page size for the request PAGESIZE. The + requested value of 1 means the default size for the architecture. */ +static inline size_t +malloc_verify_thp_pagesize (size_t pagesize) +{ + return 1UL << 21; +} + +#endif /* _MALLOC_THP_H */