From patchwork Mon Aug 23 21:57:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 44768 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A6FFA3857810 for ; Mon, 23 Aug 2021 21:58:31 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A6FFA3857810 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1629755911; bh=Q06qKnYEzWdecibjv6osXvIEPjVP7LVqhUT3s56vunU=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=WTrdTVGJ69yt/veH775MsTwcR/p2f6TE1/1oO9s6FvsVlcpKxTWNnHM0QhnWXft/e gpQbwopr/EwUyc7zMSvN07Ir8WHo3Qfjj2z8/cxxHNxI2l1MSWflTigsH9eFtgvhLs YHthQ6+BqIt9y3LUtFAoPE7upFXI3bmVr9VjCSi8= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) by sourceware.org (Postfix) with ESMTPS id 3243E3857825 for ; Mon, 23 Aug 2021 21:57:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 3243E3857825 Received: by mail-qk1-x72f.google.com with SMTP id c10so18381998qko.11 for ; Mon, 23 Aug 2021 14:57:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q06qKnYEzWdecibjv6osXvIEPjVP7LVqhUT3s56vunU=; b=fHUYsy7rN5BSDwI5OxEfM9Qg06C0A5JvdkgEKDJtq/0ZWf8WZsFKY7WcvPVrVGh1xy Pj8ahB7aXBzAS+idNnMfHkKUwK9OrjttvGbEthqAsA3lmYzd3ryYtjOcHiUfGbsDl6FQ u/BJKf+ij0UEHXbSAWrqw3BFSYh5skdyvphwLkIC6EEUPzcuD7xfxmaqAD8v+2o4F2w+ ErSHcmYqKvuxmfch5XzD5f/gokYKshQCkpwpWbvrWmL5VO2u7QkA5tAs0Z/xLdJF/pJh 41ShuLDYsxAHXBrT0lomJtOru60eUXrun4ztkGJAJ/6zUr5x6KHJlrPFUovRrTpkt1FK Yjxg== X-Gm-Message-State: AOAM530QiTsO812S7ci6MfexboM7Sj2gOVCPoC47M5XTtSmPbYEQ0yHG c310X+yMoncej3HFvI4dnpNEVeSopHMxzw== X-Google-Smtp-Source: ABdhPJyXF+d+t3eqhyBOX8FRT8PhLm19Sr5Lhcc+AcmFFLOhZu1ZOq/VsrjNxit5qXC8UNZzOMurjQ== X-Received: by 2002:a37:803:: with SMTP id 3mr23451149qki.127.1629755840659; Mon, 23 Aug 2021 14:57:20 -0700 (PDT) Received: from birita.. ([2804:431:c7ca:cd83:c38b:b50d:5d9a:43d4]) by smtp.gmail.com with ESMTPSA id g1sm7444540qti.56.2021.08.23.14.57.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Aug 2021 14:57:20 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH v3 2/5] malloc: Add THP/madvise support for sbrk Date: Mon, 23 Aug 2021 18:57:10 -0300 Message-Id: <20210823215713.3304523-3-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210823215713.3304523-1-adhemerval.zanella@linaro.org> References: <20210823215713.3304523-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-13.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=unavailable autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Guillaume Morin , Siddhesh Poyarekar Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" For the main arena, the sbrk() is used as default instead of mmap() and the granularity used when increasing the program segment is the default page size. To increase effectiveness with Transparent Huge Page with madvise, the large page size is use instead. This is enabled by setting the value '1' to the new huge pages tunable 'glibc.malloc.hugetlb'. Checked on x86_64-linux-gnu. --- include/libc-pointer-arith.h | 10 ++++++++++ malloc/malloc.c | 35 ++++++++++++++++++++++++++++++----- 2 files changed, 40 insertions(+), 5 deletions(-) diff --git a/include/libc-pointer-arith.h b/include/libc-pointer-arith.h index 04ba537617..f592cbafec 100644 --- a/include/libc-pointer-arith.h +++ b/include/libc-pointer-arith.h @@ -37,6 +37,16 @@ /* Cast an integer or a pointer VAL to integer with proper type. */ # define cast_to_integer(val) ((__integer_if_pointer_type (val)) (val)) +/* Check if SIZE is aligned on SIZE */ +#define IS_ALIGNED(base, size) \ + (((base) & (size - 1)) == 0) + +#define PTR_IS_ALIGNED(base, size) \ + ((((uintptr_t) (base)) & (size - 1)) == 0) + +#define PTR_DIFF(p1, p2) \ + ((ptrdiff_t)((uintptr_t)(p1) - (uintptr_t)(p2))) + /* Cast an integer VAL to void * pointer. */ # define cast_to_pointer(val) ((void *) (uintptr_t) (val)) diff --git a/malloc/malloc.c b/malloc/malloc.c index 81d3411560..f65e448130 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -2024,6 +2024,17 @@ madvise_thp (void *p, INTERNAL_SIZE_T size) not active. */ if (mp_.thp_pagesize == 0 || size < mp_.thp_pagesize) return; + + /* madvise() requires at least the input to be aligned to system page and + MADV_HUGEPAGE should handle unaligned address. Also unaligned inputs + should happen only for the initial data segment. */ + if (__glibc_unlikely (!PTR_IS_ALIGNED (p, GLRO (dl_pagesize)))) + { + void *q = PTR_ALIGN_DOWN (p, GLRO (dl_pagesize)); + size += PTR_DIFF (p, q); + p = q; + } + __madvise (p, size, MADV_HUGEPAGE); #endif } @@ -2610,14 +2621,25 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) size -= old_size; /* - Round to a multiple of page size. + Round to a multiple of page size or huge page size. If MORECORE is not contiguous, this ensures that we only call it with whole-page arguments. And if MORECORE is contiguous and this is not first time through, this preserves page-alignment of previous calls. Otherwise, we correct to page-align below. */ - size = ALIGN_UP (size, pagesize); +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + /* Defined in brk.c. */ + extern void *__curbrk; + if (mp_.thp_pagesize != 0) + { + uintptr_t top = ALIGN_UP ((uintptr_t) __curbrk + size, + mp_.thp_pagesize); + size = top - (uintptr_t) __curbrk; + } + else +#endif + size = ALIGN_UP (size, GLRO(dl_pagesize)); /* Don't try to call MORECORE if argument is so big as to appear @@ -2900,10 +2922,8 @@ systrim (size_t pad, mstate av) long released; /* Amount actually released */ char *current_brk; /* address returned by pre-check sbrk call */ char *new_brk; /* address returned by post-check sbrk call */ - size_t pagesize; long top_area; - pagesize = GLRO (dl_pagesize); top_size = chunksize (av->top); top_area = top_size - MINSIZE - 1; @@ -2911,7 +2931,12 @@ systrim (size_t pad, mstate av) return 0; /* Release in pagesize units and round down to the nearest page. */ - extra = ALIGN_DOWN(top_area - pad, pagesize); +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + if (mp_.thp_pagesize != 0) + extra = ALIGN_DOWN (top_area - pad, mp_.thp_pagesize); + else +#endif + extra = ALIGN_DOWN (top_area - pad, GLRO(dl_pagesize)); if (extra == 0) return 0;