From patchwork Mon Aug 30 18:52:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 44814 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id EFE3D3858429 for ; Mon, 30 Aug 2021 18:53:33 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org EFE3D3858429 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1630349614; bh=Q06qKnYEzWdecibjv6osXvIEPjVP7LVqhUT3s56vunU=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=U1FVY8eo6Jdr2ztYMLFmc80suG3KfFQTBxmpx6YJ1nJLgokHRX2k9MfFebJIjSBLO xD/krVtnhwDSD7fUS549w/NqOnxw+J9Y1QTyCybywnSU+uBv10fuVW1RfaSznDwqlj Sw2LrGCUlOEnkGvD6axIzfB+KxIz5sHDtEdEJQgY= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-qv1-xf30.google.com (mail-qv1-xf30.google.com [IPv6:2607:f8b0:4864:20::f30]) by sourceware.org (Postfix) with ESMTPS id C608A385840A for ; Mon, 30 Aug 2021 18:52:23 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org C608A385840A Received: by mail-qv1-xf30.google.com with SMTP id dt3so8911372qvb.6 for ; Mon, 30 Aug 2021 11:52:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q06qKnYEzWdecibjv6osXvIEPjVP7LVqhUT3s56vunU=; b=VPV7rBuFRhf+p4S9C8gzJ4dPr7dorm9kaXBIlejf5XagpTGkzO/G2wYCFQIB1etvpw j1sSy56sI8tDusRrx9tuf2lfNveCkKE9UobOh1iHPI/mHPS4oalVZ1KBDHiE6qqNlWjh Jxv5j3zjuy7PSWlnSwapExOY58Go3hdBWbFoPBn1z2EzgWzU+KMfwn/rTHRzNcEO4uld vjJMx4+IAhpMc4pUkTxoxVbLSfWbHqkKFRgXtUfKzJ3IkfPS6SYlYA6GfieetInwNUN3 YR0kRZRWT+tiWFyCL+xFGA0DPUbshFsQGpRck582IXtZYcaVhHuN+yafAKH9pObMyWJv E7xQ== X-Gm-Message-State: AOAM530UEDW+8/KjQVQnVxPgo7liLnTpLiBPg0doK0/61EMlz4FkYbFU Ey5QFAT5rbofooF68gXYFvpN9ucs2SWuaw== X-Google-Smtp-Source: ABdhPJzAcnegsNzAns5TD7YsCAPhHmbjrFvEl9Fu8RF29gWDm8buVKYJRdn58rl2y6G7axZHWlM+uQ== X-Received: by 2002:a05:6214:1843:: with SMTP id d3mr25231310qvy.10.1630349543180; Mon, 30 Aug 2021 11:52:23 -0700 (PDT) Received: from birita.. ([2804:431:c7ca:1a68:7647:1f41:2147:1ed2]) by smtp.gmail.com with ESMTPSA id m187sm11752338qkd.131.2021.08.30.11.52.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Aug 2021 11:52:22 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH v4 2/7] malloc: Add THP/madvise support for sbrk Date: Mon, 30 Aug 2021 15:52:10 -0300 Message-Id: <20210830185215.449572-3-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210830185215.449572-1-adhemerval.zanella@linaro.org> References: <20210830185215.449572-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-13.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=unavailable autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Guillaume Morin , Siddhesh Poyarekar Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" For the main arena, the sbrk() is used as default instead of mmap() and the granularity used when increasing the program segment is the default page size. To increase effectiveness with Transparent Huge Page with madvise, the large page size is use instead. This is enabled by setting the value '1' to the new huge pages tunable 'glibc.malloc.hugetlb'. Checked on x86_64-linux-gnu. --- include/libc-pointer-arith.h | 10 ++++++++++ malloc/malloc.c | 35 ++++++++++++++++++++++++++++++----- 2 files changed, 40 insertions(+), 5 deletions(-) diff --git a/include/libc-pointer-arith.h b/include/libc-pointer-arith.h index 04ba537617..f592cbafec 100644 --- a/include/libc-pointer-arith.h +++ b/include/libc-pointer-arith.h @@ -37,6 +37,16 @@ /* Cast an integer or a pointer VAL to integer with proper type. */ # define cast_to_integer(val) ((__integer_if_pointer_type (val)) (val)) +/* Check if SIZE is aligned on SIZE */ +#define IS_ALIGNED(base, size) \ + (((base) & (size - 1)) == 0) + +#define PTR_IS_ALIGNED(base, size) \ + ((((uintptr_t) (base)) & (size - 1)) == 0) + +#define PTR_DIFF(p1, p2) \ + ((ptrdiff_t)((uintptr_t)(p1) - (uintptr_t)(p2))) + /* Cast an integer VAL to void * pointer. */ # define cast_to_pointer(val) ((void *) (uintptr_t) (val)) diff --git a/malloc/malloc.c b/malloc/malloc.c index 81d3411560..f65e448130 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -2024,6 +2024,17 @@ madvise_thp (void *p, INTERNAL_SIZE_T size) not active. */ if (mp_.thp_pagesize == 0 || size < mp_.thp_pagesize) return; + + /* madvise() requires at least the input to be aligned to system page and + MADV_HUGEPAGE should handle unaligned address. Also unaligned inputs + should happen only for the initial data segment. */ + if (__glibc_unlikely (!PTR_IS_ALIGNED (p, GLRO (dl_pagesize)))) + { + void *q = PTR_ALIGN_DOWN (p, GLRO (dl_pagesize)); + size += PTR_DIFF (p, q); + p = q; + } + __madvise (p, size, MADV_HUGEPAGE); #endif } @@ -2610,14 +2621,25 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) size -= old_size; /* - Round to a multiple of page size. + Round to a multiple of page size or huge page size. If MORECORE is not contiguous, this ensures that we only call it with whole-page arguments. And if MORECORE is contiguous and this is not first time through, this preserves page-alignment of previous calls. Otherwise, we correct to page-align below. */ - size = ALIGN_UP (size, pagesize); +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + /* Defined in brk.c. */ + extern void *__curbrk; + if (mp_.thp_pagesize != 0) + { + uintptr_t top = ALIGN_UP ((uintptr_t) __curbrk + size, + mp_.thp_pagesize); + size = top - (uintptr_t) __curbrk; + } + else +#endif + size = ALIGN_UP (size, GLRO(dl_pagesize)); /* Don't try to call MORECORE if argument is so big as to appear @@ -2900,10 +2922,8 @@ systrim (size_t pad, mstate av) long released; /* Amount actually released */ char *current_brk; /* address returned by pre-check sbrk call */ char *new_brk; /* address returned by post-check sbrk call */ - size_t pagesize; long top_area; - pagesize = GLRO (dl_pagesize); top_size = chunksize (av->top); top_area = top_size - MINSIZE - 1; @@ -2911,7 +2931,12 @@ systrim (size_t pad, mstate av) return 0; /* Release in pagesize units and round down to the nearest page. */ - extra = ALIGN_DOWN(top_area - pad, pagesize); +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + if (mp_.thp_pagesize != 0) + extra = ALIGN_DOWN (top_area - pad, mp_.thp_pagesize); + else +#endif + extra = ALIGN_DOWN (top_area - pad, GLRO(dl_pagesize)); if (extra == 0) return 0;