From patchwork Wed Aug 18 14:19:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella X-Patchwork-Id: 44691 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A36553890421 for ; Wed, 18 Aug 2021 14:21:22 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A36553890421 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1629296482; bh=jcM+C6JZQ1A3uo9l1MlqaP46WTLcRHVMqwYvbBGqhLU=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=xB1BvSkI+Tmj18Hs6kFhQoOC6An/47pxGIPEMmNR0tURGIraoYJoAa5feEBzs9QUQ PVWsR/ABYps77IfKBUPFVnTh4iLjND6uO030zEXpv9eQiSdpjvFNyD1lRIJ8t15b0q pYsvhT4uqrAlkB6uELg8UFH1NAIRBOPSgTgUSuvU= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) by sourceware.org (Postfix) with ESMTPS id 3DE083986430 for ; Wed, 18 Aug 2021 14:20:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 3DE083986430 Received: by mail-pf1-x434.google.com with SMTP id y11so2283514pfl.13 for ; Wed, 18 Aug 2021 07:20:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jcM+C6JZQ1A3uo9l1MlqaP46WTLcRHVMqwYvbBGqhLU=; b=IGIcNskkstBe6WfVfZZYFxauG3nnehxk5/n6crtu4kF3GDm7hTe69FJZyvRGqWLast sqeTCngxMcFmZAlJgKwI2Zx4RKdT2RsMQ2p9Upmijefnz7V6SwfSrHSyfr79FgSFqSWS a/VboHkXiWCZ5Z7xe7/pzMpFqtWiosSbZ6J5iezoYYaL1y0WOul7k3MhBf/9QTRb7tFZ 23pcDjnyvhub6oD+yTfHi6X6wniHIgSqYyskP/9BwkTUT6ZdMd1MgsCeslqmiSx4M9Y1 BKNTzirWOUXBTI8Axu8QhsR9PZEee7YRVmaZ5IUJqkMGyryXkzepRT22c+WmLlbMjyv0 vXrQ== X-Gm-Message-State: AOAM533nP9jf7lr0hQC+wiYDEEYflsBFH0Z/7vAFwNc/O8H5wrqp8RwN plX1eIk4ifXcrD+awykwTZLH2wKLynt7AQ== X-Google-Smtp-Source: ABdhPJzitzoOgboIaNHJhlG3s7KdM4HTe1QQ8E6Xb6VFFV9u6+Z+kKUJDPJSwydFzWFtZXOorAR3Tg== X-Received: by 2002:aa7:9056:0:b0:3e3:332:c8b7 with SMTP id n22-20020aa79056000000b003e30332c8b7mr1593898pfo.69.1629296407955; Wed, 18 Aug 2021 07:20:07 -0700 (PDT) Received: from birita.. ([2804:431:c7ca:cd83:8c0a:d250:6dae:d807]) by smtp.gmail.com with ESMTPSA id c133sm6805015pfb.39.2021.08.18.07.20.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Aug 2021 07:20:07 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH v2 2/4] malloc: Add THP/madvise support for sbrk Date: Wed, 18 Aug 2021 11:19:58 -0300 Message-Id: <20210818142000.128752-3-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210818142000.128752-1-adhemerval.zanella@linaro.org> References: <20210818142000.128752-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=unavailable autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Guillaume Morin , Siddhesh Poyarekar Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" For the main arena, the sbrk() might the preferable syscall instead of mmap(). And the granularity used when increasing the program segment is the default page size. To increase effectiveness with Transparent Huge Page with madvise, the large page size is use instead. This is enabled with the new tunable 'glibc.malloc.thp_pagesize'. Checked on x86_64-linux-gnu. --- include/libc-pointer-arith.h | 10 ++++++++++ malloc/malloc.c | 35 ++++++++++++++++++++++++++++++----- 2 files changed, 40 insertions(+), 5 deletions(-) diff --git a/include/libc-pointer-arith.h b/include/libc-pointer-arith.h index 04ba537617..f592cbafec 100644 --- a/include/libc-pointer-arith.h +++ b/include/libc-pointer-arith.h @@ -37,6 +37,16 @@ /* Cast an integer or a pointer VAL to integer with proper type. */ # define cast_to_integer(val) ((__integer_if_pointer_type (val)) (val)) +/* Check if SIZE is aligned on SIZE */ +#define IS_ALIGNED(base, size) \ + (((base) & (size - 1)) == 0) + +#define PTR_IS_ALIGNED(base, size) \ + ((((uintptr_t) (base)) & (size - 1)) == 0) + +#define PTR_DIFF(p1, p2) \ + ((ptrdiff_t)((uintptr_t)(p1) - (uintptr_t)(p2))) + /* Cast an integer VAL to void * pointer. */ # define cast_to_pointer(val) ((void *) (uintptr_t) (val)) diff --git a/malloc/malloc.c b/malloc/malloc.c index ad3eec41ac..1a2c798a35 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -2024,6 +2024,17 @@ sysmadvise_thp (void *p, INTERNAL_SIZE_T size) not active. */ if (mp_.thp_pagesize == 0 || size < mp_.thp_pagesize) return; + + /* madvise() requires at least the input to be aligned to system page and + MADV_HUGEPAGE should handle unaligned address. Also unaligned inputs + should happen only for the initial data segment. */ + if (__glibc_unlikely (!PTR_IS_ALIGNED (p, GLRO (dl_pagesize)))) + { + void *q = PTR_ALIGN_DOWN (p, GLRO (dl_pagesize)); + size += PTR_DIFF (p, q); + p = q; + } + __madvise (p, size, MADV_HUGEPAGE); #endif } @@ -2610,14 +2621,25 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) size -= old_size; /* - Round to a multiple of page size. + Round to a multiple of page size or huge page size. If MORECORE is not contiguous, this ensures that we only call it with whole-page arguments. And if MORECORE is contiguous and this is not first time through, this preserves page-alignment of previous calls. Otherwise, we correct to page-align below. */ - size = ALIGN_UP (size, pagesize); +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + /* Defined in brk.c. */ + extern void *__curbrk; + if (mp_.thp_pagesize != 0) + { + uintptr_t top = ALIGN_UP ((uintptr_t) __curbrk + size, + mp_.thp_pagesize); + size = top - (uintptr_t) __curbrk; + } + else +#endif + size = ALIGN_UP (size, GLRO(dl_pagesize)); /* Don't try to call MORECORE if argument is so big as to appear @@ -2900,10 +2922,8 @@ systrim (size_t pad, mstate av) long released; /* Amount actually released */ char *current_brk; /* address returned by pre-check sbrk call */ char *new_brk; /* address returned by post-check sbrk call */ - size_t pagesize; long top_area; - pagesize = GLRO (dl_pagesize); top_size = chunksize (av->top); top_area = top_size - MINSIZE - 1; @@ -2911,7 +2931,12 @@ systrim (size_t pad, mstate av) return 0; /* Release in pagesize units and round down to the nearest page. */ - extra = ALIGN_DOWN(top_area - pad, pagesize); +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + if (mp_.thp_pagesize != 0) + extra = ALIGN_DOWN (top_area - pad, mp_.thp_pagesize); + else +#endif + extra = ALIGN_DOWN (top_area - pad, GLRO(dl_pagesize)); if (extra == 0) return 0;