From patchwork Fri Aug 13 21:04:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 44663 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 365833851C14 for ; Fri, 13 Aug 2021 21:05:54 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 365833851C14 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1628888754; bh=fLi5CLlxkvnLcH7KfYOhUCZf4O5gpRbn9DcBfGpMQIQ=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=Bcq+FKl8byVAPmvubfGmUNHjJT7o9xhJ0P7nnwwF5xbXVU7NdhzVpZlF8uavpjprr BDLPrveEbi1zfU8j/5YqIoFUs95eFfiQ101SFFSpOtQK3/FCHhYTE9emV47JRz4w8z DqsUfMSnxAOp24LJ0+MzIgkYe56luIg3nvdt/4xw= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by sourceware.org (Postfix) with ESMTPS id A6897386184A for ; Fri, 13 Aug 2021 21:04:38 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org A6897386184A Received: by mail-pj1-x102b.google.com with SMTP id u13-20020a17090abb0db0290177e1d9b3f7so22524124pjr.1 for ; Fri, 13 Aug 2021 14:04:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fLi5CLlxkvnLcH7KfYOhUCZf4O5gpRbn9DcBfGpMQIQ=; b=Y8ttn0UVIuVyiIBfEZ0br48yVfAk9trTmOS5ZNs3MgD9cCESIQs06vo+sJ6KG1JvnG 0SiVqtMJpIG69dFD5xce8DpVOoY6kkae1LoFEZW5m5V/aU82IxEomZHtHlrQtsReFQ5H hu/4La6H9pM8bsld7xHhR3yk6HRsOFhPST+ggtdP3d3/+ARTRF73KQ+T3quVjVA5dlrz 2HYSjTDPBTpNzbCL8vvdVI6GS3M5VgsMXCV1osgR52MNzN2c7nFrxcz0GvN3FbYY+H5e r/q/ja8Hw3WCCJLmhVykKk+NSDgJS2aD73rF7R6JRkGz3Tv0hNq2GIvBuB6JiHWvM7/D F0Vw== X-Gm-Message-State: AOAM531rDf7lLT/EIO7mnvW7Cu8zK/dsht4g0OCsg8QFJKsszZ0B1fr3 qJXjMSrLSDp+Gh3ZvoZVLgg14Zy7Ee5KEA== X-Google-Smtp-Source: ABdhPJyKWmEFpSf3incQfYjNe9ol04wATnNePEKOClQojF6sZ6vkqJQjXI/3h0LPbbZ3agEeNiIOBQ== X-Received: by 2002:a17:902:c950:b029:12d:2ada:9ef7 with SMTP id i16-20020a170902c950b029012d2ada9ef7mr3525259pla.61.1628888677639; Fri, 13 Aug 2021 14:04:37 -0700 (PDT) Received: from birita.. ([2804:431:c7cb:9dce:7cf8:4c28:b9e7:2767]) by smtp.gmail.com with ESMTPSA id v15sm3259462pff.105.2021.08.13.14.04.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Aug 2021 14:04:37 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH 2/3] malloc: Add THP/madvise support for sbrk Date: Fri, 13 Aug 2021 18:04:28 -0300 Message-Id: <20210813210429.1147112-3-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210813210429.1147112-1-adhemerval.zanella@linaro.org> References: <20210813210429.1147112-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-13.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=unavailable autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Siddhesh Poyarekar , Guillaume Morin Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" For the main arena, the sbrk() might the preferable syscall instead of mmap(). And the granularity used when increasing the program segment is the default page size. To increase effectiveness with Transparent Huge Page with madvise, the large page size is use instead. This is enabled with the new tunable 'glibc.malloc.thp_pagesize'. Checked on x86_64-linux-gnu. --- malloc/malloc.c | 40 +++++++++++++++++++++++++++++++++++----- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index 52ea84a63d..7cd586c866 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -2028,6 +2028,38 @@ sysmadvise_thp (void *p, INTERNAL_SIZE_T size) #endif } +static inline long int +thp_brk_align_up (long int size) +{ + INTERNAL_SIZE_T r = size; +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + /* Defined in brk.c. */ + extern void *__curbrk; + if (mp_.thp_pagesize != 0) + { + uintptr_t top = ALIGN_UP ((uintptr_t)__curbrk + size, mp_.thp_pagesize); + r = top - (uintptr_t)__curbrk; + } + else +#endif + r = ALIGN_UP (size, GLRO(dl_pagesize)); + return r; +} + +static inline long +thp_brk_align_down (long int top) +{ + long r; +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + if (mp_.thp_pagesize != 0) + r = ALIGN_DOWN (top, mp_.thp_pagesize); + else +#endif + r = ALIGN_DOWN (top, GLRO(dl_pagesize)); + return r; +} + + /* ------------------- Support for multiple arenas -------------------- */ #include "arena.c" @@ -2610,14 +2642,14 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) size -= old_size; /* - Round to a multiple of page size. + Round to a multiple of page size or huge page size. If MORECORE is not contiguous, this ensures that we only call it with whole-page arguments. And if MORECORE is contiguous and this is not first time through, this preserves page-alignment of previous calls. Otherwise, we correct to page-align below. */ - size = ALIGN_UP (size, pagesize); + size = thp_brk_align_up (size); /* Don't try to call MORECORE if argument is so big as to appear @@ -2900,10 +2932,8 @@ systrim (size_t pad, mstate av) long released; /* Amount actually released */ char *current_brk; /* address returned by pre-check sbrk call */ char *new_brk; /* address returned by post-check sbrk call */ - size_t pagesize; long top_area; - pagesize = GLRO (dl_pagesize); top_size = chunksize (av->top); top_area = top_size - MINSIZE - 1; @@ -2911,7 +2941,7 @@ systrim (size_t pad, mstate av) return 0; /* Release in pagesize units and round down to the nearest page. */ - extra = ALIGN_DOWN(top_area - pad, pagesize); + extra = thp_brk_align_down (top_area - pad); if (extra == 0) return 0;