From patchwork Tue Dec 14 18:58:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adhemerval Zanella Netto X-Patchwork-Id: 48908 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id EA6F5385BF80 for ; Tue, 14 Dec 2021 18:59:15 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org EA6F5385BF80 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1639508356; bh=ymiHOMoPVRruvm0h4tWF5KK8rWiDyguNR+GTcDV5xdI=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=yfh818br2zlifvq/xmFVcE1iTx9X07Fvsk5t7h1p+HaTzAcNkakphq+Koe0DoHzZC 1vsR96MGx0vrGGEo/5s3S0wbVjHA6MyvfxpHscuN1oiuyM9l5dyt6uW0AUZ9RkKPk5 ojibXRs7f8PqlxXLVqssGp66LsoCp0/d5YXt3P/E= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by sourceware.org (Postfix) with ESMTPS id 835EB385AC37 for ; Tue, 14 Dec 2021 18:58:14 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 835EB385AC37 Received: by mail-qk1-x731.google.com with SMTP id l25so8407490qkl.5 for ; Tue, 14 Dec 2021 10:58:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ymiHOMoPVRruvm0h4tWF5KK8rWiDyguNR+GTcDV5xdI=; b=JaY/BQkxHORIxsX4lSMdVNkhf/ECmeLoh+BagU3TbadxE9c6nvHkDkFrVcTWUxjjnE eL2CGFOhDAShZJ6pB8dIFFRXdIgLO1iqL2bL5/YuOQBZ/pDjy5xuIKQVVQ8EmHsPUjXP ZF9gpiyy8fWsRu0yUxx0rRRSbNGFOODj1o/JOPx5I5RsmWzPBmU+ryrgt2EgLMsc+LqW u0pYBElrt0RF9m6pvONrGDwoew5kbbah60IIWKTWr6Up8XxRrU/32f+xVXE6rAj2Z8Ci rNXQF5P1Cmp8IXnptqxxN/b7PmY5LovPQFB2eZMF1FP6t+AwS+2jz2T1NEICYio89xtK DfGw== X-Gm-Message-State: AOAM532/eR9izCMGcc8iGOHcyutILWbCjZHb5vVrj3+5eV4mfJJ30PHa Yw0Nj+eCDDRCR6f2IBxq86Hh5ZbNyIPx8Q== X-Google-Smtp-Source: ABdhPJwBYp1FBkYITu2eQeDzoTccVn4MZs4C0SjJjoNRe/cuYibgcLqnHOlMgaX4b5YlYfFNaHS7nA== X-Received: by 2002:a05:620a:45a3:: with SMTP id bp35mr5620624qkb.512.1639508293899; Tue, 14 Dec 2021 10:58:13 -0800 (PST) Received: from birita.. ([2804:431:c7ca:103f:1000:c46d:a2d6:9bed]) by smtp.gmail.com with ESMTPSA id j124sm422848qkd.98.2021.12.14.10.58.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Dec 2021 10:58:13 -0800 (PST) To: libc-alpha@sourceware.org, Siddhesh Poyarekar Subject: [PATCH v5 2/7] malloc: Add THP/madvise support for sbrk Date: Tue, 14 Dec 2021 15:58:01 -0300 Message-Id: <20211214185806.4109231-3-adhemerval.zanella@linaro.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211214185806.4109231-1-adhemerval.zanella@linaro.org> References: <20211214185806.4109231-1-adhemerval.zanella@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Adhemerval Zanella via Libc-alpha From: Adhemerval Zanella Netto Reply-To: Adhemerval Zanella Cc: Norbert Manthey , Guillaume Morin Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" To increase effectiveness with Transparent Huge Page with madvise, the large page size is use instead page size for sbrk increment for the main arena. Checked on x86_64-linux-gnu. Reviewed-by: DJ Delorie --- include/libc-pointer-arith.h | 8 ++++++++ malloc/malloc.c | 34 +++++++++++++++++++++++++++++----- 2 files changed, 37 insertions(+), 5 deletions(-) diff --git a/include/libc-pointer-arith.h b/include/libc-pointer-arith.h index 04ba537617..55dccc10ac 100644 --- a/include/libc-pointer-arith.h +++ b/include/libc-pointer-arith.h @@ -60,4 +60,12 @@ #define PTR_ALIGN_UP(base, size) \ ((__typeof__ (base)) ALIGN_UP ((uintptr_t) (base), (size))) +/* Check if BASE is aligned on SIZE */ +#define PTR_IS_ALIGNED(base, size) \ + ((((uintptr_t) (base)) & (size - 1)) == 0) + +/* Returns the ptrdiff_t diference between P1 and P2. */ +#define PTR_DIFF(p1, p2) \ + ((ptrdiff_t)((uintptr_t)(p1) - (uintptr_t)(p2))) + #endif diff --git a/malloc/malloc.c b/malloc/malloc.c index b8103aaf10..8a66012503 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -2023,6 +2023,16 @@ madvise_thp (void *p, INTERNAL_SIZE_T size) not active. */ if (mp_.thp_pagesize == 0 || size < mp_.thp_pagesize) return; + + /* Linux requires the input address to be page-aligned, and unaligned + inputs happens only for initial data segment. */ + if (__glibc_unlikely (!PTR_IS_ALIGNED (p, GLRO (dl_pagesize)))) + { + void *q = PTR_ALIGN_DOWN (p, GLRO (dl_pagesize)); + size += PTR_DIFF (p, q); + p = q; + } + __madvise (p, size, MADV_HUGEPAGE); #endif } @@ -2609,14 +2619,25 @@ sysmalloc (INTERNAL_SIZE_T nb, mstate av) size -= old_size; /* - Round to a multiple of page size. + Round to a multiple of page size or huge page size. If MORECORE is not contiguous, this ensures that we only call it with whole-page arguments. And if MORECORE is contiguous and this is not first time through, this preserves page-alignment of previous calls. Otherwise, we correct to page-align below. */ - size = ALIGN_UP (size, pagesize); +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + /* Defined in brk.c. */ + extern void *__curbrk; + if (__glibc_unlikely (mp_.thp_pagesize != 0)) + { + uintptr_t top = ALIGN_UP ((uintptr_t) __curbrk + size, + mp_.thp_pagesize); + size = top - (uintptr_t) __curbrk; + } + else +#endif + size = ALIGN_UP (size, GLRO(dl_pagesize)); /* Don't try to call MORECORE if argument is so big as to appear @@ -2899,10 +2920,8 @@ systrim (size_t pad, mstate av) long released; /* Amount actually released */ char *current_brk; /* address returned by pre-check sbrk call */ char *new_brk; /* address returned by post-check sbrk call */ - size_t pagesize; long top_area; - pagesize = GLRO (dl_pagesize); top_size = chunksize (av->top); top_area = top_size - MINSIZE - 1; @@ -2910,7 +2929,12 @@ systrim (size_t pad, mstate av) return 0; /* Release in pagesize units and round down to the nearest page. */ - extra = ALIGN_DOWN(top_area - pad, pagesize); +#if HAVE_TUNABLES && defined (MADV_HUGEPAGE) + if (__glibc_unlikely (mp_.thp_pagesize != 0)) + extra = ALIGN_DOWN (top_area - pad, mp_.thp_pagesize); + else +#endif + extra = ALIGN_DOWN (top_area - pad, GLRO(dl_pagesize)); if (extra == 0) return 0;