From patchwork Wed Jul 8 10:59:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szabolcs Nagy X-Patchwork-Id: 39959 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 67BB03861843; Wed, 8 Jul 2020 10:59:42 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00072.outbound.protection.outlook.com [40.107.0.72]) by sourceware.org (Postfix) with ESMTPS id 7A4A6386181A for ; Wed, 8 Jul 2020 10:59:37 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 7A4A6386181A Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=Szabolcs.Nagy@arm.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CRsVSFDFj75ZR30f/5MzHGqwX7KpfYu5rbQMKlYY5vs=; b=ODL1fGv3EhVhHRfKRW5idR1ZZV8+x8vTGN1fYCGBHOsLvEVh56B7LOmcWYVgBodJwTLzPhXLV1X4qgvuCm2incitYwvxWptXbgLx8PksvnYC/hkrpnS+MDrNTt5AwJB07HnURTf49fh6CX7LTPePLT3vWbbNkcE3Loktvek3Q0s= Received: from DB3PR06CA0032.eurprd06.prod.outlook.com (2603:10a6:8:1::45) by AM0PR08MB3393.eurprd08.prod.outlook.com (2603:10a6:208:e1::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.20; Wed, 8 Jul 2020 10:59:35 +0000 Received: from DB5EUR03FT007.eop-EUR03.prod.protection.outlook.com (2603:10a6:8:1:cafe::14) by DB3PR06CA0032.outlook.office365.com (2603:10a6:8:1::45) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3174.22 via Frontend Transport; Wed, 8 Jul 2020 10:59:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; sourceware.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com; sourceware.org; dmarc=bestguesspass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by DB5EUR03FT007.mail.protection.outlook.com (10.152.20.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3174.21 via Frontend Transport; Wed, 8 Jul 2020 10:59:35 +0000 Received: ("Tessian outbound 4e683f4039d5:v62"); Wed, 08 Jul 2020 10:59:35 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: f3e6f8581a211388 X-CR-MTA-TID: 64aa7808 Received: from 41aa783479d0.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id DA9BCEFE-B6CE-40BC-A0D7-1408B3938DD0.1; Wed, 08 Jul 2020 10:59:30 +0000 Received: from EUR05-AM6-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 41aa783479d0.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 08 Jul 2020 10:59:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hVMgd4zArgpD1XWlo1D4GPhrsVwXcqMIa7htphwkehjDesNlCdXjn1GUmIfIaw77GAAGIfC6AHIiI2nSbrvH/iEM2E7bRlPGLfpsksTl7yZiY4hfrgLmQg36kNoiS0rVHaAJHZncq6lIQ+j3czBP2MLO61seVFqdJNCI3f67nwUznYpXtbPHcdwXMUeVVZtAe4WjtrR8p1C0zTamSJ3jrpoPNRBjDVg1RDYdHGrv/mR9npKhJEhbHYaAuVXNr3ZCaui/ngP9U4NFmI2Vfr0V7yM3k4/tGUTvvNTYu3MOlFvZaYobrwrTai/Q7xbSvKmdDf0bX+z1sMs16Ijbij+hbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CRsVSFDFj75ZR30f/5MzHGqwX7KpfYu5rbQMKlYY5vs=; b=RRzzNaVG5eetoOCm30pr54BCvm5WNR2rWcM3PVU3C7OeT4m9jcNwCBhKD/RlM4AiZA3LTyaJ7ygY9Q90YNdMzF63mFrBeh4+X8kOP/QrTrrITH1HZz5TmTyVLXXF3o9u9KvjI3fDlZ6xs2Pu5PkdxBCU5FVmEpVFCaYyew2Xmv7ripaKcAc9kFV/r+h4t+WUKt4RYPH/WHUliu/opSOagu6u5LYko8HzFMjxuKlLqHqIsTmvJ6YL22rJuNCMizTv7Arz0/DIXzlYVNic9uCS/Ax1ag2cLvql+u0+os+MSxNZh8/ZKDjsBn65sKiuWDLPhKSWK2UH0TP//xo1MYZ8bQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=armh.onmicrosoft.com; s=selector2-armh-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CRsVSFDFj75ZR30f/5MzHGqwX7KpfYu5rbQMKlYY5vs=; b=ODL1fGv3EhVhHRfKRW5idR1ZZV8+x8vTGN1fYCGBHOsLvEVh56B7LOmcWYVgBodJwTLzPhXLV1X4qgvuCm2incitYwvxWptXbgLx8PksvnYC/hkrpnS+MDrNTt5AwJB07HnURTf49fh6CX7LTPePLT3vWbbNkcE3Loktvek3Q0s= Authentication-Results-Original: sourceware.org; dkim=none (message not signed) header.d=none;sourceware.org; dmarc=none action=none header.from=arm.com; Received: from AM6PR08MB3047.eurprd08.prod.outlook.com (2603:10a6:209:4c::23) by AM6PR08MB4133.eurprd08.prod.outlook.com (2603:10a6:20b:ab::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.27; Wed, 8 Jul 2020 10:59:28 +0000 Received: from AM6PR08MB3047.eurprd08.prod.outlook.com ([fe80::2404:de9f:78c0:313c]) by AM6PR08MB3047.eurprd08.prod.outlook.com ([fe80::2404:de9f:78c0:313c%6]) with mapi id 15.20.3153.031; Wed, 8 Jul 2020 10:59:28 +0000 From: Szabolcs Nagy To: libc-alpha@sourceware.org Subject: [PATCH v6 3/3] rtld: Avoid using up static TLS surplus for optimizations [BZ #25051] Date: Wed, 8 Jul 2020 11:59:22 +0100 Message-Id: X-Mailer: git-send-email 2.17.1 In-Reply-To: References: X-ClientProxiedBy: CWLP123CA0015.GBRP123.PROD.OUTLOOK.COM (2603:10a6:401:56::27) To AM6PR08MB3047.eurprd08.prod.outlook.com (2603:10a6:209:4c::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from localhost.localdomain (217.140.106.53) by CWLP123CA0015.GBRP123.PROD.OUTLOOK.COM (2603:10a6:401:56::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.23 via Frontend Transport; Wed, 8 Jul 2020 10:59:28 +0000 X-Mailer: git-send-email 2.17.1 X-Originating-IP: [217.140.106.53] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: cbf773f1-7a72-4a92-604a-08d8232e007f X-MS-TrafficTypeDiagnostic: AM6PR08MB4133:|AM0PR08MB3393: X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true NoDisclaimer: true X-MS-Oob-TLC-OOBClassifiers: OLM:3383;OLM:3383; X-Forefront-PRVS: 04583CED1A X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: Dp8optK8rYm1w/YVcPYTD6ijiF/sxRmGWh5vLdohKr5EVWP/N0hhouIJ8bJ3OkaAYz+HMFC6if41IZV4I6aV0L6LihZaraQVC4Ujz6T2M2Yskxng1wkFX18OWecFsk+HPEZgG+k/mIGBV+PNkxxRxzyNpORcFKL59MKeMrX7pTL0ausj1p9o96olzFywpSZeg5ffb8ROkNWD5odwvXvgZeEDSAWpGcOydUx3TKWFKj+W4Pwv1DMsQ98yDCsMxOCZwJA5WuqjwlsUcqrnBtmcLBnV+/xKn+7WcE8Lc4Bf0qlKRk7wIZeUMxgq6CAqQdnSQwN/SIVDFuvUvKNgxW0KP5ukAv3PnMhCCTQD6BL6S4N3MLSSbYa2VoW2A6Hmbnn37BaheKp3yyApgEhdLsWuAbhQgttp6rCGBRpIVkJrUT4sw5WkPd28NmRMYeHBUX4rUYp+mXEA2wy/G0APFYEap4LX4zfiVZ0DOhchrmJZrKQ= X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM6PR08MB3047.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(39860400002)(346002)(366004)(136003)(396003)(376002)(36756003)(316002)(6506007)(52116002)(6916009)(86362001)(6512007)(66946007)(66476007)(66556008)(6666004)(83380400001)(6486002)(44832011)(956004)(2616005)(2906002)(69590400007)(26005)(5660300002)(30864003)(16526019)(186003)(8936002)(8676002)(478600001)(966005)(2004002)(136400200001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: +fv3EKQdokyinSPpg92ktfp/CQJiCw+bTrorNiM8FODMU/w58MpG9knbsQJx+cEfnPEu903CtPFHaEWA1coqk93ZOj8fHjr7rKSMCP2j6h/0IpLW5DaH0SMyNSCskLp/po+Gg/SL7wV8CpfhOs6FTDM68xMVx55VQlCKbQStbrPx/g3CBsJkEMt+egTNqFGLFtY7nwZlKtRRKkB9wbYRR9spclFTo93AS4HWgBWaXq2onumsBXHjOH112rhW/xYBCHwYe1dVopvTsYIjAv2Ny/a54CBhIZeF26zDMxTLdLMxyZqRjQ/x9QpZtUf0arrcosZq912rtigCE1b7LIPQEyV5O5WeJOHqPD2ZeIdunsv+H4/ajvuDHod25nZxzp2l9wxtgFgI7ghvamIJ1dq/3NAleK79BROJTvJM8lOu61KPWWAvcDR9xASCucB6jCzM/APqq/szLRpyJvkTaQs0a75OmwXJYqsqeOo0aAfYDu8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4133 Original-Authentication-Results: sourceware.org; dkim=none (message not signed) header.d=none; sourceware.org; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: DB5EUR03FT007.eop-EUR03.prod.protection.outlook.com X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFTY:; SFS:(4636009)(136003)(396003)(39860400002)(346002)(376002)(46966005)(336012)(16526019)(186003)(47076004)(356005)(82310400002)(956004)(2616005)(81166007)(8676002)(316002)(8936002)(6666004)(82740400003)(478600001)(966005)(5660300002)(6512007)(44832011)(30864003)(69590400007)(70206006)(86362001)(6916009)(2906002)(70586007)(6506007)(26005)(36756003)(6486002)(83380400001)(136400200001)(2004002); DIR:OUT; SFP:1101; X-MS-Office365-Filtering-Correlation-Id-Prvs: b9c97f56-61f5-4647-43e0-08d8232dfc29 X-Forefront-PRVS: 04583CED1A X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZI/QHvOXkRerkBoiHYO26t1YfQ2q5KUF61EzrMLBnyficuGCFf0RRlcsV/RfHyo8KflsZ4ggzm/uBeWekPxQvQgxzJfafA4SfB0sdcmfbajWbltG77+wn/bvL5jrDo74NXrWyGSi7tt0qwrI6pE2EQVfIbfpPxJt9XNgFVF7qYm/OkC+Mur91LSBWYo29+iGR4gD0j5ZU59QvjFUKVIU2+Li3BsjnqPvqyjv4QXJQbRMyxF9V7A7a6gV+01CVoxfnSVcmw15HYxGiuyFj3tpw/1kW4Dbn5iOHdtYQTw/6aSDci6Q8qYT//KMOpF1JtuZQ6x7S2qs92sn115/22QfKcJNfGmtF/CYrQflTzlOGKCTzNLsgPVP2eYp4qPuQWJJztTbSpcCLzKPEh+l7o7kO5F77WeM2xS+bii/yP74Wd0smfl3Tom/YwnbifyQ1ektpmbpE+vQOXh1I/Ctd9UKAfE1NMsQdhpcZJ+AFVfjLhCLTJxW2Q7HGtZf5BT2TKrFlLp21X4Ui/AYlNr7lxzriQ== X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jul 2020 10:59:35.6660 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cbf773f1-7a72-4a92-604a-08d8232e007f X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: DB5EUR03FT007.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR08MB3393 X-Spam-Status: No, score=-16.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, MSGID_FROM_MTA_HEADER, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" On some targets static TLS surplus area can be used opportunistically for dynamically loaded modules such that the TLS access then becomes faster (TLSDESC and powerpc TLS optimization). However we don't want all surplus TLS to be used for this optimization because dynamically loaded modules with initial-exec model TLS can only use surplus TLS. The new contract for surplus static TLS use is: - libc.so can have up to 192 bytes of IE TLS, - other system libraries together can have up to 144 bytes of IE TLS. - Some "optional" static TLS is available for opportunistic use. The optional TLS is now tunable: rtld.optional_static_tls, so users can directly affect the allocated static TLS size. (Note that module unloading with dlclose does not reclaim static TLS. After the optional TLS runs out, TLS access is no longer optimized to use static TLS.) The default setting of rtld.optional_static_tls is 512 so the surplus TLS is 3*192 + 4*144 + 512 = 1664 by default, the same as before. Fixes BZ #25051. Tested on aarch64-linux-gnu and x86_64-linux-gnu. Reviewed-by: Carlos O'Donell --- v6: - use FAIL_EXIT1 in tests. - rebase on top of __libc_single_threaded changes. - rebase on top of audit modules accounting. - add reviewed-by. v5: - Split the patch into two: rtld.nns and tls optimization parts. - Use rtld tunable namespace. - Set tunable min val to 0. - New test using dlmopen. - Tests now have check for failing IE TLS dlopen. - Updated the tunable documentation. v4: - Rebased and moved this log out of the commit message. - Minor commit message wording changes. v3: - archived at https://sourceware.org/pipermail/libc-alpha/2020-March/111660.html - Replace TLS_STATIC_SURPLUS with GLRO(dl_tls_static_surplus) and simplify related logic. - In case of static linking, replace GL(dl_tls_static_size) with GLRO(dl_tls_static_surplus) in the code paths before the GL(dl_tls_static_size) value is actually computed. - Update comments and the test code. - Document the new tunables. - Update description, mention static linking. v2: - Add dl.nns tunable. - Add dl.optional_static_tls tunable. - New surplus TLS usage contract that works reliably up to dl.nns namespaces. --- csu/libc-tls.c | 3 + elf/Makefile | 29 +++++++++- elf/dl-reloc.c | 37 +++++++++--- elf/dl-tls.c | 9 +-- elf/dl-tunables.list | 5 ++ elf/dynamic-link.h | 5 +- elf/tst-tls-ie-dlmopen.c | 112 +++++++++++++++++++++++++++++++++++++ elf/tst-tls-ie-mod.h | 40 +++++++++++++ elf/tst-tls-ie-mod0.c | 4 ++ elf/tst-tls-ie-mod1.c | 4 ++ elf/tst-tls-ie-mod2.c | 4 ++ elf/tst-tls-ie-mod3.c | 4 ++ elf/tst-tls-ie-mod4.c | 4 ++ elf/tst-tls-ie-mod5.c | 4 ++ elf/tst-tls-ie-mod6.c | 4 ++ elf/tst-tls-ie.c | 111 ++++++++++++++++++++++++++++++++++++ manual/tunables.texi | 17 ++++++ sysdeps/generic/ldsodefs.h | 3 + 18 files changed, 382 insertions(+), 17 deletions(-) create mode 100644 elf/tst-tls-ie-dlmopen.c create mode 100644 elf/tst-tls-ie-mod.h create mode 100644 elf/tst-tls-ie-mod0.c create mode 100644 elf/tst-tls-ie-mod1.c create mode 100644 elf/tst-tls-ie-mod2.c create mode 100644 elf/tst-tls-ie-mod3.c create mode 100644 elf/tst-tls-ie-mod4.c create mode 100644 elf/tst-tls-ie-mod5.c create mode 100644 elf/tst-tls-ie-mod6.c create mode 100644 elf/tst-tls-ie.c diff --git a/csu/libc-tls.c b/csu/libc-tls.c index a67cc1dd4f..06e76bd395 100644 --- a/csu/libc-tls.c +++ b/csu/libc-tls.c @@ -56,6 +56,9 @@ size_t _dl_tls_static_align; loaded modules with IE-model TLS or for TLSDESC optimization. See comments in elf/dl-tls.c where it is initialized. */ size_t _dl_tls_static_surplus; +/* Remaining amount of static TLS that may be used for optimizing + dynamic TLS access (e.g. with TLSDESC). */ +size_t _dl_tls_static_optional; /* Generation counter for the dtv. */ size_t _dl_tls_generation; diff --git a/elf/Makefile b/elf/Makefile index 81a696c3ef..a2c3b12007 100644 --- a/elf/Makefile +++ b/elf/Makefile @@ -209,7 +209,8 @@ tests += restest1 preloadtest loadfail multiload origtest resolvfail \ tst-dlopenfail-2 \ tst-filterobj tst-filterobj-dlopen tst-auxobj tst-auxobj-dlopen \ tst-audit14 tst-audit15 tst-audit16 \ - tst-single_threaded tst-single_threaded-pthread + tst-single_threaded tst-single_threaded-pthread \ + tst-tls-ie tst-tls-ie-dlmopen # reldep9 tests-internal += loadtest unload unload2 circleload1 \ neededtest neededtest2 neededtest3 neededtest4 \ @@ -324,7 +325,11 @@ modules-names = testobj1 testobj2 testobj3 testobj4 testobj5 testobj6 \ tst-filterobj-flt tst-filterobj-aux tst-filterobj-filtee \ tst-auditlogmod-1 tst-auditlogmod-2 tst-auditlogmod-3 \ tst-single_threaded-mod1 tst-single_threaded-mod2 \ - tst-single_threaded-mod3 tst-single_threaded-mod4 + tst-single_threaded-mod3 tst-single_threaded-mod4 \ + tst-tls-ie-mod0 tst-tls-ie-mod1 tst-tls-ie-mod2 \ + tst-tls-ie-mod3 tst-tls-ie-mod4 tst-tls-ie-mod5 \ + tst-tls-ie-mod6 + # Most modules build with _ISOMAC defined, but those filtered out # depend on internal headers. modules-names-tests = $(filter-out ifuncmod% tst-libc_dlvsym-dso tst-tlsmod%,\ @@ -1769,3 +1774,23 @@ $(objpfx)tst-single_threaded-pthread.out: \ $(objpfx)tst-single_threaded-mod2.so $(objpfx)tst-single_threaded-mod3.so \ $(objpfx)tst-single_threaded-mod4.so $(objpfx)tst-single_threaded-pthread-static: $(static-thread-library) + +$(objpfx)tst-tls-ie: $(libdl) $(shared-thread-library) +$(objpfx)tst-tls-ie.out: \ + $(objpfx)tst-tls-ie-mod0.so \ + $(objpfx)tst-tls-ie-mod1.so \ + $(objpfx)tst-tls-ie-mod2.so \ + $(objpfx)tst-tls-ie-mod3.so \ + $(objpfx)tst-tls-ie-mod4.so \ + $(objpfx)tst-tls-ie-mod5.so \ + $(objpfx)tst-tls-ie-mod6.so + +$(objpfx)tst-tls-ie-dlmopen: $(libdl) $(shared-thread-library) +$(objpfx)tst-tls-ie-dlmopen.out: \ + $(objpfx)tst-tls-ie-mod0.so \ + $(objpfx)tst-tls-ie-mod1.so \ + $(objpfx)tst-tls-ie-mod2.so \ + $(objpfx)tst-tls-ie-mod3.so \ + $(objpfx)tst-tls-ie-mod4.so \ + $(objpfx)tst-tls-ie-mod5.so \ + $(objpfx)tst-tls-ie-mod6.so diff --git a/elf/dl-reloc.c b/elf/dl-reloc.c index ffcc84d396..6d32e49467 100644 --- a/elf/dl-reloc.c +++ b/elf/dl-reloc.c @@ -39,13 +39,16 @@ /* We are trying to perform a static TLS relocation in MAP, but it was dynamically loaded. This can only work if there is enough surplus in the static TLS area already allocated for each running thread. If this - object's TLS segment is too big to fit, we fail. If it fits, - we set MAP->l_tls_offset and return. - This function intentionally does not return any value but signals error - directly, as static TLS should be rare and code handling it should - not be inlined as much as possible. */ + object's TLS segment is too big to fit, we fail with -1. If it fits, + we set MAP->l_tls_offset and return 0. + A portion of the surplus static TLS can be optionally used to optimize + dynamic TLS access (with TLSDESC or powerpc TLS optimizations). + If OPTIONAL is true then TLS is allocated for such optimization and + the caller must have a fallback in case the optional portion of surplus + TLS runs out. If OPTIONAL is false then the entire surplus TLS area is + considered and the allocation only fails if that runs out. */ int -_dl_try_allocate_static_tls (struct link_map *map) +_dl_try_allocate_static_tls (struct link_map *map, bool optional) { /* If we've already used the variable with dynamic access, or if the alignment requirements are too high, fail. */ @@ -68,8 +71,14 @@ _dl_try_allocate_static_tls (struct link_map *map) size_t n = (freebytes - blsize) / map->l_tls_align; - size_t offset = GL(dl_tls_static_used) + (freebytes - n * map->l_tls_align - - map->l_tls_firstbyte_offset); + /* Account optional static TLS surplus usage. */ + size_t use = freebytes - n * map->l_tls_align - map->l_tls_firstbyte_offset; + if (optional && use > GL(dl_tls_static_optional)) + goto fail; + else if (optional) + GL(dl_tls_static_optional) -= use; + + size_t offset = GL(dl_tls_static_used) + use; map->l_tls_offset = GL(dl_tls_static_used) = offset; #elif TLS_DTV_AT_TP @@ -83,6 +92,13 @@ _dl_try_allocate_static_tls (struct link_map *map) if (used > GL(dl_tls_static_size)) goto fail; + /* Account optional static TLS surplus usage. */ + size_t use = used - GL(dl_tls_static_used); + if (optional && use > GL(dl_tls_static_optional)) + goto fail; + else if (optional) + GL(dl_tls_static_optional) -= use; + map->l_tls_offset = offset; map->l_tls_firstbyte_offset = GL(dl_tls_static_used); GL(dl_tls_static_used) = used; @@ -110,12 +126,15 @@ _dl_try_allocate_static_tls (struct link_map *map) return 0; } +/* This function intentionally does not return any value but signals error + directly, as static TLS should be rare and code handling it should + not be inlined as much as possible. */ void __attribute_noinline__ _dl_allocate_static_tls (struct link_map *map) { if (map->l_tls_offset == FORCED_DYNAMIC_TLS_OFFSET - || _dl_try_allocate_static_tls (map)) + || _dl_try_allocate_static_tls (map, false)) { _dl_signal_error (0, map->l_name, NULL, N_("\ cannot allocate memory in static TLS block")); diff --git a/elf/dl-tls.c b/elf/dl-tls.c index 4e7b10edd8..772e70d0f6 100644 --- a/elf/dl-tls.c +++ b/elf/dl-tls.c @@ -60,8 +60,6 @@ This should be large enough to cover runtime libraries of the compiler such as libgomp and libraries in libc other than libc.so. */ #define OTHER_IE_TLS 144 -/* Size of additional surplus TLS, placeholder for TLS optimizations. */ -#define OPT_SURPLUS_TLS 512 /* Calculate the size of the static TLS surplus, when the given number of audit modules are loaded. Must be called after the @@ -69,13 +67,15 @@ void _dl_tls_static_surplus_init (size_t naudit) { - size_t nns; + size_t nns, opt_tls; #if HAVE_TUNABLES nns = TUNABLE_GET (nns, size_t, NULL); + opt_tls = TUNABLE_GET (optional_static_tls, size_t, NULL); #else /* Default values of the tunables. */ nns = 4; + opt_tls = 512; #endif if (nns > DL_NNS) nns = DL_NNS; @@ -84,9 +84,10 @@ _dl_tls_static_surplus_init (size_t naudit) (unsigned long) naudit, (unsigned long) (DL_NNS - nns)); nns += naudit; + GL(dl_tls_static_optional) = opt_tls; GLRO(dl_tls_static_surplus) = ((nns - 1) * LIBC_IE_TLS + nns * OTHER_IE_TLS - + OPT_SURPLUS_TLS); + + opt_tls); } /* Out-of-memory handler. */ diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list index b07742d7b3..35634ef24d 100644 --- a/elf/dl-tunables.list +++ b/elf/dl-tunables.list @@ -134,5 +134,10 @@ glibc { maxval: 16 default: 4 } + optional_static_tls { + type: SIZE_T + minval: 0 + default: 512 + } } } diff --git a/elf/dynamic-link.h b/elf/dynamic-link.h index bb7a66f4cd..6727233e1a 100644 --- a/elf/dynamic-link.h +++ b/elf/dynamic-link.h @@ -40,9 +40,10 @@ (__builtin_expect ((sym_map)->l_tls_offset \ != FORCED_DYNAMIC_TLS_OFFSET, 1) \ && (__builtin_expect ((sym_map)->l_tls_offset != NO_TLS_OFFSET, 1) \ - || _dl_try_allocate_static_tls (sym_map) == 0)) + || _dl_try_allocate_static_tls (sym_map, true) == 0)) -int _dl_try_allocate_static_tls (struct link_map *map) attribute_hidden; +int _dl_try_allocate_static_tls (struct link_map *map, bool optional) + attribute_hidden; #include diff --git a/elf/tst-tls-ie-dlmopen.c b/elf/tst-tls-ie-dlmopen.c new file mode 100644 index 0000000000..c7b5c688e3 --- /dev/null +++ b/elf/tst-tls-ie-dlmopen.c @@ -0,0 +1,112 @@ +/* Test dlopen of modules with initial-exec TLS after dlmopen. + Copyright (C) 2016-2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* This test tries to check that surplus static TLS is not used up for + dynamic TLS optimizations and 4*144 = 576 bytes of static TLS is + still available for dlopening modules with initial-exec TLS after 3 + new dlmopen namespaces are created. It depends on rtld.nns=4 and + rtld.optional_static_tls=512 tunable settings. */ + +#include +#include +#include +#include +#include + +static int do_test (void); +#include +#include +#include +#include + +/* Have some big TLS in the main exe: should not use surplus TLS. */ +__thread char maintls[1000]; + +static pthread_barrier_t barrier; + +/* Forces multi-threaded behaviour. */ +static void * +blocked_thread_func (void *closure) +{ + xpthread_barrier_wait (&barrier); + /* TLS load and access tests run here in the main thread. */ + xpthread_barrier_wait (&barrier); + return NULL; +} + +static void * +load_and_access (Lmid_t lmid, const char *mod, const char *func) +{ + /* Load module with TLS. */ + void *p = xdlmopen (lmid, mod, RTLD_NOW); + /* Access the TLS variable to ensure it is allocated. */ + void (*f) (void) = (void (*) (void))xdlsym (p, func); + f (); + return p; +} + +static int +do_test (void) +{ + void *mods[5]; + + { + int ret = pthread_barrier_init (&barrier, NULL, 2); + if (ret != 0) + { + errno = ret; + printf ("error: pthread_barrier_init: %m\n"); + exit (1); + } + } + + pthread_t blocked_thread = xpthread_create (NULL, blocked_thread_func, NULL); + xpthread_barrier_wait (&barrier); + + printf ("maintls[%zu]:\t %p .. %p\n", + sizeof maintls, maintls, maintls + sizeof maintls); + memset (maintls, 1, sizeof maintls); + + /* Load modules with dynamic TLS (use surplus static TLS for libc + in new namespaces and may be for TLS optimizations too). */ + mods[0] = load_and_access (LM_ID_BASE, "tst-tls-ie-mod0.so", "access0"); + mods[1] = load_and_access (LM_ID_NEWLM, "tst-tls-ie-mod1.so", "access1"); + mods[2] = load_and_access (LM_ID_NEWLM, "tst-tls-ie-mod2.so", "access2"); + mods[3] = load_and_access (LM_ID_NEWLM, "tst-tls-ie-mod3.so", "access3"); + /* Load modules with initial-exec TLS (can only use surplus static TLS). */ + mods[4] = load_and_access (LM_ID_BASE, "tst-tls-ie-mod6.so", "access6"); + + /* Here 576 bytes + 3 * libc use of surplus static TLS is in use so less + than 1024 bytes are available (exact number depends on TLS optimizations + and the libc TLS use). */ + printf ("The next dlmopen should fail...\n"); + void *p = dlmopen (LM_ID_BASE, "tst-tls-ie-mod4.so", RTLD_NOW); + if (p != NULL) + FAIL_EXIT1 ("error: expected dlmopen to fail because there is " + "not enough surplus static TLS.\n"); + printf ("...OK failed with: %s.\n", dlerror ()); + + xpthread_barrier_wait (&barrier); + xpthread_join (blocked_thread); + + /* Close the modules. */ + for (int i = 0; i < 5; ++i) + xdlclose (mods[i]); + + return 0; +} diff --git a/elf/tst-tls-ie-mod.h b/elf/tst-tls-ie-mod.h new file mode 100644 index 0000000000..46b362a9b7 --- /dev/null +++ b/elf/tst-tls-ie-mod.h @@ -0,0 +1,40 @@ +/* Module with specified TLS size and model. + Copyright (C) 2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* This file is parameterized by macros N, SIZE and MODEL. */ + +#include +#include + +#define CONCATX(x, y) x ## y +#define CONCAT(x, y) CONCATX (x, y) +#define STRX(x) #x +#define STR(x) STRX (x) + +#define VAR CONCAT (var, N) + +__attribute__ ((aligned (8), tls_model (MODEL))) +__thread char VAR[SIZE]; + +void +CONCAT (access, N) (void) +{ + printf (STR (VAR) "[%d]:\t %p .. %p " MODEL "\n", SIZE, VAR, VAR + SIZE); + fflush (stdout); + memset (VAR, 1, SIZE); +} diff --git a/elf/tst-tls-ie-mod0.c b/elf/tst-tls-ie-mod0.c new file mode 100644 index 0000000000..2450686e40 --- /dev/null +++ b/elf/tst-tls-ie-mod0.c @@ -0,0 +1,4 @@ +#define N 0 +#define SIZE 480 +#define MODEL "global-dynamic" +#include "tst-tls-ie-mod.h" diff --git a/elf/tst-tls-ie-mod1.c b/elf/tst-tls-ie-mod1.c new file mode 100644 index 0000000000..849ff91e53 --- /dev/null +++ b/elf/tst-tls-ie-mod1.c @@ -0,0 +1,4 @@ +#define N 1 +#define SIZE 120 +#define MODEL "global-dynamic" +#include "tst-tls-ie-mod.h" diff --git a/elf/tst-tls-ie-mod2.c b/elf/tst-tls-ie-mod2.c new file mode 100644 index 0000000000..23915ab67b --- /dev/null +++ b/elf/tst-tls-ie-mod2.c @@ -0,0 +1,4 @@ +#define N 2 +#define SIZE 24 +#define MODEL "global-dynamic" +#include "tst-tls-ie-mod.h" diff --git a/elf/tst-tls-ie-mod3.c b/elf/tst-tls-ie-mod3.c new file mode 100644 index 0000000000..5395f844a5 --- /dev/null +++ b/elf/tst-tls-ie-mod3.c @@ -0,0 +1,4 @@ +#define N 3 +#define SIZE 16 +#define MODEL "global-dynamic" +#include "tst-tls-ie-mod.h" diff --git a/elf/tst-tls-ie-mod4.c b/elf/tst-tls-ie-mod4.c new file mode 100644 index 0000000000..93ac2eacae --- /dev/null +++ b/elf/tst-tls-ie-mod4.c @@ -0,0 +1,4 @@ +#define N 4 +#define SIZE 1024 +#define MODEL "initial-exec" +#include "tst-tls-ie-mod.h" diff --git a/elf/tst-tls-ie-mod5.c b/elf/tst-tls-ie-mod5.c new file mode 100644 index 0000000000..84b3fd285b --- /dev/null +++ b/elf/tst-tls-ie-mod5.c @@ -0,0 +1,4 @@ +#define N 5 +#define SIZE 128 +#define MODEL "initial-exec" +#include "tst-tls-ie-mod.h" diff --git a/elf/tst-tls-ie-mod6.c b/elf/tst-tls-ie-mod6.c new file mode 100644 index 0000000000..c736bf0684 --- /dev/null +++ b/elf/tst-tls-ie-mod6.c @@ -0,0 +1,4 @@ +#define N 6 +#define SIZE 576 +#define MODEL "initial-exec" +#include "tst-tls-ie-mod.h" diff --git a/elf/tst-tls-ie.c b/elf/tst-tls-ie.c new file mode 100644 index 0000000000..2dc0894480 --- /dev/null +++ b/elf/tst-tls-ie.c @@ -0,0 +1,111 @@ +/* Test dlopen of modules with initial-exec TLS. + Copyright (C) 2016-2020 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* This test tries to check that surplus static TLS is not used up for + dynamic TLS optimizations and 3*192 + 4*144 = 1152 bytes of static + TLS is available for dlopening modules with initial-exec TLS. It + depends on rtld.nns=4 and rtld.optional_static_tls=512 tunable setting. */ + +#include +#include +#include +#include +#include + +static int do_test (void); +#include +#include +#include +#include + +/* Have some big TLS in the main exe: should not use surplus TLS. */ +__thread char maintls[1000]; + +static pthread_barrier_t barrier; + +/* Forces multi-threaded behaviour. */ +static void * +blocked_thread_func (void *closure) +{ + xpthread_barrier_wait (&barrier); + /* TLS load and access tests run here in the main thread. */ + xpthread_barrier_wait (&barrier); + return NULL; +} + +static void * +load_and_access (const char *mod, const char *func) +{ + /* Load module with TLS. */ + void *p = xdlopen (mod, RTLD_NOW); + /* Access the TLS variable to ensure it is allocated. */ + void (*f) (void) = (void (*) (void))xdlsym (p, func); + f (); + return p; +} + +static int +do_test (void) +{ + void *mods[6]; + + { + int ret = pthread_barrier_init (&barrier, NULL, 2); + if (ret != 0) + { + errno = ret; + printf ("error: pthread_barrier_init: %m\n"); + exit (1); + } + } + + pthread_t blocked_thread = xpthread_create (NULL, blocked_thread_func, NULL); + xpthread_barrier_wait (&barrier); + + printf ("maintls[%zu]:\t %p .. %p\n", + sizeof maintls, maintls, maintls + sizeof maintls); + memset (maintls, 1, sizeof maintls); + + /* Load modules with dynamic TLS (may use surplus static TLS + opportunistically). */ + mods[0] = load_and_access ("tst-tls-ie-mod0.so", "access0"); + mods[1] = load_and_access ("tst-tls-ie-mod1.so", "access1"); + mods[2] = load_and_access ("tst-tls-ie-mod2.so", "access2"); + mods[3] = load_and_access ("tst-tls-ie-mod3.so", "access3"); + /* Load modules with initial-exec TLS (can only use surplus static TLS). */ + mods[4] = load_and_access ("tst-tls-ie-mod4.so", "access4"); + mods[5] = load_and_access ("tst-tls-ie-mod5.so", "access5"); + + /* Here 1152 bytes of surplus static TLS is in use and at most 512 bytes + are available (depending on TLS optimizations). */ + printf ("The next dlopen should fail...\n"); + void *p = dlopen ("tst-tls-ie-mod6.so", RTLD_NOW); + if (p != NULL) + FAIL_EXIT1 ("error: expected dlopen to fail because there is " + "not enough surplus static TLS.\n"); + printf ("...OK failed with: %s.\n", dlerror ()); + + xpthread_barrier_wait (&barrier); + xpthread_join (blocked_thread); + + /* Close the modules. */ + for (int i = 0; i < 6; ++i) + xdlclose (mods[i]); + + return 0; +} diff --git a/manual/tunables.texi b/manual/tunables.texi index d18e141db7..089cd30c43 100644 --- a/manual/tunables.texi +++ b/manual/tunables.texi @@ -247,6 +247,23 @@ e.g. LD_AUDIT, or will use more than 4 dynamic link namespaces as created by @code{dlmopen} with an lmid argument of @code{LM_ID_NEWLM}. @end deftp +@deftp Tunable glibc.rtld.optional_static_tls +Sets the amount of surplus static TLS in bytes to allocate at program +startup. Every thread created allocates this amount of specified surplus +static TLS. This is a minimum value and additional space may be allocated +for internal purposes including alignment. Optional static TLS is used for +optimizing dynamic TLS access for platforms that support such optimizations +e.g. TLS descriptors or optimized TLS access for POWER (@code{DT_PPC64_OPT} +and @code{DT_PPC_OPT}). In order to make the best use of such optimizations +the value should be as many bytes as would be required to hold all TLS +variables in all dynamic loaded shared libraries. The value cannot be known +by the dynamic loader because it doesn't know the expected set of shared +libraries which will be loaded. The existing static TLS space cannot be +changed once allocated at process startup. The default allocation of +optional static TLS is 512 bytes and is allocated in every thread. +@end deftp + + @node Elision Tunables @section Elision Tunables @cindex elision tunables diff --git a/sysdeps/generic/ldsodefs.h b/sysdeps/generic/ldsodefs.h index 64b4552653..171d325f55 100644 --- a/sysdeps/generic/ldsodefs.h +++ b/sysdeps/generic/ldsodefs.h @@ -442,6 +442,9 @@ struct rtld_global EXTERN size_t _dl_tls_static_used; /* Alignment requirement of the static TLS block. */ EXTERN size_t _dl_tls_static_align; + /* Remaining amount of static TLS that may be used for optimizing + dynamic TLS access (e.g. with TLSDESC). */ + EXTERN size_t _dl_tls_static_optional; /* Number of additional entries in the slotinfo array of each slotinfo list element. A large number makes it almost certain take we never