From patchwork Tue May 4 16:31:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Florian Weimer X-Patchwork-Id: 43257 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id AB8E4398B89F; Tue, 4 May 2021 16:31:48 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org AB8E4398B89F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1620145908; bh=2WAvbXz0vbCOHWdjA1Ko+qz5497Cj/p261dqi5TC78c=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=g1uw1sN9nB75BBbX9j+9117LIpyJTasvvHMQtMP82OOH0eTfGfvtn6UtB1FYrYyy5 5Tp6vXabQRSCSDTxs/vwCPXAhRB6Cg/hfQAC42HdWR00uKaInLoBCNc6R3yU8CC4mY qlTNNC5FPHIswoiPzyPDGm9s3riqx+Pl+32Biths= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by sourceware.org (Postfix) with ESMTP id E46C6394CC23 for ; Tue, 4 May 2021 16:31:44 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org E46C6394CC23 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-83-aV0u_hNlPHqWDRYUwMtDZQ-1; Tue, 04 May 2021 12:31:42 -0400 X-MC-Unique: aV0u_hNlPHqWDRYUwMtDZQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C6F2A1926DA1 for ; Tue, 4 May 2021 16:31:41 +0000 (UTC) Received: from oldenburg.str.redhat.com (ovpn-112-137.ams2.redhat.com [10.36.112.137]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EDC945D9C0 for ; Tue, 4 May 2021 16:31:40 +0000 (UTC) To: libc-alpha@sourceware.org Subject: [PATCH] elf, nptl: Initialize static TLS directly in ld.so Date: Tue, 04 May 2021 18:31:55 +0200 Message-ID: <87czu6ihok.fsf@oldenburg.str.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-12.8 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Florian Weimer via Libc-alpha From: Florian Weimer Reply-To: Florian Weimer Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" The stack list is available in ld.so since commit 1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move stack list variables into _rtld_global"), so it's possible to walk the stack list directly in ld.so and perform the initialization there. This eliminates an unprotected function pointer from _rtld_global and reduces the libpthread initialization code. --- elf/dl-open.c | 2 +- elf/dl-reloc.c | 5 +++-- elf/dl-support.c | 3 +-- elf/dl-tls.c | 39 +++++++++++++++++++++++++++++++++++++++ elf/rtld.c | 2 ++ nptl/allocatestack.c | 35 ----------------------------------- nptl/nptl-init.c | 2 -- nptl/pthreadP.h | 2 -- sysdeps/generic/ldsodefs.h | 19 +++++++++++++++++++ 9 files changed, 65 insertions(+), 44 deletions(-) diff --git a/elf/dl-open.c b/elf/dl-open.c index ab7aaa345e..09f0df7d38 100644 --- a/elf/dl-open.c +++ b/elf/dl-open.c @@ -426,7 +426,7 @@ TLS generation counter wrapped! Please report this.")); _dl_update_slotinfo (imap->l_tls_modid); #endif - GL(dl_init_static_tls) (imap); + dl_init_static_tls (imap); assert (imap->l_need_tls_init == 0); } } diff --git a/elf/dl-reloc.c b/elf/dl-reloc.c index c2df26deea..bb9ca1a101 100644 --- a/elf/dl-reloc.c +++ b/elf/dl-reloc.c @@ -118,7 +118,7 @@ _dl_try_allocate_static_tls (struct link_map *map, bool optional) (void) _dl_update_slotinfo (map->l_tls_modid); #endif - GL(dl_init_static_tls) (map); + dl_init_static_tls (map); } else map->l_need_tls_init = 1; @@ -141,6 +141,7 @@ cannot allocate memory in static TLS block")); } } +#if !THREAD_GSCOPE_IN_TCB /* Initialize static TLS area and DTV for current (only) thread. libpthread implementations should provide their own hook to handle all threads. */ @@ -159,7 +160,7 @@ _dl_nothread_init_static_tls (struct link_map *map) memset (__mempcpy (dest, map->l_tls_initimage, map->l_tls_initimage_size), '\0', map->l_tls_blocksize - map->l_tls_initimage_size); } - +#endif /* !THREAD_GSCOPE_IN_TCB */ void _dl_relocate_object (struct link_map *l, struct r_scope_elem *scope[], diff --git a/elf/dl-support.c b/elf/dl-support.c index 7fc2ee78e2..f966a2e7cd 100644 --- a/elf/dl-support.c +++ b/elf/dl-support.c @@ -138,8 +138,6 @@ void *_dl_random; #include #include -void (*_dl_init_static_tls) (struct link_map *) = &_dl_nothread_init_static_tls; - size_t _dl_pagesize = EXEC_PAGESIZE; size_t _dl_minsigstacksize = CONSTANT_MINSIGSTKSZ; @@ -197,6 +195,7 @@ list_t _dl_stack_user; int _dl_stack_cache_lock; #else int _dl_thread_gscope_count; +void (*_dl_init_static_tls) (struct link_map *) = &_dl_nothread_init_static_tls; #endif struct dl_scope_free_list *_dl_scope_free_list; diff --git a/elf/dl-tls.c b/elf/dl-tls.c index f8b32b3ecb..6baff0c1ea 100644 --- a/elf/dl-tls.c +++ b/elf/dl-tls.c @@ -29,6 +29,10 @@ #include #include +#if THREAD_GSCOPE_IN_TCB +# include +#endif + #define TUNABLE_NAMESPACE rtld #include @@ -1005,3 +1009,38 @@ cannot create TLS data structures")); listp->slotinfo[idx].gen = GL(dl_tls_generation) + 1; } } + +#if THREAD_GSCOPE_IN_TCB +static inline void __attribute__((always_inline)) +init_one_static_tls (struct pthread *curp, struct link_map *map) +{ +# if TLS_TCB_AT_TP + void *dest = (char *) curp - map->l_tls_offset; +# elif TLS_DTV_AT_TP + void *dest = (char *) curp + map->l_tls_offset + TLS_PRE_TCB_SIZE; +# else +# error "Either TLS_TCB_AT_TP or TLS_DTV_AT_TP must be defined" +# endif + + /* Initialize the memory. */ + memset (__mempcpy (dest, map->l_tls_initimage, map->l_tls_initimage_size), + '\0', map->l_tls_blocksize - map->l_tls_initimage_size); +} + +void +_dl_init_static_tls (struct link_map *map) +{ + lll_lock (GL (dl_stack_cache_lock), LLL_PRIVATE); + + /* Iterate over the list with system-allocated threads first. */ + list_t *runp; + list_for_each (runp, &GL (dl_stack_used)) + init_one_static_tls (list_entry (runp, struct pthread, list), map); + + /* Now the list with threads using user-allocated stacks. */ + list_for_each (runp, &GL (dl_stack_user)) + init_one_static_tls (list_entry (runp, struct pthread, list), map); + + lll_unlock (GL (dl_stack_cache_lock), LLL_PRIVATE); +} +#endif /* THREAD_GSCOPE_IN_TCB */ diff --git a/elf/rtld.c b/elf/rtld.c index 34879016ad..ad325d4c10 100644 --- a/elf/rtld.c +++ b/elf/rtld.c @@ -1139,7 +1139,9 @@ dl_main (const ElfW(Phdr) *phdr, struct dl_main_state state; dl_main_state_init (&state); +#if !THREAD_GSCOPE_IN_TCB GL(dl_init_static_tls) = &_dl_nothread_init_static_tls; +#endif #if defined SHARED && defined _LIBC_REENTRANT \ && defined __rtld_lock_default_lock_recursive diff --git a/nptl/allocatestack.c b/nptl/allocatestack.c index f1270c3109..8aaba088b1 100644 --- a/nptl/allocatestack.c +++ b/nptl/allocatestack.c @@ -951,38 +951,3 @@ __reclaim_stacks (void) GL (dl_stack_cache_lock) = LLL_LOCK_INITIALIZER; __default_pthread_attr_lock = LLL_LOCK_INITIALIZER; } - - -static inline void __attribute__((always_inline)) -init_one_static_tls (struct pthread *curp, struct link_map *map) -{ -# if TLS_TCB_AT_TP - void *dest = (char *) curp - map->l_tls_offset; -# elif TLS_DTV_AT_TP - void *dest = (char *) curp + map->l_tls_offset + TLS_PRE_TCB_SIZE; -# else -# error "Either TLS_TCB_AT_TP or TLS_DTV_AT_TP must be defined" -# endif - - /* Initialize the memory. */ - memset (__mempcpy (dest, map->l_tls_initimage, map->l_tls_initimage_size), - '\0', map->l_tls_blocksize - map->l_tls_initimage_size); -} - -void -attribute_hidden -__pthread_init_static_tls (struct link_map *map) -{ - lll_lock (GL (dl_stack_cache_lock), LLL_PRIVATE); - - /* Iterate over the list with system-allocated threads first. */ - list_t *runp; - list_for_each (runp, &GL (dl_stack_used)) - init_one_static_tls (list_entry (runp, struct pthread, list), map); - - /* Now the list with threads using user-allocated stacks. */ - list_for_each (runp, &GL (dl_stack_user)) - init_one_static_tls (list_entry (runp, struct pthread, list), map); - - lll_unlock (GL (dl_stack_cache_lock), LLL_PRIVATE); -} diff --git a/nptl/nptl-init.c b/nptl/nptl-init.c index b0879bd87e..fcab5a0904 100644 --- a/nptl/nptl-init.c +++ b/nptl/nptl-init.c @@ -191,8 +191,6 @@ __pthread_initialize_minimal_internal (void) GL(dl_make_stack_executable_hook) = &__make_stacks_executable; #endif - GL(dl_init_static_tls) = &__pthread_init_static_tls; - /* Register the fork generation counter with the libc. */ #ifndef TLS_MULTIPLE_THREADS_IN_TCB __libc_multiple_threads_ptr = diff --git a/nptl/pthreadP.h b/nptl/pthreadP.h index 00d2cfe764..0ce63672c4 100644 --- a/nptl/pthreadP.h +++ b/nptl/pthreadP.h @@ -381,8 +381,6 @@ extern int __pthread_multiple_threads attribute_hidden; extern int *__libc_multiple_threads_ptr attribute_hidden; #endif -extern void __pthread_init_static_tls (struct link_map *) attribute_hidden; - extern size_t __pthread_get_minstack (const pthread_attr_t *attr); /* Namespace save aliases. */ diff --git a/sysdeps/generic/ldsodefs.h b/sysdeps/generic/ldsodefs.h index 67c6686015..1b064c5894 100644 --- a/sysdeps/generic/ldsodefs.h +++ b/sysdeps/generic/ldsodefs.h @@ -464,7 +464,9 @@ struct rtld_global /* Generation counter for the dtv. */ EXTERN size_t _dl_tls_generation; +#if !THREAD_GSCOPE_IN_TCB EXTERN void (*_dl_init_static_tls) (struct link_map *); +#endif /* Scopes to free after next THREAD_GSCOPE_WAIT (). */ EXTERN struct dl_scope_free_list @@ -1270,6 +1272,23 @@ extern void _dl_non_dynamic_init (void) extern void _dl_aux_init (ElfW(auxv_t) *av) attribute_hidden; +/* Initialize the static TLS space for the link map in all existing + threads. */ +#if THREAD_GSCOPE_IN_TCB +void _dl_init_static_tls (struct link_map *map) attribute_hidden; +#endif +static inline void +dl_init_static_tls (struct link_map *map) +{ +#if THREAD_GSCOPE_IN_TCB + /* The stack list is available to ld.so, so the initialization can + be handled within ld.so directly. */ + _dl_init_static_tls (map); +#else + GL (dl_init_static_tls) (map); +#endif +} + /* Return true if the ld.so copy in this namespace is actually active and working. If false, the dl_open/dlfcn hooks have to be used to call into the outer dynamic linker (which happens after static