From patchwork Fri Aug 20 08:05:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fangrui Song X-Patchwork-Id: 44720 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 5BF743893C43 for ; Fri, 20 Aug 2021 08:06:03 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 5BF743893C43 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1629446763; bh=Kjoc0IwnjDQPmGiDpmuSvlq3Egv6aSERDXuCTnU/E4E=; h=Date:Subject:To:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=NvTmAHv6StmRYj4OpuOOPnyW5k7ad+l2UI7k0KJDaZv5z50Ab3K/X4OKUC7hfrnnF MxKWM+Nson6bEMxBiTBytjUS9MZ2hm3vA/WcJM1VjCku/z2jotcWxZRGKOmSAblxA9 7HhgbmIT/eHCytXpd7TU942eFD6HsqpyE2kGzjCw= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-qt1-x84a.google.com (mail-qt1-x84a.google.com [IPv6:2607:f8b0:4864:20::84a]) by sourceware.org (Postfix) with ESMTPS id 4AFAD3858432 for ; Fri, 20 Aug 2021 08:05:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 4AFAD3858432 Received: by mail-qt1-x84a.google.com with SMTP id o11-20020ac85a4b0000b029028acd99a680so4341125qta.19 for ; Fri, 20 Aug 2021 01:05:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=Kjoc0IwnjDQPmGiDpmuSvlq3Egv6aSERDXuCTnU/E4E=; b=FnMHih0eIoktqsU1QuHqHMCkFzvfGDeXKMOZFVWaWXlflbGHAdEMJyTEwAuASsb4RB tGfP9M/EkJ85AcHzztkOK3PT+fA+xv2CWOPVz9IxvMDmMJFed/AfXJOxIqVwoEYqWmMw 7l+J1p3WI1b7ue5hrmy9Q0pUWyErL/gXW7FfgmEsFPEu9gsO96mzvqGLSgBQkCAdnifj KNOPzobAAnPRlQlzOzahj37Bpi4UPt0IW2zTarMVd0q8/IIlXjIFfmdppU/N9eyKBZY3 k7h36YrBwT6OQKDm7eDEcJZXKld6qhSDuPVUgUkmMv0ja33Jp3dMzXl/C34d7xoqC0Nj D8AQ== X-Gm-Message-State: AOAM533HAQxEk1fxcUKpLgxpvOcRL9XcDfBU8w8EviHS1rSgd4qJtf1A Fo688616nA46GJLsM/it5rYeiYS/j+QfQAULo0sftJwBptZHqTA/+oRJ472YA4jedGf8x9xuUzH hDRLOwMpyMilDjw9tDhdGsZ6SDbVgY15EvDxxfdWJkaKkZPNrfoNa6o92orj0ThLdWzvZ X-Google-Smtp-Source: ABdhPJwYzgBBSbeF7YHSc/i0qqRWemBwr1juu5DlyifL41ysJiYILcIVReW9gkWmDpCDSzzbJPdTrn7PxxRo X-Received: from maskray1.svl.corp.google.com ([2620:15c:2ce:200:3aa1:6290:9c86:6a64]) (user=maskray job=sendgmr) by 2002:a05:6214:2123:: with SMTP id r3mr18949881qvc.19.1629446729895; Fri, 20 Aug 2021 01:05:29 -0700 (PDT) Date: Fri, 20 Aug 2021 01:05:13 -0700 Message-Id: <20210820080513.3004013-1-maskray@google.com> Mime-Version: 1.0 Subject: [PATCH] elf: Avoid nested functions in the loader (x86-64) [BZ #27220] To: libc-alpha@sourceware.org, "Carlos O'Donell" , Adhemerval Zanella X-Spam-Status: No, score=-18.8 required=5.0 tests=BAYES_00, DKIMWL_WL_MED, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, KAM_STOCKGEN, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Fangrui Song via Libc-alpha From: Fangrui Song Reply-To: Fangrui Song Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" dynamic-link.h is included more than once in some elf/ files (rtld.c, dl-conflict.c, dl-reloc.c, dl-reloc-static-pie.c) and uses GCC nested functions. This harms readability and the nested functions usage is the biggest obstacle prevents CC=clang (which doesn't support the feature). To un-nest elf_machine_rela, the key idea is to pass the variable in the containing scope as an extra argument. Stan Shebs implemented this in the google/grte/v5-2.27/master branch. This patch squashes and cleans up the commits. This patch just fixes x86-64: x86-64 uses the `#ifndef NESTING` branch to avoid nested functions. The `#ifdef NESTING` branch is used by all other ports whose dl-machine.h files haven't been migrated. For the time being, there is some duplicated code. `#ifdef NESTING` dispatches can be removed in the future when all arches are migrated. I can fix aarch64, powerpc64, riscv, and other arches subsequently. Migrating all ports at once is just too risky. Also, appreciate help from arch maintainers. Tested on x86_64-linux-gnu (!NESTING) and aarch64-linux-gnu (NESTING). --- elf/dl-conflict.c | 33 ++++++++++++- elf/dl-reloc-static-pie.c | 15 +++++- elf/dl-reloc.c | 51 ++++++++++++++++++- elf/do-rel.h | 30 ++++++++++-- elf/dynamic-link.h | 97 ++++++++++++++++++++++++++++++++++++- elf/get-dynamic-info.h | 10 ++++ elf/rtld.c | 30 +++++++++++- sysdeps/x86_64/dl-machine.h | 16 +++++- 8 files changed, 269 insertions(+), 13 deletions(-) diff --git a/elf/dl-conflict.c b/elf/dl-conflict.c index 31a2f90770..832cba65c1 100644 --- a/elf/dl-conflict.c +++ b/elf/dl-conflict.c @@ -27,6 +27,25 @@ #include #include "dynamic-link.h" +#ifndef NESTING + + + /* This macro is used as a callback from the ELF_DYNAMIC_RELOCATE code. */ +#define RESOLVE_MAP(ref, version, flags) (*ref = NULL, NULL) +#define RESOLVE(ref, version, flags) (*ref = NULL, 0) +#define RESOLVE_CONFLICT_FIND_MAP(map, r_offset) \ + do { \ + while ((resolve_conflict_map->l_map_end < (ElfW(Addr)) (r_offset)) \ + || (resolve_conflict_map->l_map_start > (ElfW(Addr)) (r_offset))) \ + resolve_conflict_map = resolve_conflict_map->l_next; \ + \ + (map) = resolve_conflict_map; \ + } while (0) + +#include "dynamic-link.h" + +#endif /* NESTING */ + void _dl_resolve_conflicts (struct link_map *l, ElfW(Rela) *conflict, ElfW(Rela) *conflictend) @@ -39,6 +58,8 @@ _dl_resolve_conflicts (struct link_map *l, ElfW(Rela) *conflict, /* Do the conflict relocation of the object and library GOT and other data. */ +#ifdef NESTING + /* This macro is used as a callback from the ELF_DYNAMIC_RELOCATE code. */ #define RESOLVE_MAP(ref, version, flags) (*ref = NULL, NULL) #define RESOLVE(ref, version, flags) (*ref = NULL, 0) @@ -51,13 +72,19 @@ _dl_resolve_conflicts (struct link_map *l, ElfW(Rela) *conflict, (map) = resolve_conflict_map; \ } while (0) +#endif /* NESTING */ + /* Prelinking makes no sense for anything but the main namespace. */ assert (l->l_ns == LM_ID_BASE); struct link_map *resolve_conflict_map __attribute__ ((__unused__)) = GL(dl_ns)[LM_ID_BASE]._ns_loaded; +#ifdef NESTING + #include "dynamic-link.h" +#endif /* NESTING */ + /* Override these, defined in dynamic-link.h. */ #undef CHECK_STATIC_TLS #define CHECK_STATIC_TLS(ref_map, sym_map) ((void) 0) @@ -68,7 +95,11 @@ _dl_resolve_conflicts (struct link_map *l, ElfW(Rela) *conflict, for (; conflict < conflictend; ++conflict) elf_machine_rela (l, conflict, NULL, NULL, (void *) conflict->r_offset, - 0); + 0 +#ifndef NESTING + , NULL +#endif + ); } #endif } diff --git a/elf/dl-reloc-static-pie.c b/elf/dl-reloc-static-pie.c index d5bd2f31e9..0ab4613021 100644 --- a/elf/dl-reloc-static-pie.c +++ b/elf/dl-reloc-static-pie.c @@ -23,6 +23,13 @@ #include #include "dynamic-link.h" +#ifndef NESTING +# define STATIC_PIE_BOOTSTRAP +# define BOOTSTRAP_MAP (main_map) +# define RESOLVE_MAP(sym, version, flags) BOOTSTRAP_MAP +# include "dynamic-link.h" +#endif /* n NESTING */ + /* Relocate static executable with PIE. */ void @@ -30,10 +37,12 @@ _dl_relocate_static_pie (void) { struct link_map *main_map = _dl_get_dl_main_map (); +#ifdef NESTING # define STATIC_PIE_BOOTSTRAP # define BOOTSTRAP_MAP (main_map) # define RESOLVE_MAP(sym, version, flags) BOOTSTRAP_MAP # include "dynamic-link.h" +#endif /* NESTING */ /* Figure out the run-time load address of static PIE. */ main_map->l_addr = elf_machine_load_address (); @@ -48,7 +57,11 @@ _dl_relocate_static_pie (void) /* Relocate ourselves so we can do normal function calls and data access using the global offset table. */ - ELF_DYNAMIC_RELOCATE (main_map, 0, 0, 0); + ELF_DYNAMIC_RELOCATE (main_map, 0, 0, 0 +#ifndef NESTING + , main_map +#endif + ); main_map->l_relocated = 1; /* Initialize _r_debug. */ diff --git a/elf/dl-reloc.c b/elf/dl-reloc.c index e13a672ade..6d8b64ecb9 100644 --- a/elf/dl-reloc.c +++ b/elf/dl-reloc.c @@ -162,6 +162,41 @@ _dl_nothread_init_static_tls (struct link_map *map) } #endif /* !THREAD_GSCOPE_IN_TCB */ +#ifndef NESTING + + /* String table object symbols. */ + +static struct link_map *glob_l; +static struct r_scope_elem **glob_scope; +static const char *glob_strtab; + +/* This macro is used as a callback from the ELF_DYNAMIC_RELOCATE code. */ +#define RESOLVE_MAP(ref, version, r_type) \ + ((ELFW(ST_BIND) ((*ref)->st_info) != STB_LOCAL \ + && __glibc_likely (!dl_symbol_visibility_binds_local_p (*ref))) \ + ? ((__builtin_expect ((*ref) == glob_l->l_lookup_cache.sym, 0) \ + && elf_machine_type_class (r_type) == glob_l->l_lookup_cache.type_class) \ + ? (bump_num_cache_relocations (), \ + (*ref) = glob_l->l_lookup_cache.ret, \ + glob_l->l_lookup_cache.value) \ + : ({ lookup_t _lr; \ + int _tc = elf_machine_type_class (r_type); \ + glob_l->l_lookup_cache.type_class = _tc; \ + glob_l->l_lookup_cache.sym = (*ref); \ + const struct r_found_version *v = NULL; \ + if ((version) != NULL && (version)->hash != 0) \ + v = (version); \ + _lr = _dl_lookup_symbol_x (glob_strtab + (*ref)->st_name, glob_l, (ref), \ + glob_scope, v, _tc, \ + DL_LOOKUP_ADD_DEPENDENCY, NULL); \ + glob_l->l_lookup_cache.ret = (*ref); \ + glob_l->l_lookup_cache.value = _lr; })) \ + : glob_l) + +#include "dynamic-link.h" + +#endif /* n NESTING */ + void _dl_relocate_object (struct link_map *l, struct r_scope_elem *scope[], int reloc_mode, int consider_profiling) @@ -243,6 +278,8 @@ _dl_relocate_object (struct link_map *l, struct r_scope_elem *scope[], { /* Do the actual relocation of the object's GOT and other data. */ +#ifdef NESTING + /* String table object symbols. */ const char *strtab = (const void *) D_PTR (l, l_info[DT_STRTAB]); @@ -272,7 +309,19 @@ _dl_relocate_object (struct link_map *l, struct r_scope_elem *scope[], #include "dynamic-link.h" - ELF_DYNAMIC_RELOCATE (l, lazy, consider_profiling, skip_ifunc); +#else /* n NESTING */ + + glob_l = l; + glob_scope = scope; + glob_strtab = (const void *) D_PTR (glob_l, l_info[DT_STRTAB]); + +#endif /* n NESTING */ + + ELF_DYNAMIC_RELOCATE (l, lazy, consider_profiling, skip_ifunc +#ifndef NESTING + , NULL +#endif + ); #ifndef PROF if (__glibc_unlikely (consider_profiling) diff --git a/elf/do-rel.h b/elf/do-rel.h index 321ac2b359..401a6340c8 100644 --- a/elf/do-rel.h +++ b/elf/do-rel.h @@ -41,7 +41,11 @@ auto inline void __attribute__ ((always_inline)) elf_dynamic_do_Rel (struct link_map *map, ElfW(Addr) reladdr, ElfW(Addr) relsize, __typeof (((ElfW(Dyn) *) 0)->d_un.d_val) nrelative, - int lazy, int skip_ifunc) + int lazy, int skip_ifunc +#ifndef NESTING + , struct link_map *boot_map +#endif + ) { const ElfW(Rel) *r = (const void *) reladdr; const ElfW(Rel) *end = (const void *) (reladdr + relsize); @@ -136,7 +140,11 @@ elf_dynamic_do_Rel (struct link_map *map, ElfW(Half) ndx = version[ELFW(R_SYM) (r->r_info)] & 0x7fff; elf_machine_rel (map, r, &symtab[ELFW(R_SYM) (r->r_info)], &map->l_versions[ndx], - (void *) (l_addr + r->r_offset), skip_ifunc); + (void *) (l_addr + r->r_offset), skip_ifunc +#ifndef NESTING + , boot_map +#endif + ); } #if defined ELF_MACHINE_IRELATIVE && !defined RTLD_BOOTSTRAP @@ -150,7 +158,11 @@ elf_dynamic_do_Rel (struct link_map *map, &symtab[ELFW(R_SYM) (r2->r_info)], &map->l_versions[ndx], (void *) (l_addr + r2->r_offset), - skip_ifunc); + skip_ifunc +#ifndef NESTING + , boot_map +#endif + ); } #endif } @@ -168,7 +180,11 @@ elf_dynamic_do_Rel (struct link_map *map, else # endif elf_machine_rel (map, r, &symtab[ELFW(R_SYM) (r->r_info)], NULL, - (void *) (l_addr + r->r_offset), skip_ifunc); + (void *) (l_addr + r->r_offset), skip_ifunc +#ifndef NESTING + , boot_map +#endif + ); # ifdef ELF_MACHINE_IRELATIVE if (r2 != NULL) @@ -176,7 +192,11 @@ elf_dynamic_do_Rel (struct link_map *map, if (ELFW(R_TYPE) (r2->r_info) == ELF_MACHINE_IRELATIVE) elf_machine_rel (map, r2, &symtab[ELFW(R_SYM) (r2->r_info)], NULL, (void *) (l_addr + r2->r_offset), - skip_ifunc); + skip_ifunc +#ifndef NESTING + , boot_map +#endif + ); # endif } #endif diff --git a/elf/dynamic-link.h b/elf/dynamic-link.h index 3eb24ba3a6..3493940a1e 100644 --- a/elf/dynamic-link.h +++ b/elf/dynamic-link.h @@ -16,6 +16,13 @@ License along with the GNU C Library; if not, see . */ +#if defined __x86_64__ +# define auto static +#else +/* Other arches use nested functions. */ +# define NESTING +#endif + /* This macro is used as a callback from elf_machine_rel{a,} when a static TLS reloc is about to be performed. Since (in dl-load.c) we permit dynamic loading of objects that might use such relocs, we @@ -71,7 +78,11 @@ elf_machine_rel_relative (ElfW(Addr) l_addr, const ElfW(Rel) *reloc, auto inline void __attribute__((always_inline)) elf_machine_rela (struct link_map *map, const ElfW(Rela) *reloc, const ElfW(Sym) *sym, const struct r_found_version *version, - void *const reloc_addr, int skip_ifunc); + void *const reloc_addr, int skip_ifunc +#ifndef NESTING + , struct link_map *boot_map +#endif + ); auto inline void __attribute__((always_inline)) elf_machine_rela_relative (ElfW(Addr) l_addr, const ElfW(Rela) *reloc, void *const reloc_addr); @@ -114,6 +125,60 @@ elf_machine_lazy_rel (struct link_map *map, consumes precisely the very end of the DT_REL*, or DT_JMPREL and DT_REL* are completely separate and there is a gap between them. */ +#ifndef NESTING +# define _ELF_DYNAMIC_DO_RELOC(RELOC, reloc, map, do_lazy, skip_ifunc, test_rel, boot_map) \ + do { \ + struct { ElfW(Addr) start, size; \ + __typeof (((ElfW(Dyn) *) 0)->d_un.d_val) nrelative; int lazy; } \ + ranges[2] = { { 0, 0, 0, 0 }, { 0, 0, 0, 0 } }; \ + \ + if ((map)->l_info[DT_##RELOC]) \ + { \ + ranges[0].start = D_PTR ((map), l_info[DT_##RELOC]); \ + ranges[0].size = (map)->l_info[DT_##RELOC##SZ]->d_un.d_val; \ + if (map->l_info[VERSYMIDX (DT_##RELOC##COUNT)] != NULL) \ + ranges[0].nrelative \ + = map->l_info[VERSYMIDX (DT_##RELOC##COUNT)]->d_un.d_val; \ + } \ + if ((map)->l_info[DT_PLTREL] \ + && (!test_rel || (map)->l_info[DT_PLTREL]->d_un.d_val == DT_##RELOC)) \ + { \ + ElfW(Addr) start = D_PTR ((map), l_info[DT_JMPREL]); \ + ElfW(Addr) size = (map)->l_info[DT_PLTRELSZ]->d_un.d_val; \ + \ + if (ranges[0].start + ranges[0].size == (start + size)) \ + ranges[0].size -= size; \ + if (ELF_DURING_STARTUP \ + || (!(do_lazy) \ + && (ranges[0].start + ranges[0].size) == start)) \ + { \ + /* Combine processing the sections. */ \ + ranges[0].size += size; \ + } \ + else \ + { \ + ranges[1].start = start; \ + ranges[1].size = size; \ + ranges[1].lazy = (do_lazy); \ + } \ + } \ + \ + if (ELF_DURING_STARTUP) \ + elf_dynamic_do_##reloc ((map), ranges[0].start, ranges[0].size, \ + ranges[0].nrelative, 0, skip_ifunc, boot_map); \ + else \ + { \ + int ranges_index; \ + for (ranges_index = 0; ranges_index < 2; ++ranges_index) \ + elf_dynamic_do_##reloc ((map), \ + ranges[ranges_index].start, \ + ranges[ranges_index].size, \ + ranges[ranges_index].nrelative, \ + ranges[ranges_index].lazy, \ + skip_ifunc, boot_map); \ + } \ + } while (0) +#else /* NESTING */ # define _ELF_DYNAMIC_DO_RELOC(RELOC, reloc, map, do_lazy, skip_ifunc, test_rel) \ do { \ struct { ElfW(Addr) start, size; \ @@ -166,6 +231,7 @@ elf_machine_lazy_rel (struct link_map *map, skip_ifunc); \ } \ } while (0) +#endif /* NESTING */ # if ELF_MACHINE_NO_REL || ELF_MACHINE_NO_RELA # define _ELF_CHECK_REL 0 @@ -173,6 +239,34 @@ elf_machine_lazy_rel (struct link_map *map, # define _ELF_CHECK_REL 1 # endif +#ifndef NESTING +# if ! ELF_MACHINE_NO_REL +# include "do-rel.h" +# define ELF_DYNAMIC_DO_REL(map, lazy, skip_ifunc, boot_map) \ + _ELF_DYNAMIC_DO_RELOC (REL, Rel, map, lazy, skip_ifunc, _ELF_CHECK_REL, boot_map) +# else +# define ELF_DYNAMIC_DO_REL(map, lazy, skip_ifunc, boot_map) /* Nothing to do. */ +# endif + +# if ! ELF_MACHINE_NO_RELA +# define DO_RELA +# include "do-rel.h" +# define ELF_DYNAMIC_DO_RELA(map, lazy, skip_ifunc, boot_map) \ + _ELF_DYNAMIC_DO_RELOC (RELA, Rela, map, lazy, skip_ifunc, _ELF_CHECK_REL, boot_map) +# else +# define ELF_DYNAMIC_DO_RELA(map, lazy, skip_ifunc, boot_map) /* Nothing to do. */ +# endif + +/* This can't just be an inline function because GCC is too dumb + to inline functions containing inlines themselves. */ +# define ELF_DYNAMIC_RELOCATE(map, lazy, consider_profile, skip_ifunc, boot_map) \ + do { \ + int edr_lazy = elf_machine_runtime_setup ((map), (lazy), \ + (consider_profile)); \ + ELF_DYNAMIC_DO_REL ((map), edr_lazy, skip_ifunc, boot_map); \ + ELF_DYNAMIC_DO_RELA ((map), edr_lazy, skip_ifunc, boot_map); \ + } while (0) +#else /* NESTING */ # if ! ELF_MACHINE_NO_REL # include "do-rel.h" # define ELF_DYNAMIC_DO_REL(map, lazy, skip_ifunc) \ @@ -199,5 +293,6 @@ elf_machine_lazy_rel (struct link_map *map, ELF_DYNAMIC_DO_REL ((map), edr_lazy, skip_ifunc); \ ELF_DYNAMIC_DO_RELA ((map), edr_lazy, skip_ifunc); \ } while (0) +#endif /* NESTING */ #endif diff --git a/elf/get-dynamic-info.h b/elf/get-dynamic-info.h index d8ec32377d..3bb6ab1ce4 100644 --- a/elf/get-dynamic-info.h +++ b/elf/get-dynamic-info.h @@ -22,6 +22,8 @@ #include #include +#if defined NESTING || !defined SAW_EGDI + #ifndef RESOLVE_MAP static #else @@ -180,3 +182,11 @@ elf_get_dynamic_info (struct link_map *l, ElfW(Dyn) *temp) info[DT_RPATH] = NULL; #endif } + +#endif + +#ifndef NESTING +#ifndef SAW_EGDI +# define SAW_EGDI +#endif +#endif /* n NESTING */ diff --git a/elf/rtld.c b/elf/rtld.c index 878e6480f4..167ee66cb1 100644 --- a/elf/rtld.c +++ b/elf/rtld.c @@ -499,15 +499,36 @@ _dl_start_final (void *arg, struct dl_start_final_info *info) return start_addr; } +#ifndef NESTING +#ifdef DONT_USE_BOOTSTRAP_MAP +# define bootstrap_map GL(dl_rtld_map) +#else +# define bootstrap_map info.l +#endif + + /* This #define produces dynamic linking inline functions for + bootstrap relocation instead of general-purpose relocation. + Since ld.so must not have any undefined symbols the result + is trivial: always the map of ld.so itself. */ +#define RTLD_BOOTSTRAP +#define RESOLVE_MAP(sym, version, flags) (&bootstrap_map) +#include "dynamic-link.h" +#endif /* n NESTING */ + static ElfW(Addr) __attribute_used__ _dl_start (void *arg) { +#ifndef NESTING +#ifndef DONT_USE_BOOTSTRAP_MAP + struct dl_start_final_info info; +#endif +#else /* NESTING */ #ifdef DONT_USE_BOOTSTRAP_MAP # define bootstrap_map GL(dl_rtld_map) #else struct dl_start_final_info info; # define bootstrap_map info.l -#endif +#endif /* NESTING */ /* This #define produces dynamic linking inline functions for bootstrap relocation instead of general-purpose relocation. @@ -517,6 +538,7 @@ _dl_start (void *arg) #define BOOTSTRAP_MAP (&bootstrap_map) #define RESOLVE_MAP(sym, version, flags) BOOTSTRAP_MAP #include "dynamic-link.h" +#endif /* NESTING */ #ifdef DONT_USE_BOOTSTRAP_MAP rtld_timer_start (&start_time); @@ -561,7 +583,11 @@ _dl_start (void *arg) /* Relocate ourselves so we can do normal function calls and data access using the global offset table. */ - ELF_DYNAMIC_RELOCATE (&bootstrap_map, 0, 0, 0); + ELF_DYNAMIC_RELOCATE (&bootstrap_map, 0, 0, 0 +#ifndef NESTING + , &bootstrap_map +#endif + ); } bootstrap_map.l_relocated = 1; diff --git a/sysdeps/x86_64/dl-machine.h b/sysdeps/x86_64/dl-machine.h index ceee50734e..1161f03771 100644 --- a/sysdeps/x86_64/dl-machine.h +++ b/sysdeps/x86_64/dl-machine.h @@ -255,7 +255,11 @@ auto inline void __attribute__ ((always_inline)) elf_machine_rela (struct link_map *map, const ElfW(Rela) *reloc, const ElfW(Sym) *sym, const struct r_found_version *version, - void *const reloc_addr_arg, int skip_ifunc) + void *const reloc_addr_arg, int skip_ifunc +#ifndef NESTING + , struct link_map *boot_map +#endif + ) { ElfW(Addr) *const reloc_addr = reloc_addr_arg; const unsigned long int r_type = ELFW(R_TYPE) (reloc->r_info); @@ -293,7 +297,11 @@ elf_machine_rela (struct link_map *map, const ElfW(Rela) *reloc, # ifndef RTLD_BOOTSTRAP const ElfW(Sym) *const refsym = sym; # endif +#if !defined NESTING && (defined RTLD_BOOTSTRAP || defined STATIC_PIE_BOOTSTRAP) + struct link_map *sym_map = boot_map; +#else struct link_map *sym_map = RESOLVE_MAP (&sym, version, r_type); +#endif ElfW(Addr) value = SYMBOL_ADDRESS (sym_map, sym, true); if (sym != NULL @@ -573,7 +581,11 @@ elf_machine_lazy_rel (struct link_map *map, /* Always initialize TLS descriptors completely at load time, in case static TLS is allocated for it that requires locking. */ - elf_machine_rela (map, reloc, sym, version, reloc_addr, skip_ifunc); + elf_machine_rela (map, reloc, sym, version, reloc_addr, skip_ifunc +#ifndef NESTING + , 0 +#endif + ); } else if (__glibc_unlikely (r_type == R_X86_64_IRELATIVE)) {