Message ID | 1528929896.22217.1559326257155.JavaMail.zimbra@efficios.com |
---|---|
State | Superseded |
Headers |
Received: (qmail 36741 invoked by alias); 31 May 2019 18:11:02 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: <libc-alpha.sourceware.org> List-Unsubscribe: <mailto:libc-alpha-unsubscribe-##L=##H@sourceware.org> List-Subscribe: <mailto:libc-alpha-subscribe@sourceware.org> List-Archive: <http://sourceware.org/ml/libc-alpha/> List-Post: <mailto:libc-alpha@sourceware.org> List-Help: <mailto:libc-alpha-help@sourceware.org>, <http://sourceware.org/ml/#faqs> Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 36733 invoked by uid 89); 31 May 2019 18:11:01 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-10.8 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.1 spammy=H*i:sk:87muj2k, H*f:sk:87muj2k X-HELO: mail.efficios.com DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 6F47022F452 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=default; t=1559326257; bh=5pWd1qkBb3teYvKpgXTm+KiNou/eDaH+jFixlM1uv00=; h=Date:From:To:Message-ID:MIME-Version; b=k5VLoki8BdROQLM1uUaHJqdewCx2tHy6R5ucG0eHqDTt3UWAv1n9iK2SuWXSWBFUQ vlulMZ9s32lcR5UkF7OzzNks0Vn7Qmks0QyoxoGO4Xut/3s8omXKqVY9RVZMKGHJEI TNqhwXToZDkqim9Wpy910sYbH4lsLnAPsrijTPNGLKiLLYsDQW/h96scaq1Uz6nVj8 BZnP7fvRhAFDklGPfoCS5E3wkvGtefxr4UxK/oVrXr8d2brN7j9RIel3zHh5puZV8Q /NZCvbAJaWDLBdFEiZFzMBJ91j6l7XEUzfYmXUaLL2hqS7cAoqdjCR7GuqfT8uQGFc Ahvecd3l/caGw== Date: Fri, 31 May 2019 14:10:57 -0400 (EDT) From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> To: Florian Weimer <fweimer@redhat.com> Cc: carlos <carlos@redhat.com>, Joseph Myers <joseph@codesourcery.com>, Szabolcs Nagy <szabolcs.nagy@arm.com>, libc-alpha <libc-alpha@sourceware.org>, Thomas Gleixner <tglx@linutronix.de>, Ben Maurer <bmaurer@fb.com>, Peter Zijlstra <peterz@infradead.org>, "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>, Boqun Feng <boqun.feng@gmail.com>, Will Deacon <will.deacon@arm.com>, Dave Watson <davejwatson@fb.com>, Paul Turner <pjt@google.com>, Rich Felker <dalias@libc.org>, linux-kernel <linux-kernel@vger.kernel.org>, linux-api <linux-api@vger.kernel.org> Message-ID: <1528929896.22217.1559326257155.JavaMail.zimbra@efficios.com> In-Reply-To: <87muj2k4ov.fsf@oldenburg2.str.redhat.com> References: <20190503184219.19266-1-mathieu.desnoyers@efficios.com> <87h89gjgaf.fsf@oldenburg2.str.redhat.com> <1239705947.14878.1558985272873.JavaMail.zimbra@efficios.com> <140718133.18261.1559144710554.JavaMail.zimbra@efficios.com> <2022553041.20966.1559249801435.JavaMail.zimbra@efficios.com> <875zprm4jo.fsf@oldenburg2.str.redhat.com> <732661684.21584.1559314109886.JavaMail.zimbra@efficios.com> <87muj2k4ov.fsf@oldenburg2.str.redhat.com> Subject: Re: [PATCH 1/5] glibc: Perform rseq(2) registration at C startup and thread creation (v10) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable |
Commit Message
Mathieu Desnoyers
May 31, 2019, 6:10 p.m. UTC
----- On May 31, 2019, at 11:46 AM, Florian Weimer fweimer@redhat.com wrote: > * Mathieu Desnoyers: > >> Let's break this down into the various sub-issues involved: >> >> 1) How early do we need to setup rseq ? Should it be setup before: >> - LD_PRELOAD .so constructors ? >> - Without circular dependency, >> - With circular dependency, >> - audit libraries initialization ? >> - IFUNC resolvers ? >> - other callbacks ? >> - memory allocator calls ? >> >> We may end up in a situation where we need memory allocation to be setup >> in order to initialize TLS before rseq can be registered for the main >> thread. I suspect we will end up needing a fallbacks which always work >> for the few cases that would try to use rseq too early in dl/libc startup. > > I think the answer to that depends on whether it's okay to have an > observable transition from “no rseq kernel support” to “kernel supports > rseq”. As far as my own use-cases are concerned, I only care that rseq is initialized before LD_PRELOAD .so constructors are executed. There appears to be some amount of documented limitations for what can be done by the IFUNC resolvers. It might be acceptable to document that rseq might not be initialized yet when those are executed. I'd like to hear what others think about whether we should care about IFUNC resolvers and audit libraries using restartable sequences TLS ? [...] > >> 4) Inability to touch a TLS variable (__rseq_abi) from ld-linux-*.so.2 >> - Should we extend the dynamic linker to allow such TLS variable to be >> accessed ? If so, how much effort is required ? >> - Can we find an alternative way to initialize rseq early during >> dl init stages while still performing the TLS access from a function >> implemented within libc.so ? > > This is again related to the answer for (1). There are various hacks we > could implement to make the initialization invisible (e.g., computing > the address of the variable using the equivalent of dlsym, after loading > all the initial objects and before starting relocation). If it's not > too hard to add TLS support to ld.so, we can consider that as well. > (The allocation side should be pretty easy, relocation support it could > be more tricky.) > >> So far, I got rseq to be initialized before LD_PRELOADed library >> constructors by doing the initialization in a constructor within >> libc.so. I don't particularly like this approach, because the >> constructor order is not guaranteed. > > Right. One question related to use of constructors: AFAIU, if a library depends on glibc, ELF guarantees that the glibc constructor will be executed first, before the other library. Which leaves us with the execution order of constructors within libc.so, which is not guaranteed if we just use __attribute__ ((constructor)). However, all gcc versions that are required to build recent glibc seem to support a constructor with a "priority" value (lower gets executed first, and those are executed before constructors without priority). Could we do e.g.: and csu/libc-start.c: static __attribute__ ((constructor (LIBC_CONSTRUCTOR_PRIO_RSEQ_INIT))) void __rseq_libc_init (void) { rseq_init (); /* Register rseq ABI to the kernel. */ (void) rseq_register_current_thread (); } [...] Thanks, Mathieu
Comments
* Mathieu Desnoyers: > ----- On May 31, 2019, at 11:46 AM, Florian Weimer fweimer@redhat.com wrote: > >> * Mathieu Desnoyers: >> >>> Let's break this down into the various sub-issues involved: >>> >>> 1) How early do we need to setup rseq ? Should it be setup before: >>> - LD_PRELOAD .so constructors ? >>> - Without circular dependency, >>> - With circular dependency, >>> - audit libraries initialization ? >>> - IFUNC resolvers ? >>> - other callbacks ? >>> - memory allocator calls ? >>> >>> We may end up in a situation where we need memory allocation to be setup >>> in order to initialize TLS before rseq can be registered for the main >>> thread. I suspect we will end up needing a fallbacks which always work >>> for the few cases that would try to use rseq too early in dl/libc startup. >> >> I think the answer to that depends on whether it's okay to have an >> observable transition from “no rseq kernel support” to “kernel supports >> rseq”. > > As far as my own use-cases are concerned, I only care that rseq is initialized > before LD_PRELOAD .so constructors are executed. <https://sourceware.org/bugzilla/show_bug.cgi?id=14379> is relevant in this context. It requests the opposite behavior from LD_PRELOAD. > There appears to be some amount of documented limitations for what can be > done by the IFUNC resolvers. It might be acceptable to document that rseq > might not be initialized yet when those are executed. The only obstacle is that there are so many places where we could put this information. > I'd like to hear what others think about whether we should care about IFUNC > resolvers and audit libraries using restartable sequences TLS ? In audit libraries (and after dlmopen), the inner libc will have duplicated TLS values, so it will look as if the TLS area is not active (but a registration has happened with the kernel). If we move __rseq_handled into the dynamic linker, its value will be shared along with ld.so with the inner objects. However, the inner libc still has to ensure that its registration attempt does not succeed because that would activate the wrong rseq area. The final remaining case is static dlopen. There is a copy of ld.so on the dynamic side, but it is completely inactive and has never run. I do not think we need to support that because multi-threading does not work reliably in this scenario, either. However, we should skip rseq registration in a nested libc (see the rtld_active function). >>> 4) Inability to touch a TLS variable (__rseq_abi) from ld-linux-*.so.2 >>> - Should we extend the dynamic linker to allow such TLS variable to be >>> accessed ? If so, how much effort is required ? >>> - Can we find an alternative way to initialize rseq early during >>> dl init stages while still performing the TLS access from a function >>> implemented within libc.so ? >> >> This is again related to the answer for (1). There are various hacks we >> could implement to make the initialization invisible (e.g., computing >> the address of the variable using the equivalent of dlsym, after loading >> all the initial objects and before starting relocation). If it's not >> too hard to add TLS support to ld.so, we can consider that as well. >> (The allocation side should be pretty easy, relocation support it could >> be more tricky.) >> >>> So far, I got rseq to be initialized before LD_PRELOADed library >>> constructors by doing the initialization in a constructor within >>> libc.so. I don't particularly like this approach, because the >>> constructor order is not guaranteed. >> >> Right. > > One question related to use of constructors: AFAIU, if a library depends > on glibc, ELF guarantees that the glibc constructor will be executed first, > before the other library. There are some exceptions, like DT_PREINIT_ARRAY functions and DF_1_INITFIRST. Some of these mechanisms we use in the implementation itself, so they are not really usable to end users. Cycles should not come into play here. By default, an object that uses the rseq area will have to link against libc (perhaps indirectly), and therefore the libc constructor runs first. > Which leaves us with the execution order of constructors within libc.so, > which is not guaranteed if we just use __attribute__ ((constructor)). > However, all gcc versions that are required to build recent glibc > seem to support a constructor with a "priority" value (lower gets > executed first, and those are executed before constructors without > priority). I'm not sure that's the right way to do it. If we want to happen execution in a specific order, we should write a single constructor function which is called from _init. For the time being, we can add the call to an appropriately defined inline function early in _init in elf/init-first.c (which is shared with Hurd, so Hurd will need some sort of stub function). Thanks, Florian
----- On Jun 4, 2019, at 7:46 AM, Florian Weimer fweimer@redhat.com wrote: > * Mathieu Desnoyers: > >> ----- On May 31, 2019, at 11:46 AM, Florian Weimer fweimer@redhat.com wrote: >> >>> * Mathieu Desnoyers: >>> >>>> Let's break this down into the various sub-issues involved: >>>> >>>> 1) How early do we need to setup rseq ? Should it be setup before: >>>> - LD_PRELOAD .so constructors ? >>>> - Without circular dependency, >>>> - With circular dependency, >>>> - audit libraries initialization ? >>>> - IFUNC resolvers ? >>>> - other callbacks ? >>>> - memory allocator calls ? >>>> >>>> We may end up in a situation where we need memory allocation to be setup >>>> in order to initialize TLS before rseq can be registered for the main >>>> thread. I suspect we will end up needing a fallbacks which always work >>>> for the few cases that would try to use rseq too early in dl/libc startup. >>> >>> I think the answer to that depends on whether it's okay to have an >>> observable transition from “no rseq kernel support” to “kernel supports >>> rseq”. >> >> As far as my own use-cases are concerned, I only care that rseq is initialized >> before LD_PRELOAD .so constructors are executed. > > <https://sourceware.org/bugzilla/show_bug.cgi?id=14379> is relevant in > this context. It requests the opposite behavior from LD_PRELOAD. This link is very interesting. It sheds some light into how a LD_PRELOAD user wants to override malloc. Should we plan ahead for such scheme to override which library "owns" rseq registration from a LD_PRELOAD library ? If so, then we would want glibc to set __rseq_handled _after_ LD_PRELOAD ctors are executed. However, this brings the following situation: lttng-ust can be LD_PRELOADed into applications, and I intend to make it provide rseq registration *only if* the glibc does not provide it. As a brainstorm idea, one way around this would be to turn __rseq_handled into a 4-states variable: RSEQ_REG_UNSET = 0, -> no library handles rseq RSEQ_REG_PREINIT = 1, -> libc supports RSEQ, initialization not done yet, RSEQ_REG_LIBC = 2, -> libc supports RSEQ, owns registration, RSEQ_REG_OVERRIDE = 3, -> LD_PRELOAD library owns registration. So a lttng-ust LD_PRELOAD could manage rseq registration by setting __rseq_handled = RSEQ_REG_OVERRIDE only after observing the state RSEQ_REG_UNSET. A LD_PRELOAD library wishing to override the libc rseq management should set __rseq_handled to RSEQ_REG_OVERRIDE after observing either UNSET or PREINIT. > >> There appears to be some amount of documented limitations for what can be >> done by the IFUNC resolvers. It might be acceptable to document that rseq >> might not be initialized yet when those are executed. > > The only obstacle is that there are so many places where we could put > this information. If we postpone the actual rseq registration by glibc after LD_PRELOAD ctors execution, I think it makes it clear that we have a part of the startup which executes without rseq being registered: (please let me know if I'm getting some things wrong in the following sequences) A) Startup sequence (glibc owns rseq): __rseq_handled __rseq_abi (TLS) -------------- ---------------------- RSEQ_REG_UNSET no TLS available RSEQ_REG_PREINIT IFUNC resolvers, audit libraries... TLS becomes available LD_PRELOAD ctors glibc initialization RSEQ_REG_LIBC registered to kernel by sys_rseq. B) Startup sequence (LD_PRELOAD lttng-ust owns rseq, old glibc): __rseq_handled __rseq_abi (TLS) -------------- ---------------------- RSEQ_REG_UNSET no TLS available IFUNC resolvers, audit libraries... TLS becomes available LD_PRELOAD ctors RSEQ_REG_OVERRIDE registered to kernel by sys_rseq. C) Startup sequence (LD_PRELOAD rseq override library owning rseq): __rseq_handled __rseq_abi (TLS) -------------- ---------------------- RSEQ_REG_UNSET no TLS available RSEQ_REG_PREINIT IFUNC resolvers, audit libraries... TLS becomes available LD_PRELOAD ctors RSEQ_REG_OVERRIDE registered to kernel by sys_rseq. glibc initialization > >> I'd like to hear what others think about whether we should care about IFUNC >> resolvers and audit libraries using restartable sequences TLS ? > > In audit libraries (and after dlmopen), the inner libc will have > duplicated TLS values, so it will look as if the TLS area is not active > (but a registration has happened with the kernel). If we move > __rseq_handled into the dynamic linker, its value will be shared along > with ld.so with the inner objects. However, the inner libc still has to > ensure that its registration attempt does not succeed because that would > activate the wrong rseq area. Having an intermediate RSEQ_REG_PREINIT state covering the entire duration where the inner libc is in use should do the trick to ensure the duplicated TLS area is not used at that point. The covered use-cases would be to override rseq registration ownership from LD_PRELOADed libraries, but disallow it from IFUNC resolvers and audit libraries. As a consequence of this, rseq critical sections should be prepared to use a fall-back mechanism (e.g. the cpu_opv system call I have been trying to upstream) when they notice rseq is not yet initialized for a rseq c.s. executed within a preinit stage, or very early/late in a thread's lifetime. This is a requirement I have seen coming for a while now. Testing for non-registered rseq is very straightforward and fast to do on a fast-path through the __rseq_abi.cpu_id field: it has a negative value if rseq is not registered for the current thread. > > The final remaining case is static dlopen. There is a copy of ld.so on > the dynamic side, but it is completely inactive and has never run. I do > not think we need to support that because multi-threading does not work > reliably in this scenario, either. However, we should skip rseq > registration in a nested libc (see the rtld_active function). So for SHARED, if (!rtld_active ()), we should indeed leave the state of __rseq_handled as it is, because we are within a nested inactive ld.so. > >>>> 4) Inability to touch a TLS variable (__rseq_abi) from ld-linux-*.so.2 >>>> - Should we extend the dynamic linker to allow such TLS variable to be >>>> accessed ? If so, how much effort is required ? >>>> - Can we find an alternative way to initialize rseq early during >>>> dl init stages while still performing the TLS access from a function >>>> implemented within libc.so ? >>> >>> This is again related to the answer for (1). There are various hacks we >>> could implement to make the initialization invisible (e.g., computing >>> the address of the variable using the equivalent of dlsym, after loading >>> all the initial objects and before starting relocation). If it's not >>> too hard to add TLS support to ld.so, we can consider that as well. >>> (The allocation side should be pretty easy, relocation support it could >>> be more tricky.) >>> >>>> So far, I got rseq to be initialized before LD_PRELOADed library >>>> constructors by doing the initialization in a constructor within >>>> libc.so. I don't particularly like this approach, because the >>>> constructor order is not guaranteed. >>> >>> Right. >> >> One question related to use of constructors: AFAIU, if a library depends >> on glibc, ELF guarantees that the glibc constructor will be executed first, >> before the other library. > > There are some exceptions, like DT_PREINIT_ARRAY functions and > DF_1_INITFIRST. Some of these mechanisms we use in the implementation > itself, so they are not really usable to end users. Cycles should not > come into play here. > > By default, an object that uses the rseq area will have to link against > libc (perhaps indirectly), and therefore the libc constructor runs > first. If we agree on postponing the actual TLS registration _after_ LD_PRELOAD ctors are executed, the problem becomes easier. We then only need to move __rseq_handled to ld.so, and set it to a PREINIT state until we eventually perform the TLS registration (after LD_PRELOAD ctors). > >> Which leaves us with the execution order of constructors within libc.so, >> which is not guaranteed if we just use __attribute__ ((constructor)). >> However, all gcc versions that are required to build recent glibc >> seem to support a constructor with a "priority" value (lower gets >> executed first, and those are executed before constructors without >> priority). > > I'm not sure that's the right way to do it. If we want to happen > execution in a specific order, we should write a single constructor > function which is called from _init. For the time being, we can add the > call to an appropriately defined inline function early in _init in > elf/init-first.c (which is shared with Hurd, so Hurd will need some sort > of stub function). In my attempts, there were some cases where _init was not invoked before LD_PRELOAD ctors, but I cannot remember which at this point. Anyhow, if we choose to postpone the actual TLS registration after LD_PRELOAD ctors, this becomes a non-issue. We might want to rename the __rseq_handled symbol to a better name if it becomes a 4-states variable, e.g. __rseq_reg_owner. Thoughts ? Thanks, Mathieu > > Thanks, > Florian
* Mathieu Desnoyers: > Should we plan ahead for such scheme to override which library "owns" rseq > registration from a LD_PRELOAD library ? If so, then we would want glibc to > set __rseq_handled _after_ LD_PRELOAD ctors are executed. I don't think so. The LD_PRELOAD phase is not clearly delineated from the non-preload phase. So it's not clear to me what this would even mean in practice. Let me ask the key question again: Does it matter if code observes the rseq area first without kernel support, and then with kernel support? If we don't expect any problems immediately, we do not need to worry much about the constructor ordering right now. I expect that over time, fixing this properly will become easier. >> The final remaining case is static dlopen. There is a copy of ld.so on >> the dynamic side, but it is completely inactive and has never run. I do >> not think we need to support that because multi-threading does not work >> reliably in this scenario, either. However, we should skip rseq >> registration in a nested libc (see the rtld_active function). > > So for SHARED, if (!rtld_active ()), we should indeed leave the state of > __rseq_handled as it is, because we are within a nested inactive ld.so. I think we should add __rseq_handled initialization to ld.so, so it will only run once, ever. It's the registration from libc.so which needs some care. In particular, we must not override an existing registration. Thanks, Florian
On 6/6/19 7:57 AM, Florian Weimer wrote: > Let me ask the key question again: Does it matter if code observes the > rseq area first without kernel support, and then with kernel support? > If we don't expect any problems immediately, we do not need to worry > much about the constructor ordering right now. I expect that over time, > fixing this properly will become easier. I just wanted to chime in and say that splitting this into: * Ownership (__rseq_handled) * Initialization (__rseq_abi) Makes sense to me. I agree we need an answer to this question of ownership but not yet initialized, to owned and initialized. I like the idea of having __rseq_handled in ld.so.
----- On Jun 10, 2019, at 4:43 PM, carlos carlos@redhat.com wrote: > On 6/6/19 7:57 AM, Florian Weimer wrote: >> Let me ask the key question again: Does it matter if code observes the >> rseq area first without kernel support, and then with kernel support? >> If we don't expect any problems immediately, we do not need to worry >> much about the constructor ordering right now. I expect that over time, >> fixing this properly will become easier. > > I just wanted to chime in and say that splitting this into: > > * Ownership (__rseq_handled) > > * Initialization (__rseq_abi) > > Makes sense to me. > > I agree we need an answer to this question of ownership but not yet > initialized, to owned and initialized. > > I like the idea of having __rseq_handled in ld.so. Very good, so I'll implement this approach. Sorry for the delayed feedback, I am traveling this week. Thanks, Mathieu
----- On Jun 6, 2019, at 1:57 PM, Florian Weimer fweimer@redhat.com wrote: > * Mathieu Desnoyers: > [...] > >>> The final remaining case is static dlopen. There is a copy of ld.so on >>> the dynamic side, but it is completely inactive and has never run. I do >>> not think we need to support that because multi-threading does not work >>> reliably in this scenario, either. However, we should skip rseq >>> registration in a nested libc (see the rtld_active function). >> >> So for SHARED, if (!rtld_active ()), we should indeed leave the state of >> __rseq_handled as it is, because we are within a nested inactive ld.so. > > I think we should add __rseq_handled initialization to ld.so, so it will > only run once, ever. OK > > It's the registration from libc.so which needs some care. In > particular, we must not override an existing registration. OK, so it could check if __rseq_abi.cpu_id is -1, and only perform registration if it is the case. Or do you have another approach in mind ? For the main thread, "nested" unregistration does not appear to be a problem, because we rely on program exit() to implicitly unregister. Thanks, Mathieu > > Thanks, > Florian
* Mathieu Desnoyers: >> It's the registration from libc.so which needs some care. In >> particular, we must not override an existing registration. > > OK, so it could check if __rseq_abi.cpu_id is -1, and only > perform registration if it is the case. Or do you have another > approach in mind ? No, __rseq_abi will not be shared with the outer libc, so the inner libc will always see -1 there, even if the outer libc has performed registration. libio/vtables.c has some example what you can do: /* In case this libc copy is in a non-default namespace, we always need to accept foreign vtables because there is always a possibility that FILE * objects are passed across the linking boundary. */ { Dl_info di; struct link_map *l; if (!rtld_active () || (_dl_addr (_IO_vtable_check, &di, &l, NULL) != 0 && l->l_ns != LM_ID_BASE)) return; } _IO_vtable_check would have to be replaced with your own function; the actual function doesn't really matter. The rtld_active check covers the static dlopen case, where rtld_active () is false in the inner libc. Thanks, Florian
----- On Jun 12, 2019, at 4:22 PM, Florian Weimer fweimer@redhat.com wrote: > * Mathieu Desnoyers: > >>> It's the registration from libc.so which needs some care. In >>> particular, we must not override an existing registration. >> >> OK, so it could check if __rseq_abi.cpu_id is -1, and only >> perform registration if it is the case. Or do you have another >> approach in mind ? > > No, __rseq_abi will not be shared with the outer libc, so the inner libc > will always see -1 there, even if the outer libc has performed > registration. > > libio/vtables.c has some example what you can do: > > /* In case this libc copy is in a non-default namespace, we always > need to accept foreign vtables because there is always a > possibility that FILE * objects are passed across the linking > boundary. */ > { > Dl_info di; > struct link_map *l; > if (!rtld_active () > || (_dl_addr (_IO_vtable_check, &di, &l, NULL) != 0 > && l->l_ns != LM_ID_BASE)) > return; > } > > _IO_vtable_check would have to be replaced with your own function; the > actual function doesn't really matter. > > The rtld_active check covers the static dlopen case, where > rtld_active () is false in the inner libc. Then out of curiosity, would it also work if I check for if (!__libc_multiple_libcs) in LIBC_START_MAIN ? Thanks, Mathieu
* Mathieu Desnoyers: > ----- On Jun 12, 2019, at 4:22 PM, Florian Weimer fweimer@redhat.com wrote: > >> * Mathieu Desnoyers: >> >>>> It's the registration from libc.so which needs some care. In >>>> particular, we must not override an existing registration. >>> >>> OK, so it could check if __rseq_abi.cpu_id is -1, and only >>> perform registration if it is the case. Or do you have another >>> approach in mind ? >> >> No, __rseq_abi will not be shared with the outer libc, so the inner libc >> will always see -1 there, even if the outer libc has performed >> registration. >> >> libio/vtables.c has some example what you can do: >> >> /* In case this libc copy is in a non-default namespace, we always >> need to accept foreign vtables because there is always a >> possibility that FILE * objects are passed across the linking >> boundary. */ >> { >> Dl_info di; >> struct link_map *l; >> if (!rtld_active () >> || (_dl_addr (_IO_vtable_check, &di, &l, NULL) != 0 >> && l->l_ns != LM_ID_BASE)) >> return; >> } >> >> _IO_vtable_check would have to be replaced with your own function; the >> actual function doesn't really matter. >> >> The rtld_active check covers the static dlopen case, where >> rtld_active () is false in the inner libc. > > Then out of curiosity, would it also work if I check for > > if (!__libc_multiple_libcs) > > in LIBC_START_MAIN ? In my experience, __libc_multiple_libcs is not reliable. I have not yet figured out why. Thanks, Florian
----- On Jun 12, 2019, at 4:00 PM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote: > ----- On Jun 10, 2019, at 4:43 PM, carlos carlos@redhat.com wrote: > >> On 6/6/19 7:57 AM, Florian Weimer wrote: >>> Let me ask the key question again: Does it matter if code observes the >>> rseq area first without kernel support, and then with kernel support? >>> If we don't expect any problems immediately, we do not need to worry >>> much about the constructor ordering right now. I expect that over time, >>> fixing this properly will become easier. >> >> I just wanted to chime in and say that splitting this into: >> >> * Ownership (__rseq_handled) >> >> * Initialization (__rseq_abi) >> >> Makes sense to me. >> >> I agree we need an answer to this question of ownership but not yet >> initialized, to owned and initialized. >> >> I like the idea of having __rseq_handled in ld.so. > > Very good, so I'll implement this approach. Sorry for the delayed > feedback, I am traveling this week. I had issues with cases where application or LD_PRELOAD library also define the __rseq_handled symbol. They appear not to see the same address as the one initialized by ld.so. I tried using the GL() macro in ld.so to set __rseq_handled, but it's the wrong address compared to what the preload lib and application observe. Any thoughts on how to solve this ? Thanks, Mathieu
* Mathieu Desnoyers: > ----- On Jun 12, 2019, at 4:00 PM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote: > >> ----- On Jun 10, 2019, at 4:43 PM, carlos carlos@redhat.com wrote: >> >>> On 6/6/19 7:57 AM, Florian Weimer wrote: >>>> Let me ask the key question again: Does it matter if code observes the >>>> rseq area first without kernel support, and then with kernel support? >>>> If we don't expect any problems immediately, we do not need to worry >>>> much about the constructor ordering right now. I expect that over time, >>>> fixing this properly will become easier. >>> >>> I just wanted to chime in and say that splitting this into: >>> >>> * Ownership (__rseq_handled) >>> >>> * Initialization (__rseq_abi) >>> >>> Makes sense to me. >>> >>> I agree we need an answer to this question of ownership but not yet >>> initialized, to owned and initialized. >>> >>> I like the idea of having __rseq_handled in ld.so. >> >> Very good, so I'll implement this approach. Sorry for the delayed >> feedback, I am traveling this week. > > I had issues with cases where application or LD_PRELOAD library also > define the __rseq_handled symbol. They appear not to see the same > address as the one initialized by ld.so. What exactly did you do? How did you determine the addresses? How is __rseq_handled defined in ld.so? Thanks, Florian
----- On Jun 14, 2019, at 12:06 PM, Florian Weimer fweimer@redhat.com wrote: > * Mathieu Desnoyers: > >> ----- On Jun 12, 2019, at 4:00 PM, Mathieu Desnoyers >> mathieu.desnoyers@efficios.com wrote: >> >>> ----- On Jun 10, 2019, at 4:43 PM, carlos carlos@redhat.com wrote: >>> >>>> On 6/6/19 7:57 AM, Florian Weimer wrote: >>>>> Let me ask the key question again: Does it matter if code observes the >>>>> rseq area first without kernel support, and then with kernel support? >>>>> If we don't expect any problems immediately, we do not need to worry >>>>> much about the constructor ordering right now. I expect that over time, >>>>> fixing this properly will become easier. >>>> >>>> I just wanted to chime in and say that splitting this into: >>>> >>>> * Ownership (__rseq_handled) >>>> >>>> * Initialization (__rseq_abi) >>>> >>>> Makes sense to me. >>>> >>>> I agree we need an answer to this question of ownership but not yet >>>> initialized, to owned and initialized. >>>> >>>> I like the idea of having __rseq_handled in ld.so. >>> >>> Very good, so I'll implement this approach. Sorry for the delayed >>> feedback, I am traveling this week. >> >> I had issues with cases where application or LD_PRELOAD library also >> define the __rseq_handled symbol. They appear not to see the same >> address as the one initialized by ld.so. > > What exactly did you do? How did you determine the addresses? How is > __rseq_handled defined in ld.so? The easiest way to answer these questions is through links to my github dev branch: https://github.com/compudj/glibc-dev/tree/glibc-rseq specifically this commit: https://github.com/compudj/glibc-dev/commit/c49a286497d065a7fc00aafd846e6edce14f97fc and this attempt at using GL(): https://github.com/compudj/glibc-dev/commit/8a02acfbb6943672bfa36b4fc6f61905ee4fa180 My test programs are: * a.c: #include <stdio.h> #include <linux/rseq.h> extern __thread struct rseq __rseq_abi __attribute__ ((tls_model ("initial-exec")));/* = { .cpu_id = -1, };*/ extern int __rseq_handled; int main() { fprintf(stderr, "__rseq_handled main: %d %p\n", __rseq_handled, &__rseq_handled); fprintf(stderr, "__rseq_abi.cpu_id main: %d %p\n", __rseq_abi.cpu_id, &__rseq_abi); return 0; } * s.c: #include <stdio.h> #include <linux/rseq.h> #if 0 __thread struct rseq __rseq_abi __attribute__ ((tls_model ("initial-exec"))) = { .cpu_id = -1, }; int __rseq_handled; #else extern __thread struct rseq __rseq_abi __attribute__ ((tls_model ("initial-exec"))); extern int __rseq_handled; #endif void __attribute__((constructor)) myinit(void) { fprintf(stderr, "__rseq_handled s.so: %d %p\n", __rseq_handled, &__rseq_handled); fprintf(stderr, "__rseq_abi.cpu_id s.so: %d %p\n", __rseq_abi.cpu_id, &__rseq_abi); } * Makefile: LIBCPATH=/home/efficios/glibc-test/lib KERNEL_HEADERS=/home/efficios/git/linux-percpu-dev/usr/include CFLAGS=-I${KERNEL_HEADERS} -L${LIBCPATH} -Wl,--rpath=${LIBCPATH} -Wl,--dynamic-linker=${LIBCPATH}/ld-linux-x86-64.so.2 all: gcc ${CFLAGS} -o a a.c gcc ${CFLAGS} -shared -fPIC -o s.so s.c Thanks, Mathieu > > Thanks, > Florian
* Mathieu Desnoyers: > * Makefile: > > LIBCPATH=/home/efficios/glibc-test/lib > KERNEL_HEADERS=/home/efficios/git/linux-percpu-dev/usr/include > CFLAGS=-I${KERNEL_HEADERS} -L${LIBCPATH} -Wl,--rpath=${LIBCPATH} -Wl,--dynamic-linker=${LIBCPATH}/ld-linux-x86-64.so.2 > > all: > gcc ${CFLAGS} -o a a.c > gcc ${CFLAGS} -shared -fPIC -o s.so s.c For me, that does not correctly link against the built libc because the system dynamic loader seeps into the link. > specifically this commit: > https://github.com/compudj/glibc-dev/commit/c49a286497d065a7fc00aafd846e6edce14f97fc This commit links __rseq_handled into libc.so.6 via rseq-sym.c, but does not export it from there. Thanks, Florian
----- On Jun 14, 2019, at 1:35 PM, Florian Weimer fweimer@redhat.com wrote: > * Mathieu Desnoyers: > >> * Makefile: >> >> LIBCPATH=/home/efficios/glibc-test/lib >> KERNEL_HEADERS=/home/efficios/git/linux-percpu-dev/usr/include >> CFLAGS=-I${KERNEL_HEADERS} -L${LIBCPATH} -Wl,--rpath=${LIBCPATH} >> -Wl,--dynamic-linker=${LIBCPATH}/ld-linux-x86-64.so.2 >> >> all: >> gcc ${CFLAGS} -o a a.c >> gcc ${CFLAGS} -shared -fPIC -o s.so s.c > > For me, that does not correctly link against the built libc because the > system dynamic loader seeps into the link. I have the same issue. I tried adding "-B${LIBCPATH}" as well, but it did not seem to help. I still have this ldd output: ldd a ./a: /lib64/ld-linux-x86-64.so.2: version `GLIBC_2.30' not found (required by ./a) linux-vdso.so.1 (0x00007fffaa7e9000) libc.so.6 => /home/efficios/glibc-test/lib/libc.so.6 (0x00007fac5d479000) /home/efficios/glibc-test/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fac5da33000) Still no luck there. Any idea what compiler/linker flag I am missing ? > >> specifically this commit: >> https://github.com/compudj/glibc-dev/commit/c49a286497d065a7fc00aafd846e6edce14f97fc > > This commit links __rseq_handled into libc.so.6 via rseq-sym.c, but does > not export it from there. Moving __rseq_handled to elf/dl-support.c and elf/rtld.c was part a commit on top. I've force-pushed on the dev branch, and the commit moving __rseq_handled to the dynamic linker it now appears as: https://github.com/compudj/glibc-dev/commit/f0d4e60e5d0ceb0c2642f99da5af61b6ad988531 Thanks, Mathieu
----- On Jun 14, 2019, at 2:55 PM, Mathieu Desnoyers mathieu.desnoyers@efficios.com wrote: > ----- On Jun 14, 2019, at 1:35 PM, Florian Weimer fweimer@redhat.com wrote: > >> * Mathieu Desnoyers: >> >>> * Makefile: >>> >>> LIBCPATH=/home/efficios/glibc-test/lib >>> KERNEL_HEADERS=/home/efficios/git/linux-percpu-dev/usr/include >>> CFLAGS=-I${KERNEL_HEADERS} -L${LIBCPATH} -Wl,--rpath=${LIBCPATH} >>> -Wl,--dynamic-linker=${LIBCPATH}/ld-linux-x86-64.so.2 >>> >>> all: >>> gcc ${CFLAGS} -o a a.c >>> gcc ${CFLAGS} -shared -fPIC -o s.so s.c >> >> For me, that does not correctly link against the built libc because the >> system dynamic loader seeps into the link. > > I have the same issue. I tried adding "-B${LIBCPATH}" as well, but it did > not seem to help. I still have this ldd output: > > ldd a > ./a: /lib64/ld-linux-x86-64.so.2: version `GLIBC_2.30' not found (required by > ./a) > linux-vdso.so.1 (0x00007fffaa7e9000) > libc.so.6 => /home/efficios/glibc-test/lib/libc.so.6 (0x00007fac5d479000) > /home/efficios/glibc-test/lib/ld-linux-x86-64.so.2 => > /lib64/ld-linux-x86-64.so.2 (0x00007fac5da33000) > > Still no luck there. Any idea what compiler/linker flag I am missing ? > Actually, even though ldd seems confused, running the program seems to use the right ld.so: efficios@compudjdev:~/test/libc-sym$ ./a __rseq_handled main: 1 0x55f0ec915020 __rseq_abi.cpu_id main: 28 0x7f54f6c2d4c0 efficios@compudjdev:~/test/libc-sym$ LD_PRELOAD=./s.so ./a __rseq_handled s.so: 1 0x557350bc6020 __rseq_abi.cpu_id s.so: -1 0x7fe2f30f2680 __rseq_handled main: 1 0x557350bc6020 __rseq_abi.cpu_id main: 27 0x7fe2f30f2680 But my original issue remains: if I define a variable called __rseq_handled within either the main executable or the preloaded library, it overshadows the libc one: efficios@compudjdev:~/test/libc-sym$ ./a __rseq_handled main: 0 0x56135fd5102c __rseq_abi.cpu_id main: 29 0x7fcbeca6d5a0 efficios@compudjdev:~/test/libc-sym$ LD_PRELOAD=./s.so ./a __rseq_handled s.so: 0 0x558f70aeb02c __rseq_abi.cpu_id s.so: -1 0x7fdca78b7760 __rseq_handled main: 0 0x558f70aeb02c __rseq_abi.cpu_id main: 27 0x7fdca78b7760 Which is unexpected. This is with my dev branch at this commit: https://github.com/compudj/glibc-dev/commit/f0d4e60e5d0ceb0c2642f99da5af61b6ad988531 What am I missing ? Thanks, Mathieu
* Mathieu Desnoyers: > But my original issue remains: if I define a variable called __rseq_handled > within either the main executable or the preloaded library, it overshadows > the libc one: > > efficios@compudjdev:~/test/libc-sym$ ./a > __rseq_handled main: 0 0x56135fd5102c > __rseq_abi.cpu_id main: 29 0x7fcbeca6d5a0 > efficios@compudjdev:~/test/libc-sym$ LD_PRELOAD=./s.so ./a > __rseq_handled s.so: 0 0x558f70aeb02c > __rseq_abi.cpu_id s.so: -1 0x7fdca78b7760 > __rseq_handled main: 0 0x558f70aeb02c > __rseq_abi.cpu_id main: 27 0x7fdca78b7760 > > Which is unexpected. Why is this unexpected? It has to be this way if the main program uses a copy relocation of __rseq_handled. As long as there is just one address across the entire program and ld.so initializes the copy of the variable that is actually used, everything will be fine. Thanks, Florian
----- On Jun 14, 2019, at 3:09 PM, Florian Weimer fweimer@redhat.com wrote: > * Mathieu Desnoyers: > >> But my original issue remains: if I define a variable called __rseq_handled >> within either the main executable or the preloaded library, it overshadows >> the libc one: >> >> efficios@compudjdev:~/test/libc-sym$ ./a >> __rseq_handled main: 0 0x56135fd5102c >> __rseq_abi.cpu_id main: 29 0x7fcbeca6d5a0 >> efficios@compudjdev:~/test/libc-sym$ LD_PRELOAD=./s.so ./a >> __rseq_handled s.so: 0 0x558f70aeb02c >> __rseq_abi.cpu_id s.so: -1 0x7fdca78b7760 >> __rseq_handled main: 0 0x558f70aeb02c >> __rseq_abi.cpu_id main: 27 0x7fdca78b7760 >> >> Which is unexpected. > > Why is this unexpected? It has to be this way if the main program uses > a copy relocation of __rseq_handled. As long as there is just one > address across the entire program and ld.so initializes the copy of the > variable that is actually used, everything will be fine. Here is a printout of the __rseq_handled address observed by ld.so, it does not match: LD_PRELOAD=./s.so ./a elf: __rseq_handled addr: 7f501c98a140 __rseq_handled s.so: 0 0x55817a88d02c __rseq_abi.cpu_id s.so: -1 0x7f501c983760 __rseq_handled main: 0 0x55817a88d02c __rseq_abi.cpu_id main: 27 0x7f501c983760 This is with the following in a.c: #include <stdio.h> #include <linux/rseq.h> __thread struct rseq __rseq_abi __attribute__ ((tls_model ("initial-exec"))) = { .cpu_id = -1, }; int __rseq_handled; int main() { fprintf(stderr, "__rseq_handled main: %d %p\n", __rseq_handled, &__rseq_handled); fprintf(stderr, "__rseq_abi.cpu_id main: %d %p\n", __rseq_abi.cpu_id, &__rseq_abi); return 0; } As we can see, the state of __rseq_handled observed by the preloaded lib and the program is "0", but should really be "1". This can be explained by ld.so not using the same address as the rest of the program, but how can we fix that ? Thanks, Mathieu
* Mathieu Desnoyers: > ----- On Jun 14, 2019, at 3:09 PM, Florian Weimer fweimer@redhat.com wrote: > >> * Mathieu Desnoyers: >> >>> But my original issue remains: if I define a variable called __rseq_handled >>> within either the main executable or the preloaded library, it overshadows >>> the libc one: >>> >>> efficios@compudjdev:~/test/libc-sym$ ./a >>> __rseq_handled main: 0 0x56135fd5102c >>> __rseq_abi.cpu_id main: 29 0x7fcbeca6d5a0 >>> efficios@compudjdev:~/test/libc-sym$ LD_PRELOAD=./s.so ./a >>> __rseq_handled s.so: 0 0x558f70aeb02c >>> __rseq_abi.cpu_id s.so: -1 0x7fdca78b7760 >>> __rseq_handled main: 0 0x558f70aeb02c >>> __rseq_abi.cpu_id main: 27 0x7fdca78b7760 >>> >>> Which is unexpected. >> >> Why is this unexpected? It has to be this way if the main program uses >> a copy relocation of __rseq_handled. As long as there is just one >> address across the entire program and ld.so initializes the copy of the >> variable that is actually used, everything will be fine. > > Here is a printout of the __rseq_handled address observed by ld.so, it > does not match: > > LD_PRELOAD=./s.so ./a > elf: __rseq_handled addr: 7f501c98a140 > __rseq_handled s.so: 0 0x55817a88d02c > __rseq_abi.cpu_id s.so: -1 0x7f501c983760 > __rseq_handled main: 0 0x55817a88d02c > __rseq_abi.cpu_id main: 27 0x7f501c983760 Where do you print the address? Before or after the self-relocation of the dynamic loader? The address is only correct after self-relocation. Thanks, Florian
----- On Jun 14, 2019, at 3:29 PM, David Laight David.Laight@ACULAB.COM wrote: > From: Mathieu Desnoyers >> Sent: 14 June 2019 14:02 > ... >> But my original issue remains: if I define a variable called __rseq_handled >> within either the main executable or the preloaded library, it overshadows >> the libc one: > > 1) That is the was elf symbol resolution is required to work. > Otherwise variables like 'errno' (non-thread safe form) wouldn't work. > > 2) Don't do it then :-) > Names starting with __ will be reserved (probably 'for the implementation'). > > The real 'fun' starts because, under some circumstances, looking up a symbol as: > foo = dlsym(lib_handle, "foo"); > Can find the data item instead of the function! > Usually it works (even when foo is global data) because 'lib_handle' refers > to a different symbol table. > But it can go horribly wrong. I was setting __rseq_handled too soon, before re-relocation of the dynamic linker. I moved the initialization after re-relocation and it works fine now. The purpose of __rseq_handled is to allow early adopter libraries and applications to define their own global instance of the symbol, and check whether the libc they are linked against handle rseq registration or not. libc specifies the layout of that variable (an integer). The dynamic linker chooses one of those instances so it's used in the global symbol table of the program. The important thing is that all libraries agree on that global symbol. Of course this is not compatible with libraries compiled with forced "hidden" symbols only. Thanks, Mathieu
--- a/include/libc-internal.h +++ b/include/libc-internal.h @@ -21,6 +21,12 @@ #include <hp-timing.h> +/* Libc constructor priority order. Lower is executed first. */ +enum libc_constructor_prio { + /* Priorities between 0 and 100 are reserved. */ + LIBC_CONSTRUCTOR_PRIO_RSEQ_INIT = 1000, +}; + /* Initialize the `__libc_enable_secure' flag. */ extern void __libc_init_secure (void);