[libc] __tls_get_addr with link_map * instead of modid

  Hi,

https://sourceware.org/glibc/wiki/Tools%20Interface%20NG
	From within the inferior: Functions that are inserted into a process's
	address space by the debugger need to be able to access the TLS area
	for that thread. GDB can use DWARF to to find the offset within the
	TLS of a variable and pass that in as a compile time offset into the
	function being inserted into the traced process's address space.
	However, to be able to get to the address of the TLS area for
	a particular thread, it needs to know the module id to make a call to
	__tls_get_addr(). This module id is burried in a private area of the
	link_map data structure which is subject to change. It is therefore
	proposed that we add a new function to libthread_db which extracts the
	module id from the link map. 

Other option suggested by Tom Tromey is implemented by this patch:

* A new glibc function like __tls_get_addr that takes a link_map address
  rather than a module id.

I expect I will have to also implement this Tom Tromey's suggestion to GDB but
I haven't tried to implement it yet:

* Bake more information about struct link_map into gdb (least preferred
  IMO).

Additionally one could also fetch the module id from R_X86_64_DTPMOD64 but that
would require arch specific code in GDB.  I do not plan to implement this way.


I have failed to test this patch on ia64 as I have available only RHEL-5 which
has too old ld and then I gave it up on too old Linux kernel include files.

I haven't written any GDB support for this patch yet.


Thanks,
Jan
libc/
2014-10-18  Jan Kratochvil  <jan.kratochvil@redhat.com>

	* NEWS: Add _dl_tls_get_addr_link_map note.
	* elf/Makefile (tests): Add tst-tls20.
	($(objpfx)tst-tls20): New.
	* elf/Versions (GLIBC_2.21): New.
	* elf/dl-tls.c: Conditionally define GET_ADDR_FORWARD.
	(_dl_tls_get_addr_link_map): New function.
	* elf/tst-tls20.c: New file.
	* sysdeps/ia64/dl-tls.h (GET_ADDR_FORWARD): New.
  

On Sat, Oct 18, 2014 at 10:15:40PM +0200, Jan Kratochvil wrote:
> Hi,
> 
> https://sourceware.org/glibc/wiki/Tools%20Interface%20NG
> 	From within the inferior: Functions that are inserted into a process's
> 	address space by the debugger need to be able to access the TLS area
> 	for that thread. GDB can use DWARF to to find the offset within the
> 	TLS of a variable and pass that in as a compile time offset into the
> 	function being inserted into the traced process's address space.
> 	However, to be able to get to the address of the TLS area for
> 	a particular thread, it needs to know the module id to make a call to
> 	__tls_get_addr(). This module id is burried in a private area of the
> 	link_map data structure which is subject to change. It is therefore
> 	proposed that we add a new function to libthread_db which extracts the
> 	module id from the link map. 
> 
> Other option suggested by Tom Tromey is implemented by this patch:
> 
> * A new glibc function like __tls_get_addr that takes a link_map address
>   rather than a module id.

This sounds more appealing. I don't know how much interest you have in
non-glibc systems, but from our side with musl libc, we presently
don't have a libthread_db and don't plan to, so anything that makes
gdb work better without it (or even just preserves current
functionality without it) is preferable.

> I expect I will have to also implement this Tom Tromey's suggestion to GDB but
> I haven't tried to implement it yet:
> 
> * Bake more information about struct link_map into gdb (least preferred
>   IMO).

I think this is the worst option.

> Additionally one could also fetch the module id from R_X86_64_DTPMOD64 but that
> would require arch specific code in GDB.  I do not plan to implement this way.

I don't see how this would work; how would you request the dynamic
linker to resolve that for you?

Rich
  

On Sat, 18 Oct 2014 23:20:48 +0200, Rich Felker wrote:
> > Additionally one could also fetch the module id from R_X86_64_DTPMOD64 but that
> > would require arch specific code in GDB.  I do not plan to implement this way.
> 
> I don't see how this would work; how would you request the dynamic
> linker to resolve that for you?

When the inferior is running the relocation is already resolved.  So GDB can
just read the module id from relocation's location.  The problem is that
'.rela.dyn' is not parsed by BFD so one would either have to extend BFD to
parse it or provide a separate '.rela.dyn' parser in GDB for each arch.
From '.rela.dyn' one can find the symbol name and relocation's location.

Relocation section '.rela.dyn' at offset 0x520 contains 10 entries:
    Offset             Info             Type               Symbol's Value  Symbol's Name + Addend
0000000000200fd0  0000000a00000010 R_X86_64_DTPMOD64      0000000000000000 x + 0
0000000000200fd8  0000000a00000011 R_X86_64_DTPOFF64      0000000000000000 x + 0

 7ac:   66 48 8d 3d 1c 08 20    data32 lea 0x20081c(%rip),%rdi        # 200fd0 <_DYNAMIC+0x1d8>
 7b3:   00
 7b4:   66 66 48 e8 d4 fe ff    data32 data32 callq 690 <__tls_get_addr@plt>
 7bb:   ff

But that is an offtopic discussion for this patch.


Jan
  

On Sat, Oct 18, 2014 at 11:27:27PM +0200, Jan Kratochvil wrote:
> On Sat, 18 Oct 2014 23:20:48 +0200, Rich Felker wrote:
> > > Additionally one could also fetch the module id from R_X86_64_DTPMOD64 but that
> > > would require arch specific code in GDB.  I do not plan to implement this way.
> > 
> > I don't see how this would work; how would you request the dynamic
> > linker to resolve that for you?
> 
> When the inferior is running the relocation is already resolved.  So GDB can

This is only the case if the module actually has a relocation for
local TLS contained in it. I'm not sure how you could assume this is
always the case, but maybe in cases where it's not true there's no
reasonable way you'd want to access TLS in that way.

Rich
  

On Oct 18, 2014, Jan Kratochvil <jan.kratochvil@redhat.com> wrote:

> * A new glibc function like __tls_get_addr that takes a link_map address
>   rather than a module id.

dlinfo offers operations to both map a link_map (AKA dlopen handle) to
the modid, and to get the TLS base address for that link_map.  I suppose
calling dlinfo directly is not an option, since there's no guarantee
that libdl will have been linked in.

However, the implementat of dlinfo RTLD_DI_TLS_DATA relies on
_dl_tls_get_addr_soft, that not only takes a struct link_map*, but also
refrains from assigning a TLS segment for the module in the running
thread when one isn't allocated already.  I think this would be
preferrable behavior for a debugger, to avoid heisenbugs.

This symbol is not exported by ld.so, but this shouldn't stop GDB from
using it, since it is present in the symbol table as a local (hidden)
symbol.
  

On Thu, 23 Oct 2014 12:02:34 +0200, Alexandre Oliva wrote:
> dlinfo offers operations to both map a link_map (AKA dlopen handle) to
> the modid, and to get the TLS base address for that link_map.  I suppose
> calling dlinfo directly is not an option, since there's no guarantee
> that libdl will have been linked in.

Thanks for the dlinfo() information; but I agree libdl is not an option for
the "GDB JIT" purpose.


> However, the implementat of dlinfo RTLD_DI_TLS_DATA relies on
> _dl_tls_get_addr_soft, that not only takes a struct link_map*, but also
> refrains from assigning a TLS segment for the module in the running
> thread when one isn't allocated already.  I think this would be
> preferrable behavior for a debugger, to avoid heisenbugs.

OK, I will try to use _dl_tls_get_addr_soft for GDB as is.

Although I think the TLS segment allocation would be better for the "GDB JIT"
purpose.  Imagine an inferior shared library:
	__thread int tlsvar;
And user asking for:
(gdb) compile code tlsvar = 5;

I think it is better to do the TLS segment allocation than to refuse the
command to the user.  But someone may have an opposite opinion.


> This symbol is not exported by ld.so, but this shouldn't stop GDB from
> using it, since it is present in the symbol table as a local (hidden)
> symbol.

It is OK although when we talk about it it is not perfect.  Some distros strip
all shared libraries and then they have problem libthread_db cannot find
libpthread .symtab-only _thread_db_* symbols.  _dl_tls_get_addr_soft() will
have the same problem.  But that is mostly off-topic here now.


Thanks,
Jan
  

On 18 Oct 2014 22:15, Jan Kratochvil wrote:
> I have failed to test this patch on ia64 as I have available only RHEL-5 which
> has too old ld and then I gave it up on too old Linux kernel include files.

if access to an updated ia64 system is desired, i can make it happen.
$ /lib/libc.so.6.1 | head -1
GNU C Library (Gentoo 2.19-r1 p2) stable release version 2.19, by Roland McGrath et al.
$ ld --version | head -1
GNU ld (Gentoo 2.24 p1.4) 2.24
$ gcc --version | head -1
gcc (Gentoo 4.7.4 p1.1, pie-0.5.5) 4.7.4
$ uname -r -m
3.14.14-gentoo ia64
$ gdb --version | head -1
GNU gdb (Gentoo 7.6.2 p1) 7.6.2
-mike
  

On 10/18/2014 04:15 PM, Jan Kratochvil wrote:
> Other option suggested by Tom Tromey is implemented by this patch:
> 
> * A new glibc function like __tls_get_addr that takes a link_map address
>   rather than a module id.

I do not think this is a good solution.

If you use this function in jit'd code, you've now deviated from what normal
TLS-accessing sequences look like. Now your code deviates from the published
tls.pdf in the call sequence, and is undocumented, though simple. On top of
that it's a stop-gap measure until we fix things correctly. Because of these
things I object to adding this function to ld's API.

I don't like libthread_db either, but it avoids us having to put a stop-gap
API in ld. I say stop-gap because the real solution is going to be to use
python/DWARF, not any API in ld.

I'm in favour of exactly 3 things:

* New function in libthread_db.

* Heuristics in gdb if libthread_db is not new enough.

* A python or DWARF based parser to replace libthread_db.

Cheers,
Carlos.
  

On 10/24/2014 05:38 AM, Jan Kratochvil wrote:
> On Fri, 24 Oct 2014 04:21:02 +0200, Carlos O'Donell wrote:
>> On 10/18/2014 04:15 PM, Jan Kratochvil wrote:
>>> Other option suggested by Tom Tromey is implemented by this patch:
>>>
>>> * A new glibc function like __tls_get_addr that takes a link_map address
>>>   rather than a module id.
>>
>> I do not think this is a good solution.
>>
>> If you use this function in jit'd code, you've now deviated from what normal
>> TLS-accessing sequences look like.
> 
> Normal TLS-accessing sequences depend on R_X86_64_DTPMOD64 which the GDB JIT
> module cannot depend on.  Therefore it will be always deviated.

The only purpose of that relocation is to insert the module id once known,
thus gdb has to act like a dynamic loader in this respect. What's wrong with
using libthread_db to get the module ID and then you can call the normal TLS
functions instead of adding to ld's permanent and future ABI and API?

> TBH - a bit OT - the whole GDB JIT functionality has many arch specific issues
> and deviations due to the initial decision not to use dlopen() for the GDB JIT
> module because dlopen() may do some unexpected modifications of the inferior.
> I was proposing to simply use dlopen(), Tom Tromey required the mmap(), custom
> loading, custom relocations etc.  With dlopen() TLS would not be any issue.

I don't understand the tradeoffs, but if calling dlopen() in the inferior would
have made life easy, then I would have done that first, regardless of the impact
on the inferior. Only if users complained or found use cases where things broke
would I have fallen back on the "technical purist" solution involving doing
everything yourself. Those are decisions that you, as a gdb developer need to
make, or reevaluate and make different.

What I oppose is the addition to ld's ABI and API something which is not going
to be a permanent solution, and for which we can put a function somewhere
else and get similar results. The ABI and API for ld is forever, and I change
it only with extreme circumspection.
 
>> I don't like libthread_db either, but it avoids us having to put a stop-gap
>> API in ld. I say stop-gap because the real solution is going to be to use
>> python/DWARF, not any API in ld.
>>
>> I'm in favour of exactly 3 things:
>>
>> * New function in libthread_db.
>>
>> * Heuristics in gdb if libthread_db is not new enough.
>>
>> * A python or DWARF based parser to replace libthread_db.
> 
> I do not see how libthread_db or its equivalents could be applicable.

Isn't a solution to use libthread_db to get the module ID from the link_map,
then use that with normal __tls_get_addr instead of adding to ld's API?
Please correct me if I'm wrong.

> I have attached a use case for GDB JIT to make the problem clear.
> Equivalently 'tlsvar' could be placed into a shared library instead of the
> main executable.

OK.

> When the GDB JIT code needs to access 'tlsvar' it already runs independently
> from GDB.  And inferior does not (and as directed by Tom Tromey above it
> should not) dlopen() libthread_db on its own.

If gdb knows the link map for tlsvar it can use libthread_db to lookup the
module ID, and then when compiling code to access TLS it can call __tls_get_addr?

Is that not possible because of something in the JIT?

Cheers,
Carlos.
  

On Fri, 24 Oct 2014 16:22:29 +0200, Carlos O'Donell wrote:
> What's wrong with using libthread_db to get the module ID and then you can
> call the normal TLS functions
+
> Isn't a solution to use libthread_db to get the module ID from the link_map,
> then use that with normal __tls_get_addr instead of adding to ld's API?

I forgot about this possibility - that libthread_db would provide just the
module ID (and not some per-thread address).

Therefore going to post a libthread_db patch later, instead of using
_dl_tls_get_addr_soft() which would have one disadvantage I described before
to Alex.


> > TBH - a bit OT - the whole GDB JIT functionality has many arch specific issues
> > and deviations due to the initial decision not to use dlopen() for the GDB JIT
> > module because dlopen() may do some unexpected modifications of the inferior.
> > I was proposing to simply use dlopen(), Tom Tromey required the mmap(), custom
> > loading, custom relocations etc.  With dlopen() TLS would not be any issue.
> 
> I don't understand the tradeoffs, but if calling dlopen() in the inferior would
> have made life easy, then I would have done that first, regardless of the impact
> on the inferior. Only if users complained or found use cases where things broke

For GDB the feedback is usually scarce and skewed to drive design decisions.


> would I have fallen back on the "technical purist" solution involving doing
> everything yourself. Those are decisions that you, as a gdb developer need to
> make, or reevaluate and make different.

This decision has been already made by gdb developer Tom Tromey and all the
work (except this TLS issue) has been hopefully already solved now.  I find it
a bit late to throw it all alway and choose a different set of advantages and
disadvantages.


> What I oppose is the addition to ld's ABI and API something which is not going
> to be a permanent solution, and for which we can put a function somewhere
> else and get similar results.

IMO the dlopen() way could be the first simple solution, extending it
optionally only upon demand later.  Going now back to dlopen() seems a bit
backwards to me.


Thanks,
Jan
  

On 10/24/2014 10:40 AM, Jan Kratochvil wrote:
> On Fri, 24 Oct 2014 16:22:29 +0200, Carlos O'Donell wrote:
>> What's wrong with using libthread_db to get the module ID and then you can
>> call the normal TLS functions
> +
>> Isn't a solution to use libthread_db to get the module ID from the link_map,
>> then use that with normal __tls_get_addr instead of adding to ld's API?
> 
> I forgot about this possibility - that libthread_db would provide just the
> module ID (and not some per-thread address).
> 
> Therefore going to post a libthread_db patch later, instead of using
> _dl_tls_get_addr_soft() which would have one disadvantage I described before
> to Alex.
 
That sounds good to me. If you remember I promised you that I'd implement this,
but I haven't gotten to it yet. If you want to implement it, please go ahead,
otherwise I'll get to adding the function to libthread_db and testing. Right
now though I'm handling other deeper issues (dlopen recursion). So any help
you're willing to give is appreciated. I'm happy to review patches.
 
>> What I oppose is the addition to ld's ABI and API something which is not going
>> to be a permanent solution, and for which we can put a function somewhere
>> else and get similar results.
> 
> IMO the dlopen() way could be the first simple solution, extending it
> optionally only upon demand later.  Going now back to dlopen() seems a bit
> backwards to me.

I don't judge you. That is a decision the gdb community has to make. It sounds
like you've made it. I'm here to help :-)

Cheers,
Carlos.
  

On 10/24/2014 03:22 PM, Carlos O'Donell wrote:
> I don't understand the tradeoffs, but if calling dlopen() in the inferior would
> have made life easy, then I would have done that first, regardless of the impact
> on the inferior. Only if users complained or found use cases where things broke
> would I have fallen back on the "technical purist" solution involving doing
> everything yourself. Those are decisions that you, as a gdb developer need to
> make, or reevaluate and make different.

Off the top of my head, I'm sure there are more:

- The user might want to evaluate an expression while the program itself
  has just called dlopen and is now stopped inside it.  This pesky dlopen
  recursion thing.  ;-)  It's best if GDB only calls async-signal
  safe functions behind the scenes, if possible.  Of course if the
  injected expression involves calls to async-signal unsafe code that breaks
  the inferior, the user gets what she asked for.

- The program might have not been linked with -ldl.

- I suspect there may be issues with messing with symbol resolution
  and self library walks in the inferior too.  Not sure if RTLD_LOCAL is
  enough.  dlmopen might be a better fit, but hmm, that isn't very
  well supported in GDB/glibc.

- A lower level mechanism has much better changes of working on
  more targets and runtimes of languages other than C with minimal
  changes.

Thanks,
Pedro Alves
  

On Fri, Oct 24, 2014 at 04:40:14PM +0200, Jan Kratochvil wrote:
> On Fri, 24 Oct 2014 16:22:29 +0200, Carlos O'Donell wrote:
> > What's wrong with using libthread_db to get the module ID and then you can
> > call the normal TLS functions
> +
> > Isn't a solution to use libthread_db to get the module ID from the link_map,
> > then use that with normal __tls_get_addr instead of adding to ld's API?
> 
> I forgot about this possibility - that libthread_db would provide just the
> module ID (and not some per-thread address).

Why not just use dl_iterate_phdr to get the module id?

Rich
  

On Sat, 25 Oct 2014 07:55:16 +0200, Rich Felker wrote:
> On Fri, Oct 24, 2014 at 04:40:14PM +0200, Jan Kratochvil wrote:
> > I forgot about this possibility - that libthread_db would provide just the
> > module ID (and not some per-thread address).
> 
> Why not just use dl_iterate_phdr to get the module id?

I do not see where to find l_tls_modid in the returned data structures.

dl_iterate_phdr is in libc.so.6 (and not libdl.so.2) so that would be OK.


Thanks,
Jan
  

On Sat, Oct 25, 2014 at 08:14:00AM +0200, Jan Kratochvil wrote:
> On Sat, 25 Oct 2014 07:55:16 +0200, Rich Felker wrote:
> > On Fri, Oct 24, 2014 at 04:40:14PM +0200, Jan Kratochvil wrote:
> > > I forgot about this possibility - that libthread_db would provide just the
> > > module ID (and not some per-thread address).
> > 
> > Why not just use dl_iterate_phdr to get the module id?
> 
> I do not see where to find l_tls_modid in the returned data structures.
> 
> dl_iterate_phdr is in libc.so.6 (and not libdl.so.2) so that would be OK.

Oh, it looks like it's only available in the BSD version of on the
dl_phdr_info structure. Could we get it added to glibc? See:

http://www.freebsd.org/cgi/man.cgi?query=dl_iterate_phdr&sektion=3&apropos=0&manpath=FreeBSD%2B10.0-RELEASE

This is the version of the structure I accepted for inclusion in musl,
and I don't think I was even aware that glibc lacked these fields.

Rich