RFC: Don't output symbol version requirement for non-DT_NEEDED libs

Message ID 20141127081644.GA20383@bubble.grove.modra.org
State Not applicable
Headers

Commit Message

Alan Modra Nov. 27, 2014, 8:16 a.m. UTC
  This is a fix for
Inconsistency detected by ld.so: dl-version.c: 224: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
a situation that can arise with weak references to versioned symbols
in dependent libraries.

Consider this testcase, a main that calls a weak "f", linked against a
shared library liba.so that has a dependency on libb.so where "f" is
defined as a versioned symbol.  A later revision of liba.so defines
"f" itself, without the version, and doesn't link against libb.so.
A further revision of liba.so doesn't link against libb.so and also
doesn't define "f".

cat > lib.c <<EOF
int f (void) { return 1; }
EOF
cat > lib.map <<EOF
FOO { global: f; };
EOF
cat > main.c <<EOF
int f (void) __attribute__ ((weak));
int main (void) { return f ? f() : 0; }
EOF
cat > empty.c <<EOF
EOF
gcc -o libb.so -shared -fPIC -Wl,-soname,libb.so,--version-script,lib.map lib.c
gcc -o liba.so.1 -shared -fPIC -Wl,--no-as-needed,-soname,liba.so -L. -lb
gcc -o liba.so.2 -shared -fPIC -Wl,-soname,liba.so lib.c
gcc -o liba.so.3 -shared -fPIC -Wl,-soname,liba.so empty.c
ln -sfn liba.so.1 liba.so
gcc -o pr16452 -fPIC main.c -Wl,--no-as-needed,--rpath,. -L. -la
./pr16452
echo $?
ln -sfn liba.so.2 liba.so
./pr16452
echo $?
ln -sfn liba.so.3 liba.so
./pr16452
echo $?

Currently the script output is
1
Inconsistency detected by ld.so: dl-version.c: 224: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
127
Inconsistency detected by ld.so: dl-version.c: 224: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
127

The problem here is that since binutils-2.22 we've defaulted to
--no-copy-dt-needed-entries (which was a good change despite the
number of bugreports it has generated).  So libb.so is not listed as
DT_NEEDED in the executable.  However, a version FOO from libb.so *is*
added to the executable, specifying that it came from libb.so.  When
we use rev 2 or 3 of liba.so, no libb.so is loaded and the version
check fails to find the required library.

You could argue that it's wrong for ld.so to complain about a weak
symbol version, but the ld.so check isn't being done against symbols.
ld.so is looking at .gnu.version_r, the version requirements
themselves, each possibly used by multiple symbols..

You might also argue that it's a user problem that f lost its version
in rev 2 of the testcase.  However this might be because liba.so is a
user library without any versioning and in rev 2, liba.so was linked
against a static libb.  (Typical libb in real life is libpthread.)

So, absent someone implementing a glibc fix, how about we just drop
the symbol versioning for weak symbols, when their defining library
won't be in DT_NEEDED?  Note that if "f" above was a strong symbol,
ld will still complain with "./libb.so: error adding symbols: DSO
missing from command line".

	PR 16452
	* elflink.c (_bfd_elf_link_find_version_dependencies): Exclude
	symbols from libraries that won't be listed in DT_NEEDED.
	(elf_link_output_extsym): Don't output verdefs for such symbols.
  

Comments

H.J. Lu Nov. 27, 2014, 3:56 p.m. UTC | #1
On Thu, Nov 27, 2014 at 12:16 AM, Alan Modra <amodra@gmail.com> wrote:
> This is a fix for
> Inconsistency detected by ld.so: dl-version.c: 224: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
> a situation that can arise with weak references to versioned symbols
> in dependent libraries.
>
> Consider this testcase, a main that calls a weak "f", linked against a
> shared library liba.so that has a dependency on libb.so where "f" is
> defined as a versioned symbol.  A later revision of liba.so defines
> "f" itself, without the version, and doesn't link against libb.so.
> A further revision of liba.so doesn't link against libb.so and also
> doesn't define "f".
>
> cat > lib.c <<EOF
> int f (void) { return 1; }
> EOF
> cat > lib.map <<EOF
> FOO { global: f; };
> EOF
> cat > main.c <<EOF
> int f (void) __attribute__ ((weak));
> int main (void) { return f ? f() : 0; }
> EOF
> cat > empty.c <<EOF
> EOF
> gcc -o libb.so -shared -fPIC -Wl,-soname,libb.so,--version-script,lib.map lib.c
> gcc -o liba.so.1 -shared -fPIC -Wl,--no-as-needed,-soname,liba.so -L. -lb
> gcc -o liba.so.2 -shared -fPIC -Wl,-soname,liba.so lib.c
> gcc -o liba.so.3 -shared -fPIC -Wl,-soname,liba.so empty.c
> ln -sfn liba.so.1 liba.so
> gcc -o pr16452 -fPIC main.c -Wl,--no-as-needed,--rpath,. -L. -la
> ./pr16452
> echo $?
> ln -sfn liba.so.2 liba.so
> ./pr16452
> echo $?
> ln -sfn liba.so.3 liba.so
> ./pr16452
> echo $?
>
> Currently the script output is
> 1
> Inconsistency detected by ld.so: dl-version.c: 224: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
> 127
> Inconsistency detected by ld.so: dl-version.c: 224: _dl_check_map_versions: Assertion `needed != ((void *)0)' failed!
> 127
>
> The problem here is that since binutils-2.22 we've defaulted to
> --no-copy-dt-needed-entries (which was a good change despite the
> number of bugreports it has generated).  So libb.so is not listed as
> DT_NEEDED in the executable.  However, a version FOO from libb.so *is*
> added to the executable, specifying that it came from libb.so.  When
> we use rev 2 or 3 of liba.so, no libb.so is loaded and the version
> check fails to find the required library.
>
> You could argue that it's wrong for ld.so to complain about a weak
> symbol version, but the ld.so check isn't being done against symbols.
> ld.so is looking at .gnu.version_r, the version requirements
> themselves, each possibly used by multiple symbols..
>
> You might also argue that it's a user problem that f lost its version
> in rev 2 of the testcase.  However this might be because liba.so is a
> user library without any versioning and in rev 2, liba.so was linked
> against a static libb.  (Typical libb in real life is libpthread.)
>
> So, absent someone implementing a glibc fix, how about we just drop
> the symbol versioning for weak symbols, when their defining library
> won't be in DT_NEEDED?  Note that if "f" above was a strong symbol,
> ld will still complain with "./libb.so: error adding symbols: DSO
> missing from command line".
>
>         PR 16452
>         * elflink.c (_bfd_elf_link_find_version_dependencies): Exclude
>         symbols from libraries that won't be listed in DT_NEEDED.
>         (elf_link_output_extsym): Don't output verdefs for such symbols.
>

Such a change needs a testcase to show what it does and make
sure that there is no regression in the future.
  
Carlos O'Donell Nov. 27, 2014, 4:25 p.m. UTC | #2
On 11/27/2014 03:16 AM, Alan Modra wrote:
> So, absent someone implementing a glibc fix, how about we just drop
> the symbol versioning for weak symbols, when their defining library
> won't be in DT_NEEDED?  Note that if "f" above was a strong symbol,
> ld will still complain with "./libb.so: error adding symbols: DSO
> missing from command line".

This seems like the wrong thing to do, particularly since it violates
the principle of least surprise. I would expect the versioned symbol
to stay versioned.

What's wrong with fixing this in glibc?

Cheers,
Carlos.
  
Alan Modra Nov. 27, 2014, 10:50 p.m. UTC | #3
On Thu, Nov 27, 2014 at 11:25:13AM -0500, Carlos O'Donell wrote:
> On 11/27/2014 03:16 AM, Alan Modra wrote:
> > So, absent someone implementing a glibc fix, how about we just drop
> > the symbol versioning for weak symbols, when their defining library
> > won't be in DT_NEEDED?  Note that if "f" above was a strong symbol,
> > ld will still complain with "./libb.so: error adding symbols: DSO
> > missing from command line".
> 
> This seems like the wrong thing to do, particularly since it violates
> the principle of least surprise. I would expect the versioned symbol
> to stay versioned.
> 
> What's wrong with fixing this in glibc?

That would be best.  Can I reassign the bug to you?  :)
  
Dmitry V. Levin Nov. 27, 2014, 11:42 p.m. UTC | #4
On Thu, Nov 27, 2014 at 11:25:13AM -0500, Carlos O'Donell wrote:
> On 11/27/2014 03:16 AM, Alan Modra wrote:
> > So, absent someone implementing a glibc fix, how about we just drop
> > the symbol versioning for weak symbols, when their defining library
> > won't be in DT_NEEDED?  Note that if "f" above was a strong symbol,
> > ld will still complain with "./libb.so: error adding symbols: DSO
> > missing from command line".
> 
> This seems like the wrong thing to do, particularly since it violates
> the principle of least surprise. I would expect the versioned symbol
> to stay versioned.
> 
> What's wrong with fixing this in glibc?

A few words about compatibility: ld with the proposed patch applied
would behave the same way as gold.
  
Joseph Myers Nov. 28, 2014, 12:06 a.m. UTC | #5
On Thu, 27 Nov 2014, Carlos O'Donell wrote:

> On 11/27/2014 03:16 AM, Alan Modra wrote:
> > So, absent someone implementing a glibc fix, how about we just drop
> > the symbol versioning for weak symbols, when their defining library
> > won't be in DT_NEEDED?  Note that if "f" above was a strong symbol,
> > ld will still complain with "./libb.so: error adding symbols: DSO
> > missing from command line".
> 
> This seems like the wrong thing to do, particularly since it violates
> the principle of least surprise. I would expect the versioned symbol
> to stay versioned.
> 
> What's wrong with fixing this in glibc?

Actually, I think it's a linker bug not a glibc bug.  If you don't link 
with a library providing a symbol you use, I don't think any information 
at all about how it might be resolved with some library you didn't link 
against should be embedded in the binary: not a DT_NEEDED entry, and not a 
version requirement.  I don't think you can presume at static link time, 
with a weak undefined symbol like that, "this symbol isn't needed, but if 
defined at runtime it must have this version" (as opposed to "this symbol 
isn't needed, and might have any version at runtime", which is the safe 
assumption).
  
Alan Modra Nov. 28, 2014, 1:50 a.m. UTC | #6
On Fri, Nov 28, 2014 at 12:06:09AM +0000, Joseph Myers wrote:
> On Thu, 27 Nov 2014, Carlos O'Donell wrote:
> 
> > On 11/27/2014 03:16 AM, Alan Modra wrote:
> > > So, absent someone implementing a glibc fix, how about we just drop
> > > the symbol versioning for weak symbols, when their defining library
> > > won't be in DT_NEEDED?  Note that if "f" above was a strong symbol,
> > > ld will still complain with "./libb.so: error adding symbols: DSO
> > > missing from command line".
> > 
> > This seems like the wrong thing to do, particularly since it violates
> > the principle of least surprise. I would expect the versioned symbol
> > to stay versioned.
> > 
> > What's wrong with fixing this in glibc?
> 
> Actually, I think it's a linker bug not a glibc bug.  If you don't link 
> with a library providing a symbol you use, I don't think any information 
> at all about how it might be resolved with some library you didn't link 
> against should be embedded in the binary: not a DT_NEEDED entry, and not a 
> version requirement.  I don't think you can presume at static link time, 
> with a weak undefined symbol like that, "this symbol isn't needed, but if 
> defined at runtime it must have this version" (as opposed to "this symbol 
> isn't needed, and might have any version at runtime", which is the safe 
> assumption).

This is a quibble, but in this particular case we're talking about
a weak *defined* symbol with versioning.  The definition came from a
library dependency that the user didn't specify on the command line,
which is why I call this a quibble because arguably the symbol should
be undefined.  (BFD ld looks at library DT_NEEDED and loads those
libraries.  BFD ld has done that for a long time, but gold doesn't.
That's why the testcase I posted works with gold; With gold the symbol
is left undefined and therefore unversioned.)
  
Carlos O'Donell Nov. 28, 2014, 3:27 p.m. UTC | #7
On 11/27/2014 07:06 PM, Joseph Myers wrote:
> On Thu, 27 Nov 2014, Carlos O'Donell wrote:
> 
>> On 11/27/2014 03:16 AM, Alan Modra wrote:
>>> So, absent someone implementing a glibc fix, how about we just drop
>>> the symbol versioning for weak symbols, when their defining library
>>> won't be in DT_NEEDED?  Note that if "f" above was a strong symbol,
>>> ld will still complain with "./libb.so: error adding symbols: DSO
>>> missing from command line".
>>
>> This seems like the wrong thing to do, particularly since it violates
>> the principle of least surprise. I would expect the versioned symbol
>> to stay versioned.
>>
>> What's wrong with fixing this in glibc?
> 
> Actually, I think it's a linker bug not a glibc bug.  If you don't link 
> with a library providing a symbol you use, I don't think any information 
> at all about how it might be resolved with some library you didn't link 
> against should be embedded in the binary: not a DT_NEEDED entry, and not a 
> version requirement.  I don't think you can presume at static link time, 
> with a weak undefined symbol like that, "this symbol isn't needed, but if 
> defined at runtime it must have this version" (as opposed to "this symbol 
> isn't needed, and might have any version at runtime", which is the safe 
> assumption).

I had not considered it like that. I agree with your rationale.

In which case this is clearly a bug in binutils and Alan's patch
is correct.

I tried to come up with a case where this would matter, but from
first principles I couldn't construct any sensible test cases.

Cheers,
Carlos.
  
Alan Modra Dec. 1, 2014, 3:50 a.m. UTC | #8
On Fri, Nov 28, 2014 at 10:27:26AM -0500, Carlos O'Donell wrote:
> On 11/27/2014 07:06 PM, Joseph Myers wrote:
> > Actually, I think it's a linker bug not a glibc bug.  If you don't link 
> > with a library providing a symbol you use, I don't think any information 
> > at all about how it might be resolved with some library you didn't link 
> > against should be embedded in the binary: not a DT_NEEDED entry, and not a 
> > version requirement.  I don't think you can presume at static link time, 
> > with a weak undefined symbol like that, "this symbol isn't needed, but if 
> > defined at runtime it must have this version" (as opposed to "this symbol 
> > isn't needed, and might have any version at runtime", which is the safe 
> > assumption).
> 
> I had not considered it like that. I agree with your rationale.
> 
> In which case this is clearly a bug in binutils and Alan's patch
> is correct.
> 
> I tried to come up with a case where this would matter, but from
> first principles I couldn't construct any sensible test cases.

If the version *does* matter, then mentioning on the command line
the library that defines the symbol will result in both the symbol
being versioned and the library in DT_NEEDED.

I've gone ahead and committed the linker patch, to 2.25 too.  Thanks
to all who contributed to this thread.
  

Patch

diff --git a/bfd/elflink.c b/bfd/elflink.c
index c8068c0..c964a98 100644
--- a/bfd/elflink.c
+++ b/bfd/elflink.c
@@ -1832,7 +1832,9 @@  _bfd_elf_link_find_version_dependencies (struct elf_link_hash_entry *h,
   if (!h->def_dynamic
       || h->def_regular
       || h->dynindx == -1
-      || h->verinfo.verdef == NULL)
+      || h->verinfo.verdef == NULL
+      || (elf_dyn_lib_class (h->verinfo.verdef->vd_bfd)
+	  & (DYN_AS_NEEDED | DYN_DT_NEEDED | DYN_NO_NEEDED)))
     return TRUE;
 
   /* See if we already know about this version.  */
@@ -9050,7 +9052,9 @@  elf_link_output_extsym (struct bfd_hash_entry *bh, void *data)
 
 	  if (!h->def_regular)
 	    {
-	      if (h->verinfo.verdef == NULL)
+	      if (h->verinfo.verdef == NULL
+		  || (elf_dyn_lib_class (h->verinfo.verdef->vd_bfd)
+		      & (DYN_AS_NEEDED | DYN_DT_NEEDED | DYN_NO_NEEDED)))
 		iversym.vs_vers = 0;
 	      else
 		iversym.vs_vers = h->verinfo.verdef->vd_exp_refno + 1;