[RFC,2/2] Update prototype of IFUNC resolver for MIPS

  Comments are invited on the prototype of the IFUNC resolver for MIPS. We want
something flexible enough to handle current requirements and extensions.

This patch, from RobertS, provides the resolver with 2 HWCAP registers and a
callback control function. The callback currently provides the ability to
toggle/switch between FP ABI modes.

Caveat: specific HWCAP values for MIPS are not in place yet.

ChangeLog:

* sysdeps/mips/Makefile
	(sysdep_headers): Add sys/dlifnctl.h

* sysdeps/mips/dl-ifunc-ctrl.h: New file.
	(dl_ifunc_control): New function.

* sysdeps/mips/dl-irel.h
	Include dl-ifunc-ctrl.h
	Change resolver prototype to use 2 HWCAPs and dl_ifunc_control
	callback.

* sysdeps/mips/dl-machine-reject-phdr.h
	(REJECT): Replace _dl_debug_printf with GLRO(dl_debug_printf)
	(cached_fpabi_reject_phdr_p): Restricted to SHARED build.
	(ifunc_fpabi): New variable, track the first fpabi request.
	(dl_reject_fpabi_req): New function with using compat logic from
	elf_machine_reject_phdr_p.
	(elf_machine_reject_phdr_p): Refactored to call
	dl_reject_fpabi_req; all logic related to compatibility check
	moved to dl_reject_fpabi_req.  Replace _dl_debug_printf with
	GLRO(dl_debug_printf).

* sysdeps/mips/sys/dlifnctl.h
	(R_MIPS_IRELATIVE): New file.
	(DL_IFUNC_MIPS_FR_MODE_SWITCH DL_IFUNC_MIPS_FP_ABI): New operation
	request modes for IFUNC control callback.
---
 sysdeps/mips/Makefile                 |    2 +-
 sysdeps/mips/dl-ifunc-ctrl.h          |   90 ++++++++++++++++++++++
 sysdeps/mips/dl-irel.h                |    5 +-
 sysdeps/mips/dl-machine-reject-phdr.h |  131 +++++++++++++++++++++++----------
 sysdeps/mips/sys/dlifnctl.h           |   26 +++++++
 5 files changed, 214 insertions(+), 40 deletions(-)
 create mode 100644 sysdeps/mips/dl-ifunc-ctrl.h
 create mode 100644 sysdeps/mips/sys/dlifnctl.h
  

On Thu, 3 Sep 2015, Faraz Shahbazker wrote:

> Comments are invited on the prototype of the IFUNC resolver for MIPS. We want
> something flexible enough to handle current requirements and extensions.
> 
> This patch, from RobertS, provides the resolver with 2 HWCAP registers and a
> callback control function. The callback currently provides the ability to
> toggle/switch between FP ABI modes.

Please give a much more detailed explanation of the background to and 
motivation for this patch - why you need something that doesn't appear to 
be present for any other architecture.

>  sysdep_headers += regdef.h fpregdef.h sys/regdef.h sys/fpregdef.h \
> -		  sys/asm.h sgidefs.h
> +		  sys/asm.h sys/dlifnctl.h sgidefs.h

If you're adding a new API through a public header, it needs documenting 
in the user manual.
  

On 03/09/15 23:03, Faraz Shahbazker wrote:
> Comments are invited on the prototype of the IFUNC resolver for MIPS. We want
> something flexible enough to handle current requirements and extensions.
>
> This patch, from RobertS, provides the resolver with 2 HWCAP registers and a
> callback control function. The callback currently provides the ability to
> toggle/switch between FP ABI modes.
...
> @@ -43,7 +44,9 @@ elf_ifunc_invoke (ElfW(Addr) addr)
>                               (unsigned long int)t_addr);
>       }
>
> -  return ((ElfW(Addr) (*) (unsigned long int)) (addr)) (GLRO(dl_hwcap));
> +  return ((ElfW(Addr) (*) (unsigned long int, unsigned long int,
> +                          int (int, int))) addr)
> +    (GLRO(dl_hwcap), GLRO(dl_hwcap2), dl_ifunc_control);
>   }
>

the only ifunc documentation i know of says

"The calling convention of the STT_GNU_IFUNC function, which takes
no arguments and returns a function pointer, should follow the
processor-specific ABI."
(ifunc.txt at https://sites.google.com/site/x32abi/documents )

this is for x86 and wrong for other archs.

i think the interface contracts of the resolver should be
documented if ifunc is used outside the libc.

 > +static int __attribute_used__
 > +dl_ifunc_control (int operation, int value)
 > +{
...
 > +          /* Remember the choice for the last compabitle FP ABI.  */
 > +          ifunc_fpabi = in_abi;
...
> +/* FP ABI requirement for ifunc with callback.  The new mode switch can only
> +   be requested once.  */
> +
> +static int ifunc_fpabi = -1;
> +

i think the resolver needs to be as-safe, so modified
objects should be volatile (sig_atomic_t).

the ifunc situation is quite messy, is it supposed to be
used outside of the libc? (i know on x86, gcc target libs
and function multi-versioning use it, but i'd assume other
targets would move away from it outside the libc).
  

On Fri, Sep 04, 2015 at 02:17:56PM +0100, Szabolcs Nagy wrote:
> On 03/09/15 23:03, Faraz Shahbazker wrote:
> > +static int __attribute_used__
> > +dl_ifunc_control (int operation, int value)
> > +{
> ...
> > +          /* Remember the choice for the last compabitle FP ABI.  */
> > +          ifunc_fpabi = in_abi;
> ...

also typo here

> >+/* FP ABI requirement for ifunc with callback.  The new mode switch can only
> >+   be requested once.  */
> >+
> >+static int ifunc_fpabi = -1;
> >+
> 
> i think the resolver needs to be as-safe, so modified
> objects should be volatile (sig_atomic_t).
> 
We talked about this at cauldron. This is collorary of my question if
resolver is ok. We use recursive lock around that so I asked if its ok
when singal arrives while we hold it. Carlos said that we should check
all architectures if everything stays consistent. So ifunc should also
do same.

There is also question if we should resolve ifuncs early like with
LD_BIND_NOW=1. That we could already do eager binding implies that ifunc
should return same result while application runs.

> the ifunc situation is quite messy, is it supposed to be
> used outside of the libc? (i know on x86, gcc target libs
> and function multi-versioning use it, but i'd assume other
> targets would move away from it outside the libc).

Yes, its public but there isn't clear whats allowed and what isn't.
  

Joseph Myers <joseph@codesourcery.com> writes:
> On Thu, 3 Sep 2015, Faraz Shahbazker wrote:
> 
> > Comments are invited on the prototype of the IFUNC resolver for MIPS.
> > We want something flexible enough to handle current requirements and
> extensions.
> >
> > This patch, from RobertS, provides the resolver with 2 HWCAP registers
> > and a callback control function. The callback currently provides the
> > ability to toggle/switch between FP ABI modes.
> 
> Please give a much more detailed explanation of the background to and
> motivation for this patch - why you need something that doesn't appear
> to be present for any other architecture.

I don't think there is much to debate over whether MIPS ifunc resolvers
take the hwcaps as arguments. That is common with other architectures
and it seems sensible to pass hwcap and hwcap2 for MIPS given we have
such wide variation in hardware. We will be populating hwcaps as soon
as possible for those who know we don't currently define any.

The unusual extension is the idea of a callback into the dynamic linker.
The origin of this comes from the MIPS FR0/FR1 interlinking plan which is
frequently referred to as the 'FPXX' ABI extension. The specific issue
is whether or not o32 software can rely on FR=1 mode i.e. FP64. There is
one sure fire way to do this and that is to build your module as FP64
using -mfp64 and the rest is already taken care of. However, in an ifunc
world we need the object(s) that contain ifunc resolvers and routines to
be marked with as few requirements as possible so that they will load on
the oldest/least feature-full hardware we want to support. This means
that we get no guarantee of what hardware mode is set nor whether the
mode is guaranteed not to be changed.

The primary reason for needing FR=1 is for MSA to be usable. Simply
seeing that MSA is available via a HWCAP is not enough as we have to
tell the dynamic linker to lock itself into FR=1/FP64 mode while the
ifunc's module is loaded. In fact it may not even be possible to enter
FR=1 mode because of the modules which are already loaded.

The callback interface is proposed as relatively generic to support
making different kinds of requests. The one to support MIPS FP modes is
to basically request that the current module has its FP ABI upgraded
to a specific value and report if that succeeds. I haven't reviewed the
detailed implementation of this yet but that is the general goal.

I hope that at least covers the basics of why this feature is being
proposed.

Thanks,
Matthew
  

Ondřej Bílka <neleai@seznam.cz>
> > i think the resolver needs to be as-safe, so modified objects should
> > be volatile (sig_atomic_t).
> >
> We talked about this at cauldron. This is collorary of my question if
> resolver is ok. We use recursive lock around that so I asked if its ok
> when singal arrives while we hold it. Carlos said that we should check
> all architectures if everything stays consistent. So ifunc should also
> do same.
> 
> There is also question if we should resolve ifuncs early like with
> LD_BIND_NOW=1. That we could already do eager binding implies that ifunc
> should return same result while application runs.

I'm not sure I see any specific need for the outcome of an ifunc to be the
same when resolved early/late. Apart from the resolved function needing to
be at least usable I don't think there has to be any further requirements. 

> > the ifunc situation is quite messy, is it supposed to be used outside
> > of the libc? (i know on x86, gcc target libs and function
> > multi-versioning use it, but i'd assume other targets would move away
> > from it outside the libc).
> 
> Yes, its public but there isn't clear whats allowed and what isn't.

Are you asking whether ifunc should be usable by an ordinary programmer in
their library? I'd say that it absolutely should be usable. We have some
syntactic sugar in GCC to simplify defining ifuncs and for those who do
then they need to follow a few rules about what can and cannot be done in
a resolver. You could go as far as to say that the idea proposed in this
patch could be the only way for an ifunc resolver to do anything other than
plain C code. i.e. no library calls allowed but some services are exposed
via a dynamic linker callback.

Thanks,
Matthew
  

On Mon, 14 Sep 2015, Matthew Fortune wrote:

> The primary reason for needing FR=1 is for MSA to be usable. Simply
> seeing that MSA is available via a HWCAP is not enough as we have to
> tell the dynamic linker to lock itself into FR=1/FP64 mode while the
> ifunc's module is loaded. In fact it may not even be possible to enter
> FR=1 mode because of the modules which are already loaded.

And it's not possible to have a pseudo-HWCAP bit (with a reservation so 
the kernel won't allocate it) that means "can use FP64 mode", or the IFUNC 
can't switch to FP64 mode using existing public interfaces so having such 
a bit wouldn't be sufficient?

I think the "GNU ifunc - work in progress please ignore" sections of 
<https://dmz-portal.mips.com/wiki/MIPS_O32_ABI_-_FR0_and_FR1_Interlinking> 
need expanding to explain the proposed ABI here.
  

On 14/09/15 15:33, Matthew Fortune wrote:
> Ondřej Bílka <neleai@seznam.cz>
>>> the ifunc situation is quite messy, is it supposed to be used outside
>>> of the libc? (i know on x86, gcc target libs and function
>>> multi-versioning use it, but i'd assume other targets would move away
>>> from it outside the libc).
>>
>> Yes, its public but there isn't clear whats allowed and what isn't.
>
> Are you asking whether ifunc should be usable by an ordinary programmer in
> their library? I'd say that it absolutely should be usable. We have some

yes

> syntactic sugar in GCC to simplify defining ifuncs and for those who do
> then they need to follow a few rules about what can and cannot be done in

i think the rules currently depend on dynamic
linker internals and are target specific.
(which is ok if ifunc is only used in the libc
but not ok if users write ifunc resolvers).

e.g. i don't see the fp registers saved/restored
in mips _dl_runtime_resolve, so the ifunc resolver
must not clobber fp (argument) regs.
(on x86 an ifunc resolver can clobber fp regs.)

> a resolver. You could go as far as to say that the idea proposed in this
> patch could be the only way for an ifunc resolver to do anything other than
> plain C code. i.e. no library calls allowed but some services are exposed
> via a dynamic linker callback.
>

if no library calls are allowed from a resolver
then the ifunc dispatch mechanism is limited,
if some libc calls are allowed then those should
be documented.

(some libc calls may be generated by the compiler
e.g. memcpy)

> Thanks,
> Matthew
>
  

On Mon, Sep 14, 2015 at 02:33:47PM +0000, Matthew Fortune wrote:
> Ondřej Bílka <neleai@seznam.cz>
> > > i think the resolver needs to be as-safe, so modified objects should
> > > be volatile (sig_atomic_t).
> > >
> > We talked about this at cauldron. This is collorary of my question if
> > resolver is ok. We use recursive lock around that so I asked if its ok
> > when singal arrives while we hold it. Carlos said that we should check
> > all architectures if everything stays consistent. So ifunc should also
> > do same.
> > 
> > There is also question if we should resolve ifuncs early like with
> > LD_BIND_NOW=1. That we could already do eager binding implies that ifunc
> > should return same result while application runs.
> 
> I'm not sure I see any specific need for the outcome of an ifunc to be the
> same when resolved early/late. Apart from the resolved function needing to
> be at least usable I don't think there has to be any further requirements. 
>
While not I am saying that user shouldn't rely that ifunc would be
resolved late. 
 
> > > the ifunc situation is quite messy, is it supposed to be used outside
> > > of the libc? (i know on x86, gcc target libs and function
> > > multi-versioning use it, but i'd assume other targets would move away
> > > from it outside the libc).
> > 
> > Yes, its public but there isn't clear whats allowed and what isn't.
> 
> Are you asking whether ifunc should be usable by an ordinary programmer in
> their library? 

I am not saying that, I am saying that currently one frequently finds
situations where it isnt clear if its user fault or our. For example I
wrote sample ifunc for shared library, it got seqfault and it turned out
it was because function that I returned was public and not resolved yet.

> I'd say that it absolutely should be usable. We have some
> syntactic sugar in GCC to simplify defining ifuncs and for those who do
> then they need to follow a few rules about what can and cannot be done in
> a resolver. You could go as far as to say that the idea proposed in this
> patch could be the only way for an ifunc resolver to do anything other than
> plain C code. i.e. no library calls allowed but some services are exposed
> via a dynamic linker callback.
>
I would disagree with that for two reason, first is that no library
calls limit usability. If one wants to select function based on profile
data he needs to mmap file with data or so.

Second is that ifunc should be implementation detail rather than
interface. I have on my todo list to add multiple cpu dispatch for gcc
and there are several ways to do it. I would like more fat binaries as
there is problem that to inline ifunc you need to also turn callers into
ifunc so gcc would need to understand resolver.

Other alternative that uses ifunc would add gcc attribute that tells gcc
to make variants of function for each arch, that could be now done with
macro that defines resolver, for variants uses __attribute__((target("foo"))) 
and always inline original function.