RFC: Provide a function to reset IFUNC PLTs

  Some projects snapshot+restore process images to migrate them to
a different machine. The target machine may have different (particularly
lower) set of CPU features. Restored process does crash in glibc IFUNC
functions which have been already set in PLTs due to the former more
rich CPU features on the snapshotting machine.

Providing a glibc function which can be called during the restore.

I understand the code may need more adjustments before its upstreaming
but is this an acceptable approach?

An example of the problem reproducibility with: http://criu.org

int main(void) {
  setbuf(stdout, NULL);
  for (;;) {
    time_t t = time(NULL);
    fputs(ctime(&t), stdout);
    sleep(strcmp("hello", "world") != 0);
  }
}

newcpu$ gcc -o strcmper strcmper.c -Wall -g
newcpu$ rm -rf dir;mkdir dir;./strcmper &p=$!;sleep 1;sudo criu dump --shell-job -t $p -D dir
oldcpu$ sudo criu restore --shell-job -D dir
Segmentation fault

newcpu$ gcc -o strcmper strcmper.c -Wall -g glibc/ld-linux-x86-64.so.2 glibc/libc.so.6 # patched files are in glibc/
newcpu$ rm -rf dir;mkdir dir;LD_LIBRARY_PATH=glibc ./glibc/ld-linux-x86-64.so.2 ./strcmper &p=$!;sleep 1;sudo criu dump --shell-job -t $p -D dir
oldcpu$ sudo criu restore --shell-job -D dir
running...
---
 elf/Makefile                       |   2 +
 elf/Versions                       |   9 +
 elf/dl-reloc.c                     |   2 +
 elf/dl-reset-ifunc.c               | 392 +++++++++++++++++++++++++++++
 elf/dl-tunables.c                  |  14 ++
 elf/dl-tunables.h                  |   4 +
 elf/link.h                         |   2 +
 elf/tst-reset-ifunc.c              |  70 ++++++
 sysdeps/generic/ldsodefs.h         |   4 +-
 sysdeps/x86/dl-get-cpu-features.c  |   1 +
 sysdeps/x86/include/cpu-features.h |   3 +-
 11 files changed, 500 insertions(+), 3 deletions(-)
 create mode 100644 elf/dl-reset-ifunc.c
 create mode 100644 elf/tst-reset-ifunc.c
  

* Jan Kratochvil via Libc-alpha:

> Some projects snapshot+restore process images to migrate them to
> a different machine. The target machine may have different (particularly
> lower) set of CPU features. Restored process does crash in glibc IFUNC
> functions which have been already set in PLTs due to the former more
> rich CPU features on the snapshotting machine.
>
> Providing a glibc function which can be called during the restore.
>
> I understand the code may need more adjustments before its upstreaming
> but is this an acceptable approach?

Do you have a high-level overview how this is supposed to work?  Is CRIU
expected to call the new _dl_reset_ifunc symbol after restore?

Use of <sys/platform/x86.h> is somewhat rare.  Not even GCC uses it
AFAIK.  It has its own cached CPU data used for target clones and
similar features.  Re-running those IFUNC resolvers will just give the
same results.

Distributions have started to deploy alternate builds in
glibc-hwcaps/x86-64-v3 directories:

  openSUSE Tumbleweed gains optional x86-64-v3 optimization
  <https://news.opensuse.org/2023/03/02/tw-gains-optional-optimizations/>

This might be a common situation fairly soon.  This is not IFUNC-based
at all, so any IFUNC-based approach is not going to help with that.

Practically speaking, I think cluster heterogeneity needs to be hidden
at the cluster level.

Thanks,
Florian
  

On 07/03/23 05:40, Florian Weimer via Libc-alpha wrote:
> * Jan Kratochvil via Libc-alpha:
> 
>> Some projects snapshot+restore process images to migrate them to
>> a different machine. The target machine may have different (particularly
>> lower) set of CPU features. Restored process does crash in glibc IFUNC
>> functions which have been already set in PLTs due to the former more
>> rich CPU features on the snapshotting machine.
>>
>> Providing a glibc function which can be called during the restore.
>>
>> I understand the code may need more adjustments before its upstreaming
>> but is this an acceptable approach?
> 
> Do you have a high-level overview how this is supposed to work?  Is CRIU
> expected to call the new _dl_reset_ifunc symbol after restore?
> 
> Use of <sys/platform/x86.h> is somewhat rare.  Not even GCC uses it
> AFAIK.  It has its own cached CPU data used for target clones and
> similar features.  Re-running those IFUNC resolvers will just give the
> same results.

I am not sure if the ifunc reset will be really safe without adding CRIU
migrate sync points, to avoid suspend execution in a context that the
ifunc variants are already being executed or its address is being in a
function point (for instance in PLT code).

Besides, I also not sure if adding way to remove RELRO protection won't
add more security issues (we can disable for sesuid binaries, but even
though it is not a good security practice).

> 
> Distributions have started to deploy alternate builds in
> glibc-hwcaps/x86-64-v3 directories:
> 
>   openSUSE Tumbleweed gains optional x86-64-v3 optimization
>   <https://news.opensuse.org/2023/03/02/tw-gains-optional-optimizations/>
> 
> This might be a common situation fairly soon.  This is not IFUNC-based
> at all, so any IFUNC-based approach is not going to help with that.
> 
> Practically speaking, I think cluster heterogeneity needs to be hidden
> at the cluster level.


For glibc standpoint I think heterogeneity should be handled by masking
off the higher cpu features sets so the programs always run with the 
lowest ifunc variants set.
  

On Tue, 07 Mar 2023 21:07:58 +0800, Adhemerval Zanella Netto wrote:
> I am not sure if the ifunc reset will be really safe without adding CRIU
> migrate sync points, to avoid suspend execution in a context that the
> ifunc variants are already being executed or its address is being in a
> function point (for instance in PLT code).

You are right but I left the thread safety up to the caller ("Freezer"):
	https://github.com/openjdk/crac/pull/41/files#diff-aeec57d804d56002f26a85359fc4ac8b48cfc249d57c656a30a63fc6bf3457adR6029

It could be moved to the glibc part.


> Besides, I also not sure if adding way to remove RELRO protection won't
> add more security issues (we can disable for sesuid binaries, but even
> though it is not a good security practice).

RELRO is removed only temporarily, it gets re-engaged. And that time other
threads should be even stopped (see above). Is it still a security issues?


Thanks,
Jan
  

On Tue, 07 Mar 2023 16:40:46 +0800, Florian Weimer wrote:
> Do you have a high-level overview how this is supposed to work?  Is CRIU
> expected to call the new _dl_reset_ifunc symbol after restore?

Yes, it should be called by CRIU. The current implementation was calling it
from CRaC restore (which uses CRIU underneath) but I agree it is not the
proper placement:
	https://github.com/openjdk/crac/pull/41/files#diff-aeec57d804d56002f26a85359fc4ac8b48cfc249d57c656a30a63fc6bf3457adR6274
The function is called differently there as the IFUNC reset implementation is
outside of glibc there.


> Use of <sys/platform/x86.h> is somewhat rare.  Not even GCC uses it
> AFAIK.  It has its own cached CPU data used for target clones and
> similar features.  Re-running those IFUNC resolvers will just give the
> same results.

I do not use <sys/platform/x86.h> anywhere, why do you mention it?
I am doing there
	memset((volatile uint8_t *)cpu_features, 0, sizeof(*cpu_features));
	...
	_dl_x86_init_cpu_features();
to prevent the cache of CPU data. And it does work.


> Distributions have started to deploy alternate builds in
> glibc-hwcaps/x86-64-v3 directories:
> 
>   openSUSE Tumbleweed gains optional x86-64-v3 optimization
>   <https://news.opensuse.org/2023/03/02/tw-gains-optional-optimizations/>
> 
> This might be a common situation fairly soon.  This is not IFUNC-based
> at all, so any IFUNC-based approach is not going to help with that.

What about some "dlreopen" to replace the variant of glibc?


> Practically speaking, I think cluster heterogeneity needs to be hidden
> at the cluster level.

The question is how to implement it. I have found
	https://github.com/ddcc/libcpuidoverride
but it is not compatible with latest glibc (maybe its loader code needs to
implement ELF relocations, it would be definitely fixable).

Another option is GLIBC_TUNABLES=glibc.cpu.hwcaps=... although then one has to
translate cpuid feature bits into the glibc names like SSE4_2 etc.
Also re-exec() after setenv("GLIBC_TUNABLES") is not too nice.

There are multiple projects solving the same problem such as:
	https://github.com/eclipse-openj9/openj9/issues/14253
So I was thinking some more standardized solution may be already appropriate.


Thanks,
Jan
  

* Jan Kratochvil:

> On Tue, 07 Mar 2023 16:40:46 +0800, Florian Weimer wrote:
>> Use of <sys/platform/x86.h> is somewhat rare.  Not even GCC uses it
>> AFAIK.  It has its own cached CPU data used for target clones and
>> similar features.  Re-running those IFUNC resolvers will just give the
>> same results.
>
> I do not use <sys/platform/x86.h> anywhere, why do you mention it?
> I am doing there
> 	memset((volatile uint8_t *)cpu_features, 0, sizeof(*cpu_features));
> 	...
> 	_dl_x86_init_cpu_features();
> to prevent the cache of CPU data. And it does work.

This code resets the data queried by <sys/platform/x86.h>.  But GCC and
others do not use these interfaces, they use CPUID/XGETBV directly and
cache the result, so this reset does not alter the outcome of their
IFUNC resolvers.

>> Distributions have started to deploy alternate builds in
>> glibc-hwcaps/x86-64-v3 directories:
>> 
>>   openSUSE Tumbleweed gains optional x86-64-v3 optimization
>>   <https://news.opensuse.org/2023/03/02/tw-gains-optional-optimizations/>
>> 
>> This might be a common situation fairly soon.  This is not IFUNC-based
>> at all, so any IFUNC-based approach is not going to help with that.
>
> What about some "dlreopen" to replace the variant of glibc?

The trivial implementation is not going to work because most of the
function and data addresses will change as a result of the reload.  Most
checkpoints probably occur in a blocking system call, and that would
need some form of on-stack replacement to work properly.

>> Practically speaking, I think cluster heterogeneity needs to be hidden
>> at the cluster level.
>
> The question is how to implement it. I have found
> 	https://github.com/ddcc/libcpuidoverride
> but it is not compatible with latest glibc (maybe its loader code needs to
> implement ELF relocations, it would be definitely fixable).

It needs kernel support for sure.  I do not know to what extent CPUID
can be context-switched at the silicon level, maybe the masking is
system-wide (or per-guest in a virtualized configuration). 

> Another option is GLIBC_TUNABLES=glibc.cpu.hwcaps=... although then one has to
> translate cpuid feature bits into the glibc names like SSE4_2 etc.
> Also re-exec() after setenv("GLIBC_TUNABLES") is not too nice.

Again, that's not going to work in practice because libraries and
applications do not use <sys/platform/x86.h>.

Thanks,
Florian
  

On Wed, 08 Mar 2023 18:44:45 +0800, Florian Weimer wrote:
> This code resets the data queried by <sys/platform/x86.h>.  But GCC and
> others do not use these interfaces, they use CPUID/XGETBV directly and
> cache the result, so this reset does not alter the outcome of their
> IFUNC resolvers.

We are on glibc mailing list so this patch and mail thread is about glibc.
I am aware other programs and libraries need their own reset. For OpenJDK JIT
compiler I wrote the CPU reset already; but that is offtopic here:
	https://github.com/openjdk/crac/pull/41


> > The question is how to implement it. I have found
> > 	https://github.com/ddcc/libcpuidoverride
> > but it is not compatible with latest glibc (maybe its loader code needs to
> > implement ELF relocations, it would be definitely fixable).
> 
> It needs kernel support for sure.  I do not know to what extent CPUID
> can be context-switched at the silicon level, maybe the masking is
> system-wide (or per-guest in a virtualized configuration). 

There is already its kernel support - ARCH_SET_CPUID. It is per process.


> > Another option is GLIBC_TUNABLES=glibc.cpu.hwcaps=... although then one has to
> > translate cpuid feature bits into the glibc names like SSE4_2 etc.
> > Also re-exec() after setenv("GLIBC_TUNABLES") is not too nice.
> 
> Again, that's not going to work in practice because libraries and
> applications do not use <sys/platform/x86.h>.

This mailing list is for glibc so let's talk about glibc.


Thanks,
Jan
  

On 08/03/23 07:21, Jan Kratochvil wrote:
> On Tue, 07 Mar 2023 21:07:58 +0800, Adhemerval Zanella Netto wrote:
>> I am not sure if the ifunc reset will be really safe without adding CRIU
>> migrate sync points, to avoid suspend execution in a context that the
>> ifunc variants are already being executed or its address is being in a
>> function point (for instance in PLT code).
> 
> You are right but I left the thread safety up to the caller ("Freezer"):
> 	https://github.com/openjdk/crac/pull/41/files#diff-aeec57d804d56002f26a85359fc4ac8b48cfc249d57c656a30a63fc6bf3457adR6029
> 
> It could be moved to the glibc part.
> 
> 
>> Besides, I also not sure if adding way to remove RELRO protection won't
>> add more security issues (we can disable for sesuid binaries, but even
>> though it is not a good security practice).
> 
> RELRO is removed only temporarily, it gets re-engaged. And that time other
> threads should be even stopped (see above). Is it still a security issues?

Yes, without a stop-the-world scheme where a helper thread sets PR_GET_DUMPABLE 
and PTRACE_ATTACH the process can not really be sure that any new thread will not 
be created between the time you enumerate the process threads and call the 'freeze'
function.

I really don't think glibc should provide an interface to temporary disable any
security hardening, it should always opt-in at either program startup or by
building time.  The ifunc mechanism is already full or corner cases and I think
adding a runtime mechanism to reset them is *not* a way forward.  

As I said, I think CRIU heterogeneity should be handled by masking off the higher 
cpu features.  I am not if ARCH_SET_CPUID would a solution here, it means that
we will need to handle SIGSEGV in the loader and come up with a sane subset
in case of failure (we now have x86_64-vx, so we can use it as default). 

But as Florian has said, fixing on glibc won't work consistently on other
libraries that uses cpuid instruction.  And yes, I am aware we are discussing
glibc, but I prefer a composable solution (like adding proper cpuid maskoff in 
the kernel) to add an ah-hoc one.
  

On Wed, 08 Mar 2023 21:04:59 +0800, Adhemerval Zanella Netto wrote:
> Yes, without a stop-the-world scheme where a helper thread sets PR_GET_DUMPABLE 
> and PTRACE_ATTACH the process can not really be sure that any new thread will not 
> be created between the time you enumerate the process threads and call the 'freeze'
> function.

That "Freezer" class does solve the race of new threads. I also do not see
a need for ptrace there, it is self-snapshotting/restoring.
	https://github.com/openjdk/crac/pull/41/files#diff-aeec57d804d56002f26a85359fc4ac8b48cfc249d57c656a30a63fc6bf3457adR6029


> I really don't think glibc should provide an interface to temporary disable any
> security hardening, it should always opt-in at either program startup or by
> building time.  The ifunc mechanism is already full or corner cases and I think
> adding a runtime mechanism to reset them is *not* a way forward.  
> 
> As I said, I think CRIU heterogeneity should be handled by masking off the higher 
> cpu features.  I am not if ARCH_SET_CPUID would a solution here, it means that
> we will need to handle SIGSEGV in the loader and come up with a sane subset
> in case of failure (we now have x86_64-vx, so we can use it as default). 

So the only remaining option is that all the programs will be doing
setenv("GLIBC_TUNABLES=glibc.cpu.hwcaps=...") and re-exec(). That is
a peformance kill and definitely not nice compared to any method of an IFUNC
reset.


> But as Florian has said, fixing on glibc won't work consistently on other
> libraries that uses cpuid instruction.

In Java world the other libraries (in general, there are some JNI exceptions)
do not matter as they are a Java code JIT-compiled by JVM.


Jan
  

On 09/03/23 08:32, Jan Kratochvil wrote:
> On Wed, 08 Mar 2023 21:04:59 +0800, Adhemerval Zanella Netto wrote:
>> Yes, without a stop-the-world scheme where a helper thread sets PR_GET_DUMPABLE 
>> and PTRACE_ATTACH the process can not really be sure that any new thread will not 
>> be created between the time you enumerate the process threads and call the 'freeze'
>> function.
> 
> That "Freezer" class does solve the race of new threads. I also do not see
> a need for ptrace there, it is self-snapshotting/restoring.
> 	https://github.com/openjdk/crac/pull/41/files#diff-aeec57d804d56002f26a85359fc4ac8b48cfc249d57c656a30a63fc6bf3457adR6029

I am not sure how the kernel would enumerate new tasks that are created
while iterating over /proc/self/task.  On closefrom Linux fallback we have
a similar problem, where the code iterates over /proc/self/fd, and everytime
it closes a file descriptor it lseeks back to beginning.  It works because
eventually there will be no more entries on /proc/self/fd, so either you
will need to certify that kernel adds new tasks at the end of the getdents
call (used by readdir, or lseek and keep track of all tasks already signaled.

While it might work on the JVM where you can not fully control who change 
SIGUSR1  disposition (and I am not sure JVM would prevent a JNI call to do so),
so you can't really make it generic without explicit reserve a signal to do so, 
similar to what glibc does for SIGCANCEL and SIGSETXID (used to synchronize 
setuid functions over threads).  Meaning that callers of sigaction can't 
not explicit set such reserved signal.

This is similar to what we do for SIGSETXID, so I think a proper way to
do it would to do always install a new signal handler to this on pthread_create,
on signal handle synchronize with proper async-signal-safe interface
(pthread_mutex_lock is not, you might accomplish with sem_post but most likely
you will need a atomic+futex way similar to a barrier), iterate over all 
dl_stack_used (so the interface can work without access to procfs), issue the 
signal handler or each thread, operate on the maps, then synchronize to resume
threads.  We can't really make it generic without accessing the internal
glibc thread states.

And you will also need to reallocate not only glibc, but potentially *all*
libraries (since ifunc can be used by any function).

> 
> 
>> I really don't think glibc should provide an interface to temporary disable any
>> security hardening, it should always opt-in at either program startup or by
>> building time.  The ifunc mechanism is already full or corner cases and I think
>> adding a runtime mechanism to reset them is *not* a way forward.  
>>
>> As I said, I think CRIU heterogeneity should be handled by masking off the higher 
>> cpu features.  I am not if ARCH_SET_CPUID would a solution here, it means that
>> we will need to handle SIGSEGV in the loader and come up with a sane subset
>> in case of failure (we now have x86_64-vx, so we can use it as default). 
> 
> So the only remaining option is that all the programs will be doing
> setenv("GLIBC_TUNABLES=glibc.cpu.hwcaps=...") and re-exec(). That is
> a peformance kill and definitely not nice compared to any method of an IFUNC
> reset.

Assuming you don't reset env variable on process spawning, you can set it as 
default for the session. Another option would to deploy a glibc built with 
--disable-multi-arch; it will disable ifunc generation.

And IMHO this is way nicer because this IFUNC reset as-is, without a proper 
stop-the-word support, is not safe and adds another corner case for the already
over-complicated ifunc interface.

> 
> 
>> But as Florian has said, fixing on glibc won't work consistently on other
>> libraries that uses cpuid instruction.
> 
> In Java world the other libraries (in general, there are some JNI exceptions)
> do not matter as they are a Java code JIT-compiled by JVM.

And this won't be a Java specific interface, but rather a GNU extension for C
library.  So we must make it as concise as possible, without adding any other
security or undefined behavior.
  

On 09/03/23 12:47, Adhemerval Zanella Netto wrote:
> 
> 
> On 09/03/23 08:32, Jan Kratochvil wrote:
>> On Wed, 08 Mar 2023 21:04:59 +0800, Adhemerval Zanella Netto wrote:
>>> Yes, without a stop-the-world scheme where a helper thread sets PR_GET_DUMPABLE 
>>> and PTRACE_ATTACH the process can not really be sure that any new thread will not 
>>> be created between the time you enumerate the process threads and call the 'freeze'
>>> function.
>>
>> That "Freezer" class does solve the race of new threads. I also do not see
>> a need for ptrace there, it is self-snapshotting/restoring.
>> 	https://github.com/openjdk/crac/pull/41/files#diff-aeec57d804d56002f26a85359fc4ac8b48cfc249d57c656a30a63fc6bf3457adR6029
> 
> I am not sure how the kernel would enumerate new tasks that are created
> while iterating over /proc/self/task.  On closefrom Linux fallback we have
> a similar problem, where the code iterates over /proc/self/fd, and everytime
> it closes a file descriptor it lseeks back to beginning.  It works because
> eventually there will be no more entries on /proc/self/fd, so either you
> will need to certify that kernel adds new tasks at the end of the getdents
> call (used by readdir, or lseek and keep track of all tasks already signaled.

And the 'handler' signal handler has some potential shortcomings as well: 

  * backtrace is not async-signal-safe: glibc implementation on first call
    issues dlopen, which calls malloc; and libgcc_eh.so *might* also calls malloc.
  
  * pthread calls are not async-signal-safe either.

  * it only handles libc.so, other libraries that uses ifunc for function
    selection also fails.

  * the syscall heuristics do not handle partial results (for instance if
    write syscall returns do EINTR).

So this code has the potential of deadlock, specially if you have another
thread issuing malloc.
  

* Jan Kratochvil:

> On Wed, 08 Mar 2023 21:04:59 +0800, Adhemerval Zanella Netto wrote:
>> But as Florian has said, fixing on glibc won't work consistently on other
>> libraries that uses cpuid instruction.
>
> In Java world the other libraries (in general, there are some JNI exceptions)
> do not matter as they are a Java code JIT-compiled by JVM.

That's not really true for GUI applications, which use quite a few
system libraries.  Even in headless applications, font rendering code
may be loaded, say for PDF generation.  If system versions are used,
that pulls in compression libraries and PCRE, and those tend to have
CPUID-based switching, too (but perhaps not in the form of IFUNC
resolvers).  This is just on x86-64—on aarch64, you will have to deal
with the outline atomics most distributions use.

I suggest you base your OpenJDK work on the OpenJDK musl port.  This
way, you can carefully control what you build and how you build it, and
also make it impossible to pick up system libraries by accident.  And
static JNI would be a good fit if you do not expect JNI to be used
anyway.

Thanks,
Florian
  

On Mon, 13 Mar 2023 21:59:26 +0800, Florian Weimer wrote:
> I suggest you base your OpenJDK work on the OpenJDK musl port.  This
> way, you can carefully control what you build and how you build it, and
> also make it impossible to pick up system libraries by accident.

I really cannot say OpenJDK feature CRaC ( https://github.com/CRaC/docs )
will require musl. In such case including CRaC to upstream OpenJDK is
pointless as nobody will use it anyway.


Jan Kratochvil
  

* Jan Kratochvil:

> On Mon, 13 Mar 2023 21:59:26 +0800, Florian Weimer wrote:
>> I suggest you base your OpenJDK work on the OpenJDK musl port.  This
>> way, you can carefully control what you build and how you build it, and
>> also make it impossible to pick up system libraries by accident.
>
> I really cannot say OpenJDK feature CRaC ( https://github.com/CRaC/docs )
> will require musl. In such case including CRaC to upstream OpenJDK is
> pointless as nobody will use it anyway.

You will have to rebuild everything with a custom toolchain anyway, and
avoid compiling in any incompatible features.  Use the musl port as the
baseline makes this explicit.

Thanks,
Florian
  

On Tue, 14 Mar 2023 22:49:38 +0800, Florian Weimer wrote:
> You will have to rebuild everything with a custom toolchain anyway, and
> avoid compiling in any incompatible features.

I do not see why, it normally works on Fedora and Debian systems.


Jan
  

On Thu, 09 Mar 2023 16:47:49 +0100, Adhemerval Zanella Netto wrote:
> I am not sure how the kernel would enumerate new tasks that are created
> while iterating over /proc/self/task.

I have updated the code as you have found a race there. Now this is no longer
relevant as all known tasks are already verified as stopped. So there is no
more running task to create another task. While iterating /proc/self/task:
(1) either there must be already an unstopped task when opendir() was called,
    in such case the iteration of /proc/self/task will be retried anyway.
(2) or all tasks are stopped and therefore no task can create any new task.


> On closefrom Linux fallback we have
> a similar problem, where the code iterates over /proc/self/fd, and everytime
> it closes a file descriptor it lseeks back to beginning.  It works because
> eventually there will be no more entries on /proc/self/fd, so either you
> will need to certify that kernel adds new tasks at the end of the getdents
> call (used by readdir, or lseek and keep track of all tasks already signaled.

That is not needed, see above.


> While it might work on the JVM where you can not fully control who change 
> SIGUSR1  disposition (and I am not sure JVM would prevent a JNI call to do so),
> so you can't really make it generic without explicit reserve a signal to do so, 
> similar to what glibc does for SIGCANCEL and SIGSETXID (used to synchronize 
> setuid functions over threads).  Meaning that callers of sigaction can't 
> not explicit set such reserved signal.
> 
> This is similar to what we do for SIGSETXID, so I think a proper way to
> do it would to do always install a new signal handler to this on pthread_create,
> on signal handle synchronize with proper async-signal-safe interface
> (pthread_mutex_lock is not, you might accomplish with sem_post but most likely
> you will need a atomic+futex way similar to a barrier), iterate over all 
> dl_stack_used (so the interface can work without access to procfs), issue the 
> signal handler or each thread, operate on the maps, then synchronize to resume
> threads.  We can't really make it generic without accessing the internal
> glibc thread states.

Good to know, if the patch gets a serious consideration for upstreaming
I understand the signal number needs to be handled better.


> And you will also need to reallocate not only glibc, but potentially *all*
> libraries (since ifunc can be used by any function).

This is what the patch already does by _dl_relocate_object().


> > So the only remaining option is that all the programs will be doing
> > setenv("GLIBC_TUNABLES=glibc.cpu.hwcaps=...") and re-exec(). That is
> > a peformance kill and definitely not nice compared to any method of an IFUNC
> > reset.
> 
> Assuming you don't reset env variable on process spawning, you can set it as 
> default for the session.

The /usr/bin/java program needs to setenv("GLIBC_TUNABLES=glibc.cpu.hwcaps=...")
and then it can either system("itself") or exec("itself"). This is what you
mean by the session?


> Another option would to deploy a glibc built with 
> --disable-multi-arch; it will disable ifunc generation.

That is not an option. OpenJDK must be compatible with normal existing Linux
OSes.


> And IMHO this is way nicer because this IFUNC reset as-is, without a proper 
> stop-the-word support, is not safe and adds another corner case for the already
> over-complicated ifunc interface.

stop-the-world is already implemented modulo possible bugfixes.
	https://github.com/openjdk/crac/pull/41/files#diff-aeec57d804d56002f26a85359fc4ac8b48cfc249d57c656a30a63fc6bf3457adR6029


> > In Java world the other libraries (in general, there are some JNI exceptions)
> > do not matter as they are a Java code JIT-compiled by JVM.
> 
> And this won't be a Java specific interface, but rather a GNU extension for C
> library.  So we must make it as concise as possible, without adding any other
> security or undefined behavior.  

I agree. Handling IFUNC for other libraries is also possible but it has to be
a next step. It does not make sense to handle IFUNC in other libraries when
glibc still crashes first.


On Thu, 09 Mar 2023 18:43:08 +0100, Adhemerval Zanella Netto wrote:
> And the 'handler' signal handler has some potential shortcomings as well: 
> 
>   * backtrace is not async-signal-safe: glibc implementation on first call
>     issues dlopen, which calls malloc; and libgcc_eh.so *might* also calls malloc.

I do not see it in practice:
	Temporary breakpoint 1, main () at backtrace.c:5
	5	  backtrace(buf, sizeof(buf)/sizeof(*buf));
	(gdb) b dlopen
	Breakpoint 2 at 0x7ffff7c88f20: file dlopen.c, line 77.
	(gdb) c
	Continuing.
	[Inferior 1 (process 825695) exited normally]

And neither in the sources:
glibc$ grep dlopen $(find -iname "*backtrace*")


>   * pthread calls are not async-signal-safe either.

There are no pthread_* calls, everything is based on kernel tasks.


>   * it only handles libc.so, other libraries that uses ifunc for function
>     selection also fails.

You are right, I have mostly implemented this hard-coded "libc.so.6" to make
it general (for any libraries containing at least one STT_GNU_IFUNC) although
I haven't finished this implementation due to the last paragraph below.


>   * the syscall heuristics do not handle partial results (for instance if
>     write syscall returns do EINTR).

I do not think EINTR would matter. The syscall heuristics is there expecting
that any library function which contains syscalls is not an IFUNC function.


> So this code has the potential of deadlock, specially if you have another
> thread issuing malloc.

I may have missed something but I do not see it so according to the answers
above.


According to the other reactions here I doubt this functionality would get
accepted to glibc so we have decided to give up on its upstreaming and use the
setenv("GLIBC_TUNABLES=glibc.cpu.hwcaps=...") + re-exec workaround instead.
That would need to be coded for compatibility with existing/old glibcs anyway.


Thanks,
Jan
  

On 16/03/23 11:38, Jan Kratochvil wrote:
> On Thu, 09 Mar 2023 16:47:49 +0100, Adhemerval Zanella Netto wrote:
>> I am not sure how the kernel would enumerate new tasks that are created
>> while iterating over /proc/self/task.
> 
> I have updated the code as you have found a race there. Now this is no longer
> relevant as all known tasks are already verified as stopped. So there is no
> more running task to create another task. While iterating /proc/self/task:
> (1) either there must be already an unstopped task when opendir() was called,
>     in such case the iteration of /proc/self/task will be retried anyway.
> (2) or all tasks are stopped and therefore no task can create any new task.
> 
> 
>> On closefrom Linux fallback we have
>> a similar problem, where the code iterates over /proc/self/fd, and everytime
>> it closes a file descriptor it lseeks back to beginning.  It works because
>> eventually there will be no more entries on /proc/self/fd, so either you
>> will need to certify that kernel adds new tasks at the end of the getdents
>> call (used by readdir, or lseek and keep track of all tasks already signaled.
> 
> That is not needed, see above.
> 
> 
>> While it might work on the JVM where you can not fully control who change 
>> SIGUSR1  disposition (and I am not sure JVM would prevent a JNI call to do so),
>> so you can't really make it generic without explicit reserve a signal to do so, 
>> similar to what glibc does for SIGCANCEL and SIGSETXID (used to synchronize 
>> setuid functions over threads).  Meaning that callers of sigaction can't 
>> not explicit set such reserved signal.
>>
>> This is similar to what we do for SIGSETXID, so I think a proper way to
>> do it would to do always install a new signal handler to this on pthread_create,
>> on signal handle synchronize with proper async-signal-safe interface
>> (pthread_mutex_lock is not, you might accomplish with sem_post but most likely
>> you will need a atomic+futex way similar to a barrier), iterate over all 
>> dl_stack_used (so the interface can work without access to procfs), issue the 
>> signal handler or each thread, operate on the maps, then synchronize to resume
>> threads.  We can't really make it generic without accessing the internal
>> glibc thread states.
> 
> Good to know, if the patch gets a serious consideration for upstreaming
> I understand the signal number needs to be handled better.
> 

For libc point of view I don't think this interface make sense without it itself
do the stop-the-world; since potentially calling while ifunc might be resolved 
adds another kind of concurrency that is far from ideal (we already have to handle
lazy binding, dl_profile, and other concurrent notes that touch the GOT and
associated data for symbol resolution).

> 
>> And you will also need to reallocate not only glibc, but potentially *all*
>> libraries (since ifunc can be used by any function).
> 
> This is what the patch already does by _dl_relocate_object().
> 
> 
>>> So the only remaining option is that all the programs will be doing
>>> setenv("GLIBC_TUNABLES=glibc.cpu.hwcaps=...") and re-exec(). That is
>>> a peformance kill and definitely not nice compared to any method of an IFUNC
>>> reset.
>>
>> Assuming you don't reset env variable on process spawning, you can set it as 
>> default for the session.
> 
> The /usr/bin/java program needs to setenv("GLIBC_TUNABLES=glibc.cpu.hwcaps=...")
> and then it can either system("itself") or exec("itself"). This is what you
> mean by the session?

I was thinking invoking java itself with GLIBC_TUNABLES, but I think it would
make sense to add a way to globally set tunables without the need to setting
the environment variable.

We already some ideas in mind [1], where I have in mind a system defined file
(as glibc already have for preload and ld.so.cache) that sets the defaults.
Siddhesh idea is to have a global one that is *not* overwritten by users,
while my idea is the opposite where the global sets the default, and users
can override it.  This is open to discussion on how this would work, if we
we will need an extra flag to disable it (similar to DF_1_NODEFLIB), how 
user define GLIBC_TUNABLES will interact with it, etc.

[1] https://sourceware.org/pipermail/libc-alpha/2023-March/146405.html

> 
> 
>> Another option would to deploy a glibc built with 
>> --disable-multi-arch; it will disable ifunc generation.
> 
> That is not an option. OpenJDK must be compatible with normal existing Linux
> OSes.
> 

I do not know exactly how criu works internally, but from you answer and if I
understood it correctly when criu restores a dump it will only restore private 
mappings from the process and not shared ones?  If so, how does it handle 
restoring on a system on an older glibc (or on a system with any dependency
older than the one that the process was dumped)? Does it only restore file
backed shared mappings based on naming?

Using a custom built glibc with --disable-multi-arch along with rpath, you
have a dissociated glibc from the system.  You can even tune to use an 
specific x86_64-vX abi if you see that the workload benefits from some
specific string/mem/math routine.

> 
>> And IMHO this is way nicer because this IFUNC reset as-is, without a proper 
>> stop-the-word support, is not safe and adds another corner case for the already
>> over-complicated ifunc interface.
> 
> stop-the-world is already implemented modulo possible bugfixes.
> 	https://github.com/openjdk/crac/pull/41/files#diff-aeec57d804d56002f26a85359fc4ac8b48cfc249d57c656a30a63fc6bf3457adR6029
> 

Again, from a Java perspective it might be fine.  But for a generic C interface
it still has a lot of corner cases.

>>> In Java world the other libraries (in general, there are some JNI exceptions)
>>> do not matter as they are a Java code JIT-compiled by JVM.
>>
>> And this won't be a Java specific interface, but rather a GNU extension for C
>> library.  So we must make it as concise as possible, without adding any other
>> security or undefined behavior.  
> 
> I agree. Handling IFUNC for other libraries is also possible but it has to be
> a next step. It does not make sense to handle IFUNC in other libraries when
> glibc still crashes first.
> 
> 
> On Thu, 09 Mar 2023 18:43:08 +0100, Adhemerval Zanella Netto wrote:
>> And the 'handler' signal handler has some potential shortcomings as well: 
>>
>>   * backtrace is not async-signal-safe: glibc implementation on first call
>>     issues dlopen, which calls malloc; and libgcc_eh.so *might* also calls malloc.
> 
> I do not see it in practice:
> 	Temporary breakpoint 1, main () at backtrace.c:5
> 	5	  backtrace(buf, sizeof(buf)/sizeof(*buf));
> 	(gdb) b dlopen
> 	Breakpoint 2 at 0x7ffff7c88f20: file dlopen.c, line 77.
> 	(gdb) c
> 	Continuing.
> 	[Inferior 1 (process 825695) exited normally]
> 
> And neither in the sources:
> glibc$ grep dlopen $(find -iname "*backtrace*")

Check misc/unwind-link.c:

 48   /* Initialize a copy of the data, so that we do not need about
 49      unlocking in case the dynamic loader somehow triggers
 50      unwinding.  */
 51   void *local_libgcc_handle = __libc_dlopen (LIBGCC_S_SO);
 52   if (local_libgcc_handle == NULL)
 53     {
 54       __libc_lock_unlock (lock);
 55       return NULL;
 56     }

The backtrace calls it through UNWIND_LINK_PTR macro.  You might not seeing it
because we cache libgcc_s.so loading and once the programs calls backtrace or
pthread_cancel, it won't call __libc_unwind_link_get anymore.

> 
> 
>>   * pthread calls are not async-signal-safe either.
> 
> There are no pthread_* calls, everything is based on kernel tasks.

The SIGUSR1 handler it installs on freeze() uses both mutexes and cond variables,
and this is quite easy to deadload if a thread sends a SIGUSR1 if its not suppose
to (and both pthread_mutex_lock and pthread_cond_signal are not async-signal-safe).

> 
> 
>>   * it only handles libc.so, other libraries that uses ifunc for function
>>     selection also fails.
> 
> You are right, I have mostly implemented this hard-coded "libc.so.6" to make
> it general (for any libraries containing at least one STT_GNU_IFUNC) although
> I haven't finished this implementation due to the last paragraph below.
> 
> 
>>   * the syscall heuristics do not handle partial results (for instance if
>>     write syscall returns do EINTR).
> 
> I do not think EINTR would matter. The syscall heuristics is there expecting
> that any library function which contains syscalls is not an IFUNC function.
> 
> 
>> So this code has the potential of deadlock, specially if you have another
>> thread issuing malloc.
> 
> I may have missed something but I do not see it so according to the answers
> above.
>
  

On Tue, 21 Mar 2023 00:47:53 +0800, Adhemerval Zanella Netto wrote:
> if I
> understood it correctly when criu restores a dump it will only restore private
> mappings from the process and not shared ones?  If so, how does it handle
> restoring on a system on an older glibc (or on a system with any dependency
> older than the one that the process was dumped)? Does it only restore file
> backed shared mappings based on naming?

Different OS (components) version is not supported, similar to:
	https://cr.openjdk.org/~heidinga/crac/Portability_of_checkpoints.pdf
	2. Same hardware, different distro


> Using a custom built glibc with --disable-multi-arch along with rpath, you
> have a dissociated glibc from the system.  You can even tune to use an
> specific x86_64-vX abi if you see that the workload benefits from some
> specific string/mem/math routine.

Then I need to build also all other used libraries and that is what is called
nowadays a container. Which is a perfectly valid use case. But building
a custom distribution (=glibc) is not a supported way of running any software.
[This is my personal opinion, I do not speak for any company.]


Thanks for your review,
Jan
  

On 29/03/23 09:12, Jan Kratochvil wrote:
> On Tue, 21 Mar 2023 00:47:53 +0800, Adhemerval Zanella Netto wrote:
>> if I
>> understood it correctly when criu restores a dump it will only restore private
>> mappings from the process and not shared ones?  If so, how does it handle
>> restoring on a system on an older glibc (or on a system with any dependency
>> older than the one that the process was dumped)? Does it only restore file
>> backed shared mappings based on naming?
> 
> Different OS (components) version is not supported, similar to:
> 	https://cr.openjdk.org/~heidinga/crac/Portability_of_checkpoints.pdf
> 	2. Same hardware, different distro

Thanks for the link, so it does seems that CRIU does not save anonymous
file-backed maps, this is what I understood from CRIU memory restore doc [1]
where open file mappings are done by 'files engines' (where I presume it is
either an external process or a library, correct if I am wrong).

Another issue you might see (even on same distro) is that Intel is doing a 
*lot* of performance backports on release branches which means that even 
glibc in the same release version might ended up with different sets
of ifunc variants (since this does not really affect ABI).  I am not sure
how it would reflect on distribution itself, but assuming a distro that
follows the glibc release branch it might indeed happen.

[1] https://criu.org/Memory_dumping_and_restoring	

> 
> 
>> Using a custom built glibc with --disable-multi-arch along with rpath, you
>> have a dissociated glibc from the system.  You can even tune to use an
>> specific x86_64-vX abi if you see that the workload benefits from some
>> specific string/mem/math routine.
> 
> Then I need to build also all other used libraries and that is what is called
> nowadays a container. Which is a perfectly valid use case. But building
> a custom distribution (=glibc) is not a supported way of running any software.
> [This is my personal opinion, I do not speak for any company.]

In fact I think for this specific problem is the best solution.  Along with
a JVM built with rpath, it means you will pack everything it requires and assure
that once it is installed from same package, the glibc won't enable any
optimization not supported and make sure that any distro backport won't
break these assumptions.  It also works with all scenarios you raised, assuming
that 3rd party libraries (outside the JVM package itself) does play nice
in such scenarios.