[00/18] More y2038 fixes

  Hi Carlos and Lukasz,

This patchset adds more y2038 fixes that I would like to push for 2.34.
The first two patches enable more internal y2038 usage on glibc and
also for the installed programs.  Making glibc the first consumer of
these newer interfaces should improve coverage.

The next 6 patches removes an optimization I added on some specific
syscall, where a global variable it atomically set once glibc detects
that kernel supports 64-bit time.  The problem with this approach
breaks the usage case of live migration like CRIU or similar.  Also for
some interfaces that do not query information from kernel, most usages
can be optimized away by either building glibc with a minimum 5.1 kernel
or by using the 32-bit syscall for the common case.

So the following patch implement this optimization for the interfaces
that allows it.  If the provided timeout can be fit in old 32-bit
time_t, the old syscall is used instead.  This optimization is only used
for !__ASSUME_TIME64_SYSCALLS, otherwise the newer 64-bit syscall are
used.

I checked this with 3 different configation which should cover most
usages:

  - i686-linux-gnu on a 4.15 kernel and default --enable-kernel=3.2.
    This uses the 32-bit optimization to avoid calling the newer
    64-bit syscall when possible.  It also triggers and EOVERFLOW
    when 64-bit inputs are used without proper kernel support.
  - i686-linux-gnu on a 5.11 kernel and default --enable-kernel=3.2.
    This will also use the 32-bit optimization, but it will always
    succeded due proper kernel support.
  - i686-linux-gnu on a 5.11 kernel and with --enable-kernel=5.1.
    This enables __ASSUME_TIME64_SYSCALLS and only uses 64-bit
    time_t syscalls.

Adhemerval Zanella (18):
  Use 64 bit time_t stat internally
  Use LFS and 64 bit time for installed programs
  support: Add support_create_timer
  linux: Only use 64-bit syscall if required for ppoll
  linux: Only use 64-bit syscall if required for pselect
  linux: Only use 64-bit syscall if required for select
  linux: Remove supports_time64 () from clock_getres
  linux: Remove supports_time64 () from clock_gettime
  linux: Remove time64-support
  linux: timerfd_gettime minor cleanup
  linux: Only use 64-bit syscall if required for semtimedop
  linux: Only use 64-bit syscall if required for timerfd_settime
  linux: Only use 64-bit syscall if required for mq_timedreceive
  linux: Only use 64-bit syscall if required for mq_timedsend
  linux: Only use 64-bit syscall if required for sigtimedwait
  linux: Only use 64-bit syscall if required for utimensat family
  linux: Only use 64-bit syscall if required for internal futex
  linux: Only use 64-bit syscall if required for clock_nanosleep

 Makeconfig                                    |   8 +-
 Makerules                                     |  12 +-
 csu/check_fds.c                               |   8 +-
 elf/dl-load.c                                 |   8 +-
 elf/dl-misc.c                                 |   4 +-
 elf/dl-profile.c                              |   4 +-
 iconv/gconv_cache.c                           |   4 +-
 include/dirent.h                              |   2 +-
 include/file_change_detection.h               |   6 +-
 include/sys/select.h                          |   5 +
 inet/rcmd.c                                   |   6 +-
 intl/loadmsgcat.c                             |   4 +-
 io/Makefile                                   |   4 +-
 io/file_change_detection.c                    |  16 ++-
 io/getdirname.c                               |   6 +-
 libio/filedoalloc.c                           |   2 +-
 libio/fileops.c                               |   8 +-
 libio/oldfileops.c                            |   2 +-
 libio/wfileops.c                              |   2 +-
 locale/loadarchive.c                          |   8 +-
 locale/loadlocale.c                           |   6 +-
 locale/localeinfo.h                           |   2 +-
 misc/Makefile                                 |  11 ++
 misc/tst-pselect.c                            | 120 ++++++++----------
 misc/tst-select.c                             |  39 +++---
 nptl/futex-internal.c                         |  52 +++++---
 nscd/nscd_helper.c                            |   4 +-
 nss/nss_database.c                            |   4 +-
 rt/Makefile                                   |   4 +-
 rt/tst-mqueue10-time64.c                      |   1 +
 rt/tst-mqueue10.c                             |  72 +++++++++++
 support/Makefile                              |   1 +
 support/support.h                             |  11 ++
 support/support_create_timer.c                |  69 ++++++++++
 sysdeps/nptl/futex-internal.h                 |  24 ++--
 sysdeps/posix/dl-fileid.h                     |   4 +-
 sysdeps/posix/euidaccess.c                    |   4 +-
 sysdeps/posix/getaddrinfo.c                   |  21 +--
 sysdeps/posix/getcwd.c                        |  15 ++-
 sysdeps/posix/pathconf.c                      |   4 +-
 sysdeps/posix/sysconf.c                       |   4 +-
 sysdeps/posix/tempname.c                      |   8 +-
 sysdeps/unix/sysv/linux/Makefile              |  15 ++-
 sysdeps/unix/sysv/linux/clock_getres.c        |  16 +--
 sysdeps/unix/sysv/linux/clock_gettime.c       |  14 +-
 sysdeps/unix/sysv/linux/clock_nanosleep.c     |  47 ++++---
 sysdeps/unix/sysv/linux/fdopendir.c           |   4 +-
 sysdeps/unix/sysv/linux/fexecve.c             |   4 +-
 .../unix/sysv/linux/microblaze/pselect32.c    |   3 +-
 sysdeps/unix/sysv/linux/mq_timedreceive.c     |  35 ++---
 sysdeps/unix/sysv/linux/mq_timedsend.c        |  35 ++---
 sysdeps/unix/sysv/linux/opendir.c             |   7 +-
 sysdeps/unix/sysv/linux/pathconf.c            |   5 +-
 sysdeps/unix/sysv/linux/ppoll.c               |  41 +++---
 sysdeps/unix/sysv/linux/pselect.c             |  47 ++++---
 sysdeps/unix/sysv/linux/pselect32.c           |   6 -
 sysdeps/unix/sysv/linux/select.c              |  60 +++------
 sysdeps/unix/sysv/linux/select32.c            |  58 +++++++++
 sysdeps/unix/sysv/linux/semtimedop.c          |  54 ++++----
 sysdeps/unix/sysv/linux/sigtimedwait.c        |  26 ++--
 sysdeps/unix/sysv/linux/time64-support.c      |  23 ----
 sysdeps/unix/sysv/linux/time64-support.h      |  70 ----------
 sysdeps/unix/sysv/linux/timerfd_gettime.c     |  10 +-
 sysdeps/unix/sysv/linux/timerfd_settime.c     |  25 ++--
 sysdeps/unix/sysv/linux/tst-ppoll.c           |  15 +++
 sysdeps/unix/sysv/linux/tst-sigtimedwait.c    |  18 +++
 sysdeps/unix/sysv/linux/tst-timerfd.c         |  29 ++++-
 sysdeps/unix/sysv/linux/ttyname.h             |  10 +-
 sysdeps/unix/sysv/linux/ttyname_r.c           |  16 +--
 sysdeps/unix/sysv/linux/utimensat.c           |  35 +++--
 sysvipc/Makefile                              |   9 ++
 sysvipc/ftok.c                                |   4 +-
 sysvipc/test-sysvsem.c                        |  22 +++-
 time/Makefile                                 |   9 ++
 time/tst-clock_nanosleep.c                    |  40 +++---
 time/tzfile.c                                 |   6 +-
 76 files changed, 851 insertions(+), 566 deletions(-)
 create mode 100644 rt/tst-mqueue10-time64.c
 create mode 100644 rt/tst-mqueue10.c
 create mode 100644 support/support_create_timer.c
 create mode 100644 sysdeps/unix/sysv/linux/select32.c
 delete mode 100644 sysdeps/unix/sysv/linux/time64-support.c
 delete mode 100644 sysdeps/unix/sysv/linux/time64-support.h
  

Hi Adhemerval,

> Hi Carlos and Lukasz,
> 
> This patchset adds more y2038 fixes that I would like to push for
> 2.34. The first two patches enable more internal y2038 usage on glibc
> and also for the installed programs.  Making glibc the first consumer
> of these newer interfaces should improve coverage.
> 

Those two patches were already available for review for some time.
Indeed, by using 64-bit time internally we may have more testing
(sooner).

> The next 6 patches removes an optimization I added on some specific
> syscall, where a global variable it atomically set once glibc detects
> that kernel supports 64-bit time.  The problem with this approach
> breaks the usage case of live migration like CRIU or similar.

I do recall that this optimization was to avoid performance regression
on legacy systems - to avoid issuing two syscalls (64 bit one and then
32 bit one).

>  Also
> for some interfaces that do not query information from kernel, most
> usages can be optimized away by either building glibc with a minimum
> 5.1 kernel or by using the 32-bit syscall for the common case.

I do think that this approach (with classification of use cases based
on specified kernel version) aligns with how people will build setups
for those systems (at least I do it in this way for OE/Yocto).

> 
> So the following patch implement this optimization for the interfaces
> that allows it.  If the provided timeout can be fit in old 32-bit
> time_t, the old syscall is used instead.  This optimization is only
> used for !__ASSUME_TIME64_SYSCALLS, otherwise the newer 64-bit
> syscall are used.

Ok.

> 
> I checked this with 3 different configation which should cover most
> usages:
> 
>   - i686-linux-gnu on a 4.15 kernel and default --enable-kernel=3.2.
>     This uses the 32-bit optimization to avoid calling the newer
>     64-bit syscall when possible.  It also triggers and EOVERFLOW
>     when 64-bit inputs are used without proper kernel support.

Ok.

>   - i686-linux-gnu on a 5.11 kernel and default --enable-kernel=3.2.
>     This will also use the 32-bit optimization, but it will always
>     succeded due proper kernel support.

Ok.

>   - i686-linux-gnu on a 5.11 kernel and with --enable-kernel=5.1.
>     This enables __ASSUME_TIME64_SYSCALLS and only uses 64-bit
>     time_t syscalls.

Ok.

BTW: I'm wondering when the minimal, supported Linux kernel version is
going to be moved forward?

> 
> Adhemerval Zanella (18):
>   Use 64 bit time_t stat internally
>   Use LFS and 64 bit time for installed programs
>   support: Add support_create_timer
>   linux: Only use 64-bit syscall if required for ppoll
>   linux: Only use 64-bit syscall if required for pselect
>   linux: Only use 64-bit syscall if required for select
>   linux: Remove supports_time64 () from clock_getres
>   linux: Remove supports_time64 () from clock_gettime
>   linux: Remove time64-support
>   linux: timerfd_gettime minor cleanup
>   linux: Only use 64-bit syscall if required for semtimedop
>   linux: Only use 64-bit syscall if required for timerfd_settime
>   linux: Only use 64-bit syscall if required for mq_timedreceive
>   linux: Only use 64-bit syscall if required for mq_timedsend
>   linux: Only use 64-bit syscall if required for sigtimedwait
>   linux: Only use 64-bit syscall if required for utimensat family
>   linux: Only use 64-bit syscall if required for internal futex
>   linux: Only use 64-bit syscall if required for clock_nanosleep
> 
>  Makeconfig                                    |   8 +-
>  Makerules                                     |  12 +-
>  csu/check_fds.c                               |   8 +-
>  elf/dl-load.c                                 |   8 +-
>  elf/dl-misc.c                                 |   4 +-
>  elf/dl-profile.c                              |   4 +-
>  iconv/gconv_cache.c                           |   4 +-
>  include/dirent.h                              |   2 +-
>  include/file_change_detection.h               |   6 +-
>  include/sys/select.h                          |   5 +
>  inet/rcmd.c                                   |   6 +-
>  intl/loadmsgcat.c                             |   4 +-
>  io/Makefile                                   |   4 +-
>  io/file_change_detection.c                    |  16 ++-
>  io/getdirname.c                               |   6 +-
>  libio/filedoalloc.c                           |   2 +-
>  libio/fileops.c                               |   8 +-
>  libio/oldfileops.c                            |   2 +-
>  libio/wfileops.c                              |   2 +-
>  locale/loadarchive.c                          |   8 +-
>  locale/loadlocale.c                           |   6 +-
>  locale/localeinfo.h                           |   2 +-
>  misc/Makefile                                 |  11 ++
>  misc/tst-pselect.c                            | 120
> ++++++++---------- misc/tst-select.c                             |
> 39 +++--- nptl/futex-internal.c                         |  52 +++++---
>  nscd/nscd_helper.c                            |   4 +-
>  nss/nss_database.c                            |   4 +-
>  rt/Makefile                                   |   4 +-
>  rt/tst-mqueue10-time64.c                      |   1 +
>  rt/tst-mqueue10.c                             |  72 +++++++++++
>  support/Makefile                              |   1 +
>  support/support.h                             |  11 ++
>  support/support_create_timer.c                |  69 ++++++++++
>  sysdeps/nptl/futex-internal.h                 |  24 ++--
>  sysdeps/posix/dl-fileid.h                     |   4 +-
>  sysdeps/posix/euidaccess.c                    |   4 +-
>  sysdeps/posix/getaddrinfo.c                   |  21 +--
>  sysdeps/posix/getcwd.c                        |  15 ++-
>  sysdeps/posix/pathconf.c                      |   4 +-
>  sysdeps/posix/sysconf.c                       |   4 +-
>  sysdeps/posix/tempname.c                      |   8 +-
>  sysdeps/unix/sysv/linux/Makefile              |  15 ++-
>  sysdeps/unix/sysv/linux/clock_getres.c        |  16 +--
>  sysdeps/unix/sysv/linux/clock_gettime.c       |  14 +-
>  sysdeps/unix/sysv/linux/clock_nanosleep.c     |  47 ++++---
>  sysdeps/unix/sysv/linux/fdopendir.c           |   4 +-
>  sysdeps/unix/sysv/linux/fexecve.c             |   4 +-
>  .../unix/sysv/linux/microblaze/pselect32.c    |   3 +-
>  sysdeps/unix/sysv/linux/mq_timedreceive.c     |  35 ++---
>  sysdeps/unix/sysv/linux/mq_timedsend.c        |  35 ++---
>  sysdeps/unix/sysv/linux/opendir.c             |   7 +-
>  sysdeps/unix/sysv/linux/pathconf.c            |   5 +-
>  sysdeps/unix/sysv/linux/ppoll.c               |  41 +++---
>  sysdeps/unix/sysv/linux/pselect.c             |  47 ++++---
>  sysdeps/unix/sysv/linux/pselect32.c           |   6 -
>  sysdeps/unix/sysv/linux/select.c              |  60 +++------
>  sysdeps/unix/sysv/linux/select32.c            |  58 +++++++++
>  sysdeps/unix/sysv/linux/semtimedop.c          |  54 ++++----
>  sysdeps/unix/sysv/linux/sigtimedwait.c        |  26 ++--
>  sysdeps/unix/sysv/linux/time64-support.c      |  23 ----
>  sysdeps/unix/sysv/linux/time64-support.h      |  70 ----------
>  sysdeps/unix/sysv/linux/timerfd_gettime.c     |  10 +-
>  sysdeps/unix/sysv/linux/timerfd_settime.c     |  25 ++--
>  sysdeps/unix/sysv/linux/tst-ppoll.c           |  15 +++
>  sysdeps/unix/sysv/linux/tst-sigtimedwait.c    |  18 +++
>  sysdeps/unix/sysv/linux/tst-timerfd.c         |  29 ++++-
>  sysdeps/unix/sysv/linux/ttyname.h             |  10 +-
>  sysdeps/unix/sysv/linux/ttyname_r.c           |  16 +--
>  sysdeps/unix/sysv/linux/utimensat.c           |  35 +++--
>  sysvipc/Makefile                              |   9 ++
>  sysvipc/ftok.c                                |   4 +-
>  sysvipc/test-sysvsem.c                        |  22 +++-
>  time/Makefile                                 |   9 ++
>  time/tst-clock_nanosleep.c                    |  40 +++---
>  time/tzfile.c                                 |   6 +-
>  76 files changed, 851 insertions(+), 566 deletions(-)
>  create mode 100644 rt/tst-mqueue10-time64.c
>  create mode 100644 rt/tst-mqueue10.c
>  create mode 100644 support/support_create_timer.c
>  create mode 100644 sysdeps/unix/sysv/linux/select32.c
>  delete mode 100644 sysdeps/unix/sysv/linux/time64-support.c
>  delete mode 100644 sysdeps/unix/sysv/linux/time64-support.h
> 



Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de
  

On 17/06/2021 11:20, Lukasz Majewski wrote:
> Hi Adhemerval,
> 
>> Hi Carlos and Lukasz,
>>
>> This patchset adds more y2038 fixes that I would like to push for
>> 2.34. The first two patches enable more internal y2038 usage on glibc
>> and also for the installed programs.  Making glibc the first consumer
>> of these newer interfaces should improve coverage.
>>
> 
> Those two patches were already available for review for some time.
> Indeed, by using 64-bit time internally we may have more testing
> (sooner).

Yes, but I changed slight the one that enabled the y2038 symbol on
installed program to use apply the CFLAGS on module, instead of
setting in each sub Makefile.

> 
>> The next 6 patches removes an optimization I added on some specific
>> syscall, where a global variable it atomically set once glibc detects
>> that kernel supports 64-bit time.  The problem with this approach
>> breaks the usage case of live migration like CRIU or similar.
> 
> I do recall that this optimization was to avoid performance regression
> on legacy systems - to avoid issuing two syscalls (64 bit one and then
> 32 bit one).

Yes, but as I noted some interfaces don't really need to use the newer
syscall in the compatibility mode (where glibc is build to support older
kernels). This optimization is really tricky, since it incurs in some
data races (I think to fully avoid it would require seq-cst atomic instead
of relaxed ones) and it might break live migration.

> 
>>  Also
>> for some interfaces that do not query information from kernel, most
>> usages can be optimized away by either building glibc with a minimum
>> 5.1 kernel or by using the 32-bit syscall for the common case.
> 
> I do think that this approach (with classification of use cases based
> on specified kernel version) aligns with how people will build setups
> for those systems (at least I do it in this way for OE/Yocto).

I think there will be specific cases where kernel update is not an
option, but I think these will be really niches. 

> 
>>
>> So the following patch implement this optimization for the interfaces
>> that allows it.  If the provided timeout can be fit in old 32-bit
>> time_t, the old syscall is used instead.  This optimization is only
>> used for !__ASSUME_TIME64_SYSCALLS, otherwise the newer 64-bit
>> syscall are used.
> 
> Ok.
> 
>>
>> I checked this with 3 different configation which should cover most
>> usages:
>>
>>   - i686-linux-gnu on a 4.15 kernel and default --enable-kernel=3.2.
>>     This uses the 32-bit optimization to avoid calling the newer
>>     64-bit syscall when possible.  It also triggers and EOVERFLOW
>>     when 64-bit inputs are used without proper kernel support.
> 
> Ok.
> 
>>   - i686-linux-gnu on a 5.11 kernel and default --enable-kernel=3.2.
>>     This will also use the 32-bit optimization, but it will always
>>     succeded due proper kernel support.
> 
> Ok.
> 
>>   - i686-linux-gnu on a 5.11 kernel and with --enable-kernel=5.1.
>>     This enables __ASSUME_TIME64_SYSCALLS and only uses 64-bit
>>     time_t syscalls.
> 
> Ok.
> 
> BTW: I'm wondering when the minimal, supported Linux kernel version is
> going to be moved forward?

We usually follow the minimum LTS supported by Linux kernel developers.
  

On Thu, 17 Jun 2021, Adhemerval Zanella via Libc-alpha wrote:

> > BTW: I'm wondering when the minimal, supported Linux kernel version is
> > going to be moved forward?
> 
> We usually follow the minimum LTS supported by Linux kernel developers.

However, updates are usually only done when they actually allow 
significant cleanups in glibc.  And given there isn't much benefit on 
x86_64 to increasing the minimum version, it seems like a good idea to 
first implement Carlos's proposal to remove the error at glibc startup for 
an old kernel version number (so people trying to use new glibc in 
containers on old kernels don't automatically get everything failing at 
startup, and may well have things work especially if a few syscalls have 
been backported to the old kernel).

Once Carlos's proposal is implemented, a 4.4 minimum *would* allow 
significant cleanups, such as removing most of the socketcall support.  
But it would be important to *check* the kernel support in 4.4 for each 
feature carefully before assuming it to be present globally.  (For 
example, when removing the socketcall support, we can't do so for 
getpeername or getsockname until the minimum kernel version is 4.20 or 
later (so not until 2025 based on the current EOL dates at 
<https://www.kernel.org/category/releases.html>), because that's when 
those syscalls were added to the compat syscall table for 32-bit SPARC 
programs running on 64-bit kernels.)