time: Use CLOCK_REALTIME for time (BZ #30200)

  Different than gettimeofday and timespec_get, time uses
CLOCK_REALTIME_COARSE instead of CLOCK_REALTIME on Linux.  The
coarse time is used mostly as optimization, but it may show
divergence progression due the clock resolution.

For x86_64 and powerpc64, it should add slight more latency since
it would call now clock_gettime internally.

Checked on x86_64-linux-gnu.
---
 include/time.h                           |  5 ++-
 manual/time.texi                         |  4 +--
 sysdeps/generic/time-clockid.h           | 20 ------------
 sysdeps/mach/hurd/bits/errno.h           |  1 -
 sysdeps/unix/sysv/linux/dl-vdso-setup.c  |  3 --
 sysdeps/unix/sysv/linux/dl-vdso-setup.h  |  3 --
 sysdeps/unix/sysv/linux/i386/sysdep.h    |  1 -
 sysdeps/unix/sysv/linux/powerpc/sysdep.h |  1 -
 sysdeps/unix/sysv/linux/powerpc/time.c   | 22 -------------
 sysdeps/unix/sysv/linux/time-clockid.h   | 22 -------------
 sysdeps/unix/sysv/linux/time.c           | 41 +++---------------------
 sysdeps/unix/sysv/linux/x86/time.c       | 22 -------------
 sysdeps/unix/sysv/linux/x86_64/sysdep.h  |  1 -
 time/time.c                              |  3 +-
 14 files changed, 8 insertions(+), 141 deletions(-)
 delete mode 100644 sysdeps/generic/time-clockid.h
 delete mode 100644 sysdeps/unix/sysv/linux/powerpc/time.c
 delete mode 100644 sysdeps/unix/sysv/linux/time-clockid.h
 delete mode 100644 sysdeps/unix/sysv/linux/x86/time.c
  

* Adhemerval Zanella:

> Different than gettimeofday and timespec_get, time uses
> CLOCK_REALTIME_COARSE instead of CLOCK_REALTIME on Linux.  The
> coarse time is used mostly as optimization, but it may show
> divergence progression due the clock resolution.
>
> For x86_64 and powerpc64, it should add slight more latency since
> it would call now clock_gettime internally.

It seems really significant on x86-64.

Before:

min:        14 ns
25%:        16 ns
50%:        17 ns
75%:        17 ns
95%:        18 ns
99%:        18 ns
max:     18722 ns
avg:   16.6606 ns

After:

min:        29 ns
25%:        31 ns
50%:        31 ns
75%:        32 ns
95%:        32 ns
99%:        33 ns
max:     12161 ns
avg:   31.2205 ns

And of those original 17 ns, quite a bit is overhead from the
benchmarking loop.  I guess applications could work around it by having
a background timer thread that increments a global variable and use that
instead of the time function call, but that seems not a great approach.

Based on previous feedback, I expect we'd have to carry a downstream
revert of this patch indefinitely, so I'm rather strongly against
applying it upstrean.

Thanks,
Florian
  

On 07/03/23 08:11, Florian Weimer wrote:
> * Adhemerval Zanella:
> 
>> Different than gettimeofday and timespec_get, time uses
>> CLOCK_REALTIME_COARSE instead of CLOCK_REALTIME on Linux.  The
>> coarse time is used mostly as optimization, but it may show
>> divergence progression due the clock resolution.
>>
>> For x86_64 and powerpc64, it should add slight more latency since
>> it would call now clock_gettime internally.
> 
> It seems really significant on x86-64.
> 
> Before:
> 
> min:        14 ns
> 25%:        16 ns
> 50%:        17 ns
> 75%:        17 ns
> 95%:        18 ns
> 99%:        18 ns
> max:     18722 ns
> avg:   16.6606 ns
> 
> After:
> 
> min:        29 ns
> 25%:        31 ns
> 50%:        31 ns
> 75%:        32 ns
> 95%:        32 ns
> 99%:        33 ns
> max:     12161 ns
> avg:   31.2205 ns
> 
> And of those original 17 ns, quite a bit is overhead from the
> benchmarking loop.  I guess applications could work around it by having
> a background timer thread that increments a global variable and use that
> instead of the time function call, but that seems not a great approach.

Yes, this is expected since time call will be route through clock_gettime.
Another fix would be to convince kernels developers to use CLOCK_REALTIME
on vDSO as well.

> 
> Based on previous feedback, I expect we'd have to carry a downstream
> revert of this patch indefinitely, so I'm rather strongly against
> applying it upstrean.

To me it really seems like a over-optimization specially because 'time'
has only second resolution.
  

* Adhemerval Zanella Netto:

>> And of those original 17 ns, quite a bit is overhead from the
>> benchmarking loop.  I guess applications could work around it by having
>> a background timer thread that increments a global variable and use that
>> instead of the time function call, but that seems not a great approach.
>
> Yes, this is expected since time call will be route through clock_gettime.
> Another fix would be to convince kernels developers to use CLOCK_REALTIME
> on vDSO as well.

No, that won't help.  I think the crucial aspect for good x86-64
performance is that by using the time entry point, we tell the kernel
that it does not have to obtain microsecond or millisecond precision.

>> Based on previous feedback, I expect we'd have to carry a downstream
>> revert of this patch indefinitely, so I'm rather strongly against
>> applying it upstrean.
>
> To me it really seems like a over-optimization specially because 'time'
> has only second resolution.

I'm afraid that this will impact logging performance significantly in
some scenarios.

Thanks,
Florian
  

On 07/03/23 08:51, Florian Weimer wrote:
> * Adhemerval Zanella Netto:
> 
>>> And of those original 17 ns, quite a bit is overhead from the
>>> benchmarking loop.  I guess applications could work around it by having
>>> a background timer thread that increments a global variable and use that
>>> instead of the time function call, but that seems not a great approach.
>>
>> Yes, this is expected since time call will be route through clock_gettime.
>> Another fix would be to convince kernels developers to use CLOCK_REALTIME
>> on vDSO as well.
> 
> No, that won't help.  I think the crucial aspect for good x86-64
> performance is that by using the time entry point, we tell the kernel
> that it does not have to obtain microsecond or millisecond precision.
> 
>>> Based on previous feedback, I expect we'd have to carry a downstream
>>> revert of this patch indefinitely, so I'm rather strongly against
>>> applying it upstrean.
>>
>> To me it really seems like a over-optimization specially because 'time'
>> has only second resolution.
> 
> I'm afraid that this will impact logging performance significantly in
> some scenarios.

Alright, so I think we should close BZ#30200 as wontfix and state it is
done for performance reasons.
  

* Adhemerval Zanella Netto:

> Alright, so I think we should close BZ#30200 as wontfix and state it is
> done for performance reasons.

Probably RESOLVED/MOVED because we get whatever time the kernel
provides.  It seems a deliberate kernel decision to me.

Thanks,
Florian
  

On 2023-03-07 04:07, Florian Weimer via Libc-alpha wrote:
> Probably RESOLVED/MOVED because we get whatever time the kernel
> provides.  It seems a deliberate kernel decision to me.

For my own information, is 'time' the only standard function that uses a 
clock other than CLOCK_REALTIME and so can be out-of-sync with 
timespec_get? What about gettimeofday, futimens with UTIME_NOW, or 
timestamps set by modifying files?

My idea is to go through the apps I help maintain, and make sure that 
they never call 'time' anywhere that it's important that a timestamp be 
in sync with with the rest of the system, since Glibc 'time' is out of 
sync in that way.
  

* Paul Eggert:

> On 2023-03-07 04:07, Florian Weimer via Libc-alpha wrote:
>> Probably RESOLVED/MOVED because we get whatever time the kernel
>> provides.  It seems a deliberate kernel decision to me.
>
> For my own information, is 'time' the only standard function that uses
> a clock other than CLOCK_REALTIME and so can be out-of-sync with
> timespec_get? What about gettimeofday, futimens with UTIME_NOW, or
> timestamps set by modifying files?

Even if the same underlying clock is used, rounding may cause interface
which use different precision for the fractional seconds part to report
different integral seconds.  The standard does not require rounding
towards minus infinity.

Thanks,
Florian
  

Paul Eggert wrote:
> My idea is to go through the apps I help maintain, and make sure that 
> they never call 'time' anywhere that it's important that a timestamp be 
> in sync with with the rest of the system

Alternatively, these applications can continue to call 'time', if the
package uses the Gnulib module 'time' that provides a workaround against
https://sourceware.org/bugzilla/show_bug.cgi?id=30200 .

Bruno
  

On 08/03/23 13:23, Bruno Haible wrote:
> Paul Eggert wrote:
>> My idea is to go through the apps I help maintain, and make sure that 
>> they never call 'time' anywhere that it's important that a timestamp be 
>> in sync with with the rest of the system
> 
> Alternatively, these applications can continue to call 'time', if the
> package uses the Gnulib module 'time' that provides a workaround against
> https://sourceware.org/bugzilla/show_bug.cgi?id=30200 .
> 
> Bruno

Florian, do you really think that and latency increase of roughly 15ns 
is really worth all the trouble gnulib is pushing? It means that we will 
end up with programs that use CLOCK_REALTIME, while other use 
CLOCK_REALTIME_COURSE.  

If users really to squeeze more performance, they can use clock_gettime
with CLOCK_REALTIME_COURSE.  It should have similar performance to
time vDSO.
  

* Adhemerval Zanella Netto:

> On 08/03/23 13:23, Bruno Haible wrote:
>> Paul Eggert wrote:
>>> My idea is to go through the apps I help maintain, and make sure that 
>>> they never call 'time' anywhere that it's important that a timestamp be 
>>> in sync with with the rest of the system
>> 
>> Alternatively, these applications can continue to call 'time', if the
>> package uses the Gnulib module 'time' that provides a workaround against
>> https://sourceware.org/bugzilla/show_bug.cgi?id=30200 .
>> 
>> Bruno
>
> Florian, do you really think that and latency increase of roughly 15ns 
> is really worth all the trouble gnulib is pushing? It means that we will 
> end up with programs that use CLOCK_REALTIME, while other use 
> CLOCK_REALTIME_COURSE.  
>
> If users really to squeeze more performance, they can use clock_gettime
> with CLOCK_REALTIME_COURSE.  It should have similar performance to
> time vDSO.

Eh, I think the difference derives from using the time vDSO entrypoint
(which your patch removed as well on x86-64).  I don't see a performance
difference between CLOCK_REALTIME and CLOCK_REALTIME_COARSE, so that's
not it.

Thanks,
Florian
  

Adhemerval Zanella Netto wrote:
> ... all the trouble gnulib is pushing?

The trouble does not originate in gnulib; it originate through the
use of inconsistent time APIs.

On a typical system, there are
  - hundreds of programs that call gettimeofday() or timespec_get(),
  - hundreds of programs that call time().
Find attached the lists of programs in /usr/bin on my machine:
  - 425 programs that call gettimeofday() or timespec_get(),
  - 730 programs that call time().

Each time a program calls gettimeofday() or timespec_get(), and then,
within 1 to 3 milliseconds, a program (the same or a different one)
calls time(), there is the potential for trouble because the time
appears to go backwards. This trouble can occur with probability
between 0.1% and 0.3%, according to the data in
https://sourceware.org/bugzilla/show_bug.cgi?id=30200 .

Gnulib attempts to reduce the trouble for specific programs, by
moving them from the "call time()" camp to the "call gettimeofday() or
timespec_get()" camp.

Bruno
/usr/bin/aarch64-linux-gnu-as
/usr/bin/aarch64-linux-gnu-cpp
/usr/bin/aarch64-linux-gnu-cpp-11
/usr/bin/aarch64-linux-gnu-g++
/usr/bin/aarch64-linux-gnu-g++-11
/usr/bin/aarch64-linux-gnu-gcc
/usr/bin/aarch64-linux-gnu-gcc-11
/usr/bin/aarch64-linux-gnu-gcov
/usr/bin/aarch64-linux-gnu-gcov-11
/usr/bin/aarch64-linux-gnu-gcov-dump
/usr/bin/aarch64-linux-gnu-gcov-dump-11
/usr/bin/aarch64-linux-gnu-gcov-tool
/usr/bin/aarch64-linux-gnu-gcov-tool-11
/usr/bin/aarch64-linux-gnu-lto-dump-11
/usr/bin/ab
/usr/bin/afl-fuzz
/usr/bin/airscan-discover
/usr/bin/akonadi_pop3_resource
/usr/bin/aleph
/usr/bin/alpha-linux-gnu-as
/usr/bin/alpha-linux-gnu-cpp
/usr/bin/alpha-linux-gnu-cpp-11
/usr/bin/alpha-linux-gnu-g++
/usr/bin/alpha-linux-gnu-g++-11
/usr/bin/alpha-linux-gnu-gcc
/usr/bin/alpha-linux-gnu-gcc-11
/usr/bin/alpha-linux-gnu-gcov
/usr/bin/alpha-linux-gnu-gcov-11
/usr/bin/alpha-linux-gnu-gcov-dump
/usr/bin/alpha-linux-gnu-gcov-dump-11
/usr/bin/alpha-linux-gnu-gcov-tool
/usr/bin/alpha-linux-gnu-gcov-tool-11
/usr/bin/alpha-linux-gnu-lto-dump-11
/usr/bin/aplay
/usr/bin/apt-ftparchive
/usr/bin/aptitude
/usr/bin/aptitude-curses
/usr/bin/arecord
/usr/bin/arj
/usr/bin/arm-linux-gnueabi-as
/usr/bin/arm-linux-gnueabi-cpp
/usr/bin/arm-linux-gnueabi-cpp-11
/usr/bin/arm-linux-gnueabi-g++
/usr/bin/arm-linux-gnueabi-g++-11
/usr/bin/arm-linux-gnueabi-gcc
/usr/bin/arm-linux-gnueabi-gcc-11
/usr/bin/arm-linux-gnueabi-gcov
/usr/bin/arm-linux-gnueabi-gcov-11
/usr/bin/arm-linux-gnueabi-gcov-dump
/usr/bin/arm-linux-gnueabi-gcov-dump-11
/usr/bin/arm-linux-gnueabi-gcov-tool
/usr/bin/arm-linux-gnueabi-gcov-tool-11
/usr/bin/arm-linux-gnueabihf-as
/usr/bin/arm-linux-gnueabihf-cpp
/usr/bin/arm-linux-gnueabihf-cpp-11
/usr/bin/arm-linux-gnueabihf-g++
/usr/bin/arm-linux-gnueabihf-g++-11
/usr/bin/arm-linux-gnueabihf-gcc
/usr/bin/arm-linux-gnueabihf-gcc-11
/usr/bin/arm-linux-gnueabihf-gcov
/usr/bin/arm-linux-gnueabihf-gcov-11
/usr/bin/arm-linux-gnueabihf-gcov-dump
/usr/bin/arm-linux-gnueabihf-gcov-dump-11
/usr/bin/arm-linux-gnueabihf-gcov-tool
/usr/bin/arm-linux-gnueabihf-gcov-tool-11
/usr/bin/arm-linux-gnueabihf-lto-dump-11
/usr/bin/arm-linux-gnueabi-lto-dump-11
/usr/bin/as
/usr/bin/awk
/usr/bin/bash
/usr/bin/bison
/usr/bin/bsdcpio
/usr/bin/bsdtar
/usr/bin/btrfs
/usr/bin/btrfsck
/usr/bin/btrfs-convert
/usr/bin/btrfs-find-root
/usr/bin/btrfs-image
/usr/bin/btrfs-map-logical
/usr/bin/btrfs-select-super
/usr/bin/btrfstune
/usr/bin/c++
/usr/bin/cabal
/usr/bin/cacaclock
/usr/bin/cal
/usr/bin/cc
/usr/bin/cdrdao
/usr/bin/chage
/usr/bin/class-11d.x
/usr/bin/class-4d.x
/usr/bin/class-5d.x
/usr/bin/class-6d.x
/usr/bin/class.x
/usr/bin/cli
/usr/bin/cli-ildasm
/usr/bin/cmake
/usr/bin/cpack
/usr/bin/cpio
/usr/bin/cpp
/usr/bin/cpp-11
/usr/bin/crontab
/usr/bin/cryfs
/usr/bin/cryfs-unmount
/usr/bin/ctest
/usr/bin/curl
/usr/bin/date
/usr/bin/db5.3_checkpoint
/usr/bin/db5.3_deadlock
/usr/bin/db5.3_hotbackup
/usr/bin/db5.3_recover
/usr/bin/db5.3_replicate
/usr/bin/db_checkpoint
/usr/bin/db_deadlock
/usr/bin/db_hotbackup
/usr/bin/db_recover
/usr/bin/db_replicate
/usr/bin/debian-distro-info
/usr/bin/dig
/usr/bin/dirmngr
/usr/bin/dirmngr-client
/usr/bin/distro-info
/usr/bin/dmesg
/usr/bin/dpkg
/usr/bin/dpkg-deb
/usr/bin/dpkg-divert
/usr/bin/dpkg-query
/usr/bin/dpkg-split
/usr/bin/dumpimage
/usr/bin/dvilualatex
/usr/bin/dvilualatex-dev
/usr/bin/dviluatex
/usr/bin/dvipdfm
/usr/bin/dvipdfmx
/usr/bin/dvipos
/usr/bin/dvips
/usr/bin/dvisvgm
/usr/bin/dvitomp
/usr/bin/ebb
/usr/bin/ecm
/usr/bin/editor
/usr/bin/encfs
/usr/bin/eptex
/usr/bin/etex
/usr/bin/euptex
/usr/bin/ex
/usr/bin/expiry
/usr/bin/extractbb
/usr/bin/f77
/usr/bin/f95
/usr/bin/faillog
/usr/bin/faked-sysv
/usr/bin/ffmpeg
/usr/bin/ffplay
/usr/bin/ffprobe
/usr/bin/find
/usr/bin/fiwalk
/usr/bin/fluid
/usr/bin/fonttosfnt
/usr/bin/foo2ddst
/usr/bin/foo2hiperc
/usr/bin/foo2lava
/usr/bin/foo2oak
/usr/bin/foo2qpdl
/usr/bin/foo2xqx
/usr/bin/foo2zjs
/usr/bin/foomatic-rip
/usr/bin/g++
/usr/bin/g++-11
/usr/bin/gawk
/usr/bin/gcal
/usr/bin/gcc
/usr/bin/gcc-11
/usr/bin/gdb
/usr/bin/gdb-multiarch
/usr/bin/gdc
/usr/bin/gdc-11
/usr/bin/gdomap
/usr/bin/genisoimage
/usr/bin/genrb
/usr/bin/gfan
/usr/bin/gfan_bases
/usr/bin/gfan_buchberger
/usr/bin/gfan_combinerays
/usr/bin/gfan_doesidealcontain
/usr/bin/gfan_fancommonrefinement
/usr/bin/gfan_fanhomology
/usr/bin/gfan_fanisbalanced
/usr/bin/gfan_fanlink
/usr/bin/gfan_fanproduct
/usr/bin/gfan_fansubfan
/usr/bin/gfan_genericlinearchange
/usr/bin/gfan_groebnercone
/usr/bin/gfan_groebnerfan
/usr/bin/gfan_homogeneityspace
/usr/bin/gfan_homogenize
/usr/bin/gfan_initialforms
/usr/bin/gfan_interactive
/usr/bin/gfan_ismarkedgroebnerbasis
/usr/bin/gfan_krulldimension
/usr/bin/gfan_latticeideal
/usr/bin/gfan_leadingterms
/usr/bin/gfan_list
/usr/bin/gfan_markpolynomialset
/usr/bin/gfan_minkowskisum
/usr/bin/gfan_minors
/usr/bin/gfan_mixedvolume
/usr/bin/gfan_overintegers
/usr/bin/gfan_padic
/usr/bin/gfan_polynomialsetunion
/usr/bin/gfan_render
/usr/bin/gfan_renderstaircase
/usr/bin/gfan_resultantfan
/usr/bin/gfan_saturation
/usr/bin/gfan_secondaryfan
/usr/bin/gfan_stats
/usr/bin/gfan_substitute
/usr/bin/gfan_symmetries
/usr/bin/gfan_tolatex
/usr/bin/gfan_topolyhedralfan
/usr/bin/gfan_tropicalbasis
/usr/bin/gfan_tropicalbruteforce
/usr/bin/gfan_tropicalcurve
/usr/bin/gfan_tropicalevaluation
/usr/bin/gfan_tropicalfunction
/usr/bin/gfan_tropicalhypersurface
/usr/bin/gfan_tropicalintersection
/usr/bin/gfan_tropicallifting
/usr/bin/gfan_tropicallinearspace
/usr/bin/gfan_tropicalmultiplicity
/usr/bin/gfan_tropicalrank
/usr/bin/gfan_tropicalstartingcone
/usr/bin/gfan_tropicaltraverse
/usr/bin/gfan_tropicalweildivisor
/usr/bin/gfan_version
/usr/bin/gfortran
/usr/bin/gfortran-11
/usr/bin/giac
/usr/bin/gimp
/usr/bin/gimp-2.10
/usr/bin/gimp-console
/usr/bin/gimp-console-2.10
/usr/bin/git
/usr/bin/git-receive-pack
/usr/bin/git-shell
/usr/bin/git-upload-archive
/usr/bin/git-upload-pack
/usr/bin/gmake
/usr/bin/gp
/usr/bin/gp-2.13
/usr/bin/gperf
/usr/bin/gpg
/usr/bin/gpg2
/usr/bin/gpg-agent
/usr/bin/gpgcompose
/usr/bin/gpgconf
/usr/bin/gpg-connect-agent
/usr/bin/gpgsm
/usr/bin/gpgsplit
/usr/bin/gpgtar
/usr/bin/gpgv
/usr/bin/gpg-wks-server
/usr/bin/gpu-manager
/usr/bin/grops
/usr/bin/grub-mkrescue
/usr/bin/gvgen
/usr/bin/hpls
/usr/bin/hppa-linux-gnu-as
/usr/bin/hppa-linux-gnu-cpp
/usr/bin/hppa-linux-gnu-cpp-11
/usr/bin/hppa-linux-gnu-g++
/usr/bin/hppa-linux-gnu-g++-11
/usr/bin/hppa-linux-gnu-gcc
/usr/bin/hppa-linux-gnu-gcc-11
/usr/bin/hppa-linux-gnu-gcov
/usr/bin/hppa-linux-gnu-gcov-11
/usr/bin/hppa-linux-gnu-gcov-dump
/usr/bin/hppa-linux-gnu-gcov-dump-11
/usr/bin/hppa-linux-gnu-gcov-tool
/usr/bin/hppa-linux-gnu-gcov-tool-11
/usr/bin/hppa-linux-gnu-lto-dump-11
/usr/bin/i686-linux-gnu-as
/usr/bin/i686-linux-gnu-cpp
/usr/bin/i686-linux-gnu-cpp-11
/usr/bin/i686-linux-gnu-g++
/usr/bin/i686-linux-gnu-g++-11
/usr/bin/i686-linux-gnu-gcc
/usr/bin/i686-linux-gnu-gcc-11
/usr/bin/i686-linux-gnu-gcov
/usr/bin/i686-linux-gnu-gcov-11
/usr/bin/i686-linux-gnu-gcov-dump
/usr/bin/i686-linux-gnu-gcov-dump-11
/usr/bin/i686-linux-gnu-gcov-tool
/usr/bin/i686-linux-gnu-gcov-tool-11
/usr/bin/i686-linux-gnu-ld
/usr/bin/i686-linux-gnu-ld.bfd
/usr/bin/i686-linux-gnu-lto-dump-11
/usr/bin/icas
/usr/bin/ico
/usr/bin/inimf
/usr/bin/initex
/usr/bin/innochecksum
/usr/bin/ip
/usr/bin/ipptool
/usr/bin/iptables-xml
/usr/bin/isdv4-serial-debugger
/usr/bin/kaidan
/usr/bin/kbxutil
/usr/bin/ksmserver
/usr/bin/kvm
/usr/bin/kwalletd5
/usr/bin/last
/usr/bin/lastb
/usr/bin/lastlog
/usr/bin/latex
/usr/bin/latex-dev
/usr/bin/ld
/usr/bin/ld.bfd
/usr/bin/less
/usr/bin/llvm-cvtres
/usr/bin/llvm-cvtres-13
/usr/bin/llvm-cvtres-14
/usr/bin/llvm-exegesis
/usr/bin/llvm-exegesis-13
/usr/bin/llvm-exegesis-14
/usr/bin/llvm-ml-13
/usr/bin/llvm-ml-14
/usr/bin/login
/usr/bin/lsar
/usr/bin/lsof
/usr/bin/lto-dump-11
/usr/bin/lua
/usr/bin/lua5.4
/usr/bin/luac
/usr/bin/luac5.4
/usr/bin/luahbtex
/usr/bin/luajithbtex
/usr/bin/luajittex
/usr/bin/lualatex
/usr/bin/lualatex-dev
/usr/bin/luatex
/usr/bin/lzop
/usr/bin/m68k-linux-gnu-as
/usr/bin/m68k-linux-gnu-cpp
/usr/bin/m68k-linux-gnu-cpp-11
/usr/bin/m68k-linux-gnu-g++
/usr/bin/m68k-linux-gnu-g++-11
/usr/bin/m68k-linux-gnu-gcc
/usr/bin/m68k-linux-gnu-gcc-11
/usr/bin/m68k-linux-gnu-gcov
/usr/bin/m68k-linux-gnu-gcov-11
/usr/bin/m68k-linux-gnu-gcov-dump
/usr/bin/m68k-linux-gnu-gcov-dump-11
/usr/bin/m68k-linux-gnu-gcov-tool
/usr/bin/m68k-linux-gnu-gcov-tool-11
/usr/bin/m68k-linux-gnu-lto-dump-11
/usr/bin/make
/usr/bin/mariadb
/usr/bin/mariadb-check
/usr/bin/mattrib
/usr/bin/mawk
/usr/bin/mbadblocks
/usr/bin/mcat
/usr/bin/mcd
/usr/bin/mclasserase
/usr/bin/mcopy
/usr/bin/mdel
/usr/bin/mdeltree
/usr/bin/mdir
/usr/bin/mdu
/usr/bin/mf
/usr/bin/mflua
/usr/bin/mfluajit
/usr/bin/mfluajit-nowin
/usr/bin/mflua-nowin
/usr/bin/mf-nowin
/usr/bin/mformat
/usr/bin/mfplain
/usr/bin/minfo
/usr/bin/mips64-linux-gnuabi64-as
/usr/bin/mips64-linux-gnuabi64-cpp
/usr/bin/mips64-linux-gnuabi64-cpp-10
/usr/bin/mips64-linux-gnuabi64-g++
/usr/bin/mips64-linux-gnuabi64-g++-10
/usr/bin/mips64-linux-gnuabi64-gcc
/usr/bin/mips64-linux-gnuabi64-gcc-10
/usr/bin/mips64-linux-gnuabi64-gcov
/usr/bin/mips64-linux-gnuabi64-gcov-10
/usr/bin/mips64-linux-gnuabi64-gcov-dump
/usr/bin/mips64-linux-gnuabi64-gcov-dump-10
/usr/bin/mips64-linux-gnuabi64-gcov-tool
/usr/bin/mips64-linux-gnuabi64-gcov-tool-10
/usr/bin/mips64-linux-gnuabi64-lto-dump-10
/usr/bin/mips-linux-gnu-as
/usr/bin/mips-linux-gnu-cpp
/usr/bin/mips-linux-gnu-cpp-10
/usr/bin/mips-linux-gnu-g++
/usr/bin/mips-linux-gnu-g++-10
/usr/bin/mips-linux-gnu-gcc
/usr/bin/mips-linux-gnu-gcc-10
/usr/bin/mips-linux-gnu-gcov
/usr/bin/mips-linux-gnu-gcov-10
/usr/bin/mips-linux-gnu-gcov-dump
/usr/bin/mips-linux-gnu-gcov-dump-10
/usr/bin/mips-linux-gnu-gcov-tool
/usr/bin/mips-linux-gnu-gcov-tool-10
/usr/bin/mips-linux-gnu-lto-dump-10
/usr/bin/mkimage
/usr/bin/mkisofs
/usr/bin/mksquashfs
/usr/bin/mlabel
/usr/bin/mmcli
/usr/bin/mmd
/usr/bin/mmount
/usr/bin/mmove
/usr/bin/mono
/usr/bin/monodis
/usr/bin/mono-sgen
/usr/bin/mpartition
/usr/bin/mpost
/usr/bin/mrd
/usr/bin/mren
/usr/bin/msginit
/usr/bin/mshortname
/usr/bin/mshowfat
/usr/bin/msxlint
/usr/bin/mtools
/usr/bin/mtoolstest
/usr/bin/mtr
/usr/bin/mtype
/usr/bin/mysql
/usr/bin/mzip
/usr/bin/nano
/usr/bin/nauty-cubhamg
/usr/bin/nawk
/usr/bin/ncal
/usr/bin/ncftp
/usr/bin/ncftp3
/usr/bin/ncftpbatch
/usr/bin/ncftpbookmarks
/usr/bin/ncftpget
/usr/bin/ncftpls
/usr/bin/ncftpput
/usr/bin/ncftpspooler
/usr/bin/nef-11d.x
/usr/bin/nef-4d.x
/usr/bin/nef-5d.x
/usr/bin/nef-6d.x
/usr/bin/nef.x
/usr/bin/net
/usr/bin/nmblookup
/usr/bin/nstat
/usr/bin/ntfssecaudit
/usr/bin/ntfswipe
/usr/bin/ocamldoc.opt
/usr/bin/oclock
/usr/bin/oggCat
/usr/bin/oggCut
/usr/bin/oggSilence
/usr/bin/oggSlideshow
/usr/bin/oggTranscode
/usr/bin/openssl
/usr/bin/pager
/usr/bin/pandoc
/usr/bin/pbmreduce
/usr/bin/pdfetex
/usr/bin/pdflatex
/usr/bin/pdflatex-dev
/usr/bin/pdftex
/usr/bin/pdvitomp
/usr/bin/pedump
/usr/bin/perl
/usr/bin/perl5.34.0
/usr/bin/pgmcrater
/usr/bin/pgmnoise
/usr/bin/pgmtopbm
/usr/bin/pgrep
/usr/bin/php
/usr/bin/php8.1
/usr/bin/pico
/usr/bin/pidwait
/usr/bin/pinentry
/usr/bin/pinentry-qt
/usr/bin/pinentry-x11
/usr/bin/pinky
/usr/bin/pkill
/usr/bin/pmpost
/usr/bin/pnmremap
/usr/bin/pnmtorle
/usr/bin/post-grohtml
/usr/bin/powerpc64le-linux-gnu-as
/usr/bin/powerpc64le-linux-gnu-cpp
/usr/bin/powerpc64le-linux-gnu-cpp-11
/usr/bin/powerpc64le-linux-gnu-g++
/usr/bin/powerpc64le-linux-gnu-g++-11
/usr/bin/powerpc64le-linux-gnu-gcc
/usr/bin/powerpc64le-linux-gnu-gcc-11
/usr/bin/powerpc64le-linux-gnu-gcov
/usr/bin/powerpc64le-linux-gnu-gcov-11
/usr/bin/powerpc64le-linux-gnu-gcov-dump
/usr/bin/powerpc64le-linux-gnu-gcov-dump-11
/usr/bin/powerpc64le-linux-gnu-gcov-tool
/usr/bin/powerpc64le-linux-gnu-gcov-tool-11
/usr/bin/powerpc64le-linux-gnu-lto-dump-11
/usr/bin/powerpc64-linux-gnu-as
/usr/bin/powerpc64-linux-gnu-cpp
/usr/bin/powerpc64-linux-gnu-cpp-11
/usr/bin/powerpc64-linux-gnu-g++
/usr/bin/powerpc64-linux-gnu-g++-11
/usr/bin/powerpc64-linux-gnu-gcc
/usr/bin/powerpc64-linux-gnu-gcc-11
/usr/bin/powerpc64-linux-gnu-gcov
/usr/bin/powerpc64-linux-gnu-gcov-11
/usr/bin/powerpc64-linux-gnu-gcov-dump
/usr/bin/powerpc64-linux-gnu-gcov-dump-11
/usr/bin/powerpc64-linux-gnu-gcov-tool
/usr/bin/powerpc64-linux-gnu-gcov-tool-11
/usr/bin/powerpc64-linux-gnu-lto-dump-11
/usr/bin/powerpc-linux-gnu-as
/usr/bin/powerpc-linux-gnu-cpp
/usr/bin/powerpc-linux-gnu-cpp-11
/usr/bin/powerpc-linux-gnu-g++
/usr/bin/powerpc-linux-gnu-g++-11
/usr/bin/powerpc-linux-gnu-gcc
/usr/bin/powerpc-linux-gnu-gcc-11
/usr/bin/powerpc-linux-gnu-gcov
/usr/bin/powerpc-linux-gnu-gcov-11
/usr/bin/powerpc-linux-gnu-gcov-dump
/usr/bin/powerpc-linux-gnu-gcov-dump-11
/usr/bin/powerpc-linux-gnu-gcov-tool
/usr/bin/powerpc-linux-gnu-gcov-tool-11
/usr/bin/powerpc-linux-gnu-lto-dump-11
/usr/bin/ppmforge
/usr/bin/ppmpat
/usr/bin/ppmquant
/usr/bin/ppmshift
/usr/bin/ppmspread
/usr/bin/ppmtompeg
/usr/bin/ps
/usr/bin/ptex
/usr/bin/python3
/usr/bin/python3.10
/usr/bin/qemu-img
/usr/bin/qemu-io
/usr/bin/qemu-nbd
/usr/bin/qemu-pr-helper
/usr/bin/qemu-storage-daemon
/usr/bin/qemu-system-i386
/usr/bin/qemu-system-x86_64
/usr/bin/qemu-system-x86_64-microvm
/usr/bin/rbash
/usr/bin/rdesktop-vrdp
/usr/bin/rdma
/usr/bin/rearj
/usr/bin/redcheck
/usr/bin/redcheck_gmp
/usr/bin/riscv64-linux-gnu-as
/usr/bin/riscv64-linux-gnu-cpp
/usr/bin/riscv64-linux-gnu-cpp-11
/usr/bin/riscv64-linux-gnu-g++
/usr/bin/riscv64-linux-gnu-g++-11
/usr/bin/riscv64-linux-gnu-gcc
/usr/bin/riscv64-linux-gnu-gcc-11
/usr/bin/riscv64-linux-gnu-gcov
/usr/bin/riscv64-linux-gnu-gcov-11
/usr/bin/riscv64-linux-gnu-gcov-dump
/usr/bin/riscv64-linux-gnu-gcov-dump-11
/usr/bin/riscv64-linux-gnu-gcov-tool
/usr/bin/riscv64-linux-gnu-gcov-tool-11
/usr/bin/riscv64-linux-gnu-lto-dump-11
/usr/bin/rlogin
/usr/bin/r-mpost
/usr/bin/rnano
/usr/bin/rpcclient
/usr/bin/r-pmpost
/usr/bin/rsh
/usr/bin/rsync
/usr/bin/r-upmpost
/usr/bin/rview
/usr/bin/rvim
/usr/bin/rzsh
/usr/bin/s390x-linux-gnu-as
/usr/bin/s390x-linux-gnu-cpp
/usr/bin/s390x-linux-gnu-cpp-11
/usr/bin/s390x-linux-gnu-g++
/usr/bin/s390x-linux-gnu-g++-11
/usr/bin/s390x-linux-gnu-gcc
/usr/bin/s390x-linux-gnu-gcc-11
/usr/bin/s390x-linux-gnu-gcov
/usr/bin/s390x-linux-gnu-gcov-11
/usr/bin/s390x-linux-gnu-gcov-dump
/usr/bin/s390x-linux-gnu-gcov-dump-11
/usr/bin/s390x-linux-gnu-gcov-tool
/usr/bin/s390x-linux-gnu-gcov-tool-11
/usr/bin/s390x-linux-gnu-lto-dump-11
/usr/bin/sbvarsign
/usr/bin/scanimage
/usr/bin/script
/usr/bin/session-migration
/usr/bin/sessreg
/usr/bin/sftp
/usr/bin/sh4-linux-gnu-as
/usr/bin/sh4-linux-gnu-cpp
/usr/bin/sh4-linux-gnu-cpp-11
/usr/bin/sh4-linux-gnu-g++
/usr/bin/sh4-linux-gnu-g++-11
/usr/bin/sh4-linux-gnu-gcc
/usr/bin/sh4-linux-gnu-gcc-11
/usr/bin/sh4-linux-gnu-gcov
/usr/bin/sh4-linux-gnu-gcov-11
/usr/bin/sh4-linux-gnu-gcov-dump
/usr/bin/sh4-linux-gnu-gcov-dump-11
/usr/bin/sh4-linux-gnu-gcov-tool
/usr/bin/sh4-linux-gnu-gcov-tool-11
/usr/bin/sh4-linux-gnu-lto-dump-11
/usr/bin/shred
/usr/bin/slogin
/usr/bin/snapfuse
/usr/bin/sparc64-linux-gnu-as
/usr/bin/sparc64-linux-gnu-cpp
/usr/bin/sparc64-linux-gnu-cpp-11
/usr/bin/sparc64-linux-gnu-g++
/usr/bin/sparc64-linux-gnu-g++-11
/usr/bin/sparc64-linux-gnu-gcc
/usr/bin/sparc64-linux-gnu-gcc-11
/usr/bin/sparc64-linux-gnu-gcov
/usr/bin/sparc64-linux-gnu-gcov-11
/usr/bin/sparc64-linux-gnu-gcov-dump
/usr/bin/sparc64-linux-gnu-gcov-dump-11
/usr/bin/sparc64-linux-gnu-gcov-tool
/usr/bin/sparc64-linux-gnu-gcov-tool-11
/usr/bin/sparc64-linux-gnu-lto-dump-11
/usr/bin/sqfstar
/usr/bin/sqlite3
/usr/bin/ss
/usr/bin/ssh
/usr/bin/ssh-agent
/usr/bin/sshfs
/usr/bin/ssh-keygen
/usr/bin/sudoreplay
/usr/bin/supermin
/usr/bin/surf-alggeo
/usr/bin/surf-alggeo-nox
/usr/bin/t1mac
/usr/bin/tasm
/usr/bin/tcal
/usr/bin/tcpdump
/usr/bin/tex
/usr/bin/tex2aspc
/usr/bin/texlua
/usr/bin/texluac
/usr/bin/texluajit
/usr/bin/texluajitc
/usr/bin/timedatectl
/usr/bin/top
/usr/bin/touch
/usr/bin/troff
/usr/bin/ttf2afm
/usr/bin/ubuntu-distro-info
/usr/bin/unar
/usr/bin/unpack200
/usr/bin/update-alternatives
/usr/bin/updvitomp
/usr/bin/upmpost
/usr/bin/upower
/usr/bin/uptex
/usr/bin/valgrind-di-server
/usr/bin/valgrind-listener
/usr/bin/vi
/usr/bin/view
/usr/bin/vim
/usr/bin/vim.basic
/usr/bin/vimdiff
/usr/bin/vim.tiny
/usr/bin/vmstat
/usr/bin/w
/usr/bin/w3m
/usr/bin/wall
/usr/bin/watch
/usr/bin/watchgnupg
/usr/bin/wget
/usr/bin/who
/usr/bin/whoopsie
/usr/bin/write
/usr/bin/write.ul
/usr/bin/www-browser
/usr/bin/wxrc
/usr/bin/x11perf
/usr/bin/x86_64-linux-gnu-as
/usr/bin/x86_64-linux-gnu-cpp
/usr/bin/x86_64-linux-gnu-cpp-11
/usr/bin/x86_64-linux-gnu-g++
/usr/bin/x86_64-linux-gnu-g++-11
/usr/bin/x86_64-linux-gnu-gcc
/usr/bin/x86_64-linux-gnu-gcc-11
/usr/bin/x86_64-linux-gnu-gdc
/usr/bin/x86_64-linux-gnu-gdc-11
/usr/bin/x86_64-linux-gnu-gfortran
/usr/bin/x86_64-linux-gnu-gfortran-11
/usr/bin/x86_64-linux-gnu-ld
/usr/bin/x86_64-linux-gnu-ld.bfd
/usr/bin/x86_64-linux-gnu-lto-dump-11
/usr/bin/x86_64-linux-gnux32-as
/usr/bin/x86_64-linux-gnux32-cpp
/usr/bin/x86_64-linux-gnux32-cpp-11
/usr/bin/x86_64-linux-gnux32-g++
/usr/bin/x86_64-linux-gnux32-g++-11
/usr/bin/x86_64-linux-gnux32-gcc
/usr/bin/x86_64-linux-gnux32-gcc-11
/usr/bin/x86_64-linux-gnux32-gcov
/usr/bin/x86_64-linux-gnux32-gcov-11
/usr/bin/x86_64-linux-gnux32-gcov-dump
/usr/bin/x86_64-linux-gnux32-gcov-dump-11
/usr/bin/x86_64-linux-gnux32-gcov-tool
/usr/bin/x86_64-linux-gnux32-gcov-tool-11
/usr/bin/x86_64-linux-gnux32-ld
/usr/bin/x86_64-linux-gnux32-ld.bfd
/usr/bin/x86_64-linux-gnux32-lto-dump-11
/usr/bin/xcas
/usr/bin/xclock
/usr/bin/xdvipdfmx
/usr/bin/xedit
/usr/bin/xetex
/usr/bin/xgettext
/usr/bin/xterm
/usr/bin/yasm
/usr/bin/ytasm
/usr/bin/zdump
/usr/bin/zip
/usr/bin/zipcloak
/usr/bin/zsh
/usr/bin/aarch64-linux-gnu-cpp
/usr/bin/aarch64-linux-gnu-cpp-11
/usr/bin/aarch64-linux-gnu-g++
/usr/bin/aarch64-linux-gnu-g++-11
/usr/bin/aarch64-linux-gnu-gcc
/usr/bin/aarch64-linux-gnu-gcc-11
/usr/bin/aarch64-linux-gnu-lto-dump-11
/usr/bin/afl-c++
/usr/bin/afl-cc
/usr/bin/afl-clang-fast
/usr/bin/afl-clang-fast++
/usr/bin/afl-fuzz
/usr/bin/afl-g++
/usr/bin/afl-gcc
/usr/bin/afl-gotcpu
/usr/bin/afl-network-server
/usr/bin/airscan-discover
/usr/bin/alpha-linux-gnu-cpp
/usr/bin/alpha-linux-gnu-cpp-11
/usr/bin/alpha-linux-gnu-g++
/usr/bin/alpha-linux-gnu-g++-11
/usr/bin/alpha-linux-gnu-gcc
/usr/bin/alpha-linux-gnu-gcc-11
/usr/bin/alpha-linux-gnu-lto-dump-11
/usr/bin/alsaloop
/usr/bin/aplay
/usr/bin/aptitude
/usr/bin/aptitude-curses
/usr/bin/arecord
/usr/bin/arj
/usr/bin/arm-linux-gnueabi-cpp
/usr/bin/arm-linux-gnueabi-cpp-11
/usr/bin/arm-linux-gnueabi-g++
/usr/bin/arm-linux-gnueabi-g++-11
/usr/bin/arm-linux-gnueabi-gcc
/usr/bin/arm-linux-gnueabi-gcc-11
/usr/bin/arm-linux-gnueabihf-cpp
/usr/bin/arm-linux-gnueabihf-cpp-11
/usr/bin/arm-linux-gnueabihf-g++
/usr/bin/arm-linux-gnueabihf-g++-11
/usr/bin/arm-linux-gnueabihf-gcc
/usr/bin/arm-linux-gnueabihf-gcc-11
/usr/bin/arm-linux-gnueabihf-lto-dump-11
/usr/bin/arm-linux-gnueabi-lto-dump-11
/usr/bin/bash
/usr/bin/btmon
/usr/bin/btrfs
/usr/bin/btrfsck
/usr/bin/btrfs-convert
/usr/bin/btrfs-find-root
/usr/bin/btrfs-image
/usr/bin/btrfs-map-logical
/usr/bin/btrfs-select-super
/usr/bin/btrfstune
/usr/bin/c++
/usr/bin/cabal
/usr/bin/cc
/usr/bin/cli
/usr/bin/cli-ildasm
/usr/bin/cmake
/usr/bin/cp
/usr/bin/cpack
/usr/bin/cpio
/usr/bin/cpp
/usr/bin/cpp-11
/usr/bin/cryfs
/usr/bin/ctest
/usr/bin/ctstat
/usr/bin/curl
/usr/bin/cvs
/usr/bin/devdump
/usr/bin/dirmngr
/usr/bin/dmesg
/usr/bin/dreadnaut
/usr/bin/dvilualatex
/usr/bin/dvilualatex-dev
/usr/bin/dviluatex
/usr/bin/dvisvgm
/usr/bin/dvitomp
/usr/bin/emacs
/usr/bin/emacs-gtk
/usr/bin/encfs
/usr/bin/eptex
/usr/bin/es2gears_wayland
/usr/bin/es2gears_wayland.x86_64-linux-gnu
/usr/bin/es2gears_x11
/usr/bin/es2gears_x11.x86_64-linux-gnu
/usr/bin/etex
/usr/bin/euptex
/usr/bin/ex
/usr/bin/f77
/usr/bin/f95
/usr/bin/find
/usr/bin/fiwalk
/usr/bin/ftp
/usr/bin/g++
/usr/bin/g++-11
/usr/bin/gcc
/usr/bin/gcc-11
/usr/bin/gdb
/usr/bin/gdb-multiarch
/usr/bin/gdc
/usr/bin/gdc-11
/usr/bin/genisoimage
/usr/bin/gfortran
/usr/bin/gfortran-11
/usr/bin/giac
/usr/bin/git
/usr/bin/git-receive-pack
/usr/bin/git-shell
/usr/bin/git-upload-archive
/usr/bin/git-upload-pack
/usr/bin/glxgears
/usr/bin/glxgears.x86_64-linux-gnu
/usr/bin/gmake
/usr/bin/gp
/usr/bin/gp-2.13
/usr/bin/gpasswd
/usr/bin/grub-editenv
/usr/bin/grub-file
/usr/bin/grub-fstest
/usr/bin/grub-glue-efi
/usr/bin/grub-menulst2cfg
/usr/bin/grub-mkfont
/usr/bin/grub-mkimage
/usr/bin/grub-mklayout
/usr/bin/grub-mknetdir
/usr/bin/grub-mkpasswd-pbkdf2
/usr/bin/grub-mkrelpath
/usr/bin/grub-mkrescue
/usr/bin/grub-mkstandalone
/usr/bin/grub-mount
/usr/bin/grub-render-label
/usr/bin/grub-script-check
/usr/bin/grub-syslinux2cfg
/usr/bin/h5perf_serial
/usr/bin/hppa-linux-gnu-cpp
/usr/bin/hppa-linux-gnu-cpp-11
/usr/bin/hppa-linux-gnu-g++
/usr/bin/hppa-linux-gnu-g++-11
/usr/bin/hppa-linux-gnu-gcc
/usr/bin/hppa-linux-gnu-gcc-11
/usr/bin/hppa-linux-gnu-lto-dump-11
/usr/bin/i686-linux-gnu-cpp
/usr/bin/i686-linux-gnu-cpp-11
/usr/bin/i686-linux-gnu-g++
/usr/bin/i686-linux-gnu-g++-11
/usr/bin/i686-linux-gnu-gcc
/usr/bin/i686-linux-gnu-gcc-11
/usr/bin/i686-linux-gnu-lto-dump-11
/usr/bin/icas
/usr/bin/install
/usr/bin/ip
/usr/bin/ipcmk
/usr/bin/ippfind
/usr/bin/isodump
/usr/bin/isoinfo
/usr/bin/isovfy
/usr/bin/kmod
/usr/bin/kvm
/usr/bin/l2ping
/usr/bin/l2test
/usr/bin/lame
/usr/bin/last
/usr/bin/lastb
/usr/bin/latex
/usr/bin/latex-dev
/usr/bin/lefty
/usr/bin/ln
/usr/bin/lnstat
/usr/bin/logger
/usr/bin/login
/usr/bin/lsar
/usr/bin/lsipc
/usr/bin/lslogins
/usr/bin/lsmod
/usr/bin/lto-dump-11
/usr/bin/ltrace
/usr/bin/luahbtex
/usr/bin/luajithbtex
/usr/bin/luajittex
/usr/bin/lualatex
/usr/bin/lualatex-dev
/usr/bin/luatex
/usr/bin/m68k-linux-gnu-cpp
/usr/bin/m68k-linux-gnu-cpp-11
/usr/bin/m68k-linux-gnu-g++
/usr/bin/m68k-linux-gnu-g++-11
/usr/bin/m68k-linux-gnu-gcc
/usr/bin/m68k-linux-gnu-gcc-11
/usr/bin/m68k-linux-gnu-lto-dump-11
/usr/bin/make
/usr/bin/mcookie
/usr/bin/mfplain
/usr/bin/mips64-linux-gnuabi64-cpp
/usr/bin/mips64-linux-gnuabi64-cpp-10
/usr/bin/mips64-linux-gnuabi64-g++
/usr/bin/mips64-linux-gnuabi64-g++-10
/usr/bin/mips64-linux-gnuabi64-gcc
/usr/bin/mips64-linux-gnuabi64-gcc-10
/usr/bin/mips64-linux-gnuabi64-lto-dump-10
/usr/bin/mips-linux-gnu-cpp
/usr/bin/mips-linux-gnu-cpp-10
/usr/bin/mips-linux-gnu-g++
/usr/bin/mips-linux-gnu-g++-10
/usr/bin/mips-linux-gnu-gcc
/usr/bin/mips-linux-gnu-gcc-10
/usr/bin/mips-linux-gnu-lto-dump-10
/usr/bin/mkisofs
/usr/bin/mktemp
/usr/bin/mokutil
/usr/bin/mono
/usr/bin/monodis
/usr/bin/mono-sgen
/usr/bin/mpost
/usr/bin/mtr
/usr/bin/mtr-packet
/usr/bin/mv
/usr/bin/nauty-genrang
/usr/bin/nauty-hamheuristic
/usr/bin/nauty-ranlabg
/usr/bin/nauty-twohamg
/usr/bin/ncftp
/usr/bin/ncftp3
/usr/bin/ncftpbatch
/usr/bin/ncftpbookmarks
/usr/bin/ncftpget
/usr/bin/ncftpls
/usr/bin/ncftpput
/usr/bin/ncftpspooler
/usr/bin/nstat
/usr/bin/ntfs-3g
/usr/bin/ocamlc
/usr/bin/ocamlcmt
/usr/bin/ocamlc.opt
/usr/bin/ocamlcp
/usr/bin/ocamlcp.opt
/usr/bin/ocamldep
/usr/bin/ocamldep.opt
/usr/bin/ocamldoc.opt
/usr/bin/ocamllex
/usr/bin/ocamllex.opt
/usr/bin/ocamlmklib
/usr/bin/ocamlmklib.opt
/usr/bin/ocamlmktop
/usr/bin/ocamlmktop.opt
/usr/bin/ocamlobjinfo
/usr/bin/ocamlobjinfo.opt
/usr/bin/ocamlopt
/usr/bin/ocamlopt.opt
/usr/bin/ocamloptp
/usr/bin/ocamloptp.opt
/usr/bin/ocamlprof
/usr/bin/ocamlprof.opt
/usr/bin/ocamlrun
/usr/bin/ocamlrund
/usr/bin/ocamlruni
/usr/bin/pandoc
/usr/bin/patch
/usr/bin/pdfetex
/usr/bin/pdflatex
/usr/bin/pdflatex-dev
/usr/bin/pdftex
/usr/bin/pdvitomp
/usr/bin/pedump
/usr/bin/perl
/usr/bin/perl5.34.0
/usr/bin/php
/usr/bin/php8.1
/usr/bin/ping
/usr/bin/ping4
/usr/bin/ping6
/usr/bin/pmpost
/usr/bin/powerpc64le-linux-gnu-cpp
/usr/bin/powerpc64le-linux-gnu-cpp-11
/usr/bin/powerpc64le-linux-gnu-g++
/usr/bin/powerpc64le-linux-gnu-g++-11
/usr/bin/powerpc64le-linux-gnu-gcc
/usr/bin/powerpc64le-linux-gnu-gcc-11
/usr/bin/powerpc64le-linux-gnu-lto-dump-11
/usr/bin/powerpc64-linux-gnu-cpp
/usr/bin/powerpc64-linux-gnu-cpp-11
/usr/bin/powerpc64-linux-gnu-g++
/usr/bin/powerpc64-linux-gnu-g++-11
/usr/bin/powerpc64-linux-gnu-gcc
/usr/bin/powerpc64-linux-gnu-gcc-11
/usr/bin/powerpc64-linux-gnu-lto-dump-11
/usr/bin/powerpc-linux-gnu-cpp
/usr/bin/powerpc-linux-gnu-cpp-11
/usr/bin/powerpc-linux-gnu-g++
/usr/bin/powerpc-linux-gnu-g++-11
/usr/bin/powerpc-linux-gnu-gcc
/usr/bin/powerpc-linux-gnu-gcc-11
/usr/bin/powerpc-linux-gnu-lto-dump-11
/usr/bin/python3
/usr/bin/python3.10
/usr/bin/qemu-img
/usr/bin/qemu-io
/usr/bin/qemu-nbd
/usr/bin/qemu-pr-helper
/usr/bin/qemu-storage-daemon
/usr/bin/qemu-system-i386
/usr/bin/qemu-system-x86_64
/usr/bin/qemu-system-x86_64-microvm
/usr/bin/rbash
/usr/bin/rcp
/usr/bin/rctest
/usr/bin/rdesktop-vrdp
/usr/bin/rearj
/usr/bin/riscv64-linux-gnu-cpp
/usr/bin/riscv64-linux-gnu-cpp-11
/usr/bin/riscv64-linux-gnu-g++
/usr/bin/riscv64-linux-gnu-g++-11
/usr/bin/riscv64-linux-gnu-gcc
/usr/bin/riscv64-linux-gnu-gcc-11
/usr/bin/riscv64-linux-gnu-lto-dump-11
/usr/bin/rlogin
/usr/bin/r-mpost
/usr/bin/r-pmpost
/usr/bin/rsh
/usr/bin/rsync
/usr/bin/rtstat
/usr/bin/r-upmpost
/usr/bin/rview
/usr/bin/rvim
/usr/bin/rzsh
/usr/bin/s390x-linux-gnu-cpp
/usr/bin/s390x-linux-gnu-cpp-11
/usr/bin/s390x-linux-gnu-g++
/usr/bin/s390x-linux-gnu-g++-11
/usr/bin/s390x-linux-gnu-gcc
/usr/bin/s390x-linux-gnu-gcc-11
/usr/bin/s390x-linux-gnu-lto-dump-11
/usr/bin/sane-find-scanner
/usr/bin/scp
/usr/bin/scrub
/usr/bin/sftp
/usr/bin/sh4-linux-gnu-cpp
/usr/bin/sh4-linux-gnu-cpp-11
/usr/bin/sh4-linux-gnu-g++
/usr/bin/sh4-linux-gnu-g++-11
/usr/bin/sh4-linux-gnu-gcc
/usr/bin/sh4-linux-gnu-gcc-11
/usr/bin/sh4-linux-gnu-lto-dump-11
/usr/bin/shred
/usr/bin/shuf
/usr/bin/slogin
/usr/bin/socat
/usr/bin/sort
/usr/bin/sparc64-linux-gnu-cpp
/usr/bin/sparc64-linux-gnu-cpp-11
/usr/bin/sparc64-linux-gnu-g++
/usr/bin/sparc64-linux-gnu-g++-11
/usr/bin/sparc64-linux-gnu-gcc
/usr/bin/sparc64-linux-gnu-gcc-11
/usr/bin/sparc64-linux-gnu-lto-dump-11
/usr/bin/speaker-test
/usr/bin/sqlite3
/usr/bin/ssh
/usr/bin/ssh-agent
/usr/bin/sshfs
/usr/bin/ssh-keygen
/usr/bin/ssh-keyscan
/usr/bin/su
/usr/bin/sudo
/usr/bin/sudoedit
/usr/bin/supermin
/usr/bin/tar
/usr/bin/texlua
/usr/bin/texluac
/usr/bin/texluajit
/usr/bin/texluajitc
/usr/bin/time
/usr/bin/tnftp
/usr/bin/ubuntu-core-launcher
/usr/bin/umax_pp
/usr/bin/unar
/usr/bin/updatedb
/usr/bin/updvitomp
/usr/bin/upmpost
/usr/bin/upower
/usr/bin/uptime
/usr/bin/usbhid-dump
/usr/bin/vbox-img
/usr/bin/vgdb
/usr/bin/vi
/usr/bin/view
/usr/bin/vim
/usr/bin/vim.basic
/usr/bin/vimdiff
/usr/bin/vim.tiny
/usr/bin/watch
/usr/bin/x11perf
/usr/bin/x86_64-linux-gnu-cpp
/usr/bin/x86_64-linux-gnu-cpp-11
/usr/bin/x86_64-linux-gnu-g++
/usr/bin/x86_64-linux-gnu-g++-11
/usr/bin/x86_64-linux-gnu-gcc
/usr/bin/x86_64-linux-gnu-gcc-11
/usr/bin/x86_64-linux-gnu-gdc
/usr/bin/x86_64-linux-gnu-gdc-11
/usr/bin/x86_64-linux-gnu-gfortran
/usr/bin/x86_64-linux-gnu-gfortran-11
/usr/bin/x86_64-linux-gnu-lto-dump-11
/usr/bin/x86_64-linux-gnux32-cpp
/usr/bin/x86_64-linux-gnux32-cpp-11
/usr/bin/x86_64-linux-gnux32-g++
/usr/bin/x86_64-linux-gnux32-g++-11
/usr/bin/x86_64-linux-gnux32-gcc
/usr/bin/x86_64-linux-gnux32-gcc-11
/usr/bin/x86_64-linux-gnux32-lto-dump-11
/usr/bin/xcas
/usr/bin/xclock
/usr/bin/xdvi.bin
/usr/bin/xdvi-xaw
/usr/bin/xedit
/usr/bin/xetex
/usr/bin/xgc
/usr/bin/xmllint
/usr/bin/xterm
/usr/bin/zsh
/usr/bin/unzstd
/usr/bin/zstd
/usr/bin/zstdcat
/usr/bin/zstdmt
  

On 08/03/23 14:09, Florian Weimer wrote:
> * Adhemerval Zanella Netto:
> 
>> On 08/03/23 13:23, Bruno Haible wrote:
>>> Paul Eggert wrote:
>>>> My idea is to go through the apps I help maintain, and make sure that 
>>>> they never call 'time' anywhere that it's important that a timestamp be 
>>>> in sync with with the rest of the system
>>>
>>> Alternatively, these applications can continue to call 'time', if the
>>> package uses the Gnulib module 'time' that provides a workaround against
>>> https://sourceware.org/bugzilla/show_bug.cgi?id=30200 .
>>>
>>> Bruno
>>
>> Florian, do you really think that and latency increase of roughly 15ns 
>> is really worth all the trouble gnulib is pushing? It means that we will 
>> end up with programs that use CLOCK_REALTIME, while other use 
>> CLOCK_REALTIME_COURSE.  
>>
>> If users really to squeeze more performance, they can use clock_gettime
>> with CLOCK_REALTIME_COURSE.  It should have similar performance to
>> time vDSO.
> 
> Eh, I think the difference derives from using the time vDSO entrypoint
> (which your patch removed as well on x86-64).  I don't see a performance
> difference between CLOCK_REALTIME and CLOCK_REALTIME_COARSE, so that's
> not it.

I just did some benchmarks using vdso test with time vdso support [1]

$ ./vdsotest clock-gettime-realtime bench
clock-gettime-realtime: syscall: 82 nsec/call
clock-gettime-realtime:    libc: 18 nsec/call
clock-gettime-realtime:    vdso: 18 nsec/call

$ ./vdsotest clock-gettime-realtime-coarse bench
clock-gettime-realtime-coarse: syscall: 65 nsec/call
clock-gettime-realtime-coarse:    libc: 5 nsec/call
clock-gettime-realtime-coarse:    vdso: 3 nsec/call

$ ./vdsotest time bench
time: syscall: 41 nsec/call
time:    libc: 2 nsec/call
time:    vdso: 1 nsec/call

So while time vdso still show small improvement, I hardly think it is a 
dealbreaker. 

[1] git@github.com:zatrazz/vdsotest.git
  

On 08/03/23 14:44, Bruno Haible wrote:
> Adhemerval Zanella Netto wrote:
>> ... all the trouble gnulib is pushing?
> 
> The trouble does not originate in gnulib; it originate through the
> use of inconsistent time APIs.

Yes I understand it and I agree with that this inconsistency is not
desirable, that exactly why I have sent this patch.  The 'trouble'
I meant all this boilerplate wrapper code you are adding on gnulib
to 'fix' this glibc optimization.

> 
> On a typical system, there are
>   - hundreds of programs that call gettimeofday() or timespec_get(),
>   - hundreds of programs that call time().
> Find attached the lists of programs in /usr/bin on my machine:
>   - 425 programs that call gettimeofday() or timespec_get(),
>   - 730 programs that call time().
> 
> Each time a program calls gettimeofday() or timespec_get(), and then,
> within 1 to 3 milliseconds, a program (the same or a different one)
> calls time(), there is the potential for trouble because the time
> appears to go backwards. This trouble can occur with probability
> between 0.1% and 0.3%, according to the data in
> https://sourceware.org/bugzilla/show_bug.cgi?id=30200 .
> 
> Gnulib attempts to reduce the trouble for specific programs, by
> moving them from the "call time()" camp to the "call gettimeofday() or
> timespec_get()" camp.
> 
> Bruno
>
  

On 3/8/23 00:59, Florian Weimer wrote:
> Even if the same underlying clock is used, rounding may cause interface
> which use different precision for the fractional seconds part to report
> different integral seconds.  The standard does not require rounding
> towards minus infinity.
Actually, POSIX does require rounding timestamps towards minus infinity 
for file timestamps, in its spec for utimensat. This requirement was put 
in because some implementations played fast and loose with these 
timestamps, with some rounding up and some rounding down, and this 
confusion caused more trouble than it was worth.

Perhaps it would help if POSIX had a similar requirement for the 'time' 
function, as my impression from this discussion is that the confusion 
with 'time' being out-of-sync with CLOCK_REALTIME is also more trouble 
than it's worth.

Anyway, thanks for clarifying what Glibc does.