[0/7] stdio-common: Verify printf format tests with Python rather than AWK

  The formatted printf output tests verify their records against GNU AWK, on
the grounds that it supplies an implementation of format processing that is
independent from ours.  It does not do that as cleanly as one would like.

AWK has to run in the bignum mode for the floating-point conversions,
because otherwise it uses the system sprintf(3) internally and we end up
verifying our code against itself.  That in turn makes gawk compiled with
MPFR support a requirement for testing the library at all.  Beyond that
gawk mishandles a number of cases which the AWK script then has to undo by
hand: the extraneous leading 0 produced for the alternative form with the
octal conversion, the 0 produced where no characters are expected for the
hexadecimal conversions, the missing + and space characters for a zero
value with the precision of zero, and a collection of sign, flag and field
width anomalies for Inf and NaN values.  Each such workaround suppresses
whatever we might get wrong in the same place.  The a, A, b, and B
conversions cannot be verified at all, because gawk either does not handle
them or produces different output.  And where gawk was built without the f
and F conversions the tests reported unsupported rather than checking
anything.

The first patch replaces the AWK script with an equivalent one written in
Python, which is already a requirement for building the library.  Rather
than calling into any formatting routine it computes the reference output
directly, using exact integer and rational arithmetic.  Working exactly
means the result does not depend on the range or precision of any host
floating-point type, so the wider types are handled without
arbitrary-precision arithmetic having to be built into the interpreter, and
none of the workarounds listed above are needed: the corner cases they
cover are computed correctly.  The capability probes go away along with the
AWK script, so f and F are now always verified, and the note on MPFR is
dropped from the installation instructions.

The rest of the series spends the freedom that buys:

  2  stops iterating over the huge width for eleven of the twelve printf
     family functions -- see below;
  3  adds the b and B conversions, which gawk does not handle;
  4  adds the a and A conversions, which gawk renders differently;
  5  adds subnormal values, which needed the minimum exponent for the type
     to be reported to the verification;
  6  fixes a printf_fp bug the added coverage exposed: with the alternative
     form, a g or G conversion that rounds into a new decade dropped the
     trailing zeros it is required to keep, so that printf ("%#.2g", 99.9)
     gave "1.e+02" rather than "1.0e+02";
  7  marks a and A unsupported where long double has the IBM extended
     format, which is a pair of doubles rather than a significand of a
     single fixed width and so cannot be modelled the way the verification
     does it.

This changes both what the tests cover and what they cost, so here are the
numbers.  Measured on x86_64-linux-gnu with the tests run serially,
processor time for the tst-printf-format-* tests, which varies by under 2%
between runs:

                                 tests  results      records      CPU
  before the series                288      576   32,588,304     935s
  1  verify with Python            288      576   32,588,304     684s
  2  huge width for printf only    288      576   27,563,064     375s
  3  add b and B                   288      576   33,879,864     392s
  4  add a and A                   336      672   37,258,224     402s
  5  add subnormal values          336      672   38,609,568     411s
  6  keep %#g trailing zeros       336      672   39,960,912     416s
  7  skip a and A for IBM          336      672   39,960,912     416s

So the suite ends up checking 23% more records, spread over 48 more tests,
in 2.25 times less processor time.  All 672 results pass, as do all 576
before the series.

Two things account for the time.  Python turns out to be faster than gawk
in the bignum mode to begin with, once rendered digits are memoized per
value, which is what patch 1 alone buys; without the memoization the exact
arithmetic makes the long double conversions slower than AWK instead.  The
rest is patch 2.  HUGE_WIDTH is chosen so that none of the strings produced
are truncated, which for the floating-point types means every record it
takes part in carries hundreds or thousands of digits, and those records
dominate the cost of this whole family of tests.  The digits are produced
by the same conversion code whichever of the printf family of functions is
used; what differs between the twelve of them is the sink the result is
written to, which the smaller widths cover already.  Iterating over the
huge width for printf alone therefore gives up no coverage worth having.

Records are not equal in cost, which is why the two columns move
independently: the 15% of records patch 2 drops are the most expensive in
the suite and take 45% of the run time with them, while the b and B records
patch 3 adds are integer ones, and the a and A records patch 4 adds are
short as well, the hexadecimal significand having a fixed width.  What
remains is mostly intrinsic -- the f and F conversions print the whole
integer part regardless of the precision requested, so LDBL_MAX runs to
some 4932 digits even at MID_WIDTH.

Patch 7 costs nothing here, as expected: powerpc64le is the one target with
the IBM extended format and it builds everything with the IEEE format
instead, so the conversions are only skipped where shared libraries have
been turned off.

Tested on x86_64-linux-gnu and powerpc64le-linux-gnu, and with the long
double conversions forced to the IBM extended format, where the a and A
conversions report unsupported and the rest continue to pass.
  

On 24/08/26 19:16, Matt Turner wrote:
> The formatted printf output tests verify their records against GNU AWK, on
> the grounds that it supplies an implementation of format processing that is
> independent from ours.  It does not do that as cleanly as one would like.
> 
> AWK has to run in the bignum mode for the floating-point conversions,
> because otherwise it uses the system sprintf(3) internally and we end up
> verifying our code against itself.  That in turn makes gawk compiled with
> MPFR support a requirement for testing the library at all.  Beyond that
> gawk mishandles a number of cases which the AWK script then has to undo by
> hand: the extraneous leading 0 produced for the alternative form with the
> octal conversion, the 0 produced where no characters are expected for the
> hexadecimal conversions, the missing + and space characters for a zero
> value with the precision of zero, and a collection of sign, flag and field
> width anomalies for Inf and NaN values.  Each such workaround suppresses
> whatever we might get wrong in the same place.  The a, A, b, and B
> conversions cannot be verified at all, because gawk either does not handle
> them or produces different output.  And where gawk was built without the f
> and F conversions the tests reported unsupported rather than checking
> anything.

Hi Matt,

Thanks for working on this. It turned out that using awk as a printf oracle
was not the best approach, and the -M being on parole is not a good indication
to keep relying on it (even though there is no actual plan to remove it). 

The Python multi-precision is also builtin, so one less conditional knob to
kind in mind during testing.

But I think the main advantage of this approach is it increases coverage
(stdio-common goes from 1515 tests to 1611) and it is way faster (on my
box stdio-common tests time is cut in half).

> 
> The first patch replaces the AWK script with an equivalent one written in
> Python, which is already a requirement for building the library.  Rather
> than calling into any formatting routine it computes the reference output
> directly, using exact integer and rational arithmetic.  Working exactly
> means the result does not depend on the range or precision of any host
> floating-point type, so the wider types are handled without
> arbitrary-precision arithmetic having to be built into the interpreter, and
> none of the workarounds listed above are needed: the corner cases they
> cover are computed correctly.  The capability probes go away along with the
> AWK script, so f and F are now always verified, and the note on MPFR is
> dropped from the installation instructions.

I though we had Python as requirement only for testing, but we now do use
for gen-as-const-headers since 7e1d42400c1b8f03316fe14176133c8853cd3bbe.
> 
> The rest of the series spends the freedom that buys:
> 
>   2  stops iterating over the huge width for eleven of the twelve printf
>      family functions -- see below;

This can be pushed independently. 

>   3  adds the b and B conversions, which gawk does not handle;
>   4  adds the a and A conversions, which gawk renders differently;
>   5  adds subnormal values, which needed the minimum exponent for the type
>      to be reported to the verification;
>   6  fixes a printf_fp bug the added coverage exposed: with the alternative
>      form, a g or G conversion that rounds into a new decade dropped the
>      trailing zeros it is required to keep, so that printf ("%#.2g", 99.9)
>      gave "1.e+02" rather than "1.0e+02";

This can done independently and I think it would be useful to open a bug
report to track it.

>   7  marks a and A unsupported where long double has the IBM extended
>      format, which is a pair of doubles rather than a significand of a
>      single fixed width and so cannot be modelled the way the verification
>      does it.
> 
> This changes both what the tests cover and what they cost, so here are the
> numbers.  Measured on x86_64-linux-gnu with the tests run serially,
> processor time for the tst-printf-format-* tests, which varies by under 2%
> between runs:
> 
>                                  tests  results      records      CPU
>   before the series                288      576   32,588,304     935s
>   1  verify with Python            288      576   32,588,304     684s
>   2  huge width for printf only    288      576   27,563,064     375s
>   3  add b and B                   288      576   33,879,864     392s
>   4  add a and A                   336      672   37,258,224     402s
>   5  add subnormal values          336      672   38,609,568     411s
>   6  keep %#g trailing zeros       336      672   39,960,912     416s
>   7  skip a and A for IBM          336      672   39,960,912     416s
> 
> So the suite ends up checking 23% more records, spread over 48 more tests,
> in 2.25 times less processor time.  All 672 results pass, as do all 576
> before the series.
> 
> Two things account for the time.  Python turns out to be faster than gawk
> in the bignum mode to begin with, once rendered digits are memoized per
> value, which is what patch 1 alone buys; without the memoization the exact
> arithmetic makes the long double conversions slower than AWK instead.  The
> rest is patch 2.  HUGE_WIDTH is chosen so that none of the strings produced
> are truncated, which for the floating-point types means every record it
> takes part in carries hundreds or thousands of digits, and those records
> dominate the cost of this whole family of tests.  The digits are produced
> by the same conversion code whichever of the printf family of functions is
> used; what differs between the twelve of them is the sink the result is
> written to, which the smaller widths cover already.  Iterating over the
> huge width for printf alone therefore gives up no coverage worth having.

On my box the main difference of stdio-common tests time while running
in parallel was the 'stdio-common: Iterate over the huge width for one printf
function only'.

But at same time, with the new Python testing framework it does enable
a bunch a more tests.

> 
> Records are not equal in cost, which is why the two columns move
> independently: the 15% of records patch 2 drops are the most expensive in
> the suite and take 45% of the run time with them, while the b and B records
> patch 3 adds are integer ones, and the a and A records patch 4 adds are
> short as well, the hexadecimal significand having a fixed width.  What
> remains is mostly intrinsic -- the f and F conversions print the whole
> integer part regardless of the precision requested, so LDBL_MAX runs to
> some 4932 digits even at MID_WIDTH.
> 
> Patch 7 costs nothing here, as expected: powerpc64le is the one target with
> the IBM extended format and it builds everything with the IEEE format
> instead, so the conversions are only skipped where shared libraries have
> been turned off.
> 
> Tested on x86_64-linux-gnu and powerpc64le-linux-gnu, and with the long
> double conversions forced to the IBM extended format, where the a and A
> conversions report unsupported and the rest continue to pass.

So, as a side note, I started to check whether using mpfr/gmp directly would 
result in a simpler way to check this. The mpfr/gmp also had some corner cases
that we need to handle (like no standard %a/%A support, missing sign for
infinities and NaNs), and it required adding an additional build step in
build-many-glibcs.py to enable it.

It turned out to *not* be simpler than this approach. I will review the
rest of this patchset.