Use gen-libm-test.py to generate ulps table for manual
Commit Message
This patch, relative to a tree with
<https://sourceware.org/ml/libc-alpha/2018-08/msg00133.html> (pending
review) applied, extends gen-libm-test.py to generate the ulps table
for the manual, so meaning there is only a single ulps file parser
needed and another Perl script is eliminated. As with the
introduction of gen-libm-test.py, this is designed to generate exactly
the same libm-err.texi as libm-err-tab.pl did. (gen-libm-test.py is
still shorter in lines than the old gen-libm-test.pl even after this
patch.) Note that this introduces a Python dependency for building
the manual, which is thus noted in install.texi and NEWS.
Tested building html / info / pdf versions of the manual.
2018-08-09 Joseph Myers <joseph@codesourcery.com>
* math/gen-libm-test.py: Import os.
(ALL_FLOATS_MANUAL): New constant.
(ALL_FLOATS_SUFFIX): Likewise.
(Ulps.all_functions): New function.
(real_all_ulps): Likewise.
(generate_err_table_sub): Likewise.
(generate_err_table): Likewise.
(main): Handle -s and -m options.
* manual/libm-err-tab.pl: Remove.
* manual/Makefile ($(objpfx)stamp-libm-err): Use gen-libm-test.py
instead of libm-err-tab.pl.
[$(PERL) != no]: Change condition to [$(if $(PYTHON),$(PERL),no)
!= no].
* manual/install.texi (Tools for Compilation): Document
requirement for Python to build manual.
* INSTALL: Regenerated.
Comments
Ping. This patch
<https://sourceware.org/ml/libc-alpha/2018-08/msg00190.html> is pending
review (for the principle of introducing the Python dependency for
building the manual, at least).
On Aug 09 2018, Joseph Myers <joseph@codesourcery.com> wrote:
> patch.) Note that this introduces a Python dependency for building
> the manual, which is thus noted in install.texi and NEWS.
This would add python to the bootstrap cycle which is a big problem.
Andreas.
On Mon, 13 Aug 2018, Andreas Schwab wrote:
> On Aug 09 2018, Joseph Myers <joseph@codesourcery.com> wrote:
>
> > patch.) Note that this introduces a Python dependency for building
> > the manual, which is thus noted in install.texi and NEWS.
>
> This would add python to the bootstrap cycle which is a big problem.
Why is it a big problem? I'd expect it to be at most a one-off addition
to an automated bootstrap process.
(a) If you are bootstrapping purely from sources and a minimum set of
binaries (e.g. starting from GNU Mes), e.g. to defend against Thompson
attacks, you'll inevitably need various techniques to break cycles. In
this case it's very simple: an early glibc build might not build the
manual (or run tests, etc.), but once Python has been built a full glibc
build can build the manual and run tests. Even with Python required for
the build, such a bootstrap from a minimum set of binaries would probably
in practice need to start with building older, simpler versions of various
tools with reduced dependencies (GNU Mes does).
(b) If you are bootstrapping a new architecture by cross-compilation, (i)
Python on the build system is fully sufficient for building the
cross-compiled glibc including its manual, and (ii) either you can
cross-compile Python, or you can build it natively later.
(c) If you are repeatedly rebuilding packages with themselves - in the
full configuration building everything from those packages intended to be
in a release - to ensure the rebuilds converge to a set of binary packages
that are byte-for-byte reproducible when rebuilt with themselves (a
stronger version of reproducible builds than the ones where reproducibly
building package X in a release depends on having an older version of
package Y than the one included in that release), it doesn't even affect
the set of packages that depend on each other because Python is already
used in glibc testing.
For both (a) and (b), having automation of the bootstrap process is
clearly strongly desirable, and maybe a one-off change to that automation
would be needed to insert a few extra builds. For (c) I'd expect existing
automation to be able to work automatically with the build-dependencies so
only those need updating for it to determine the updated set of packages
to rebuild (and as noted, Python is already a dependency for running the
full set of tests).
There was clear support in the discussion descending from
<https://sourceware.org/ml/libc-alpha/2018-07/msg00549.html> for replacing
both perl and awk code in the build with python for the reasons described
at <https://sourceware.org/ml/libc-alpha/2018-07/msg00559.html>: it's now
better for the current community maintaining glibc. For the bootstrap
cases discussed, I don't think Python is significantly different from any
other widely-available build dependency - glibc is very careful about
cyclic build dependencies in the actual cross-toolchain bootstrap for (b)
(glibc built with a static-libgcc-only C-only GCC should be identical to
one built in a cycle iterated more times, although you need to rebuild
glibc with the shared-libgcc C++-supporting GCC to get a build tree
appropriately configured to run the tests), but for tools that only need
to be present on the build system to build glibc (not to be built
specially for the glibc architecture), and where there aren't strong
dependencies on particular versions, I think such bootstrap considerations
are really matters for other projects not for glibc.
On Aug 13 2018, Joseph Myers <joseph@codesourcery.com> wrote:
> (c) If you are repeatedly rebuilding packages with themselves - in the
> full configuration building everything from those packages intended to be
> in a release - to ensure the rebuilds converge to a set of binary packages
> that are byte-for-byte reproducible when rebuilt with themselves (a
> stronger version of reproducible builds than the ones where reproducibly
> building package X in a release depends on having an older version of
> package Y than the one included in that release), it doesn't even affect
> the set of packages that depend on each other because Python is already
> used in glibc testing.
glibc testing is not part of the bootstrap cycle.
Andreas.
On Mon, 13 Aug 2018, Andreas Schwab wrote:
> On Aug 13 2018, Joseph Myers <joseph@codesourcery.com> wrote:
>
> > (c) If you are repeatedly rebuilding packages with themselves - in the
> > full configuration building everything from those packages intended to be
> > in a release - to ensure the rebuilds converge to a set of binary packages
> > that are byte-for-byte reproducible when rebuilt with themselves (a
> > stronger version of reproducible builds than the ones where reproducibly
> > building package X in a release depends on having an older version of
> > package Y than the one included in that release), it doesn't even affect
> > the set of packages that depend on each other because Python is already
> > used in glibc testing.
>
> glibc testing is not part of the bootstrap cycle.
In that case you could also make the glibc manual build not part of the
bootstrap cycle (until we generally move awk scripts to python as well,
which was supported in the previous discussion). Or simply include python
and its build-dependencies in that bootstrap cycle, much like perl and its
build-dependencies (even if we successfully eliminate all direct perl uses
in glibc, there will still be the indirect use from makeinfo).
On 08/13/2018 08:46 AM, Andreas Schwab wrote:
> On Aug 09 2018, Joseph Myers <joseph@codesourcery.com> wrote:
>
>> patch.) Note that this introduces a Python dependency for building
>> the manual, which is thus noted in install.texi and NEWS.
>
> This would add python to the bootstrap cycle which is a big problem.
Why is this a big problem?
Can you expand on your particular use case?
Even in Fedora where we do stage1/2/3 boostrapping, we have to build
glibc much later to enable manual building because we haven't yet
built the tex packages.
On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
> Even in Fedora where we do stage1/2/3 boostrapping, we have to build
> glibc much later to enable manual building because we haven't yet
> built the tex packages.
texinfo does not require tex.
Currently the following packages are part of the bootstrap cycle:
<https://build.opensuse.org/project/repository_state/openSUSE:Factory/standard>
acl, attr, audit, autoconf, automake, bc, binutils, bison, bzip2, cpio,
cracklib, dejagnu, dwz, e2fsprogs, ed, elfutils, expect, fdupes, file,
findutils, flex, gawk, gcc, gcc8, gdbm, gettext-runtime-mini, glibc,
gmp, gpm, gzip, help2man, isl, keyutils, krb5-mini, libcap, libcap-ng,
libdb-4_8, libnsl, libselinux, libsemanage, libsepol, libtirpc, libtool,
libustr, libverto, libzio, linux-glibc-devel, lua-macros, lua53, m4,
make, mpc, mpfr, ncurses, pam, patch, pcre, perl, perl-Text-Unidecode,
perl-gettext, pkg-config, popt, readline, rpm, screen, shadow,
system-user-root, system-users, systemd-rpm-macros, systemtap-headers,
sysuser-tools, tcl, tcpd, texinfo, update-alternatives, utempter,
util-linux, which, xz, zlib
Andreas.
On 08/13/2018 12:01 PM, Andreas Schwab wrote:
> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>
>> Even in Fedora where we do stage1/2/3 boostrapping, we have to build
>> glibc much later to enable manual building because we haven't yet
>> built the tex packages.
>
> texinfo does not require tex.
Sorry, I meant to write texinfo.
> Currently the following packages are part of the bootstrap cycle:
> <https://build.opensuse.org/project/repository_state/openSUSE:Factory/standard>
>
> acl, attr, audit, autoconf, automake, bc, binutils, bison, bzip2, cpio,
> cracklib, dejagnu, dwz, e2fsprogs, ed, elfutils, expect, fdupes, file,
> findutils, flex, gawk, gcc, gcc8, gdbm, gettext-runtime-mini, glibc,
> gmp, gpm, gzip, help2man, isl, keyutils, krb5-mini, libcap, libcap-ng,
> libdb-4_8, libnsl, libselinux, libsemanage, libsepol, libtirpc, libtool,
> libustr, libverto, libzio, linux-glibc-devel, lua-macros, lua53, m4,
> make, mpc, mpfr, ncurses, pam, patch, pcre, perl, perl-Text-Unidecode,
> perl-gettext, pkg-config, popt, readline, rpm, screen, shadow,
> system-user-root, system-users, systemd-rpm-macros, systemtap-headers,
> sysuser-tools, tcl, tcpd, texinfo, update-alternatives, utempter,
> util-linux, which, xz, zlib
I'm impressed, you've done a really good job keeping this minimal, but what's
the problem with including python here?
Are you simply disappointed that your minimal list will need to grow a bit?
Is there a technical issue at hand?
* Carlos O'Donell:
> I'm impressed, you've done a really good job keeping this minimal, but what's
> the problem with including python here?
It would need a special bootstrap build, reducing external
dependencies to the bare minimum. A typical distribution build of
Python pulls in X, Java and perhaps more.
On 08/13/2018 04:04 PM, Florian Weimer wrote:
> * Carlos O'Donell:
>
>> I'm impressed, you've done a really good job keeping this minimal, but what's
>> the problem with including python here?
>
> It would need a special bootstrap build, reducing external
> dependencies to the bare minimum. A typical distribution build of
> Python pulls in X, Java and perhaps more.
You still haven't answered the "Why?"
Why does your bootstrap need to be smaller?
What is wrong with "typical?"
* Carlos O'Donell:
> On 08/13/2018 04:04 PM, Florian Weimer wrote:
>> * Carlos O'Donell:
>>
>>> I'm impressed, you've done a really good job keeping this minimal, but what's
>>> the problem with including python here?
>>
>> It would need a special bootstrap build, reducing external
>> dependencies to the bare minimum. A typical distribution build of
>> Python pulls in X, Java and perhaps more.
>
> You still haven't answered the "Why?"
>
> Why does your bootstrap need to be smaller?
>
> What is wrong with "typical?"
When we did this in Fedora, the bootstrap set was pretty much the
whole distribution, so fully bootstrapped builds are extremely rare
and generally happen only during architecture bringup, and often not
even then.
Andreas' setup is very good for detecting certain types of build
issues, and I wouldn't want to lose that.
On 08/13/2018 04:12 PM, Florian Weimer wrote:
> * Carlos O'Donell:
>
>> On 08/13/2018 04:04 PM, Florian Weimer wrote:
>>> * Carlos O'Donell:
>>>
>>>> I'm impressed, you've done a really good job keeping this minimal, but what's
>>>> the problem with including python here?
>>>
>>> It would need a special bootstrap build, reducing external
>>> dependencies to the bare minimum. A typical distribution build of
>>> Python pulls in X, Java and perhaps more.
>>
>> You still haven't answered the "Why?"
>>
>> Why does your bootstrap need to be smaller?
>>
>> What is wrong with "typical?"
>
> When we did this in Fedora, the bootstrap set was pretty much the
> whole distribution, so fully bootstrapped builds are extremely rare
> and generally happen only during architecture bringup, and often not
> even then.
It is a question of interest really. Fedora has the technical ability
to deliver a bootstrap that happens routinely, but I don't see the
motivation to do so. Only mass rebuilds happen routinely when Fedora
makes a release (don't have bootstrap issues).
> Andreas' setup is very good for detecting certain types of build
> issues, and I wouldn't want to lose that.
I assume you are arguing that a full bootstrap would take much longer
than the set of packages Andreas identifies. Please make such assertions
clear. I don't know if Andreas has automation to run the builds in
parallel across lots of machines and so it doesn't take that long to
add more packages.
The honest truth is that I'd like to see Python usable in the build
process for glibc, so I think we'd have to bite the bullet on a minimal
python at some point. $0.02.
* Carlos O'Donell:
> It is a question of interest really. Fedora has the technical ability
> to deliver a bootstrap that happens routinely, but I don't see the
> motivation to do so.
Sorry, this sounds … very unconvincing.
This is a capability that deteriorates very quickly if you do not
continuously exercise it.
We can't even do periodic (say weekly) mass rebuilds in Fedora, and we
know that there are uses for that, such as generating accurate markup
(be it annobin or CET) based on the latest toolchain fixes.
>> Andreas' setup is very good for detecting certain types of build
>> issues, and I wouldn't want to lose that.
>
> I assume you are arguing that a full bootstrap would take much longer
> than the set of packages Andreas identifies. Please make such assertions
> clear.
It's also much more brittle because as the Fedora experience shows, it
is very difficult to keep even the core of the distribution in a
permanently buildable state. So it's not purely about speed.
On 08/13/2018 04:29 PM, Florian Weimer wrote:
> * Carlos O'Donell:
>
>> It is a question of interest really. Fedora has the technical ability
>> to deliver a bootstrap that happens routinely, but I don't see the
>> motivation to do so.
>
> Sorry, this sounds … very unconvincing.
Could you clarify what you were not convinced about? The technical ability
or lack of motivation?
> This is a capability that deteriorates very quickly if you do not
> continuously exercise it.
Agreed. How does adding python prevent you from continuously exercising
the bootstrap? Do you argue that it makes the whole build cycle too long?
> We can't even do periodic (say weekly) mass rebuilds in Fedora, and we
> know that there are uses for that, such as generating accurate markup
> (be it annobin or CET) based on the latest toolchain fixes.
Either we can't do it technically (not true, we can).
Or we aren't sufficiently motivated to do it (the real case).
I don't know why Andreas wants the core bootstrap small.
I don't understand Andreas' motivation.
If glibc upstream can't understand the reasons why downstream want a
minimal bootstrap then we can't ourselves support the position either.
You appear to argue that it would be nice if downstream Fedora had a
minimal bootstrap (without python) to test:
* Annobin (binary annotation work we are doing).
* Intel CET (ABI flag day changes for *context() routines).
That would be really nice, but the reality is that we don't value that
and so we haven't invested in it. Why should glibc upstream value it?
Using python in upstream glibc makes it easier to maintain, and the only
downside is the bootstrap issue. The solution *today* is to not build the
subsystems in glibc that need python (manual, benchtests) during bootstrap.
If more things require python then we may need a minimal python for
the bootstrap. Investing in that is a good idea only if you value what
we have just discussed as a bootstrap.
Other more useful bootstraps are IMO something like "Build all C++ programs
with the latest C++ compiler changes, but do it in a side tag" which would
allow the Fedora toolchain team to test C++ changes. These kinds of CI-esque
workflows don't need a minimal python or a minimal bootstrap since the C++
applications you are building will likely already require much of the distro
to be present.
>>> Andreas' setup is very good for detecting certain types of build
>>> issues, and I wouldn't want to lose that.
>>
>> I assume you are arguing that a full bootstrap would take much longer
>> than the set of packages Andreas identifies. Please make such assertions
>> clear.
>
> It's also much more brittle because as the Fedora experience shows, it
> is very difficult to keep even the core of the distribution in a
> permanently buildable state. So it's not purely about speed.
Agreed. This still doesn't convince me that you have a compelling enough
use case for minimal bootstraps that upstream glibc must not adopt python
as a bootstrap requirement (which it isn't yet).
To convince myself that the real solution is a python3-minimal solution I
started with the python3 in Fedora Rawhide, and with maybe 5 minutes of
spec file hacking I had a Python3 which built (in a clean rawhide mock
chroot) *without* all the dependencies enabled. It appears to work for
simple bootstrap things. Tests are still running, but languages tests
are mostly passing (one _haslib failure).
Is there a technical problem with needing python3-minimal? It seems like
such a package would solve all sorts of early bootstrap issues with needing
python?
* Carlos O'Donell:
> On 08/13/2018 04:29 PM, Florian Weimer wrote:
>> * Carlos O'Donell:
>>
>>> It is a question of interest really. Fedora has the technical ability
>>> to deliver a bootstrap that happens routinely, but I don't see the
>>> motivation to do so.
>>
>> Sorry, this sounds … very unconvincing.
>
> Could you clarify what you were not convinced about? The technical ability
> or lack of motivation?
The ability, both at a technical and organizational level.
>> We can't even do periodic (say weekly) mass rebuilds in Fedora, and we
>> know that there are uses for that, such as generating accurate markup
>> (be it annobin or CET) based on the latest toolchain fixes.
>
> Either we can't do it technically (not true, we can).
I doubt that. It would be a significant engineering effort.
On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
> I'm impressed, you've done a really good job keeping this minimal, but what's
> the problem with including python here?
We want to keep the cycle as small as possible. For example, even
though rpmlint is included in every build root and it is written in
python, it is not part of the bootstrap cycle because it is replaced by
a special package rpmlint-mini that contains copies of the python files
and dependencies from the rpmlint package. That shows how important it
is to keep python out of the equation.
Andreas.
On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
> It is a question of interest really. Fedora has the technical ability
> to deliver a bootstrap that happens routinely, but I don't see the
> motivation to do so. Only mass rebuilds happen routinely when Fedora
> makes a release (don't have bootstrap issues).
In openSUSE Factory, every package update goes through a staging build
where it is built and tested using the core subset of Factory (see
<https://build.opensuse.org/project/dashboard/openSUSE:Factory>). Only
after the testing succeeded the package is added to Factory.
Andreas.
On Tue, 14 Aug 2018, Andreas Schwab wrote:
> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>
> > I'm impressed, you've done a really good job keeping this minimal, but what's
> > the problem with including python here?
>
> We want to keep the cycle as small as possible. For example, even
> though rpmlint is included in every build root and it is written in
> python, it is not part of the bootstrap cycle because it is replaced by
> a special package rpmlint-mini that contains copies of the python files
> and dependencies from the rpmlint package. That shows how important it
> is to keep python out of the equation.
Given that you're already using special different packages in this
bootstrap cycle, a special minimum python package is the obvious way to
deal with uses in the glibc build.
On Aug 14 2018, Joseph Myers <joseph@codesourcery.com> wrote:
> On Tue, 14 Aug 2018, Andreas Schwab wrote:
>
>> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>>
>> > I'm impressed, you've done a really good job keeping this minimal, but what's
>> > the problem with including python here?
>>
>> We want to keep the cycle as small as possible. For example, even
>> though rpmlint is included in every build root and it is written in
>> python, it is not part of the bootstrap cycle because it is replaced by
>> a special package rpmlint-mini that contains copies of the python files
>> and dependencies from the rpmlint package. That shows how important it
>> is to keep python out of the equation.
>
> Given that you're already using special different packages in this
> bootstrap cycle, a special minimum python package is the obvious way to
> deal with uses in the glibc build.
That already exists, it is called python{3}-base.
Andreas.
On Tue, Aug 14, 2018 at 3:10 AM, Andreas Schwab <schwab@suse.de> wrote:
> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>
>> I'm impressed, you've done a really good job keeping this minimal, but what's
>> the problem with including python here?
>
> We want to keep the cycle as small as possible. For example, even
> though rpmlint is included in every build root and it is written in
> python, it is not part of the bootstrap cycle because it is replaced by
> a special package rpmlint-mini that contains copies of the python files
> and dependencies from the rpmlint package. That shows how important it
> is to keep python out of the equation.
I deeply sympathize with the desire to keep the bootstrap dependency
graph as small as possible, but I also deeply sympathize with the
glibc community's desire to move away from Perl and AWK. Especially
AWK.
Let me ask you: how many other packages (besides glibc) in the
bootstrap depend on Perl? Is it within our reach, perhaps, to
*replace* Perl with Python?
zw
On Aug 14 2018, Zack Weinberg <zackw@panix.com> wrote:
> Let me ask you: how many other packages (besides glibc) in the
> bootstrap depend on Perl? Is it within our reach, perhaps, to
> *replace* Perl with Python?
You would need to rewrite autoconf and automake in python. Also, a lot
of testsuites are using perl. Perl is ubiquitous, whereas python is
used almost nowhere.
Andreas.
On 08/14/2018 09:03 AM, Andreas Schwab wrote:
> On Aug 14 2018, Joseph Myers <joseph@codesourcery.com> wrote:
>
>> On Tue, 14 Aug 2018, Andreas Schwab wrote:
>>
>>> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>>>
>>>> I'm impressed, you've done a really good job keeping this minimal, but what's
>>>> the problem with including python here?
>>>
>>> We want to keep the cycle as small as possible. For example, even
>>> though rpmlint is included in every build root and it is written in
>>> python, it is not part of the bootstrap cycle because it is replaced by
>>> a special package rpmlint-mini that contains copies of the python files
>>> and dependencies from the rpmlint package. That shows how important it
>>> is to keep python out of the equation.
>>
>> Given that you're already using special different packages in this
>> bootstrap cycle, a special minimum python package is the obvious way to
>> deal with uses in the glibc build.
>
> That already exists, it is called python{3}-base.
What is your incremental cost to adding python3-base to the bootstrap?
On Tue, 14 Aug 2018, Zack Weinberg wrote:
> Let me ask you: how many other packages (besides glibc) in the
> bootstrap depend on Perl? Is it within our reach, perhaps, to
> *replace* Perl with Python?
We can't even replace it for building the glibc manual (makeinfo is
written in Perl). What we can do is eliminate it as a *direct* dependency
in glibc (so reducing the number of languages we need to deal with as
glibc developers and the number of build-dependencies we need to
document).
On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote:
> What is your incremental cost to adding python3-base to the bootstrap?
It's the cost of having another package in the bootstrap cycle, which
means building it at least three times.
Andreas.
On 08/15/2018 03:02 AM, Andreas Schwab wrote:
> On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>
>> What is your incremental cost to adding python3-base to the bootstrap?
>
> It's the cost of having another package in the bootstrap cycle, which
> means building it at least three times.
Does python3-base depend on many other things?
On Aug 15 2018, Carlos O'Donell <carlos@redhat.com> wrote:
> On 08/15/2018 03:02 AM, Andreas Schwab wrote:
>> On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>>
>>> What is your incremental cost to adding python3-base to the bootstrap?
>>
>> It's the cost of having another package in the bootstrap cycle, which
>> means building it at least three times.
>
> Does python3-base depend on many other things?
Yes, it drags in more packages into the bootstrap cycle.
Andreas.
On 08/15/2018 10:42 AM, Andreas Schwab wrote:
> On Aug 15 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>
>> On 08/15/2018 03:02 AM, Andreas Schwab wrote:
>>> On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>>>
>>>> What is your incremental cost to adding python3-base to the bootstrap?
>>>
>>> It's the cost of having another package in the bootstrap cycle, which
>>> means building it at least three times.
>>
>> Does python3-base depend on many other things?
>
> Yes, it drags in more packages into the bootstrap cycle.
How many more packages?
How is the boostrap build related to the testing a package has to go
through before being added to Factory?
Does the core subset of Factory, which is used in the package testing,
get rebuilt every time the package is built, just to test that the
core is still buildable?
Hi,
[answering to a bounced mail, I'm not on libc-alpha@, so keep me CCed]
I'm randomly using one mail of this thread as it covers most topics, but I
use it for making general points, not only specifically answering to your
mail Carlos.
On Tue, 14 Aug 2018, Carlos O'Donell wrote:
> > This is a capability that deteriorates very quickly if you do not
> > continuously exercise it.
>
> Agreed. How does adding python prevent you from continuously exercising
> the bootstrap? Do you argue that it makes the whole build cycle too
> long?
It's technically not (very) difficult to include python into the bootstrap
set. But that's not the most important part. All SUSE distros since
forever use this scheme of building the distro, a bootstrap set (iterating
builds until stability), and once the bootstrap set is finished building
all dependend packages in a topological order (breaking cycles in that
other non-bootstrap set, if they exist, at defined points).
That implies the following thing: every change in any package of the
bootstrap set implies a whole-distro rebuild (any change in any of the
other packages implies only a rebuild on the depending packages). So
enlarging the bootstrap set makes full distro rebuild more common, and
given that this takes about two to four days (depending on architecture)
it's something you don't want to do lightly. So changes to packages in
the bootstrap set are done more seldom than for other packages.
> > We can't even do periodic (say weekly) mass rebuilds in Fedora, and we
> > know that there are uses for that, such as generating accurate markup
> > (be it annobin or CET) based on the latest toolchain fixes.
>
> Either we can't do it technically (not true, we can).
I bet you theoretically can, in reality you don't. Such bootstrap
capability detoriates within weeks if not done regularly (we for instance
see this with old enterprise products where this is not regularly done;
once you try this you go "oh my god, what is this, what happened here?").
> I don't know why Andreas wants the core bootstrap small.
It's not Andreas, it's all (open)SUSE distros. It's the way we do this
since ages and it's not small work (occasionally) to keep the bootstrap
set small. Which is why we try very hard to avoid extending it, including
having these very discussions.
> I don't understand Andreas' motivation.
I can bring up the original philosophy of how this came to be in the SUSE
distros: whenever there's a change in any package, all other packages
depending on it (per buildrequires, and transitively so) should be rebuilt
(I won't got into the reasoning for this policy here). That requires a
topological sort, and as such can't handle cycles. So we allowed exactly
one cycle of packages, the bootstrap cycle. That one builds until a fixed
point (or well, in reality all packages a max of three times). But all
other packages are build in topological order. Obviously it's a good idea
to let this cycle be small, so that at least the rest of packages can be
built according to the golden rule. So, that's where we're coming from.
As you can see in Andreas' list the bootstrap cycle is small (and with
minor additions over time contains still the same things since 20 years;
though I think we could remove some more things from it with a little
work).
Over time we of course got more cycles in BuildRequires than this (and the
reason is often programmers not taking these things into account, e.g.
adding a dependency on java from within gettext; just an example, not
necessarily what happened), but still only the bootstrap cycle is iterated
and implies a full distro rebuild.
> Using python in upstream glibc makes it easier to maintain, and the only
> downside is the bootstrap issue.
(sidenote: if I were a glibc maintainer I'd disagree; I know perl better
than python. Alas, I'm not :) Though one point I'll make is that
everybody sufficiently intelligent can learn any reasonable language; and
hence the like or dislike for specific languages should never in itself be
a reason to choose one or the other. Rather all other reasons to prefer
one over the other should trump such personal preferences. Such
considerations include: preexisting use of language $X, wideness of
availability for language $X interpreters/compilers and so on)
Another point I made over at gcc@ is that for base tools like gcc and
glibc the list of dependencies should be as minimal as possible, even at
the expense of a little more work. In my world the former is of much
higher importance than a little convenience for authors of helper scripts.
To my liking even depending on perl should be avoided (surely a little
text processing can just as well be done with awk).
> The solution *today* is to not build the subsystems in glibc that need
> python (manual, benchtests) during bootstrap.
To note, for all of the above I'm not talking about use of anything for
building optional stuff. We do the above, and if python is to be used for
building the manual we'll split that off as well from the main glibc
build. All my points above only apply to dependencies for
building essential part, which for glibc would be, ... well, libc ;-)
> If more things require python then we may need a minimal python for the
> bootstrap. Investing in that is a good idea only if you value what we
> have just discussed as a bootstrap.
We split python already into a base and a full python (which requires
tk and hence all of X). It might be that we "only" need to add openssl to
the bootstrap cycle in addition to python3-base. It would be good if we
can avoid all of this though.
> Other more useful bootstraps are IMO something like "Build all C++
> programs with the latest C++ compiler changes, but do it in a side tag"
> which would allow the Fedora toolchain team to test C++ changes. These
> kinds of CI-esque workflows don't need a minimal python or a minimal
> bootstrap since the C++ applications you are building will likely
> already require much of the distro to be present.
Actually at SUSE we do something similar to this as well. All compiler
changes are going into a staging project where many packages are rebuilt.
A new compiler version (and new glibc and similar things) goes through
larger stagings. That is indeed nice, but orthogonal to the bootstrap
topic.
> Agreed. This still doesn't convince me that you have a compelling enough
> use case for minimal bootstraps that upstream glibc must not adopt
> python as a bootstrap requirement (which it isn't yet).
What I'm always wondering in these discussion about switching languages
for helper scripts (of all!) are compelling reasons to switch _to_ new
infrastructures. Shouldn't the onus of finding such reasons be on those
suggesting a change? (I realize that one stated reason is "easier
maintenance", I merely disagree that even if this is true (which is at
least questionable) that it'd be a very compelling reason).
> To convince myself that the real solution is a python3-minimal solution
> I started with the python3 in Fedora Rawhide, and with maybe 5 minutes
> of spec file hacking I had a Python3 which built (in a clean rawhide
> mock chroot) *without* all the dependencies enabled.
As without any dependencies no C compiler would be installed I assume the
mock chroot contains a multitude of things by default. So is your
experiment for creating a -minimal variant valid?
> Is there a technical problem with needing python3-minimal? It seems like
> such a package would solve all sorts of early bootstrap issues with
> needing python?
As we currently don't have any such problems including it wouldn't solve
anything. If you chose to create such problem, then yes, including a
python3-base will be the technical solution to it.
Ciao,
Michael.
On Thu, 16 Aug 2018, Michael Matz wrote:
> a reason to choose one or the other. Rather all other reasons to prefer
> one over the other should trump such personal preferences. Such
> considerations include: preexisting use of language $X, wideness of
> availability for language $X interpreters/compilers and so on)
I think popularity of a language is a perfectly relevant consideration
(not the only one, of course). It's not simply among glibc developers
that python is preferred to perl; various language rankings show python
has much higher popularity than perl nowadays, which serves as external
evidence that new contributors as well are more likely to be comfortable
with build infrastructure in python than in perl.
> Another point I made over at gcc@ is that for base tools like gcc and
> glibc the list of dependencies should be as minimal as possible, even at
I think the considerations for glibc are separate from those for GCC
(given, for example, that glibc is almost certainly built, even when being
cross-compiled, on a GNU system for which a range of build tools are
likely already available, including python, whereas GCC allows for being
built natively on a non-GNU system that may not have such tools, as a
starting point for then building such other tools which might not build
with random vendor compilers on proprietary Unices).
Building glibc is also much more specialized than building GCC (if you're
building glibc you're probably either a glibc developer or a system
integrator using it as part of building a whole system, rather than just
building a newer version locally for your own use, and using a different
version of glibc than that coming with the system is tricky, whereas
building your own GCC more recent than any that comes with the OS is an
expected and supported normal use of GCC).
This discussion seems to have died down. How can we move further towards
consensus on the use of Python in building the manual (taking account of
the desires also expressed to use it to replace Awk scripts used in the
build, and so make it a required build tool for building glibc at all)?
On 08/21/2018 01:20 PM, Joseph Myers wrote:
> This discussion seems to have died down. How can we move further towards
> consensus on the use of Python in building the manual (taking account of
> the desires also expressed to use it to replace Awk scripts used in the
> build, and so make it a required build tool for building glibc at all)?
https://sourceware.org/glibc/wiki/Consensus#How_do_I_build_consensus.3F
1. Define or reiterate success criteria.
Decide on the suitability of Python for core glibc scripts.
2. Identify a technical problems.
SUSE does not want python to be used in core glibc scripts because it would
increase the size of their bootstrap, costing them time, machine use, etc.
3. Look for resolution on a singular technical problem.
SUSE says it will cost them to add python to the bootstrap.
I expect that Joseph and I and perhaps others see Python is the strongest
viable option to making the core glibc scripts maintainable from now into
the future. Python is a growing language, users will be familiar with it,
and all of that plays into new developer adoption for the project.
The fact that SUSE has python-minimal to use is already indicative of that,
and while I'm empathic to SUSE's need here, I think they are going to loose
at some point and admit defeat that the minimal bootstrap needs some kind
of full-featured interpreted language and that language is Python or Perl.
Many professional development environments use LUA for this to limit
license and dependency exposure, but in glibc we've been using Python.
I'd like to hear that glibc's use of Python is going to cause SUSE
"undue hardship" in accommodating this use of Python, otherwise I'd like
to see us move forward with using Python.
4. Summarize the resolved list.
Nobody has object to non-core uses of Python in glibc. We use python to
run microbenchmarks, we use it to rebuild Unicode data sets, etc.
The question that remains is the use of python for core glibc scripts.
On Fri, 24 Aug 2018, Carlos O'Donell wrote:
> I'd like to hear that glibc's use of Python is going to cause SUSE
> "undue hardship" in accommodating this use of Python, otherwise I'd like
> to see us move forward with using Python.
Do any SUSE people wish to comment further on this issue?
On 08/29/2018 12:56 PM, Joseph Myers wrote:
> On Fri, 24 Aug 2018, Carlos O'Donell wrote:
>
>> I'd like to hear that glibc's use of Python is going to cause SUSE
>> "undue hardship" in accommodating this use of Python, otherwise I'd like
>> to see us move forward with using Python.
>
> Do any SUSE people wish to comment further on this issue?
Joseph,
I will ping this again on Friday, since that will have been a week since
my request for information.
I'm not sure we can move forward on this issue until Andreas makes his
position clear, either a sustained objection (blocks consensus), or a
recorded objection to the change (does not block consensus).
Paul,
I'd like your opinion on this also. Do you have any opinion on the use
of python more uniformly within glibc to replace existing awk, perl,
and shell scripts? It would require a minimal python in the bootstrap,
which is what SUSE is objecting to.
Discussion starts here:
https://www.sourceware.org/ml/libc-alpha/2018-08/msg00218.html
Hi,
On Wed, 29 Aug 2018, Carlos O'Donell wrote:
> >> I'd like to hear that glibc's use of Python is going to cause SUSE
> >> "undue hardship" in accommodating this use of Python, otherwise I'd
> >> like to see us move forward with using Python.
> >
> > Do any SUSE people wish to comment further on this issue?
>
> Joseph,
> I will ping this again on Friday, since that will have been a week since
> my request for information.
I'm sorry, I haven't seen the above followup(s) as I'm not on libc-alpha@
and Andreas is on vacation right now. I think we've said all we had to
say. As engineers we can deal with most things (though I'd have expected
this from new contributors to glibc as well), so we'll manage. Just try
to not use python modules that aren't in the base distro of python or
require native code parts.
Ciao,
Michael.
P.S: not looking forward to the helper scripts being rewritten into
javascript 10 years from now because "it's more popular" ;-/
On 08/30/2018 08:59 AM, Michael Matz wrote:
> Hi,
>
> On Wed, 29 Aug 2018, Carlos O'Donell wrote:
>
>>>> I'd like to hear that glibc's use of Python is going to cause SUSE
>>>> "undue hardship" in accommodating this use of Python, otherwise I'd
>>>> like to see us move forward with using Python.
>>>
>>> Do any SUSE people wish to comment further on this issue?
>>
>> Joseph,
>> I will ping this again on Friday, since that will have been a week since
>> my request for information.
>
> I'm sorry, I haven't seen the above followup(s) as I'm not on libc-alpha@
> and Andreas is on vacation right now. I think we've said all we had to
> say. As engineers we can deal with most things (though I'd have expected
> this from new contributors to glibc as well), so we'll manage. Just try
> to not use python modules that aren't in the base distro of python or
> require native code parts.
Does your position represent that of SUSE?
Andreas is a project steward, and GNU project maintainer, so we still need
his input on this issue. We will need to wait for him to come back and
respond to this.
Lastly, your objection is recorded, and that you will not block consensus
on this issue e.g. "deal with most things." Thank you for your input.
> P.S: not looking forward to the helper scripts being rewritten into
> javascript 10 years from now because "it's more popular" ;-/
Are you kidding? We'll have a native WebAssembly port by then. You'll need a
whole browser in the bootstrap ;-)
Hi,
On Thu, 30 Aug 2018, Carlos O'Donell wrote:
> > I'm sorry, I haven't seen the above followup(s) as I'm not on
> > libc-alpha@ and Andreas is on vacation right now. I think we've said
> > all we had to say. As engineers we can deal with most things (though
> > I'd have expected this from new contributors to glibc as well), so
> > we'll manage. Just try to not use python modules that aren't in the
> > base distro of python or require native code parts.
>
> Does your position represent that of SUSE?
Effectively, yes. Though of course, for real official positions of a
company I guess I'd need to ask our press officer :) (The ones that will
have most practical daytoday problems with including python in the
bootstrap cycle aren't actually us, the toolchain team, but the build
people and project managers trying to fit build schedules for their
products; and of course the python packagers who then are under much
stricter constraints in updating their package. For them I only speak out
of experience not because we have had large meetings about this topic.)
> Andreas is a project steward, and GNU project maintainer, so we still
> need his input on this issue. We will need to wait for him to come back
> and respond to this.
Yes, definitely.
> Lastly, your objection is recorded, and that you will not block
> consensus on this issue e.g. "deal with most things." Thank you for your
> input.
I will also reserve the right to say "haha, told you so!" five years from
now when people start complaining ;-)
> > P.S: not looking forward to the helper scripts being rewritten into
> > javascript 10 years from now because "it's more popular" ;-/
>
> Are you kidding? We'll have a native WebAssembly port by then. You'll
> need a whole browser in the bootstrap ;-)
/me ogles musl ;)
Ciao,
Michael.
On 08/30/2018 10:14 AM, Michael Matz wrote:
> On Thu, 30 Aug 2018, Carlos O'Donell wrote:
>> Are you kidding? We'll have a native WebAssembly port by then. You'll
>> need a whole browser in the bootstrap ;-)
>
> /me ogles musl ;)
So will musl...
https://github.com/jfbastien/musl
On Thu, 30 Aug 2018, Carlos O'Donell wrote:
> Are you kidding? We'll have a native WebAssembly port by then. You'll need a
It exists (complete with copyright assignments on file at the FSF), just
hasn't been updated lately (nor has the corresponding GCC port, and I
don't know how complete the binutils port - which *is* upstream - is).
https://github.com/pipcet/glibc
https://github.com/pipcet/gcc
I withdraw my objection, under the condition that the use of python is
restricted to the base python language.
Andreas.
On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote:
>
> I withdraw my objection, under the condition that the use of python is
> restricted to the base python language.
Can you define "base python language" more precisely, please? There
are a lot of modules in the Python standard library, and many of them
depend on externally-maintained libraries, and it's conceivable that
we would find a use for 'shelve' for instance...
zw
On Sep 04 2018, Zack Weinberg <zackw@panix.com> wrote:
> On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote:
>>
>> I withdraw my objection, under the condition that the use of python is
>> restricted to the base python language.
>
> Can you define "base python language" more precisely, please?
Basically what is part of the openSUSE python3-base package.
Andreas.
On Tue, 4 Sep 2018, Andreas Schwab wrote:
> On Sep 04 2018, Zack Weinberg <zackw@panix.com> wrote:
>
> > On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote:
> >>
> >> I withdraw my objection, under the condition that the use of python is
> >> restricted to the base python language.
> >
> > Can you define "base python language" more precisely, please?
>
> Basically what is part of the openSUSE python3-base package.
Based on the description at
<https://software.opensuse.org/package/python3-base> I've updated
<https://sourceware.org/glibc/wiki/Style_and_Conventions> to say:
* Scripts used in the build of glibc (a standard {{{make}}}, possibly
regenerating files in the source tree if timestamps are out of order and
the dependencies are present in the normal makefiles) need to avoid
modules outside the Python standard library, and standard library modules
depending on external libraries (such as XML, database and UI libraries).
If a script used in the testsuite has other dependencies (e.g., the pretty
printer tests using PExpect), the relevant tests should return UNSUPPORTED
if those dependencies are not present.
* Scripts used only by glibc developers may have other dependencies (e.g.
requiring Python 3), where justified in a particular case.
Are there other cases to add of standard library modules depending on
external libraries that should be avoided? (There are external library
dependencies for data compression modules and hashlib, for example; do we
wish to exclude those, or say they are allowed?)
Paul, Carlos asked for your input. Do you wish to comment on the use of
Python in the glibc build (whether just for the manual, or more
generally)?
Do we wish to make Python a requirement for the build process, not just
for the manual (require it in configure, remove PYTHON conditionals in the
makefiles), so that we can use it to replace awk scripts required for the
build?
(My expectation is that on 1 January 2020 - or rather 1 Feburary 2020,
since 1 January is the start of a release freeze - we can cease supporting
Python 2 - that we can use the table at <https://devguide.python.org/> as
the basis of what Python versions should be supported for building glibc.
But we don't need to decide that now.)
On Tue, Sep 4, 2018 at 10:30 AM Joseph Myers <joseph@codesourcery.com> wrote:
> On Tue, 4 Sep 2018, Andreas Schwab wrote:
>
> > On Sep 04 2018, Zack Weinberg <zackw@panix.com> wrote:
> >
> > > On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote:
> > >>
> > >> I withdraw my objection, under the condition that the use of python is
> > >> restricted to the base python language.
> > >
> > > Can you define "base python language" more precisely, please?
> >
> > Basically what is part of the openSUSE python3-base package.
>
> Based on the description at
> <https://software.opensuse.org/package/python3-base> I've updated
> <https://sourceware.org/glibc/wiki/Style_and_Conventions> to say:
>
> * Scripts used in the build of glibc (a standard {{{make}}}, possibly
> regenerating files in the source tree if timestamps are out of order and
> the dependencies are present in the normal makefiles) need to avoid
> modules outside the Python standard library, and standard library modules
> depending on external libraries (such as XML, database and UI libraries).
The Debian equivalent, (lib)python2.7-minimal, has a README with an
exact list of modules that may be used. This appears to be a subset
of the stuff in SUSE's python3-base I've reproduced the contents of
that file (from unstable; specifically, 2.7.15-3_amd64.deb) below.
If SUSE has already gotten to the point where Python 2 is not part of
their bootstrap, maybe we should ditch Python 2 now rather than
waiting for its EOL. Debian currently has both python2.7-minimal and
python3.6-minimal packages. The sets of standard library modules
available from each appear to be roughly the same, after accounting
for all the Python 3 module renames.
May I additionally suggest that it's not obvious that this stuff is
documented under "Code formatting in python sources", and the section
should be renamed to indicate it's general policy for Python usage?
zw
----
Contents of the python2.7-minimal package
-----------------------------------------
python2.7-minimal consists of a minimum set of modules which may be needed
for python scripts used during the boot process. If other packages
are needed in these scripts, don't work around the missing module, but
file a bug report against this package. The modules in this package
are:
__builtin__ builtin
__future__ module
_abcoll module
_bisect extension
_bytesio extension
_codecs builtin
_collections extension
_fileio extension
_functools extension
_hashlib extensionx
_heapq extension
_io extension
_locale extension
_md5 extension
_random extension
_sha extension
_sha256 extension
_sha512 extension
_socket extension
_sre builtin
_ssl extensionx
_struct extension
_symtable builtin
_sysconfigdata module
_types builtin
_warnings builtin
_weakref extension
_weakrefset module
abc module
atexit module
ConfigParser module
StringIO module
UserDict module
cPickle extension
cStringIO extension
array extension
base64 module
binascii extension
bisect module
cmath extension
calendar module
codecs module
collections module
compileall module
contextlib module
copy module
copy_reg module
datetime extension
dis module
encodings package
errno builtin
exceptions builtin
fcntl extension
fnmatch module
functools module
gc builtin
genericpath module
getopt module
glob module
grp extension
hashlib module
heapq module
imp builtin
inspect module
io module
itertools extension
keyword module
linecache module
locale module
logging package
marshal builtin
math extension
md5 module
opcode module
operator extension
optparse module
os module
pickle module
platform module
popen2 module
posix builtin
posixpath module
pkgutil module
pwd builtin
py_compile module
random module
re module
repr module
runpy module
select extension
sha module
shutil module
signal builtin
socket module
spwd extension
sre module
sre_compile module
sre_constants module
sre_parse module
ssl module
stat module
string module
strop extension
struct module
subprocess module
sys builtin
syslog extension
sysconfig module
tempfile module
textwrap module
time extension
token module
thread builtin
token module
tokenize module
traceback module
types module
unicodedata extension
weakref module
warnings module
zipimport extension
zlib extension
Included are as well the codecs and stringprep modules, and the encodings
modules for all encodings except the multibyte encodings and the bz2 codec.
The following modules are excluded, their import is guarded from the
importing module:
Used in Excluded
------------ ------------------------------------
os nt ntpath os2 os2emxpath mac macpath
riscos riscospath riscosenviron
optparse gettext
pickle doctest
subprocess threading
This list was derived by looking at the modules in the perl-base package,
then adding python specific "core modules".
TODO's
------
- time.strptime cannot be used. The required _strptime module is not
included in the -minimal package yet. _strptime, locale, _locale and
calendar have to be added.
- modules used very often in the testsuite: copy, cPickle, operator.
On Tue, 4 Sep 2018, Zack Weinberg wrote:
> The Debian equivalent, (lib)python2.7-minimal, has a README with an
> exact list of modules that may be used. This appears to be a subset
> of the stuff in SUSE's python3-base I've reproduced the contents of
> that file (from unstable; specifically, 2.7.15-3_amd64.deb) below.
That list does not include argparse. I think we should be able to use
argparse in the glibc build.
On Tue, Sep 4, 2018 at 11:55 AM Joseph Myers <joseph@codesourcery.com> wrote:
> On Tue, 4 Sep 2018, Zack Weinberg wrote:
> > The Debian equivalent, (lib)python2.7-minimal, has a README with an
> > exact list of modules that may be used. This appears to be a subset
> > of the stuff in SUSE's python3-base I've reproduced the contents of
> > that file (from unstable; specifically, 2.7.15-3_amd64.deb) below.
>
> That list does not include argparse. I think we should be able to use
> argparse in the glibc build.
I agree. argparse is a pure-Python module and it doesn't look like it
depends on any C extensions that aren't already included in the
"minimal" subset. I have filed wishlist bugs against python-minimal
(#907977) and python3-minimal (#907976) and we'll see what happens.
zw
On Tue, 4 Sep 2018, Zack Weinberg wrote:
> On Tue, Sep 4, 2018 at 11:55 AM Joseph Myers <joseph@codesourcery.com> wrote:
> > On Tue, 4 Sep 2018, Zack Weinberg wrote:
> > > The Debian equivalent, (lib)python2.7-minimal, has a README with an
> > > exact list of modules that may be used. This appears to be a subset
> > > of the stuff in SUSE's python3-base I've reproduced the contents of
> > > that file (from unstable; specifically, 2.7.15-3_amd64.deb) below.
> >
> > That list does not include argparse. I think we should be able to use
> > argparse in the glibc build.
>
> I agree. argparse is a pure-Python module and it doesn't look like it
> depends on any C extensions that aren't already included in the
> "minimal" subset. I have filed wishlist bugs against python-minimal
> (#907977) and python3-minimal (#907976) and we'll see what happens.
Ubuntu 18.04 /usr/share/doc/python3.6-minimal/README.Debian *does* include
argparse. Is that some Ubuntu-specific addition?
On Tue, Sep 4, 2018 at 1:29 PM Joseph Myers <joseph@codesourcery.com> wrote:
> On Tue, 4 Sep 2018, Zack Weinberg wrote:
> > On Tue, Sep 4, 2018 at 11:55 AM Joseph Myers <joseph@codesourcery.com> wrote:
> > > On Tue, 4 Sep 2018, Zack Weinberg wrote:
> > > > The Debian equivalent, (lib)python2.7-minimal, has a README with an
> > > > exact list of modules that may be used. This appears to be a subset
> > > > of the stuff in SUSE's python3-base I've reproduced the contents of
> > > > that file (from unstable; specifically, 2.7.15-3_amd64.deb) below.
> > >
> > > That list does not include argparse. I think we should be able to use
> > > argparse in the glibc build.
> >
> > I agree. argparse is a pure-Python module and it doesn't look like it
> > depends on any C extensions that aren't already included in the
> > "minimal" subset. I have filed wishlist bugs against python-minimal
> > (#907977) and python3-minimal (#907976) and we'll see what happens.
>
> Ubuntu 18.04 /usr/share/doc/python3.6-minimal/README.Debian *does* include
> argparse. Is that some Ubuntu-specific addition?
Doh! No, I just missed it in the list. argparse is present in the
python3.6-minimal list in version 3.6.6-3 of the package.
zw
Dmitry, do you wish to make any comments on the use of Python in building
glibc?
On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
> Dmitry, do you wish to make any comments on the use of Python in building
> glibc?
My concern with making python a requirement to build glibc is that it will
pull in more packages not required to build glibc. If there is a "minimal"
subset of python that does not bring indirect dependencies to glibc build,
and at the same time is sufficient to build glibc, then adding this subset
of python as a requirement to build glibc shouldn't be a problem.
I don't know yet whether SUSE or Debian take on the minimal subset pulls
in extra dependencies, I'll check this when time permits.
On Mon, 10 Sep 2018, Dmitry V. Levin wrote:
> On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
> > Dmitry, do you wish to make any comments on the use of Python in building
> > glibc?
>
> My concern with making python a requirement to build glibc is that it will
> pull in more packages not required to build glibc. If there is a "minimal"
> subset of python that does not bring indirect dependencies to glibc build,
> and at the same time is sufficient to build glibc, then adding this subset
> of python as a requirement to build glibc shouldn't be a problem.
>
> I don't know yet whether SUSE or Debian take on the minimal subset pulls
> in extra dependencies, I'll check this when time permits.
Any conclusions there?
I'm not sure it really matters for these purposes what a particular
distribution's package pulls in, if they are happy with what works for
them for the distribution build or bootstrap. The question for us would
be more about what constraints this imposes on distributions in general
(e.g. if it means libffi will be needed for a native build of glibc,
depending on how far the indirect dependencies on ctypes in the Python
standard library go), and whether we are OK with those constraints. The
dependencies of the minimal distribution packages are simply useful
information for us to help in determining those constraints.
On Mon, 17 Sep 2018, Joseph Myers wrote:
> On Mon, 10 Sep 2018, Dmitry V. Levin wrote:
>
> > On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
> > > Dmitry, do you wish to make any comments on the use of Python in building
> > > glibc?
> >
> > My concern with making python a requirement to build glibc is that it will
> > pull in more packages not required to build glibc. If there is a "minimal"
> > subset of python that does not bring indirect dependencies to glibc build,
> > and at the same time is sufficient to build glibc, then adding this subset
> > of python as a requirement to build glibc shouldn't be a problem.
> >
> > I don't know yet whether SUSE or Debian take on the minimal subset pulls
> > in extra dependencies, I'll check this when time permits.
>
> Any conclusions there?
>
> I'm not sure it really matters for these purposes what a particular
> distribution's package pulls in, if they are happy with what works for
> them for the distribution build or bootstrap. The question for us would
> be more about what constraints this imposes on distributions in general
> (e.g. if it means libffi will be needed for a native build of glibc,
> depending on how far the indirect dependencies on ctypes in the Python
> standard library go), and whether we are OK with those constraints. The
> dependencies of the minimal distribution packages are simply useful
> information for us to help in determining those constraints.
Ping. Any conclusion on this question about indirect dependencies
introduced by use of Python?
On 25/09/18 17:54, Joseph Myers wrote:
> On Mon, 17 Sep 2018, Joseph Myers wrote:
>> On Mon, 10 Sep 2018, Dmitry V. Levin wrote:
>>> On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
>>>> Dmitry, do you wish to make any comments on the use of Python in building
>>>> glibc?
>>>
>>> My concern with making python a requirement to build glibc is that it will
>>> pull in more packages not required to build glibc. If there is a "minimal"
>>> subset of python that does not bring indirect dependencies to glibc build,
>>> and at the same time is sufficient to build glibc, then adding this subset
>>> of python as a requirement to build glibc shouldn't be a problem.
>>>
>>> I don't know yet whether SUSE or Debian take on the minimal subset pulls
>>> in extra dependencies, I'll check this when time permits.
>>
>> Any conclusions there?
>>
>> I'm not sure it really matters for these purposes what a particular
>> distribution's package pulls in, if they are happy with what works for
>> them for the distribution build or bootstrap. The question for us would
>> be more about what constraints this imposes on distributions in general
>> (e.g. if it means libffi will be needed for a native build of glibc,
>> depending on how far the indirect dependencies on ctypes in the Python
>> standard library go), and whether we are OK with those constraints. The
>> dependencies of the minimal distribution packages are simply useful
>> information for us to help in determining those constraints.
>
> Ping. Any conclusion on this question about indirect dependencies
> introduced by use of Python?
using python3 in the build would be a bit disruptive for
me, because the current systems where i build glibc don't
have it.
last time when make 4.0 requirement was added i had to
build it on several machines until our infrastructure
got updated to have make 4.0 in the path everywhere.
(and our regular testing machines did not produce useful
results for several days because of broken glibc build)
installing python3 manually is obviously more effort and
time, so please do not add such dependency anywhere near
release time (like it was done with make 4).
i don't have other concerns regarding indirect deps if
the python3.4-minimal debian/ubuntu package works.
On Tue, 25 Sep 2018, Joseph Myers wrote:
> On Mon, 17 Sep 2018, Joseph Myers wrote:
>
> > On Mon, 10 Sep 2018, Dmitry V. Levin wrote:
> >
> > > On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
> > > > Dmitry, do you wish to make any comments on the use of Python in building
> > > > glibc?
> > >
> > > My concern with making python a requirement to build glibc is that it will
> > > pull in more packages not required to build glibc. If there is a "minimal"
> > > subset of python that does not bring indirect dependencies to glibc build,
> > > and at the same time is sufficient to build glibc, then adding this subset
> > > of python as a requirement to build glibc shouldn't be a problem.
> > >
> > > I don't know yet whether SUSE or Debian take on the minimal subset pulls
> > > in extra dependencies, I'll check this when time permits.
> >
> > Any conclusions there?
> >
> > I'm not sure it really matters for these purposes what a particular
> > distribution's package pulls in, if they are happy with what works for
> > them for the distribution build or bootstrap. The question for us would
> > be more about what constraints this imposes on distributions in general
> > (e.g. if it means libffi will be needed for a native build of glibc,
> > depending on how far the indirect dependencies on ctypes in the Python
> > standard library go), and whether we are OK with those constraints. The
> > dependencies of the minimal distribution packages are simply useful
> > information for us to help in determining those constraints.
>
> Ping. Any conclusion on this question about indirect dependencies
> introduced by use of Python?
Another ping on this question....
On 10/1/18 11:06 AM, Joseph Myers wrote:
> On Tue, 25 Sep 2018, Joseph Myers wrote:
>
>> On Mon, 17 Sep 2018, Joseph Myers wrote:
>>
>>> On Mon, 10 Sep 2018, Dmitry V. Levin wrote:
>>>
>>>> On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
>>>>> Dmitry, do you wish to make any comments on the use of Python in building
>>>>> glibc?
>>>>
>>>> My concern with making python a requirement to build glibc is that it will
>>>> pull in more packages not required to build glibc. If there is a "minimal"
>>>> subset of python that does not bring indirect dependencies to glibc build,
>>>> and at the same time is sufficient to build glibc, then adding this subset
>>>> of python as a requirement to build glibc shouldn't be a problem.
>>>>
>>>> I don't know yet whether SUSE or Debian take on the minimal subset pulls
>>>> in extra dependencies, I'll check this when time permits.
>>>
>>> Any conclusions there?
>>>
>>> I'm not sure it really matters for these purposes what a particular
>>> distribution's package pulls in, if they are happy with what works for
>>> them for the distribution build or bootstrap. The question for us would
>>> be more about what constraints this imposes on distributions in general
>>> (e.g. if it means libffi will be needed for a native build of glibc,
>>> depending on how far the indirect dependencies on ctypes in the Python
>>> standard library go), and whether we are OK with those constraints. The
>>> dependencies of the minimal distribution packages are simply useful
>>> information for us to help in determining those constraints.
>>
>> Ping. Any conclusion on this question about indirect dependencies
>> introduced by use of Python?
>
> Another ping on this question....
Could you help me to gather consensus?
If you could write up the following:
* Exact version of python to use.
* Exact set of imports we need.
I'll take that and try to get an acknowledgment from each of the
distributions. I'll email all the maintainers and follow up with
them (all on libc-alpha) in a new thread starting with the information
you will put together for me.
I'll also include some of the resolution about the concerns regarding
building python on an old distribution.
On Mon, Oct 1, 2018 at 11:21 AM Carlos O'Donell <carlos@redhat.com> wrote:
> On 10/1/18 11:06 AM, Joseph Myers wrote:
> > On Tue, 25 Sep 2018, Joseph Myers wrote:
> >> Ping. Any conclusion on this question about indirect dependencies
> >> introduced by use of Python?
> >
> > Another ping on this question....
> Could you help me to gather consensus?
>
> If you could write up the following:
>
> * Exact version of python to use.
>
> * Exact set of imports we need.
>
> I'll take that and try to get an acknowledgment from each of the
> distributions. I'll email all the maintainers and follow up with
> them (all on libc-alpha) in a new thread starting with the information
> you will put together for me.
>
> I'll also include some of the resolution about the concerns regarding
> building python on an old distribution.
For the record, although I offered some details earlier about how
Debian packages Python, I have no official relationship to Debian and
cannot speak for any of their teams that might care about this
proposed change.
As a glibc developer who might well take on some of the work of
rewriting existing build logic in Python, I would like to put in a
word for requiring Python 3.(>=4) rather than allowing 2.7 as well.
The 2.x series is close to end-of-life, writing scripts that work with
both major versions of the language is significantly harder than
writing 3.x-only code, and I fear 2.x build compatibility will not be
adequately tested. 3.4 is the oldest version of the 3.x series still
in wide use AFAIK.
zw
On Mon, 1 Oct 2018, Carlos O'Donell wrote:
> If you could write up the following:
>
> * Exact version of python to use.
Any non-EOL Python 3 series. So 3.4 or later at present, 3.5 or later
could be required from 2019-03-16 (but that doesn't mean we have to
increase the requirement then, if there aren't any new features we want to
use), 3.6 or later could be required from 2020-09-13, as per the table at
<https://devguide.python.org/>.
> * Exact set of imports we need.
Only modules in the Python (3.4) standard library, and excluding modules
such as XML, database and UI libraries that depend on external libraries.
I don't think we need anything depending on OpenSSL (hashlib appears to
work without it, in Python 3.7, and we probably don't need hashlib
anyway), but it's possible there may be dependencies on libffi - however,
I think it's likely we can avoid ctypes which is what brings in the libffi
dependency, and indirect dependencies on it from e.g. multiprocessing.
I think trying to list exhaustively exactly which modules may be used is a
bad idea, since there are lots of pure-Python modules in the standard
library without any external C library dependencies (direct or indirect)
and it's hard to predict exactly which could be useful at some point.
gen-libm-test.py uses argparse, collections, os (after the present patch
that would require it to build the manual) and re. The scripts used in
the conform/ tests (not of course strictly relevant to build requirements,
exact in regard to what tests can use unconditionally rather than needing
to be conditionally UNSUPPORTED) also use shutil, subprocess, tempfile. I
expect there would be plenty more pure-Python standard library modules
that would turn out to be useful for some particular script.
On Mon, 1 Oct 2018, Carlos O'Donell wrote:
> I'll take that and try to get an acknowledgment from each of the
> distributions. I'll email all the maintainers and follow up with
> them (all on libc-alpha) in a new thread starting with the information
> you will put together for me.
>
> I'll also include some of the resolution about the concerns regarding
> building python on an old distribution.
Ping? I'd like to be able to move the gen-as-const machinery from awk +
sed + makefile logic to python + simpler makefile logic, with a view to it
being the starting point of general python infrastructure for extracting
constants, macros etc. from C headers (there are various existing tests,
e.g. for the syscall list and for signal numbers, that could be rewritten
on top of such shared infrastructure, and I'd like to be able to add a lot
more tests of consistency with uapi headers using such infrastructure).
On Thu, 11 Oct 2018, Joseph Myers wrote:
> On Mon, 1 Oct 2018, Carlos O'Donell wrote:
>
> > I'll take that and try to get an acknowledgment from each of the
> > distributions. I'll email all the maintainers and follow up with
> > them (all on libc-alpha) in a new thread starting with the information
> > you will put together for me.
> >
> > I'll also include some of the resolution about the concerns regarding
> > building python on an old distribution.
>
> Ping? I'd like to be able to move the gen-as-const machinery from awk +
> sed + makefile logic to python + simpler makefile logic, with a view to it
> being the starting point of general python infrastructure for extracting
> constants, macros etc. from C headers (there are various existing tests,
> e.g. for the syscall list and for signal numbers, that could be rewritten
> on top of such shared infrastructure, and I'd like to be able to add a lot
> more tests of consistency with uapi headers using such infrastructure).
Ping^4? Dmitry, Carlos, any other distributors, any further comments on
requiring Python 3.4 (standard library modules not involving external C
library dependencies) or later to build glibc?
* Joseph Myers:
> Ping^4? Dmitry, Carlos, any other distributors, any further comments on
> requiring Python 3.4 (standard library modules not involving external C
> library dependencies) or later to build glibc?
Fedora and Red Hat Enterprise Linux have sufficiently new versions of
Python 3 readily available, so from a distribution perspective, it is
not an issue for us.
Debian 8/jessie is the oldest (LTS-)supported version, and it has Python
3.4, too. Debian wheezy does not, but I don't think you can reasonably
use it for glibc development anyway.
Thanks,
Florian
On Wed, 17 Oct 2018, Florian Weimer wrote:
> * Joseph Myers:
>
> > Ping^4? Dmitry, Carlos, any other distributors, any further comments on
> > requiring Python 3.4 (standard library modules not involving external C
> > library dependencies) or later to build glibc?
>
> Fedora and Red Hat Enterprise Linux have sufficiently new versions of
> Python 3 readily available, so from a distribution perspective, it is
> not an issue for us.
Thanks. Dmitry? Your previous comments were at
<https://sourceware.org/ml/libc-alpha/2018-09/msg00107.html>. Have you
reached any conclusions regarding dependencies from minimal Python?
On Wed, Oct 17, 2018 at 02:20:45PM +0000, Joseph Myers wrote:
> On Thu, 11 Oct 2018, Joseph Myers wrote:
> > On Mon, 1 Oct 2018, Carlos O'Donell wrote:
> >
> > > I'll take that and try to get an acknowledgment from each of the
> > > distributions. I'll email all the maintainers and follow up with
> > > them (all on libc-alpha) in a new thread starting with the information
> > > you will put together for me.
> > >
> > > I'll also include some of the resolution about the concerns regarding
> > > building python on an old distribution.
> >
> > Ping? I'd like to be able to move the gen-as-const machinery from awk +
> > sed + makefile logic to python + simpler makefile logic, with a view to it
> > being the starting point of general python infrastructure for extracting
> > constants, macros etc. from C headers (there are various existing tests,
> > e.g. for the syscall list and for signal numbers, that could be rewritten
> > on top of such shared infrastructure, and I'd like to be able to add a lot
> > more tests of consistency with uapi headers using such infrastructure).
>
> Ping^4? Dmitry, Carlos, any other distributors, any further comments on
> requiring Python 3.4 (standard library modules not involving external C
> library dependencies) or later to build glibc?
Thanks, I think this should be fine for us, too.
@@ -514,8 +514,9 @@ build the GNU C Library:
* Python 2.7/3.4 or later
- Python is required to run some tests. As of release time, Python
- 3.6 is the newest verified to work for testing the GNU C Library.
+ Python is required to build the GNU C Library manual and to run
+ some tests. As of release time, Python 3.6 is the newest verified
+ to work for testing the GNU C Library.
* PExpect 4.0
@@ -23,7 +23,7 @@ Deprecated and removed features, and other changes affecting compatibility:
Changes to build and runtime requirements:
- [Add changes to build and runtime requirements here]
+* Python is required to build the GNU C Library manual.
Security related changes:
@@ -99,10 +99,11 @@ $(objpfx)dir-add.texi: xtract-typefun.awk $(texis-path)
# The table with the math errors is generated.
$(objpfx)libm-err.texi: $(objpfx)stamp-libm-err
-$(objpfx)stamp-libm-err: libm-err-tab.pl $(wildcard $(foreach dir,$(sysdirs),\
- $(dir)/libm-test-ulps))
+$(objpfx)stamp-libm-err: $(..)math/gen-libm-test.py \
+ $(wildcard $(foreach dir,$(sysdirs),\
+ $(dir)/libm-test-ulps))
pwd=`pwd`; \
- $(PERL) $< $$pwd/.. > $(objpfx)libm-err-tmp
+ $(PYTHON) $< -s $$pwd/.. -m $(objpfx)libm-err-tmp
$(move-if-change) $(objpfx)libm-err-tmp $(objpfx)libm-err.texi
touch $@
@@ -169,8 +170,9 @@ include ../Rules
.PHONY: install subdir_install install-data
install-data subdir_install: install
-# Generated files requiring perl: libm-err.texi, summary.texi
-ifneq ($(PERL),no)
+# Generated files requiring python: libm-err.texi
+# Generated files requiring perl: summary.texi
+ifneq ($(if $(PYTHON),$(PERL),no),no)
ifneq ($(strip $(MAKEINFO)),:)
install: $(inst_infodir)/libc.info
@if $(SHELL) -c '$(INSTALL_INFO) --version' >/dev/null 2>&1; then \
@@ -560,8 +560,9 @@ with any version of @code{sed}. As of release time, @code{sed} version
@item
Python 2.7/3.4 or later
-Python is required to run some tests. As of release time, Python 3.6
-is the newest verified to work for testing @theglibc{}.
+Python is required to build the @glibcadj{} manual and to run some
+tests. As of release time, Python 3.6 is the newest verified to work
+for testing @theglibc{}.
@item PExpect 4.0
deleted file mode 100755
@@ -1,206 +0,0 @@
-#!/usr/bin/perl -w
-# Copyright (C) 1999-2018 Free Software Foundation, Inc.
-# This file is part of the GNU C Library.
-# Contributed by Andreas Jaeger <aj@suse.de>, 1999.
-
-# The GNU C Library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-
-# The GNU C Library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-# Lesser General Public License for more details.
-
-# You should have received a copy of the GNU Lesser General Public
-# License along with the GNU C Library; if not, see
-# <http://www.gnu.org/licenses/>.
-
-# Information about tests are stored in: %results
-# $results{$test}{"type"} is the result type, e.g. normal or complex.
-# In the following description $platform, $type and $float are:
-# - $platform is the used platform
-# - $type is either "normal", "real" (for the real part of a complex number)
-# or "imag" (for the imaginary part # of a complex number).
-# - $float is either of float, ifloat, double, idouble, ldouble, ildouble;
-# It represents the underlying floating point type (float, double or long
-# double) and if inline functions (the leading i stands for inline)
-# are used.
-# $results{$test}{$platform}{$type}{$float} is defined and has a delta
-# or 'fail' as value.
-
-use File::Find;
-
-use strict;
-
-use vars qw ($sources @platforms %pplatforms);
-use vars qw (%results @all_floats %suffices %all_functions);
-
-
-# all_floats is in output order and contains all recognised float types that
-# we're going to output
-@all_floats = ('float', 'double', 'ldouble', 'float128');
-%suffices =
- ( 'float' => 'f',
- 'double' => '',
- 'ldouble' => 'l',
- 'float128' => 'f128'
- );
-
-# Pretty description of platform
-%pplatforms = ();
-
-%all_functions = ();
-
-if ($#ARGV == 0) {
- $sources = $ARGV[0];
-} else {
- $sources = '/usr/src/cvs/libc';
-}
-
-find (\&find_files, $sources);
-
-@platforms = sort by_platforms @platforms;
-
-&print_all;
-
-sub find_files {
- if ($_ eq 'libm-test-ulps') {
- # print "Parsing $File::Find::name\n";
- push @platforms, $File::Find::dir;
- my ($file, $name);
- $file = "${File::Find::name}-name";
- open NAME, $file or die ("Can't open $file: $!");
- $name = <NAME>;
- chomp $name;
- close NAME;
- $pplatforms{$File::Find::dir} = $name;
- &parse_ulps ($File::Find::name, $File::Find::dir);
- }
-}
-
-# Parse ulps file
-sub parse_ulps {
- my ($file, $platform) = @_;
- my ($test, $type, $float, $eps, $ignore_fn);
-
- # $type has the following values:
- # "normal": No complex variable
- # "real": Real part of complex result
- # "imag": Imaginary part of complex result
- open ULP, $file or die ("Can't open $file: $!");
- while (<ULP>) {
- chop;
- # ignore comments and empty lines
- next if /^#/;
- next if /^\s*$/;
- if (/^Function: /) {
- if (/Real part of/) {
- s/Real part of //;
- $type = 'real';
- } elsif (/Imaginary part of/) {
- s/Imaginary part of //;
- $type = 'imag';
- } else {
- $type = 'normal';
- }
- ($test) = ($_ =~ /^Function:\s*\"([a-zA-Z0-9_]+)\"/);
- next;
- }
- if ($test =~ /_(downward|towardzero|upward|vlen)/) {
- $ignore_fn = 1;
- } else {
- $ignore_fn = 0;
- $all_functions{$test} = 1;
- }
- if (/^i?(float|double|ldouble|float128):/) {
- ($float, $eps) = split /\s*:\s*/,$_,2;
- if ($ignore_fn) {
- next;
- } elsif ($eps eq 'fail') {
- $results{$test}{$platform}{$type}{$float} = 'fail';
- } elsif ($eps eq "0") {
- # ignore
- next;
- } elsif (!exists $results{$test}{$platform}{$type}{$float}
- || $results{$test}{$platform}{$type}{$float} ne 'fail') {
- $results{$test}{$platform}{$type}{$float} = $eps;
- }
- if ($type =~ /^real|imag$/) {
- $results{$test}{'type'} = 'complex';
- } elsif ($type eq 'normal') {
- $results{$test}{'type'} = 'normal';
- }
- next;
- }
- print "Skipping unknown entry: `$_'\n";
- }
- close ULP;
-}
-
-sub get_value {
- my ($fct, $platform, $type, $float) = @_;
-
- return (exists $results{$fct}{$platform}{$type}{$float}
- ? $results{$fct}{$platform}{$type}{$float} : "0");
-}
-
-sub print_platforms {
- my (@p) = @_;
- my ($fct, $platform, $float, $first, $i, $platform_no, $platform_total);
-
- print '@multitable {nexttowardf} ';
- foreach (@p) {
- print ' {1000 + i 1000}';
- }
- print "\n";
-
- print '@item Function ';
- foreach (@p) {
- print ' @tab ';
- print $pplatforms{$_};
- }
- print "\n";
-
-
- foreach $fct (sort keys %all_functions) {
- foreach $float (@all_floats) {
- print "\@item $fct$suffices{$float} ";
- foreach $platform (@p) {
- print ' @tab ';
- if (exists $results{$fct}{$platform}{'normal'}{$float}
- || exists $results{$fct}{$platform}{'real'}{$float}
- || exists $results{$fct}{$platform}{'imag'}{$float}) {
- if ($results{$fct}{'type'} eq 'complex') {
- print &get_value ($fct, $platform, 'real', $float),
- ' + i ', &get_value ($fct, $platform, 'imag', $float);
- } else {
- print $results{$fct}{$platform}{'normal'}{$float};
- }
- } else {
- print '-';
- }
- }
- print "\n";
- }
- }
-
- print "\@end multitable\n";
-}
-
-sub print_all {
- my ($i, $max);
-
- my ($columns) = 5;
-
- # Print only 5 platforms at a time.
- for ($i=0; $i < $#platforms; $i+=$columns) {
- $max = $i+$columns-1 > $#platforms ? $#platforms : $i+$columns-1;
- print_platforms (@platforms[$i .. $max]);
- }
-}
-
-sub by_platforms {
- return $pplatforms{$a} cmp $pplatforms{$b};
-}
@@ -19,6 +19,7 @@
import argparse
from collections import defaultdict
+import os
import re
@@ -32,6 +33,16 @@ ALL_FLOATS_PFX = {'double': 'DBL',
'float': 'FLT',
'float128': 'FLT128'}
+# Float types in the order used in the generated ulps tables in the
+# manual.
+ALL_FLOATS_MANUAL = ('float', 'double', 'ldouble', 'float128')
+
+# Map float types in ulps files to C function suffix.
+ALL_FLOATS_SUFFIX = {'double': '',
+ 'ldouble': 'l',
+ 'float': 'f',
+ 'float128': 'f128'}
+
# Number of arguments in structure (as opposed to arguments that are
# pointers to return values) for an argument descriptor.
DESCR_NUM_ARGS = {'f': 1, 'a': 1, 'j': 1, 'i': 1, 'u': 1, 'l': 1, 'L': 1,
@@ -161,6 +172,17 @@ class Ulps(object):
ulps_dict[ulps_fn][line_first],
ulps_val)
+ def all_functions(self):
+ """Return the set of functions with ulps and whether they are
+ complex."""
+ funcs = set()
+ complex = {}
+ for k_prefix, k_dict in self.ulps_kinds:
+ for f in k_dict:
+ funcs.add(f)
+ complex[f] = True if k_prefix else False
+ return funcs, complex
+
def write(self, ulps_file):
"""Write ulps back out as a sorted ulps file."""
# Output is sorted first by function name, then by (real,
@@ -231,6 +253,18 @@ class Ulps(object):
f.write(header_text)
+def read_all_ulps(srcdir):
+ """Read all platforms' libm-test-ulps files."""
+ all_ulps = {}
+ for dirpath, dirnames, filenames in os.walk(srcdir):
+ if 'libm-test-ulps' in filenames:
+ with open(os.path.join(dirpath, 'libm-test-ulps-name')) as f:
+ name = f.read().rstrip()
+ all_ulps[name] = Ulps()
+ all_ulps[name].read(os.path.join(dirpath, 'libm-test-ulps'))
+ return all_ulps
+
+
def read_auto_tests(test_file):
"""Read tests from auto-libm-test-out-<function> (possibly None)."""
auto_tests = defaultdict(lambda: defaultdict(dict))
@@ -571,6 +605,60 @@ def generate_testfile(inc_input, auto_tests, c_output):
f.write(''.join(test_list))
+def generate_err_table_sub(all_ulps, all_functions, fns_complex, platforms):
+ """Generate a single table within the overall ulps table section."""
+ plat_width = [' {1000 + i 1000}' for p in platforms]
+ plat_header = [' @tab %s' % p for p in platforms]
+ table_list = ['@multitable {nexttowardf} %s\n' % ''.join(plat_width),
+ '@item Function %s\n' % ''.join(plat_header)]
+ for func in all_functions:
+ for flt in ALL_FLOATS_MANUAL:
+ func_ulps = []
+ for p in platforms:
+ p_ulps = all_ulps[p]
+ if fns_complex[func]:
+ ulp_real = p_ulps.real[func][flt]
+ ulp_imag = p_ulps.imag[func][flt]
+ ulp_str = '%d + i %d' % (ulp_real, ulp_imag)
+ ulp_str = ulp_str if ulp_real or ulp_imag else '-'
+ else:
+ ulp = p_ulps.normal[func][flt]
+ ulp_str = str(ulp) if ulp else '-'
+ func_ulps.append(ulp_str)
+ table_list.append('@item %s%s @tab %s\n'
+ % (func, ALL_FLOATS_SUFFIX[flt],
+ ' @tab '.join(func_ulps)))
+ table_list.append('@end multitable\n')
+ return ''.join(table_list)
+
+
+def generate_err_table(all_ulps, err_table):
+ """Generate ulps table for manual."""
+ all_platforms = sorted(all_ulps.keys())
+ functions_set = set()
+ functions_complex = {}
+ for p in all_platforms:
+ p_functions, p_complex = all_ulps[p].all_functions()
+ functions_set.update(p_functions)
+ functions_complex.update(p_complex)
+ all_functions = sorted([f for f in functions_set
+ if ('_downward' not in f
+ and '_towardzero' not in f
+ and '_upward' not in f
+ and '_vlen' not in f)])
+ err_table_list = []
+ # Print five platforms at a time.
+ num_platforms = len(all_platforms)
+ for i in range((num_platforms + 4) // 5):
+ start = i * 5
+ end = i * 5 + 5 if num_platforms >= i * 5 + 5 else num_platforms
+ err_table_list.append(generate_err_table_sub(all_ulps, all_functions,
+ functions_complex,
+ all_platforms[start:end]))
+ with open(err_table, 'w') as f:
+ f.write(''.join(err_table_list))
+
+
def main():
"""The main entry point."""
parser = argparse.ArgumentParser(description='Generate libm tests.')
@@ -580,23 +668,31 @@ def main():
help='input file .inc file with tests')
parser.add_argument('-u', dest='ulps_file', metavar='FILE',
help='input file with ulps')
+ parser.add_argument('-s', dest='srcdir', metavar='DIR',
+ help='input source directory with all ulps')
parser.add_argument('-n', dest='ulps_output', metavar='FILE',
help='generate sorted ulps file FILE')
parser.add_argument('-C', dest='c_output', metavar='FILE',
help='generate output C file FILE from .inc file')
parser.add_argument('-H', dest='ulps_header', metavar='FILE',
help='generate output ulps header FILE')
+ parser.add_argument('-m', dest='err_table', metavar='FILE',
+ help='generate output ulps table for manual FILE')
args = parser.parse_args()
ulps = Ulps()
if args.ulps_file is not None:
ulps.read(args.ulps_file)
auto_tests = read_auto_tests(args.auto_input)
+ if args.srcdir is not None:
+ all_ulps = read_all_ulps(args.srcdir)
if args.ulps_output is not None:
ulps.write(args.ulps_output)
if args.ulps_header is not None:
ulps.write_header(args.ulps_header)
if args.c_output is not None:
generate_testfile(args.inc_input, auto_tests, args.c_output)
+ if args.err_table is not None:
+ generate_err_table(all_ulps, args.err_table)
if __name__ == '__main__':