diff mbox

Use gen-libm-test.py to generate ulps table for manual

Message ID alpine.DEB.2.20.1808092025590.28500@digraph.polyomino.org.uk
State New, archived
Headers show

Commit Message

Joseph Myers Aug. 9, 2018, 8:26 p.m. UTC
This patch, relative to a tree with
<https://sourceware.org/ml/libc-alpha/2018-08/msg00133.html> (pending
review) applied, extends gen-libm-test.py to generate the ulps table
for the manual, so meaning there is only a single ulps file parser
needed and another Perl script is eliminated.  As with the
introduction of gen-libm-test.py, this is designed to generate exactly
the same libm-err.texi as libm-err-tab.pl did.  (gen-libm-test.py is
still shorter in lines than the old gen-libm-test.pl even after this
patch.)  Note that this introduces a Python dependency for building
the manual, which is thus noted in install.texi and NEWS.

Tested building html / info / pdf versions of the manual.

2018-08-09  Joseph Myers  <joseph@codesourcery.com>

	* math/gen-libm-test.py: Import os.
	(ALL_FLOATS_MANUAL): New constant.
	(ALL_FLOATS_SUFFIX): Likewise.
	(Ulps.all_functions): New function.
	(real_all_ulps): Likewise.
	(generate_err_table_sub): Likewise.
	(generate_err_table): Likewise.
	(main): Handle -s and -m options.
	* manual/libm-err-tab.pl: Remove.
	* manual/Makefile ($(objpfx)stamp-libm-err): Use gen-libm-test.py
	instead of libm-err-tab.pl.
	[$(PERL) != no]: Change condition to [$(if $(PYTHON),$(PERL),no)
	!= no].
	* manual/install.texi (Tools for Compilation): Document
	requirement for Python to build manual.
	* INSTALL: Regenerated.

Comments

Joseph Myers Aug. 13, 2018, 12:15 p.m. UTC | #1
Ping.  This patch 
<https://sourceware.org/ml/libc-alpha/2018-08/msg00190.html> is pending 
review (for the principle of introducing the Python dependency for 
building the manual, at least).
Andreas Schwab Aug. 13, 2018, 12:46 p.m. UTC | #2
On Aug 09 2018, Joseph Myers <joseph@codesourcery.com> wrote:

> patch.)  Note that this introduces a Python dependency for building
> the manual, which is thus noted in install.texi and NEWS.

This would add python to the bootstrap cycle which is a big problem.

Andreas.
Joseph Myers Aug. 13, 2018, 3:17 p.m. UTC | #3
On Mon, 13 Aug 2018, Andreas Schwab wrote:

> On Aug 09 2018, Joseph Myers <joseph@codesourcery.com> wrote:
> 
> > patch.)  Note that this introduces a Python dependency for building
> > the manual, which is thus noted in install.texi and NEWS.
> 
> This would add python to the bootstrap cycle which is a big problem.

Why is it a big problem?  I'd expect it to be at most a one-off addition 
to an automated bootstrap process.

(a) If you are bootstrapping purely from sources and a minimum set of 
binaries (e.g. starting from GNU Mes), e.g. to defend against Thompson 
attacks, you'll inevitably need various techniques to break cycles.  In 
this case it's very simple: an early glibc build might not build the 
manual (or run tests, etc.), but once Python has been built a full glibc 
build can build the manual and run tests.  Even with Python required for 
the build, such a bootstrap from a minimum set of binaries would probably 
in practice need to start with building older, simpler versions of various 
tools with reduced dependencies (GNU Mes does).

(b) If you are bootstrapping a new architecture by cross-compilation, (i) 
Python on the build system is fully sufficient for building the 
cross-compiled glibc including its manual, and (ii) either you can 
cross-compile Python, or you can build it natively later.

(c) If you are repeatedly rebuilding packages with themselves - in the 
full configuration building everything from those packages intended to be 
in a release - to ensure the rebuilds converge to a set of binary packages 
that are byte-for-byte reproducible when rebuilt with themselves (a 
stronger version of reproducible builds than the ones where reproducibly 
building package X in a release depends on having an older version of 
package Y than the one included in that release), it doesn't even affect 
the set of packages that depend on each other because Python is already 
used in glibc testing.

For both (a) and (b), having automation of the bootstrap process is 
clearly strongly desirable, and maybe a one-off change to that automation 
would be needed to insert a few extra builds.  For (c) I'd expect existing 
automation to be able to work automatically with the build-dependencies so 
only those need updating for it to determine the updated set of packages 
to rebuild (and as noted, Python is already a dependency for running the 
full set of tests).

There was clear support in the discussion descending from 
<https://sourceware.org/ml/libc-alpha/2018-07/msg00549.html> for replacing 
both perl and awk code in the build with python for the reasons described 
at <https://sourceware.org/ml/libc-alpha/2018-07/msg00559.html>: it's now 
better for the current community maintaining glibc.  For the bootstrap 
cases discussed, I don't think Python is significantly different from any 
other widely-available build dependency - glibc is very careful about 
cyclic build dependencies in the actual cross-toolchain bootstrap for (b) 
(glibc built with a static-libgcc-only C-only GCC should be identical to 
one built in a cycle iterated more times, although you need to rebuild 
glibc with the shared-libgcc C++-supporting GCC to get a build tree 
appropriately configured to run the tests), but for tools that only need 
to be present on the build system to build glibc (not to be built 
specially for the glibc architecture), and where there aren't strong 
dependencies on particular versions, I think such bootstrap considerations 
are really matters for other projects not for glibc.
Andreas Schwab Aug. 13, 2018, 3:23 p.m. UTC | #4
On Aug 13 2018, Joseph Myers <joseph@codesourcery.com> wrote:

> (c) If you are repeatedly rebuilding packages with themselves - in the 
> full configuration building everything from those packages intended to be 
> in a release - to ensure the rebuilds converge to a set of binary packages 
> that are byte-for-byte reproducible when rebuilt with themselves (a 
> stronger version of reproducible builds than the ones where reproducibly 
> building package X in a release depends on having an older version of 
> package Y than the one included in that release), it doesn't even affect 
> the set of packages that depend on each other because Python is already 
> used in glibc testing.

glibc testing is not part of the bootstrap cycle.

Andreas.
Joseph Myers Aug. 13, 2018, 3:39 p.m. UTC | #5
On Mon, 13 Aug 2018, Andreas Schwab wrote:

> On Aug 13 2018, Joseph Myers <joseph@codesourcery.com> wrote:
> 
> > (c) If you are repeatedly rebuilding packages with themselves - in the 
> > full configuration building everything from those packages intended to be 
> > in a release - to ensure the rebuilds converge to a set of binary packages 
> > that are byte-for-byte reproducible when rebuilt with themselves (a 
> > stronger version of reproducible builds than the ones where reproducibly 
> > building package X in a release depends on having an older version of 
> > package Y than the one included in that release), it doesn't even affect 
> > the set of packages that depend on each other because Python is already 
> > used in glibc testing.
> 
> glibc testing is not part of the bootstrap cycle.

In that case you could also make the glibc manual build not part of the 
bootstrap cycle (until we generally move awk scripts to python as well, 
which was supported in the previous discussion).  Or simply include python 
and its build-dependencies in that bootstrap cycle, much like perl and its 
build-dependencies (even if we successfully eliminate all direct perl uses 
in glibc, there will still be the indirect use from makeinfo).
Carlos O'Donell Aug. 13, 2018, 3:39 p.m. UTC | #6
On 08/13/2018 08:46 AM, Andreas Schwab wrote:
> On Aug 09 2018, Joseph Myers <joseph@codesourcery.com> wrote:
> 
>> patch.)  Note that this introduces a Python dependency for building
>> the manual, which is thus noted in install.texi and NEWS.
> 
> This would add python to the bootstrap cycle which is a big problem.

Why is this a big problem?

Can you expand on your particular use case?

Even in Fedora where we do stage1/2/3 boostrapping, we have to build
glibc much later to enable manual building because we haven't yet
built the tex packages.
Andreas Schwab Aug. 13, 2018, 4:01 p.m. UTC | #7
On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:

> Even in Fedora where we do stage1/2/3 boostrapping, we have to build
> glibc much later to enable manual building because we haven't yet
> built the tex packages.

texinfo does not require tex.

Currently the following packages are part of the bootstrap cycle:
<https://build.opensuse.org/project/repository_state/openSUSE:Factory/standard>

acl, attr, audit, autoconf, automake, bc, binutils, bison, bzip2, cpio,
cracklib, dejagnu, dwz, e2fsprogs, ed, elfutils, expect, fdupes, file,
findutils, flex, gawk, gcc, gcc8, gdbm, gettext-runtime-mini, glibc,
gmp, gpm, gzip, help2man, isl, keyutils, krb5-mini, libcap, libcap-ng,
libdb-4_8, libnsl, libselinux, libsemanage, libsepol, libtirpc, libtool,
libustr, libverto, libzio, linux-glibc-devel, lua-macros, lua53, m4,
make, mpc, mpfr, ncurses, pam, patch, pcre, perl, perl-Text-Unidecode,
perl-gettext, pkg-config, popt, readline, rpm, screen, shadow,
system-user-root, system-users, systemd-rpm-macros, systemtap-headers,
sysuser-tools, tcl, tcpd, texinfo, update-alternatives, utempter,
util-linux, which, xz, zlib

Andreas.
Carlos O'Donell Aug. 13, 2018, 5:06 p.m. UTC | #8
On 08/13/2018 12:01 PM, Andreas Schwab wrote:
> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
> 
>> Even in Fedora where we do stage1/2/3 boostrapping, we have to build
>> glibc much later to enable manual building because we haven't yet
>> built the tex packages.
> 
> texinfo does not require tex.

Sorry, I meant to write texinfo.
 
> Currently the following packages are part of the bootstrap cycle:
> <https://build.opensuse.org/project/repository_state/openSUSE:Factory/standard>
> 
> acl, attr, audit, autoconf, automake, bc, binutils, bison, bzip2, cpio,
> cracklib, dejagnu, dwz, e2fsprogs, ed, elfutils, expect, fdupes, file,
> findutils, flex, gawk, gcc, gcc8, gdbm, gettext-runtime-mini, glibc,
> gmp, gpm, gzip, help2man, isl, keyutils, krb5-mini, libcap, libcap-ng,
> libdb-4_8, libnsl, libselinux, libsemanage, libsepol, libtirpc, libtool,
> libustr, libverto, libzio, linux-glibc-devel, lua-macros, lua53, m4,
> make, mpc, mpfr, ncurses, pam, patch, pcre, perl, perl-Text-Unidecode,
> perl-gettext, pkg-config, popt, readline, rpm, screen, shadow,
> system-user-root, system-users, systemd-rpm-macros, systemtap-headers,
> sysuser-tools, tcl, tcpd, texinfo, update-alternatives, utempter,
> util-linux, which, xz, zlib

I'm impressed, you've done a really good job keeping this minimal, but what's
the problem with including python here?

Are you simply disappointed that your minimal list will need to grow a bit?

Is there a technical issue at hand?
Florian Weimer Aug. 13, 2018, 8:04 p.m. UTC | #9
* Carlos O'Donell:

> I'm impressed, you've done a really good job keeping this minimal, but what's
> the problem with including python here?

It would need a special bootstrap build, reducing external
dependencies to the bare minimum.  A typical distribution build of
Python pulls in X, Java and perhaps more.
Carlos O'Donell Aug. 13, 2018, 8:10 p.m. UTC | #10
On 08/13/2018 04:04 PM, Florian Weimer wrote:
> * Carlos O'Donell:
> 
>> I'm impressed, you've done a really good job keeping this minimal, but what's
>> the problem with including python here?
> 
> It would need a special bootstrap build, reducing external
> dependencies to the bare minimum.  A typical distribution build of
> Python pulls in X, Java and perhaps more.
 
You still haven't answered the "Why?"

Why does your bootstrap need to be smaller?

What is wrong with "typical?"
Florian Weimer Aug. 13, 2018, 8:12 p.m. UTC | #11
* Carlos O'Donell:

> On 08/13/2018 04:04 PM, Florian Weimer wrote:
>> * Carlos O'Donell:
>> 
>>> I'm impressed, you've done a really good job keeping this minimal, but what's
>>> the problem with including python here?
>> 
>> It would need a special bootstrap build, reducing external
>> dependencies to the bare minimum.  A typical distribution build of
>> Python pulls in X, Java and perhaps more.
>  
> You still haven't answered the "Why?"
>
> Why does your bootstrap need to be smaller?
>
> What is wrong with "typical?"

When we did this in Fedora, the bootstrap set was pretty much the
whole distribution, so fully bootstrapped builds are extremely rare
and generally happen only during architecture bringup, and often not
even then.

Andreas' setup is very good for detecting certain types of build
issues, and I wouldn't want to lose that.
Carlos O'Donell Aug. 13, 2018, 8:22 p.m. UTC | #12
On 08/13/2018 04:12 PM, Florian Weimer wrote:
> * Carlos O'Donell:
> 
>> On 08/13/2018 04:04 PM, Florian Weimer wrote:
>>> * Carlos O'Donell:
>>>
>>>> I'm impressed, you've done a really good job keeping this minimal, but what's
>>>> the problem with including python here?
>>>
>>> It would need a special bootstrap build, reducing external
>>> dependencies to the bare minimum.  A typical distribution build of
>>> Python pulls in X, Java and perhaps more.
>>  
>> You still haven't answered the "Why?"
>>
>> Why does your bootstrap need to be smaller?
>>
>> What is wrong with "typical?"
> 
> When we did this in Fedora, the bootstrap set was pretty much the
> whole distribution, so fully bootstrapped builds are extremely rare
> and generally happen only during architecture bringup, and often not
> even then.

It is a question of interest really. Fedora has the technical ability
to deliver a bootstrap that happens routinely, but I don't see the
motivation to do so. Only mass rebuilds happen routinely when Fedora
makes a release (don't have bootstrap issues).
 
> Andreas' setup is very good for detecting certain types of build
> issues, and I wouldn't want to lose that.

I assume you are arguing that a full bootstrap would take much longer
than the set of packages Andreas identifies. Please make such assertions
clear. I don't know if Andreas has automation to run the builds in
parallel across lots of machines and so it doesn't take that long to
add more packages.

The honest truth is that I'd like to see Python usable in the build
process for glibc, so I think we'd have to bite the bullet on a minimal
python at some point. $0.02.
Florian Weimer Aug. 13, 2018, 8:29 p.m. UTC | #13
* Carlos O'Donell:

> It is a question of interest really. Fedora has the technical ability
> to deliver a bootstrap that happens routinely, but I don't see the
> motivation to do so.

Sorry, this sounds … very unconvincing.

This is a capability that deteriorates very quickly if you do not
continuously exercise it.

We can't even do periodic (say weekly) mass rebuilds in Fedora, and we
know that there are uses for that, such as generating accurate markup
(be it annobin or CET) based on the latest toolchain fixes.

>> Andreas' setup is very good for detecting certain types of build
>> issues, and I wouldn't want to lose that.
>
> I assume you are arguing that a full bootstrap would take much longer
> than the set of packages Andreas identifies. Please make such assertions
> clear.

It's also much more brittle because as the Fedora experience shows, it
is very difficult to keep even the core of the distribution in a
permanently buildable state.  So it's not purely about speed.
Carlos O'Donell Aug. 14, 2018, 4:26 a.m. UTC | #14
On 08/13/2018 04:29 PM, Florian Weimer wrote:
> * Carlos O'Donell:
> 
>> It is a question of interest really. Fedora has the technical ability
>> to deliver a bootstrap that happens routinely, but I don't see the
>> motivation to do so.
> 
> Sorry, this sounds … very unconvincing.

Could you clarify what you were not convinced about? The technical ability
or lack of motivation?
 
> This is a capability that deteriorates very quickly if you do not
> continuously exercise it.

Agreed. How does adding python prevent you from continuously exercising
the bootstrap? Do you argue that it makes the whole build cycle too long?

> We can't even do periodic (say weekly) mass rebuilds in Fedora, and we
> know that there are uses for that, such as generating accurate markup
> (be it annobin or CET) based on the latest toolchain fixes.

Either we can't do it technically (not true, we can).

Or we aren't sufficiently motivated to do it (the real case).

I don't know why Andreas wants the core bootstrap small.

I don't understand Andreas' motivation.

If glibc upstream can't understand the reasons why downstream want a
minimal bootstrap then we can't ourselves support the position either.

You appear to argue that it would be nice if downstream Fedora had a
minimal bootstrap (without python) to test:

* Annobin (binary annotation work we are doing).

* Intel CET (ABI flag day changes for *context() routines).

That would be really nice, but the reality is that we don't value that
and so we haven't invested in it. Why should glibc upstream value it?

Using python in upstream glibc makes it easier to maintain, and the only
downside is the bootstrap issue. The solution *today* is to not build the
subsystems in glibc that need python (manual, benchtests) during bootstrap.

If more things require python then we may need a minimal python for
the bootstrap. Investing in that is a good idea only if you value what
we have just discussed as a bootstrap.

Other more useful bootstraps are IMO something like "Build all C++ programs
with the latest C++ compiler changes, but do it in a side tag" which would
allow the Fedora toolchain team to test C++ changes. These kinds of CI-esque
workflows don't need a minimal python or a minimal bootstrap since the C++
applications you are building will likely already require much of the distro
to be present.

>>> Andreas' setup is very good for detecting certain types of build
>>> issues, and I wouldn't want to lose that.
>>
>> I assume you are arguing that a full bootstrap would take much longer
>> than the set of packages Andreas identifies. Please make such assertions
>> clear.
> 
> It's also much more brittle because as the Fedora experience shows, it
> is very difficult to keep even the core of the distribution in a
> permanently buildable state.  So it's not purely about speed.

Agreed. This still doesn't convince me that you have a compelling enough
use case for minimal bootstraps that upstream glibc must not adopt python
as a bootstrap requirement (which it isn't yet).

To convince myself that the real solution is a python3-minimal solution I
started with the python3 in Fedora Rawhide, and with maybe 5 minutes of
spec file hacking I had a Python3 which built (in a clean rawhide mock
chroot) *without* all the dependencies enabled. It appears to work for 
simple bootstrap things. Tests are still running, but languages tests
are mostly passing (one _haslib failure).

Is there a technical problem with needing python3-minimal? It seems like
such a package would solve all sorts of early bootstrap issues with needing
python?
Florian Weimer Aug. 14, 2018, 5:43 a.m. UTC | #15
* Carlos O'Donell:

> On 08/13/2018 04:29 PM, Florian Weimer wrote:
>> * Carlos O'Donell:
>> 
>>> It is a question of interest really. Fedora has the technical ability
>>> to deliver a bootstrap that happens routinely, but I don't see the
>>> motivation to do so.
>> 
>> Sorry, this sounds … very unconvincing.
>
> Could you clarify what you were not convinced about? The technical ability
> or lack of motivation?

The ability, both at a technical and organizational level.

>> We can't even do periodic (say weekly) mass rebuilds in Fedora, and we
>> know that there are uses for that, such as generating accurate markup
>> (be it annobin or CET) based on the latest toolchain fixes.
>
> Either we can't do it technically (not true, we can).

I doubt that.  It would be a significant engineering effort.
Andreas Schwab Aug. 14, 2018, 7:10 a.m. UTC | #16
On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:

> I'm impressed, you've done a really good job keeping this minimal, but what's
> the problem with including python here?

We want to keep the cycle as small as possible.  For example, even
though rpmlint is included in every build root and it is written in
python, it is not part of the bootstrap cycle because it is replaced by
a special package rpmlint-mini that contains copies of the python files
and dependencies from the rpmlint package.  That shows how important it
is to keep python out of the equation.

Andreas.
Andreas Schwab Aug. 14, 2018, 7:18 a.m. UTC | #17
On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:

> It is a question of interest really. Fedora has the technical ability
> to deliver a bootstrap that happens routinely, but I don't see the
> motivation to do so. Only mass rebuilds happen routinely when Fedora
> makes a release (don't have bootstrap issues).

In openSUSE Factory, every package update goes through a staging build
where it is built and tested using the core subset of Factory (see
<https://build.opensuse.org/project/dashboard/openSUSE:Factory>).  Only
after the testing succeeded the package is added to Factory.

Andreas.
Joseph Myers Aug. 14, 2018, 12:57 p.m. UTC | #18
On Tue, 14 Aug 2018, Andreas Schwab wrote:

> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
> 
> > I'm impressed, you've done a really good job keeping this minimal, but what's
> > the problem with including python here?
> 
> We want to keep the cycle as small as possible.  For example, even
> though rpmlint is included in every build root and it is written in
> python, it is not part of the bootstrap cycle because it is replaced by
> a special package rpmlint-mini that contains copies of the python files
> and dependencies from the rpmlint package.  That shows how important it
> is to keep python out of the equation.

Given that you're already using special different packages in this 
bootstrap cycle, a special minimum python package is the obvious way to 
deal with uses in the glibc build.
Andreas Schwab Aug. 14, 2018, 1:03 p.m. UTC | #19
On Aug 14 2018, Joseph Myers <joseph@codesourcery.com> wrote:

> On Tue, 14 Aug 2018, Andreas Schwab wrote:
>
>> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>> 
>> > I'm impressed, you've done a really good job keeping this minimal, but what's
>> > the problem with including python here?
>> 
>> We want to keep the cycle as small as possible.  For example, even
>> though rpmlint is included in every build root and it is written in
>> python, it is not part of the bootstrap cycle because it is replaced by
>> a special package rpmlint-mini that contains copies of the python files
>> and dependencies from the rpmlint package.  That shows how important it
>> is to keep python out of the equation.
>
> Given that you're already using special different packages in this 
> bootstrap cycle, a special minimum python package is the obvious way to 
> deal with uses in the glibc build.

That already exists, it is called python{3}-base.

Andreas.
Zack Weinberg Aug. 14, 2018, 1:07 p.m. UTC | #20
On Tue, Aug 14, 2018 at 3:10 AM, Andreas Schwab <schwab@suse.de> wrote:
> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>
>> I'm impressed, you've done a really good job keeping this minimal, but what's
>> the problem with including python here?
>
> We want to keep the cycle as small as possible.  For example, even
> though rpmlint is included in every build root and it is written in
> python, it is not part of the bootstrap cycle because it is replaced by
> a special package rpmlint-mini that contains copies of the python files
> and dependencies from the rpmlint package.  That shows how important it
> is to keep python out of the equation.

I deeply sympathize with the desire to keep the bootstrap dependency
graph as small as possible, but I also deeply sympathize with the
glibc community's desire to move away from Perl and AWK.  Especially
AWK.

Let me ask you: how many other packages (besides glibc) in the
bootstrap depend on Perl?  Is it within our reach, perhaps, to
*replace* Perl with Python?

zw
Andreas Schwab Aug. 14, 2018, 1:20 p.m. UTC | #21
On Aug 14 2018, Zack Weinberg <zackw@panix.com> wrote:

> Let me ask you: how many other packages (besides glibc) in the
> bootstrap depend on Perl?  Is it within our reach, perhaps, to
> *replace* Perl with Python?

You would need to rewrite autoconf and automake in python.  Also, a lot
of testsuites are using perl.  Perl is ubiquitous, whereas python is
used almost nowhere.

Andreas.
Carlos O'Donell Aug. 14, 2018, 5:45 p.m. UTC | #22
On 08/14/2018 09:03 AM, Andreas Schwab wrote:
> On Aug 14 2018, Joseph Myers <joseph@codesourcery.com> wrote:
> 
>> On Tue, 14 Aug 2018, Andreas Schwab wrote:
>>
>>> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>>>
>>>> I'm impressed, you've done a really good job keeping this minimal, but what's
>>>> the problem with including python here?
>>>
>>> We want to keep the cycle as small as possible.  For example, even
>>> though rpmlint is included in every build root and it is written in
>>> python, it is not part of the bootstrap cycle because it is replaced by
>>> a special package rpmlint-mini that contains copies of the python files
>>> and dependencies from the rpmlint package.  That shows how important it
>>> is to keep python out of the equation.
>>
>> Given that you're already using special different packages in this 
>> bootstrap cycle, a special minimum python package is the obvious way to 
>> deal with uses in the glibc build.
> 
> That already exists, it is called python{3}-base.

What is your incremental cost to adding python3-base to the bootstrap?
Joseph Myers Aug. 14, 2018, 7:28 p.m. UTC | #23
On Tue, 14 Aug 2018, Zack Weinberg wrote:

> Let me ask you: how many other packages (besides glibc) in the
> bootstrap depend on Perl?  Is it within our reach, perhaps, to
> *replace* Perl with Python?

We can't even replace it for building the glibc manual (makeinfo is 
written in Perl).  What we can do is eliminate it as a *direct* dependency 
in glibc (so reducing the number of languages we need to deal with as 
glibc developers and the number of build-dependencies we need to 
document).
Andreas Schwab Aug. 15, 2018, 7:02 a.m. UTC | #24
On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote:

> What is your incremental cost to adding python3-base to the bootstrap?

It's the cost of having another package in the bootstrap cycle, which
means building it at least three times.

Andreas.
Carlos O'Donell Aug. 15, 2018, 2:20 p.m. UTC | #25
On 08/15/2018 03:02 AM, Andreas Schwab wrote:
> On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote:
> 
>> What is your incremental cost to adding python3-base to the bootstrap?
> 
> It's the cost of having another package in the bootstrap cycle, which
> means building it at least three times.

Does python3-base depend on many other things?
Andreas Schwab Aug. 15, 2018, 2:42 p.m. UTC | #26
On Aug 15 2018, Carlos O'Donell <carlos@redhat.com> wrote:

> On 08/15/2018 03:02 AM, Andreas Schwab wrote:
>> On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>> 
>>> What is your incremental cost to adding python3-base to the bootstrap?
>> 
>> It's the cost of having another package in the bootstrap cycle, which
>> means building it at least three times.
>
> Does python3-base depend on many other things?

Yes, it drags in more packages into the bootstrap cycle.

Andreas.
Carlos O'Donell Aug. 15, 2018, 3:09 p.m. UTC | #27
On 08/15/2018 10:42 AM, Andreas Schwab wrote:
> On Aug 15 2018, Carlos O'Donell <carlos@redhat.com> wrote:
> 
>> On 08/15/2018 03:02 AM, Andreas Schwab wrote:
>>> On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote:
>>>
>>>> What is your incremental cost to adding python3-base to the bootstrap?
>>>
>>> It's the cost of having another package in the bootstrap cycle, which
>>> means building it at least three times.
>>
>> Does python3-base depend on many other things?
> 
> Yes, it drags in more packages into the bootstrap cycle.

How many more packages?

How is the boostrap build related to the testing a package has to go
through before being added to Factory?

Does the core subset of Factory, which is used in the package testing,
get rebuilt every time the package is built, just to test that the
core is still buildable?
Michael Matz Aug. 16, 2018, 12:45 p.m. UTC | #28
Hi,

[answering to a bounced mail, I'm not on libc-alpha@, so keep me CCed]

I'm randomly using one mail of this thread as it covers most topics, but I 
use it for making general points, not only specifically answering to your 
mail Carlos.

On Tue, 14 Aug 2018, Carlos O'Donell wrote:

> > This is a capability that deteriorates very quickly if you do not 
> > continuously exercise it.
> 
> Agreed. How does adding python prevent you from continuously exercising 
> the bootstrap? Do you argue that it makes the whole build cycle too 
> long?

It's technically not (very) difficult to include python into the bootstrap 
set.  But that's not the most important part.  All SUSE distros since 
forever use this scheme of building the distro, a bootstrap set (iterating 
builds until stability), and once the bootstrap set is finished building 
all dependend packages in a topological order (breaking cycles in that 
other non-bootstrap set, if they exist, at defined points).

That implies the following thing: every change in any package of the 
bootstrap set implies a whole-distro rebuild (any change in any of the 
other packages implies only a rebuild on the depending packages).  So 
enlarging the bootstrap set makes full distro rebuild more common, and 
given that this takes about two to four days (depending on architecture) 
it's something you don't want to do lightly.  So changes to packages in 
the bootstrap set are done more seldom than for other packages.

> > We can't even do periodic (say weekly) mass rebuilds in Fedora, and we
> > know that there are uses for that, such as generating accurate markup
> > (be it annobin or CET) based on the latest toolchain fixes.
> 
> Either we can't do it technically (not true, we can).

I bet you theoretically can, in reality you don't.  Such bootstrap 
capability detoriates within weeks if not done regularly (we for instance 
see this with old enterprise products where this is not regularly done; 
once you try this you go "oh my god, what is this, what happened here?").

> I don't know why Andreas wants the core bootstrap small.

It's not Andreas, it's all (open)SUSE distros.  It's the way we do this 
since ages and it's not small work (occasionally) to keep the bootstrap 
set small.  Which is why we try very hard to avoid extending it, including 
having these very discussions.

> I don't understand Andreas' motivation.

I can bring up the original philosophy of how this came to be in the SUSE 
distros: whenever there's a change in any package, all other packages 
depending on it (per buildrequires, and transitively so) should be rebuilt 
(I won't got into the reasoning for this policy here).  That requires a 
topological sort, and as such can't handle cycles.  So we allowed exactly 
one cycle of packages, the bootstrap cycle.  That one builds until a fixed 
point (or well, in reality all packages a max of three times).  But all 
other packages are build in topological order.  Obviously it's a good idea 
to let this cycle be small, so that at least the rest of packages can be 
built according to the golden rule.  So, that's where we're coming from.

As you can see in Andreas' list the bootstrap cycle is small (and with 
minor additions over time contains still the same things since 20 years; 
though I think we could remove some more things from it with a little 
work).

Over time we of course got more cycles in BuildRequires than this (and the 
reason is often programmers not taking these things into account, e.g. 
adding a dependency on java from within gettext; just an example, not 
necessarily what happened), but still only the bootstrap cycle is iterated 
and implies a full distro rebuild.

> Using python in upstream glibc makes it easier to maintain, and the only
> downside is the bootstrap issue.

(sidenote: if I were a glibc maintainer I'd disagree; I know perl better 
than python.  Alas, I'm not :)  Though one point I'll make is that 
everybody sufficiently intelligent can learn any reasonable language; and 
hence the like or dislike for specific languages should never in itself be 
a reason to choose one or the other.  Rather all other reasons to prefer 
one over the other should trump such personal preferences.  Such 
considerations include: preexisting use of language $X, wideness of 
availability for language $X interpreters/compilers and so on)

Another point I made over at gcc@ is that for base tools like gcc and 
glibc the list of dependencies should be as minimal as possible, even at 
the expense of a little more work.  In my world the former is of much 
higher importance than a little convenience for authors of helper scripts.
To my liking even depending on perl should be avoided (surely a little 
text processing can just as well be done with awk).

> The solution *today* is to not build the subsystems in glibc that need 
> python (manual, benchtests) during bootstrap.

To note, for all of the above I'm not talking about use of anything for 
building optional stuff.  We do the above, and if python is to be used for 
building the manual we'll split that off as well from the main glibc 
build.  All my points above only apply to dependencies for 
building essential part, which for glibc would be, ... well, libc ;-)

> If more things require python then we may need a minimal python for the 
> bootstrap. Investing in that is a good idea only if you value what we 
> have just discussed as a bootstrap.

We split python already into a base and a full python (which requires 
tk and hence all of X).  It might be that we "only" need to add openssl to 
the bootstrap cycle in addition to python3-base.  It would be good if we 
can avoid all of this though.

> Other more useful bootstraps are IMO something like "Build all C++ 
> programs with the latest C++ compiler changes, but do it in a side tag" 
> which would allow the Fedora toolchain team to test C++ changes. These 
> kinds of CI-esque workflows don't need a minimal python or a minimal 
> bootstrap since the C++ applications you are building will likely 
> already require much of the distro to be present.

Actually at SUSE we do something similar to this as well.  All compiler 
changes are going into a staging project where many packages are rebuilt.  
A new compiler version (and new glibc and similar things) goes through 
larger stagings.  That is indeed nice, but orthogonal to the bootstrap 
topic.

> Agreed. This still doesn't convince me that you have a compelling enough 
> use case for minimal bootstraps that upstream glibc must not adopt 
> python as a bootstrap requirement (which it isn't yet).

What I'm always wondering in these discussion about switching languages 
for helper scripts (of all!) are compelling reasons to switch _to_ new 
infrastructures.  Shouldn't the onus of finding such reasons be on those 
suggesting a change?  (I realize that one stated reason is "easier 
maintenance", I merely disagree that even if this is true (which is at 
least questionable) that it'd be a very compelling reason).

> To convince myself that the real solution is a python3-minimal solution 
> I started with the python3 in Fedora Rawhide, and with maybe 5 minutes 
> of spec file hacking I had a Python3 which built (in a clean rawhide 
> mock chroot) *without* all the dependencies enabled.

As without any dependencies no C compiler would be installed I assume the 
mock chroot contains a multitude of things by default.  So is your 
experiment for creating a -minimal variant valid?

> Is there a technical problem with needing python3-minimal? It seems like 
> such a package would solve all sorts of early bootstrap issues with 
> needing python?

As we currently don't have any such problems including it wouldn't solve 
anything.  If you chose to create such problem, then yes, including a 
python3-base will be the technical solution to it.


Ciao,
Michael.
Joseph Myers Aug. 16, 2018, 1:04 p.m. UTC | #29
On Thu, 16 Aug 2018, Michael Matz wrote:

> a reason to choose one or the other.  Rather all other reasons to prefer 
> one over the other should trump such personal preferences.  Such 
> considerations include: preexisting use of language $X, wideness of 
> availability for language $X interpreters/compilers and so on)

I think popularity of a language is a perfectly relevant consideration 
(not the only one, of course).  It's not simply among glibc developers 
that python is preferred to perl; various language rankings show python 
has much higher popularity than perl nowadays, which serves as external 
evidence that new contributors as well are more likely to be comfortable 
with build infrastructure in python than in perl.

> Another point I made over at gcc@ is that for base tools like gcc and 
> glibc the list of dependencies should be as minimal as possible, even at 

I think the considerations for glibc are separate from those for GCC 
(given, for example, that glibc is almost certainly built, even when being 
cross-compiled, on a GNU system for which a range of build tools are 
likely already available, including python, whereas GCC allows for being 
built natively on a non-GNU system that may not have such tools, as a 
starting point for then building such other tools which might not build 
with random vendor compilers on proprietary Unices).

Building glibc is also much more specialized than building GCC (if you're 
building glibc you're probably either a glibc developer or a system 
integrator using it as part of building a whole system, rather than just 
building a newer version locally for your own use, and using a different 
version of glibc than that coming with the system is tricky, whereas 
building your own GCC more recent than any that comes with the OS is an 
expected and supported normal use of GCC).
Joseph Myers Aug. 21, 2018, 5:20 p.m. UTC | #30
This discussion seems to have died down.  How can we move further towards 
consensus on the use of Python in building the manual (taking account of 
the desires also expressed to use it to replace Awk scripts used in the 
build, and so make it a required build tool for building glibc at all)?
Carlos O'Donell Aug. 25, 2018, 2 a.m. UTC | #31
On 08/21/2018 01:20 PM, Joseph Myers wrote:
> This discussion seems to have died down.  How can we move further towards 
> consensus on the use of Python in building the manual (taking account of 
> the desires also expressed to use it to replace Awk scripts used in the 
> build, and so make it a required build tool for building glibc at all)?

https://sourceware.org/glibc/wiki/Consensus#How_do_I_build_consensus.3F

1. Define or reiterate success criteria.

Decide on the suitability of Python for core glibc scripts.

2. Identify a technical problems.

SUSE does not want python to be used in core glibc scripts because it would
increase the size of their bootstrap, costing them time, machine use, etc.

3. Look for resolution on a singular technical problem.

SUSE says it will cost them to add python to the bootstrap.

I expect that Joseph and I and perhaps others see Python is the strongest
viable option to making the core glibc scripts maintainable from now into
the future. Python is a growing language, users will be familiar with it,
and all of that plays into new developer adoption for the project.

The fact that SUSE has python-minimal to use is already indicative of that,
and while I'm empathic to SUSE's need here, I think they are going to loose
at some point and admit defeat that the minimal bootstrap needs some kind
of full-featured interpreted language and that language is Python or Perl.
Many professional development environments use LUA for this to limit
license and dependency exposure, but in glibc we've been using Python.

I'd like to hear that glibc's use of Python is going to cause SUSE
"undue hardship" in accommodating this use of Python, otherwise I'd like
to see us move forward with using Python.

4. Summarize the resolved list.

Nobody has object to non-core uses of Python in glibc. We use python to
run microbenchmarks, we use it to rebuild Unicode data sets, etc.

The question that remains is the use of python for core glibc scripts.
Joseph Myers Aug. 29, 2018, 4:56 p.m. UTC | #32
On Fri, 24 Aug 2018, Carlos O'Donell wrote:

> I'd like to hear that glibc's use of Python is going to cause SUSE
> "undue hardship" in accommodating this use of Python, otherwise I'd like
> to see us move forward with using Python.

Do any SUSE people wish to comment further on this issue?
Carlos O'Donell Aug. 29, 2018, 5:08 p.m. UTC | #33
On 08/29/2018 12:56 PM, Joseph Myers wrote:
> On Fri, 24 Aug 2018, Carlos O'Donell wrote:
> 
>> I'd like to hear that glibc's use of Python is going to cause SUSE
>> "undue hardship" in accommodating this use of Python, otherwise I'd like
>> to see us move forward with using Python.
> 
> Do any SUSE people wish to comment further on this issue?

Joseph,
 I will ping this again on Friday, since that will have been a week since
my request for information.

I'm not sure we can move forward on this issue until Andreas makes his
position clear, either a sustained objection (blocks consensus), or a
recorded objection to the change (does not block consensus).

Paul,

I'd like your opinion on this also. Do you have any opinion on the use
of python more uniformly within glibc to replace existing awk, perl,
and shell scripts? It would require a minimal python in the bootstrap,
which is what SUSE is objecting to.

Discussion starts here:
https://www.sourceware.org/ml/libc-alpha/2018-08/msg00218.html
Michael Matz Aug. 30, 2018, 12:59 p.m. UTC | #34
Hi,

On Wed, 29 Aug 2018, Carlos O'Donell wrote:

> >> I'd like to hear that glibc's use of Python is going to cause SUSE 
> >> "undue hardship" in accommodating this use of Python, otherwise I'd 
> >> like to see us move forward with using Python.
> > 
> > Do any SUSE people wish to comment further on this issue?
> 
> Joseph,
>  I will ping this again on Friday, since that will have been a week since
> my request for information.

I'm sorry, I haven't seen the above followup(s) as I'm not on libc-alpha@ 
and Andreas is on vacation right now.  I think we've said all we had to 
say.  As engineers we can deal with most things (though I'd have expected 
this from new contributors to glibc as well), so we'll manage.  Just try 
to not use python modules that aren't in the base distro of python or 
require native code parts.


Ciao,
Michael.
P.S: not looking forward to the helper scripts being rewritten into 
javascript 10 years from now because "it's more popular" ;-/
Carlos O'Donell Aug. 30, 2018, 1:53 p.m. UTC | #35
On 08/30/2018 08:59 AM, Michael Matz wrote:
> Hi,
> 
> On Wed, 29 Aug 2018, Carlos O'Donell wrote:
> 
>>>> I'd like to hear that glibc's use of Python is going to cause SUSE 
>>>> "undue hardship" in accommodating this use of Python, otherwise I'd 
>>>> like to see us move forward with using Python.
>>>
>>> Do any SUSE people wish to comment further on this issue?
>>
>> Joseph,
>>  I will ping this again on Friday, since that will have been a week since
>> my request for information.
> 
> I'm sorry, I haven't seen the above followup(s) as I'm not on libc-alpha@ 
> and Andreas is on vacation right now.  I think we've said all we had to 
> say.  As engineers we can deal with most things (though I'd have expected 
> this from new contributors to glibc as well), so we'll manage.  Just try 
> to not use python modules that aren't in the base distro of python or 
> require native code parts.

Does your position represent that of SUSE?

Andreas is a project steward, and GNU project maintainer, so we still need
his input on this issue. We will need to wait for him to come back and
respond to this.

Lastly, your objection is recorded, and that you will not block consensus
on this issue e.g. "deal with most things." Thank you for your input.

> P.S: not looking forward to the helper scripts being rewritten into 
> javascript 10 years from now because "it's more popular" ;-/

Are you kidding? We'll have a native WebAssembly port by then. You'll need a
whole browser in the bootstrap ;-)
Michael Matz Aug. 30, 2018, 2:14 p.m. UTC | #36
Hi,

On Thu, 30 Aug 2018, Carlos O'Donell wrote:

> > I'm sorry, I haven't seen the above followup(s) as I'm not on 
> > libc-alpha@ and Andreas is on vacation right now.  I think we've said 
> > all we had to say.  As engineers we can deal with most things (though 
> > I'd have expected this from new contributors to glibc as well), so 
> > we'll manage.  Just try to not use python modules that aren't in the 
> > base distro of python or require native code parts.
> 
> Does your position represent that of SUSE?

Effectively, yes.  Though of course, for real official positions of a 
company I guess I'd need to ask our press officer :)  (The ones that will 
have most practical daytoday problems with including python in the 
bootstrap cycle aren't actually us, the toolchain team, but the build 
people and project managers trying to fit build schedules for their 
products; and of course the python packagers who then are under much 
stricter constraints in updating their package.  For them I only speak out 
of experience not because we have had large meetings about this topic.)

> Andreas is a project steward, and GNU project maintainer, so we still 
> need his input on this issue. We will need to wait for him to come back 
> and respond to this.

Yes, definitely.

> Lastly, your objection is recorded, and that you will not block 
> consensus on this issue e.g. "deal with most things." Thank you for your 
> input.

I will also reserve the right to say "haha, told you so!" five years from 
now when people start complaining ;-)

> > P.S: not looking forward to the helper scripts being rewritten into 
> > javascript 10 years from now because "it's more popular" ;-/
> 
> Are you kidding? We'll have a native WebAssembly port by then. You'll 
> need a whole browser in the bootstrap ;-)

/me ogles musl  ;)


Ciao,
Michael.
Carlos O'Donell Aug. 30, 2018, 2:18 p.m. UTC | #37
On 08/30/2018 10:14 AM, Michael Matz wrote:
> On Thu, 30 Aug 2018, Carlos O'Donell wrote:
>> Are you kidding? We'll have a native WebAssembly port by then. You'll 
>> need a whole browser in the bootstrap ;-)
> 
> /me ogles musl  ;)

So will musl...
https://github.com/jfbastien/musl
Joseph Myers Aug. 30, 2018, 2:50 p.m. UTC | #38
On Thu, 30 Aug 2018, Carlos O'Donell wrote:

> Are you kidding? We'll have a native WebAssembly port by then. You'll need a

It exists (complete with copyright assignments on file at the FSF), just 
hasn't been updated lately (nor has the corresponding GCC port, and I 
don't know how complete the binutils port - which *is* upstream - is).

https://github.com/pipcet/glibc
https://github.com/pipcet/gcc
Andreas Schwab Sept. 4, 2018, 1:34 p.m. UTC | #39
I withdraw my objection, under the condition that the use of python is
restricted to the base python language.

Andreas.
Zack Weinberg Sept. 4, 2018, 1:52 p.m. UTC | #40
On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote:
>
> I withdraw my objection, under the condition that the use of python is
> restricted to the base python language.

Can you define "base python language" more precisely, please?  There
are a lot of modules in the Python standard library, and many of them
depend on externally-maintained libraries, and it's conceivable that
we would find a use for 'shelve' for instance...

zw
Andreas Schwab Sept. 4, 2018, 2:01 p.m. UTC | #41
On Sep 04 2018, Zack Weinberg <zackw@panix.com> wrote:

> On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote:
>>
>> I withdraw my objection, under the condition that the use of python is
>> restricted to the base python language.
>
> Can you define "base python language" more precisely, please?

Basically what is part of the openSUSE python3-base package.

Andreas.
Joseph Myers Sept. 4, 2018, 2:30 p.m. UTC | #42
On Tue, 4 Sep 2018, Andreas Schwab wrote:

> On Sep 04 2018, Zack Weinberg <zackw@panix.com> wrote:
> 
> > On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote:
> >>
> >> I withdraw my objection, under the condition that the use of python is
> >> restricted to the base python language.
> >
> > Can you define "base python language" more precisely, please?
> 
> Basically what is part of the openSUSE python3-base package.

Based on the description at 
<https://software.opensuse.org/package/python3-base> I've updated 
<https://sourceware.org/glibc/wiki/Style_and_Conventions> to say:

 * Scripts used in the build of glibc (a standard {{{make}}}, possibly 
regenerating files in the source tree if timestamps are out of order and 
the dependencies are present in the normal makefiles) need to avoid 
modules outside the Python standard library, and standard library modules 
depending on external libraries (such as XML, database and UI libraries).  
If a script used in the testsuite has other dependencies (e.g., the pretty 
printer tests using PExpect), the relevant tests should return UNSUPPORTED 
if those dependencies are not present.
 * Scripts used only by glibc developers may have other dependencies (e.g. 
requiring Python 3), where justified in a particular case.

Are there other cases to add of standard library modules depending on 
external libraries that should be avoided?  (There are external library 
dependencies for data compression modules and hashlib, for example; do we 
wish to exclude those, or say they are allowed?)

Paul, Carlos asked for your input.  Do you wish to comment on the use of 
Python in the glibc build (whether just for the manual, or more 
generally)?

Do we wish to make Python a requirement for the build process, not just 
for the manual (require it in configure, remove PYTHON conditionals in the 
makefiles), so that we can use it to replace awk scripts required for the 
build?

(My expectation is that on 1 January 2020 - or rather 1 Feburary 2020, 
since 1 January is the start of a release freeze - we can cease supporting 
Python 2 - that we can use the table at <https://devguide.python.org/> as 
the basis of what Python versions should be supported for building glibc.  
But we don't need to decide that now.)
Zack Weinberg Sept. 4, 2018, 2:44 p.m. UTC | #43
On Tue, Sep 4, 2018 at 10:30 AM Joseph Myers <joseph@codesourcery.com> wrote:
> On Tue, 4 Sep 2018, Andreas Schwab wrote:
>
> > On Sep 04 2018, Zack Weinberg <zackw@panix.com> wrote:
> >
> > > On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote:
> > >>
> > >> I withdraw my objection, under the condition that the use of python is
> > >> restricted to the base python language.
> > >
> > > Can you define "base python language" more precisely, please?
> >
> > Basically what is part of the openSUSE python3-base package.
>
> Based on the description at
> <https://software.opensuse.org/package/python3-base> I've updated
> <https://sourceware.org/glibc/wiki/Style_and_Conventions> to say:
>
>  * Scripts used in the build of glibc (a standard {{{make}}}, possibly
> regenerating files in the source tree if timestamps are out of order and
> the dependencies are present in the normal makefiles) need to avoid
> modules outside the Python standard library, and standard library modules
> depending on external libraries (such as XML, database and UI libraries).

The Debian equivalent, (lib)python2.7-minimal, has a README with an
exact list of modules that may be used.  This appears to be a subset
of the stuff in SUSE's python3-base  I've reproduced the contents of
that file (from unstable; specifically, 2.7.15-3_amd64.deb) below.

If SUSE has already gotten to the point where Python 2 is not part of
their bootstrap, maybe we should ditch Python 2 now rather than
waiting for its EOL.  Debian currently has both python2.7-minimal and
python3.6-minimal packages.  The sets of standard library modules
available from each appear to be roughly the same, after accounting
for all the Python 3 module renames.

May I additionally suggest that it's not obvious that this stuff is
documented under "Code formatting in python sources", and the section
should be renamed to indicate it's general policy for Python usage?

zw

----

Contents of the python2.7-minimal package
-----------------------------------------

python2.7-minimal consists of a minimum set of modules which may be needed
for python scripts used during the boot process.  If other packages
are needed in these scripts, don't work around the missing module, but
file a bug report against this package. The modules in this package
are:

  __builtin__        builtin
  __future__        module
  _abcoll        module
  _bisect        extension
  _bytesio        extension
  _codecs        builtin
  _collections        extension
  _fileio        extension
  _functools        extension
  _hashlib        extensionx
  _heapq        extension
  _io            extension
  _locale        extension
  _md5            extension
  _random        extension
  _sha            extension
  _sha256        extension
  _sha512        extension
  _socket        extension
  _sre            builtin
  _ssl            extensionx
  _struct        extension
  _symtable        builtin
  _sysconfigdata    module
  _types        builtin
  _warnings        builtin
  _weakref        extension
  _weakrefset        module
  abc            module
  atexit        module
  ConfigParser        module
  StringIO        module
  UserDict        module
  cPickle        extension
  cStringIO        extension
  array            extension
  base64        module
  binascii        extension
  bisect        module
  cmath            extension
  calendar        module
  codecs        module
  collections        module
  compileall        module
  contextlib        module
  copy            module
  copy_reg        module
  datetime        extension
  dis            module
  encodings        package
  errno            builtin
  exceptions        builtin
  fcntl            extension
  fnmatch        module
  functools        module
  gc            builtin
  genericpath        module
  getopt        module
  glob            module
  grp            extension
  hashlib        module
  heapq            module
  imp            builtin
  inspect        module
  io            module
  itertools        extension
  keyword        module
  linecache        module
  locale        module
  logging        package
  marshal        builtin
  math            extension
  md5            module
  opcode        module
  operator        extension
  optparse        module
  os            module
  pickle        module
  platform        module
  popen2        module
  posix            builtin
  posixpath        module
  pkgutil        module
  pwd            builtin
  py_compile        module
  random        module
  re            module
  repr            module
  runpy            module
  select        extension
  sha            module
  shutil        module
  signal        builtin
  socket        module
  spwd            extension
  sre            module
  sre_compile        module
  sre_constants        module
  sre_parse        module
  ssl            module
  stat            module
  string        module
  strop            extension
  struct        module
  subprocess        module
  sys            builtin
  syslog        extension
  sysconfig        module
  tempfile        module
  textwrap        module
  time            extension
  token            module
  thread        builtin
  token            module
  tokenize        module
  traceback        module
  types            module
  unicodedata        extension
  weakref        module
  warnings        module
  zipimport        extension
  zlib            extension

Included are as well the codecs and stringprep modules, and the encodings
modules for all encodings except the multibyte encodings and the bz2 codec.

The following modules are excluded, their import is guarded from the
importing module:

  Used in       Excluded
  ------------  ------------------------------------
  os        nt ntpath os2 os2emxpath mac macpath
        riscos riscospath riscosenviron
  optparse    gettext
  pickle    doctest
  subprocess    threading

This list was derived by looking at the modules in the perl-base package,
then adding python specific "core modules".

TODO's
------

- time.strptime cannot be used. The required _strptime module is not
  included in the -minimal package yet. _strptime, locale, _locale and
  calendar have to be added.

- modules used very often in the testsuite: copy, cPickle, operator.
Joseph Myers Sept. 4, 2018, 3:55 p.m. UTC | #44
On Tue, 4 Sep 2018, Zack Weinberg wrote:

> The Debian equivalent, (lib)python2.7-minimal, has a README with an
> exact list of modules that may be used.  This appears to be a subset
> of the stuff in SUSE's python3-base  I've reproduced the contents of
> that file (from unstable; specifically, 2.7.15-3_amd64.deb) below.

That list does not include argparse.  I think we should be able to use 
argparse in the glibc build.
Zack Weinberg Sept. 4, 2018, 5:11 p.m. UTC | #45
On Tue, Sep 4, 2018 at 11:55 AM Joseph Myers <joseph@codesourcery.com> wrote:
> On Tue, 4 Sep 2018, Zack Weinberg wrote:
> > The Debian equivalent, (lib)python2.7-minimal, has a README with an
> > exact list of modules that may be used.  This appears to be a subset
> > of the stuff in SUSE's python3-base  I've reproduced the contents of
> > that file (from unstable; specifically, 2.7.15-3_amd64.deb) below.
>
> That list does not include argparse.  I think we should be able to use
> argparse in the glibc build.

I agree. argparse is a pure-Python module and it doesn't look like it
depends on any C extensions that aren't already included in the
"minimal" subset.  I have filed wishlist bugs against python-minimal
(#907977) and python3-minimal (#907976) and we'll see what happens.

zw
Joseph Myers Sept. 4, 2018, 5:29 p.m. UTC | #46
On Tue, 4 Sep 2018, Zack Weinberg wrote:

> On Tue, Sep 4, 2018 at 11:55 AM Joseph Myers <joseph@codesourcery.com> wrote:
> > On Tue, 4 Sep 2018, Zack Weinberg wrote:
> > > The Debian equivalent, (lib)python2.7-minimal, has a README with an
> > > exact list of modules that may be used.  This appears to be a subset
> > > of the stuff in SUSE's python3-base  I've reproduced the contents of
> > > that file (from unstable; specifically, 2.7.15-3_amd64.deb) below.
> >
> > That list does not include argparse.  I think we should be able to use
> > argparse in the glibc build.
> 
> I agree. argparse is a pure-Python module and it doesn't look like it
> depends on any C extensions that aren't already included in the
> "minimal" subset.  I have filed wishlist bugs against python-minimal
> (#907977) and python3-minimal (#907976) and we'll see what happens.

Ubuntu 18.04 /usr/share/doc/python3.6-minimal/README.Debian *does* include 
argparse.  Is that some Ubuntu-specific addition?
Zack Weinberg Sept. 4, 2018, 5:55 p.m. UTC | #47
On Tue, Sep 4, 2018 at 1:29 PM Joseph Myers <joseph@codesourcery.com> wrote:
> On Tue, 4 Sep 2018, Zack Weinberg wrote:
> > On Tue, Sep 4, 2018 at 11:55 AM Joseph Myers <joseph@codesourcery.com> wrote:
> > > On Tue, 4 Sep 2018, Zack Weinberg wrote:
> > > > The Debian equivalent, (lib)python2.7-minimal, has a README with an
> > > > exact list of modules that may be used.  This appears to be a subset
> > > > of the stuff in SUSE's python3-base  I've reproduced the contents of
> > > > that file (from unstable; specifically, 2.7.15-3_amd64.deb) below.
> > >
> > > That list does not include argparse.  I think we should be able to use
> > > argparse in the glibc build.
> >
> > I agree. argparse is a pure-Python module and it doesn't look like it
> > depends on any C extensions that aren't already included in the
> > "minimal" subset.  I have filed wishlist bugs against python-minimal
> > (#907977) and python3-minimal (#907976) and we'll see what happens.
>
> Ubuntu 18.04 /usr/share/doc/python3.6-minimal/README.Debian *does* include
> argparse.  Is that some Ubuntu-specific addition?

Doh! No, I just missed it in the list.  argparse is present in the
python3.6-minimal list in version 3.6.6-3 of the package.

zw
Joseph Myers Sept. 10, 2018, 12:41 p.m. UTC | #48
Dmitry, do you wish to make any comments on the use of Python in building 
glibc?
Dmitry V. Levin Sept. 10, 2018, 6:12 p.m. UTC | #49
On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
> Dmitry, do you wish to make any comments on the use of Python in building 
> glibc?

My concern with making python a requirement to build glibc is that it will
pull in more packages not required to build glibc.  If there is a "minimal"
subset of python that does not bring indirect dependencies to glibc build,
and at the same time is sufficient to build glibc, then adding this subset
of python as a requirement to build glibc shouldn't be a problem.

I don't know yet whether SUSE or Debian take on the minimal subset pulls
in extra dependencies, I'll check this when time permits.
Joseph Myers Sept. 17, 2018, 12:03 p.m. UTC | #50
On Mon, 10 Sep 2018, Dmitry V. Levin wrote:

> On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
> > Dmitry, do you wish to make any comments on the use of Python in building 
> > glibc?
> 
> My concern with making python a requirement to build glibc is that it will
> pull in more packages not required to build glibc.  If there is a "minimal"
> subset of python that does not bring indirect dependencies to glibc build,
> and at the same time is sufficient to build glibc, then adding this subset
> of python as a requirement to build glibc shouldn't be a problem.
> 
> I don't know yet whether SUSE or Debian take on the minimal subset pulls
> in extra dependencies, I'll check this when time permits.

Any conclusions there?

I'm not sure it really matters for these purposes what a particular 
distribution's package pulls in, if they are happy with what works for 
them for the distribution build or bootstrap.  The question for us would 
be more about what constraints this imposes on distributions in general 
(e.g. if it means libffi will be needed for a native build of glibc, 
depending on how far the indirect dependencies on ctypes in the Python 
standard library go), and whether we are OK with those constraints.  The 
dependencies of the minimal distribution packages are simply useful 
information for us to help in determining those constraints.
Joseph Myers Sept. 25, 2018, 4:54 p.m. UTC | #51
On Mon, 17 Sep 2018, Joseph Myers wrote:

> On Mon, 10 Sep 2018, Dmitry V. Levin wrote:
> 
> > On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
> > > Dmitry, do you wish to make any comments on the use of Python in building 
> > > glibc?
> > 
> > My concern with making python a requirement to build glibc is that it will
> > pull in more packages not required to build glibc.  If there is a "minimal"
> > subset of python that does not bring indirect dependencies to glibc build,
> > and at the same time is sufficient to build glibc, then adding this subset
> > of python as a requirement to build glibc shouldn't be a problem.
> > 
> > I don't know yet whether SUSE or Debian take on the minimal subset pulls
> > in extra dependencies, I'll check this when time permits.
> 
> Any conclusions there?
> 
> I'm not sure it really matters for these purposes what a particular 
> distribution's package pulls in, if they are happy with what works for 
> them for the distribution build or bootstrap.  The question for us would 
> be more about what constraints this imposes on distributions in general 
> (e.g. if it means libffi will be needed for a native build of glibc, 
> depending on how far the indirect dependencies on ctypes in the Python 
> standard library go), and whether we are OK with those constraints.  The 
> dependencies of the minimal distribution packages are simply useful 
> information for us to help in determining those constraints.

Ping.  Any conclusion on this question about indirect dependencies 
introduced by use of Python?
Szabolcs Nagy Sept. 26, 2018, 10:04 a.m. UTC | #52
On 25/09/18 17:54, Joseph Myers wrote:
> On Mon, 17 Sep 2018, Joseph Myers wrote:
>> On Mon, 10 Sep 2018, Dmitry V. Levin wrote:
>>> On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
>>>> Dmitry, do you wish to make any comments on the use of Python in building
>>>> glibc?
>>>
>>> My concern with making python a requirement to build glibc is that it will
>>> pull in more packages not required to build glibc.  If there is a "minimal"
>>> subset of python that does not bring indirect dependencies to glibc build,
>>> and at the same time is sufficient to build glibc, then adding this subset
>>> of python as a requirement to build glibc shouldn't be a problem.
>>>
>>> I don't know yet whether SUSE or Debian take on the minimal subset pulls
>>> in extra dependencies, I'll check this when time permits.
>>
>> Any conclusions there?
>>
>> I'm not sure it really matters for these purposes what a particular
>> distribution's package pulls in, if they are happy with what works for
>> them for the distribution build or bootstrap.  The question for us would
>> be more about what constraints this imposes on distributions in general
>> (e.g. if it means libffi will be needed for a native build of glibc,
>> depending on how far the indirect dependencies on ctypes in the Python
>> standard library go), and whether we are OK with those constraints.  The
>> dependencies of the minimal distribution packages are simply useful
>> information for us to help in determining those constraints.
> 
> Ping.  Any conclusion on this question about indirect dependencies
> introduced by use of Python?

using python3 in the build would be a bit disruptive for
me, because the current systems where i build glibc don't
have it.

last time when make 4.0 requirement was added i had to
build it on several machines until our infrastructure
got updated to have make 4.0 in the path everywhere.
(and our regular testing machines did not produce useful
results for several days because of broken glibc build)

installing python3 manually is obviously more effort and
time, so please do not add such dependency anywhere near
release time (like it was done with make 4).

i don't have other concerns regarding indirect deps if
the python3.4-minimal debian/ubuntu package works.
Joseph Myers Oct. 1, 2018, 3:06 p.m. UTC | #53
On Tue, 25 Sep 2018, Joseph Myers wrote:

> On Mon, 17 Sep 2018, Joseph Myers wrote:
> 
> > On Mon, 10 Sep 2018, Dmitry V. Levin wrote:
> > 
> > > On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
> > > > Dmitry, do you wish to make any comments on the use of Python in building 
> > > > glibc?
> > > 
> > > My concern with making python a requirement to build glibc is that it will
> > > pull in more packages not required to build glibc.  If there is a "minimal"
> > > subset of python that does not bring indirect dependencies to glibc build,
> > > and at the same time is sufficient to build glibc, then adding this subset
> > > of python as a requirement to build glibc shouldn't be a problem.
> > > 
> > > I don't know yet whether SUSE or Debian take on the minimal subset pulls
> > > in extra dependencies, I'll check this when time permits.
> > 
> > Any conclusions there?
> > 
> > I'm not sure it really matters for these purposes what a particular 
> > distribution's package pulls in, if they are happy with what works for 
> > them for the distribution build or bootstrap.  The question for us would 
> > be more about what constraints this imposes on distributions in general 
> > (e.g. if it means libffi will be needed for a native build of glibc, 
> > depending on how far the indirect dependencies on ctypes in the Python 
> > standard library go), and whether we are OK with those constraints.  The 
> > dependencies of the minimal distribution packages are simply useful 
> > information for us to help in determining those constraints.
> 
> Ping.  Any conclusion on this question about indirect dependencies 
> introduced by use of Python?

Another ping on this question....
Carlos O'Donell Oct. 1, 2018, 3:21 p.m. UTC | #54
On 10/1/18 11:06 AM, Joseph Myers wrote:
> On Tue, 25 Sep 2018, Joseph Myers wrote:
> 
>> On Mon, 17 Sep 2018, Joseph Myers wrote:
>>
>>> On Mon, 10 Sep 2018, Dmitry V. Levin wrote:
>>>
>>>> On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote:
>>>>> Dmitry, do you wish to make any comments on the use of Python in building 
>>>>> glibc?
>>>>
>>>> My concern with making python a requirement to build glibc is that it will
>>>> pull in more packages not required to build glibc.  If there is a "minimal"
>>>> subset of python that does not bring indirect dependencies to glibc build,
>>>> and at the same time is sufficient to build glibc, then adding this subset
>>>> of python as a requirement to build glibc shouldn't be a problem.
>>>>
>>>> I don't know yet whether SUSE or Debian take on the minimal subset pulls
>>>> in extra dependencies, I'll check this when time permits.
>>>
>>> Any conclusions there?
>>>
>>> I'm not sure it really matters for these purposes what a particular 
>>> distribution's package pulls in, if they are happy with what works for 
>>> them for the distribution build or bootstrap.  The question for us would 
>>> be more about what constraints this imposes on distributions in general 
>>> (e.g. if it means libffi will be needed for a native build of glibc, 
>>> depending on how far the indirect dependencies on ctypes in the Python 
>>> standard library go), and whether we are OK with those constraints.  The 
>>> dependencies of the minimal distribution packages are simply useful 
>>> information for us to help in determining those constraints.
>>
>> Ping.  Any conclusion on this question about indirect dependencies 
>> introduced by use of Python?
> 
> Another ping on this question....
 Could you help me to gather consensus?

If you could write up the following:

* Exact version of python to use.

* Exact set of imports we need.

I'll take that and try to get an acknowledgment from each of the
distributions. I'll email all the maintainers and follow up with
them (all on libc-alpha) in a new thread starting with the information
you will put together for me.

I'll also include some of the resolution about the concerns regarding
building python on an old distribution.
Zack Weinberg Oct. 1, 2018, 3:34 p.m. UTC | #55
On Mon, Oct 1, 2018 at 11:21 AM Carlos O'Donell <carlos@redhat.com> wrote:
> On 10/1/18 11:06 AM, Joseph Myers wrote:
> > On Tue, 25 Sep 2018, Joseph Myers wrote:
> >> Ping.  Any conclusion on this question about indirect dependencies
> >> introduced by use of Python?
> >
> > Another ping on this question....
>  Could you help me to gather consensus?
>
> If you could write up the following:
>
> * Exact version of python to use.
>
> * Exact set of imports we need.
>
> I'll take that and try to get an acknowledgment from each of the
> distributions. I'll email all the maintainers and follow up with
> them (all on libc-alpha) in a new thread starting with the information
> you will put together for me.
>
> I'll also include some of the resolution about the concerns regarding
> building python on an old distribution.

For the record, although I offered some details earlier about how
Debian packages Python, I have no official relationship to Debian and
cannot speak for any of their teams that might care about this
proposed change.

As a glibc developer who might well take on some of the work of
rewriting existing build logic in Python, I would like to put in a
word for requiring Python 3.(>=4) rather than allowing 2.7 as well.
The 2.x series is close to end-of-life, writing scripts that work with
both major versions of the language is significantly harder than
writing 3.x-only code, and I fear 2.x build compatibility will not be
adequately tested.  3.4 is the oldest version of the 3.x series still
in wide use AFAIK.

zw
Joseph Myers Oct. 1, 2018, 3:44 p.m. UTC | #56
On Mon, 1 Oct 2018, Carlos O'Donell wrote:

> If you could write up the following:
> 
> * Exact version of python to use.

Any non-EOL Python 3 series.  So 3.4 or later at present, 3.5 or later 
could be required from 2019-03-16 (but that doesn't mean we have to 
increase the requirement then, if there aren't any new features we want to 
use), 3.6 or later could be required from 2020-09-13, as per the table at 
<https://devguide.python.org/>.

> * Exact set of imports we need.

Only modules in the Python (3.4) standard library, and excluding modules 
such as XML, database and UI libraries that depend on external libraries.  
I don't think we need anything depending on OpenSSL (hashlib appears to 
work without it, in Python 3.7, and we probably don't need hashlib 
anyway), but it's possible there may be dependencies on libffi - however, 
I think it's likely we can avoid ctypes which is what brings in the libffi 
dependency, and indirect dependencies on it from e.g. multiprocessing.

I think trying to list exhaustively exactly which modules may be used is a 
bad idea, since there are lots of pure-Python modules in the standard 
library without any external C library dependencies (direct or indirect) 
and it's hard to predict exactly which could be useful at some point.  
gen-libm-test.py uses argparse, collections, os (after the present patch 
that would require it to build the manual) and re.  The scripts used in 
the conform/ tests (not of course strictly relevant to build requirements, 
exact in regard to what tests can use unconditionally rather than needing 
to be conditionally UNSUPPORTED) also use shutil, subprocess, tempfile.  I 
expect there would be plenty more pure-Python standard library modules 
that would turn out to be useful for some particular script.
Joseph Myers Oct. 11, 2018, 8:44 p.m. UTC | #57
On Mon, 1 Oct 2018, Carlos O'Donell wrote:

> I'll take that and try to get an acknowledgment from each of the
> distributions. I'll email all the maintainers and follow up with
> them (all on libc-alpha) in a new thread starting with the information
> you will put together for me.
> 
> I'll also include some of the resolution about the concerns regarding
> building python on an old distribution.

Ping?  I'd like to be able to move the gen-as-const machinery from awk + 
sed + makefile logic to python + simpler makefile logic, with a view to it 
being the starting point of general python infrastructure for extracting 
constants, macros etc. from C headers (there are various existing tests, 
e.g. for the syscall list and for signal numbers, that could be rewritten 
on top of such shared infrastructure, and I'd like to be able to add a lot 
more tests of consistency with uapi headers using such infrastructure).
Joseph Myers Oct. 17, 2018, 2:20 p.m. UTC | #58
On Thu, 11 Oct 2018, Joseph Myers wrote:

> On Mon, 1 Oct 2018, Carlos O'Donell wrote:
> 
> > I'll take that and try to get an acknowledgment from each of the
> > distributions. I'll email all the maintainers and follow up with
> > them (all on libc-alpha) in a new thread starting with the information
> > you will put together for me.
> > 
> > I'll also include some of the resolution about the concerns regarding
> > building python on an old distribution.
> 
> Ping?  I'd like to be able to move the gen-as-const machinery from awk + 
> sed + makefile logic to python + simpler makefile logic, with a view to it 
> being the starting point of general python infrastructure for extracting 
> constants, macros etc. from C headers (there are various existing tests, 
> e.g. for the syscall list and for signal numbers, that could be rewritten 
> on top of such shared infrastructure, and I'd like to be able to add a lot 
> more tests of consistency with uapi headers using such infrastructure).

Ping^4?  Dmitry, Carlos, any other distributors, any further comments on 
requiring Python 3.4 (standard library modules not involving external C 
library dependencies) or later to build glibc?
Florian Weimer Oct. 17, 2018, 2:42 p.m. UTC | #59
* Joseph Myers:

> Ping^4?  Dmitry, Carlos, any other distributors, any further comments on 
> requiring Python 3.4 (standard library modules not involving external C 
> library dependencies) or later to build glibc?

Fedora and Red Hat Enterprise Linux have sufficiently new versions of
Python 3 readily available, so from a distribution perspective, it is
not an issue for us.

Debian 8/jessie is the oldest (LTS-)supported version, and it has Python
3.4, too.  Debian wheezy does not, but I don't think you can reasonably
use it for glibc development anyway.

Thanks,
Florian
Joseph Myers Oct. 18, 2018, 9:24 p.m. UTC | #60
On Wed, 17 Oct 2018, Florian Weimer wrote:

> * Joseph Myers:
> 
> > Ping^4?  Dmitry, Carlos, any other distributors, any further comments on 
> > requiring Python 3.4 (standard library modules not involving external C 
> > library dependencies) or later to build glibc?
> 
> Fedora and Red Hat Enterprise Linux have sufficiently new versions of
> Python 3 readily available, so from a distribution perspective, it is
> not an issue for us.

Thanks.  Dmitry?  Your previous comments were at 
<https://sourceware.org/ml/libc-alpha/2018-09/msg00107.html>.  Have you 
reached any conclusions regarding dependencies from minimal Python?
Dmitry V. Levin Oct. 18, 2018, 9:49 p.m. UTC | #61
On Wed, Oct 17, 2018 at 02:20:45PM +0000, Joseph Myers wrote:
> On Thu, 11 Oct 2018, Joseph Myers wrote:
> > On Mon, 1 Oct 2018, Carlos O'Donell wrote:
> > 
> > > I'll take that and try to get an acknowledgment from each of the
> > > distributions. I'll email all the maintainers and follow up with
> > > them (all on libc-alpha) in a new thread starting with the information
> > > you will put together for me.
> > > 
> > > I'll also include some of the resolution about the concerns regarding
> > > building python on an old distribution.
> > 
> > Ping?  I'd like to be able to move the gen-as-const machinery from awk + 
> > sed + makefile logic to python + simpler makefile logic, with a view to it 
> > being the starting point of general python infrastructure for extracting 
> > constants, macros etc. from C headers (there are various existing tests, 
> > e.g. for the syscall list and for signal numbers, that could be rewritten 
> > on top of such shared infrastructure, and I'd like to be able to add a lot 
> > more tests of consistency with uapi headers using such infrastructure).
> 
> Ping^4?  Dmitry, Carlos, any other distributors, any further comments on 
> requiring Python 3.4 (standard library modules not involving external C 
> library dependencies) or later to build glibc?

Thanks, I think this should be fine for us, too.
diff mbox

Patch

diff --git a/INSTALL b/INSTALL
index 603b23b..518ed58 100644
--- a/INSTALL
+++ b/INSTALL
@@ -514,8 +514,9 @@  build the GNU C Library:
 
    * Python 2.7/3.4 or later
 
-     Python is required to run some tests.  As of release time, Python
-     3.6 is the newest verified to work for testing the GNU C Library.
+     Python is required to build the GNU C Library manual and to run
+     some tests.  As of release time, Python 3.6 is the newest verified
+     to work for testing the GNU C Library.
 
    * PExpect 4.0
 
diff --git a/NEWS b/NEWS
index 7868c84..b208404 100644
--- a/NEWS
+++ b/NEWS
@@ -23,7 +23,7 @@  Deprecated and removed features, and other changes affecting compatibility:
 
 Changes to build and runtime requirements:
 
-  [Add changes to build and runtime requirements here]
+* Python is required to build the GNU C Library manual.
 
 Security related changes:
 
diff --git a/manual/Makefile b/manual/Makefile
index c275664..9c35c9d 100644
--- a/manual/Makefile
+++ b/manual/Makefile
@@ -99,10 +99,11 @@  $(objpfx)dir-add.texi: xtract-typefun.awk $(texis-path)
 
 # The table with the math errors is generated.
 $(objpfx)libm-err.texi: $(objpfx)stamp-libm-err
-$(objpfx)stamp-libm-err: libm-err-tab.pl $(wildcard $(foreach dir,$(sysdirs),\
-						     $(dir)/libm-test-ulps))
+$(objpfx)stamp-libm-err: $(..)math/gen-libm-test.py \
+			 $(wildcard $(foreach dir,$(sysdirs),\
+						  $(dir)/libm-test-ulps))
 	pwd=`pwd`; \
-	$(PERL) $< $$pwd/.. > $(objpfx)libm-err-tmp
+	$(PYTHON) $< -s $$pwd/.. -m $(objpfx)libm-err-tmp
 	$(move-if-change) $(objpfx)libm-err-tmp $(objpfx)libm-err.texi
 	touch $@
 
@@ -169,8 +170,9 @@  include ../Rules
 
 .PHONY: install subdir_install install-data
 install-data subdir_install: install
-# Generated files requiring perl: libm-err.texi, summary.texi
-ifneq ($(PERL),no)
+# Generated files requiring python: libm-err.texi
+# Generated files requiring perl: summary.texi
+ifneq ($(if $(PYTHON),$(PERL),no),no)
 ifneq ($(strip $(MAKEINFO)),:)
 install: $(inst_infodir)/libc.info
 	@if $(SHELL) -c '$(INSTALL_INFO) --version' >/dev/null 2>&1; then \
diff --git a/manual/install.texi b/manual/install.texi
index df32b39..5372bc4 100644
--- a/manual/install.texi
+++ b/manual/install.texi
@@ -560,8 +560,9 @@  with any version of @code{sed}.  As of release time, @code{sed} version
 @item
 Python 2.7/3.4 or later
 
-Python is required to run some tests.  As of release time, Python 3.6
-is the newest verified to work for testing @theglibc{}.
+Python is required to build the @glibcadj{} manual and to run some
+tests.  As of release time, Python 3.6 is the newest verified to work
+for testing @theglibc{}.
 
 @item PExpect 4.0
 
diff --git a/manual/libm-err-tab.pl b/manual/libm-err-tab.pl
deleted file mode 100755
index 768d2da..0000000
--- a/manual/libm-err-tab.pl
+++ /dev/null
@@ -1,206 +0,0 @@ 
-#!/usr/bin/perl -w
-# Copyright (C) 1999-2018 Free Software Foundation, Inc.
-# This file is part of the GNU C Library.
-# Contributed by Andreas Jaeger <aj@suse.de>, 1999.
-
-# The GNU C Library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-
-# The GNU C Library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-# Lesser General Public License for more details.
-
-# You should have received a copy of the GNU Lesser General Public
-# License along with the GNU C Library; if not, see
-# <http://www.gnu.org/licenses/>.
-
-# Information about tests are stored in: %results
-# $results{$test}{"type"} is the result type, e.g. normal or complex.
-# In the following description $platform, $type and $float are:
-# - $platform is the used platform
-# - $type is either "normal", "real" (for the real part of a complex number)
-#   or "imag" (for the imaginary part # of a complex number).
-# - $float is either of float, ifloat, double, idouble, ldouble, ildouble;
-#   It represents the underlying floating point type (float, double or long
-#   double) and if inline functions (the leading i stands for inline)
-#   are used.
-# $results{$test}{$platform}{$type}{$float} is defined and has a delta
-# or 'fail' as value.
-
-use File::Find;
-
-use strict;
-
-use vars qw ($sources @platforms %pplatforms);
-use vars qw (%results @all_floats %suffices %all_functions);
-
-
-# all_floats is in output order and contains all recognised float types that
-# we're going to output
-@all_floats = ('float', 'double', 'ldouble', 'float128');
-%suffices =
-  ( 'float' => 'f',
-    'double' => '',
-    'ldouble' => 'l',
-    'float128' => 'f128'
-  );
-
-# Pretty description of platform
-%pplatforms = ();
-
-%all_functions = ();
-
-if ($#ARGV == 0) {
-  $sources = $ARGV[0];
-} else {
-  $sources = '/usr/src/cvs/libc';
-}
-
-find (\&find_files, $sources);
-
-@platforms = sort by_platforms @platforms;
-
-&print_all;
-
-sub find_files {
-  if ($_ eq 'libm-test-ulps') {
-    # print "Parsing $File::Find::name\n";
-    push @platforms, $File::Find::dir;
-    my ($file, $name);
-    $file = "${File::Find::name}-name";
-    open NAME, $file or die ("Can't open $file: $!");
-    $name = <NAME>;
-    chomp $name;
-    close NAME;
-    $pplatforms{$File::Find::dir} = $name;
-    &parse_ulps ($File::Find::name, $File::Find::dir);
-  }
-}
-
-# Parse ulps file
-sub parse_ulps {
-  my ($file, $platform) = @_;
-  my ($test, $type, $float, $eps, $ignore_fn);
-
-  # $type has the following values:
-  # "normal": No complex variable
-  # "real": Real part of complex result
-  # "imag": Imaginary part of complex result
-  open ULP, $file  or die ("Can't open $file: $!");
-  while (<ULP>) {
-    chop;
-    # ignore comments and empty lines
-    next if /^#/;
-    next if /^\s*$/;
-    if (/^Function: /) {
-      if (/Real part of/) {
-	s/Real part of //;
-	$type = 'real';
-      } elsif (/Imaginary part of/) {
-	s/Imaginary part of //;
-	$type = 'imag';
-      } else {
-	$type = 'normal';
-      }
-      ($test) = ($_ =~ /^Function:\s*\"([a-zA-Z0-9_]+)\"/);
-      next;
-    }
-    if ($test =~ /_(downward|towardzero|upward|vlen)/) {
-      $ignore_fn = 1;
-    } else {
-      $ignore_fn = 0;
-      $all_functions{$test} = 1;
-    }
-    if (/^i?(float|double|ldouble|float128):/) {
-      ($float, $eps) = split /\s*:\s*/,$_,2;
-      if ($ignore_fn) {
-	next;
-      } elsif ($eps eq 'fail') {
-	$results{$test}{$platform}{$type}{$float} = 'fail';
-      } elsif ($eps eq "0") {
-	# ignore
-	next;
-      } elsif (!exists $results{$test}{$platform}{$type}{$float}
-	    || $results{$test}{$platform}{$type}{$float} ne 'fail') {
-	$results{$test}{$platform}{$type}{$float} = $eps;
-      }
-      if ($type =~ /^real|imag$/) {
-	$results{$test}{'type'} = 'complex';
-      } elsif ($type eq 'normal') {
-	$results{$test}{'type'} = 'normal';
-      }
-      next;
-    }
-    print "Skipping unknown entry: `$_'\n";
-  }
-  close ULP;
-}
-
-sub get_value {
-  my ($fct, $platform, $type, $float) = @_;
-
-  return (exists $results{$fct}{$platform}{$type}{$float}
-	  ? $results{$fct}{$platform}{$type}{$float} : "0");
-}
-
-sub print_platforms {
-  my (@p) = @_;
-  my ($fct, $platform, $float, $first, $i, $platform_no, $platform_total);
-
-  print '@multitable {nexttowardf} ';
-  foreach (@p) {
-    print ' {1000 + i 1000}';
-  }
-  print "\n";
-
-  print '@item Function ';
-  foreach (@p) {
-    print ' @tab ';
-    print $pplatforms{$_};
-  }
-  print "\n";
-
-
-  foreach $fct (sort keys %all_functions) {
-    foreach $float (@all_floats) {
-      print "\@item $fct$suffices{$float} ";
-      foreach $platform (@p) {
-	print ' @tab ';
-	if (exists $results{$fct}{$platform}{'normal'}{$float}
-	    || exists $results{$fct}{$platform}{'real'}{$float}
-	    || exists $results{$fct}{$platform}{'imag'}{$float}) {
-	  if ($results{$fct}{'type'} eq 'complex') {
-	    print &get_value ($fct, $platform, 'real', $float),
-	    ' + i ', &get_value ($fct, $platform, 'imag', $float);
-	  } else {
-	    print $results{$fct}{$platform}{'normal'}{$float};
-	  }
-	} else {
-	  print '-';
-	}
-      }
-      print "\n";
-    }
-  }
-
-  print "\@end multitable\n";
-}
-
-sub print_all {
-  my ($i, $max);
-
-  my ($columns) = 5;
-
-  # Print only 5 platforms at a time.
-  for ($i=0; $i < $#platforms; $i+=$columns) {
-    $max = $i+$columns-1 > $#platforms ? $#platforms : $i+$columns-1;
-    print_platforms (@platforms[$i .. $max]);
-  }
-}
-
-sub by_platforms {
-  return $pplatforms{$a} cmp $pplatforms{$b};
-}
diff --git a/math/gen-libm-test.py b/math/gen-libm-test.py
index 8d156ea..0b25a90 100755
--- a/math/gen-libm-test.py
+++ b/math/gen-libm-test.py
@@ -19,6 +19,7 @@ 
 
 import argparse
 from collections import defaultdict
+import os
 import re
 
 
@@ -32,6 +33,16 @@  ALL_FLOATS_PFX = {'double': 'DBL',
                   'float': 'FLT',
                   'float128': 'FLT128'}
 
+# Float types in the order used in the generated ulps tables in the
+# manual.
+ALL_FLOATS_MANUAL = ('float', 'double', 'ldouble', 'float128')
+
+# Map float types in ulps files to C function suffix.
+ALL_FLOATS_SUFFIX = {'double': '',
+                     'ldouble': 'l',
+                     'float': 'f',
+                     'float128': 'f128'}
+
 # Number of arguments in structure (as opposed to arguments that are
 # pointers to return values) for an argument descriptor.
 DESCR_NUM_ARGS = {'f': 1, 'a': 1, 'j': 1, 'i': 1, 'u': 1, 'l': 1, 'L': 1,
@@ -161,6 +172,17 @@  class Ulps(object):
                             ulps_dict[ulps_fn][line_first],
                             ulps_val)
 
+    def all_functions(self):
+        """Return the set of functions with ulps and whether they are
+        complex."""
+        funcs = set()
+        complex = {}
+        for k_prefix, k_dict in self.ulps_kinds:
+            for f in k_dict:
+                funcs.add(f)
+                complex[f] = True if k_prefix else False
+        return funcs, complex
+
     def write(self, ulps_file):
         """Write ulps back out as a sorted ulps file."""
         # Output is sorted first by function name, then by (real,
@@ -231,6 +253,18 @@  class Ulps(object):
             f.write(header_text)
 
 
+def read_all_ulps(srcdir):
+    """Read all platforms' libm-test-ulps files."""
+    all_ulps = {}
+    for dirpath, dirnames, filenames in os.walk(srcdir):
+        if 'libm-test-ulps' in filenames:
+            with open(os.path.join(dirpath, 'libm-test-ulps-name')) as f:
+                name = f.read().rstrip()
+            all_ulps[name] = Ulps()
+            all_ulps[name].read(os.path.join(dirpath, 'libm-test-ulps'))
+    return all_ulps
+
+
 def read_auto_tests(test_file):
     """Read tests from auto-libm-test-out-<function> (possibly None)."""
     auto_tests = defaultdict(lambda: defaultdict(dict))
@@ -571,6 +605,60 @@  def generate_testfile(inc_input, auto_tests, c_output):
         f.write(''.join(test_list))
 
 
+def generate_err_table_sub(all_ulps, all_functions, fns_complex, platforms):
+    """Generate a single table within the overall ulps table section."""
+    plat_width = [' {1000 + i 1000}' for p in platforms]
+    plat_header = [' @tab %s' % p for p in platforms]
+    table_list = ['@multitable {nexttowardf} %s\n' % ''.join(plat_width),
+                  '@item Function %s\n' % ''.join(plat_header)]
+    for func in all_functions:
+        for flt in ALL_FLOATS_MANUAL:
+            func_ulps = []
+            for p in platforms:
+                p_ulps = all_ulps[p]
+                if fns_complex[func]:
+                    ulp_real = p_ulps.real[func][flt]
+                    ulp_imag = p_ulps.imag[func][flt]
+                    ulp_str = '%d + i %d' % (ulp_real, ulp_imag)
+                    ulp_str = ulp_str if ulp_real or ulp_imag else '-'
+                else:
+                    ulp = p_ulps.normal[func][flt]
+                    ulp_str = str(ulp) if ulp else '-'
+                func_ulps.append(ulp_str)
+            table_list.append('@item %s%s  @tab %s\n'
+                              % (func, ALL_FLOATS_SUFFIX[flt],
+                                 ' @tab '.join(func_ulps)))
+    table_list.append('@end multitable\n')
+    return ''.join(table_list)
+
+
+def generate_err_table(all_ulps, err_table):
+    """Generate ulps table for manual."""
+    all_platforms = sorted(all_ulps.keys())
+    functions_set = set()
+    functions_complex = {}
+    for p in all_platforms:
+        p_functions, p_complex = all_ulps[p].all_functions()
+        functions_set.update(p_functions)
+        functions_complex.update(p_complex)
+    all_functions = sorted([f for f in functions_set
+                            if ('_downward' not in f
+                                and '_towardzero' not in f
+                                and '_upward' not in f
+                                and '_vlen' not in f)])
+    err_table_list = []
+    # Print five platforms at a time.
+    num_platforms = len(all_platforms)
+    for i in range((num_platforms + 4) // 5):
+        start = i * 5
+        end = i * 5 + 5 if num_platforms >= i * 5 + 5 else num_platforms
+        err_table_list.append(generate_err_table_sub(all_ulps, all_functions,
+                                                     functions_complex,
+                                                     all_platforms[start:end]))
+    with open(err_table, 'w') as f:
+        f.write(''.join(err_table_list))
+
+
 def main():
     """The main entry point."""
     parser = argparse.ArgumentParser(description='Generate libm tests.')
@@ -580,23 +668,31 @@  def main():
                         help='input file .inc file with tests')
     parser.add_argument('-u', dest='ulps_file', metavar='FILE',
                         help='input file with ulps')
+    parser.add_argument('-s', dest='srcdir', metavar='DIR',
+                        help='input source directory with all ulps')
     parser.add_argument('-n', dest='ulps_output', metavar='FILE',
                         help='generate sorted ulps file FILE')
     parser.add_argument('-C', dest='c_output', metavar='FILE',
                         help='generate output C file FILE from .inc file')
     parser.add_argument('-H', dest='ulps_header', metavar='FILE',
                         help='generate output ulps header FILE')
+    parser.add_argument('-m', dest='err_table', metavar='FILE',
+                        help='generate output ulps table for manual FILE')
     args = parser.parse_args()
     ulps = Ulps()
     if args.ulps_file is not None:
         ulps.read(args.ulps_file)
     auto_tests = read_auto_tests(args.auto_input)
+    if args.srcdir is not None:
+        all_ulps = read_all_ulps(args.srcdir)
     if args.ulps_output is not None:
         ulps.write(args.ulps_output)
     if args.ulps_header is not None:
         ulps.write_header(args.ulps_header)
     if args.c_output is not None:
         generate_testfile(args.inc_input, auto_tests, args.c_output)
+    if args.err_table is not None:
+        generate_err_table(all_ulps, args.err_table)
 
 
 if __name__ == '__main__':