Provide Solaris 11 buildbots

  If the proposed Solaris 11 buildbots

	https://sourceware.org/ml/gdb/2018-09/msg00004.html

are considered worthwhile, here's a patch to introduce them.  It's
completely untested of course (I cannot even check syntactic validity),
and there are a couple of caveats:

* While a regular gdb build succeeds, a -g -D_GLIBCXX_DEBUG build as
  used by the buildbot fails as reported in PR build/23676.  This can be
  avoided either by performing a -g -O build or with
  --disable-unit-tests from Sergio's proposed patch.

* The buildslaves are configured to be compile-only at the moment: at
  -j4, a build takes ca. 15 minutes, while make check takes 1h 15 due to
  many timeouts (mostly in gdb.threads).  Until those are resolved, it's
  probably useless to run the tests.

* I couldn't find proper documentation for at least two fields:

** arch in config.json (slaves), seems to be unused AFAICT

** tags in config.json (builders)

Comments?

	Rainer
  

On Wednesday, September 19 2018, Rainer Orth wrote:

> If the proposed Solaris 11 buildbots
>
> 	https://sourceware.org/ml/gdb/2018-09/msg00004.html

Thanks for the offer, and for the patch!

IMO, "the more the merrier".  So yeah, Solaris 11 buildslaves are
absolutely welcome.

> are considered worthwhile, here's a patch to introduce them.  It's
> completely untested of course (I cannot even check syntactic validity),
> and there are a couple of caveats:
>
> * While a regular gdb build succeeds, a -g -D_GLIBCXX_DEBUG build as
>   used by the buildbot fails as reported in PR build/23676.  This can be
>   avoided either by performing a -g -O build or with
>   --disable-unit-tests from Sergio's proposed patch.

Oh, cool, I saw your comment regarding my patch on the bug.  I'm happy
it helps to solve other problems.

> * The buildslaves are configured to be compile-only at the moment: at
>   -j4, a build takes ca. 15 minutes, while make check takes 1h 15 due to
>   many timeouts (mostly in gdb.threads).  Until those are resolved, it's
>   probably useless to run the tests.

Right.  Until I implement a way to enable only a subset of tests from
our testsuite, I agree that it's not a good idea to have builds taking
that long to finish.

> * I couldn't find proper documentation for at least two fields:
>
> ** arch in config.json (slaves), seems to be unused AFAICT
>
> ** tags in config.json (builders)

Sorry about that.

The "arch" field is indeed unused.  The initial plan was to use it to
offer some filtering capabilities in the web interface, but that's been
replaced by "tags".

The "tags" filter is just a collection of tags that are used to
categorize the builders.  If you go to
<https://gdb-build.sergiodj.net/waterfall>, at the top of the page
you'll see a "Tags:" header, with a bunch of tags.  You can use them to
selectively display just the builders associated with a certain tag.
For example, if you want to list the x86_64 builders, you'd go to
<https://gdb-build.sergiodj.net/waterfall?tag=x86_64>.

There are two "special" tags: "MAIL" and "TEST".  When a builder is
marked as "MAIL", the BuildBot master will send e-mail notifications
about it to gdb-testers/gdb-patches.  If it's marked as "TEST", then all
e-mail notifications are supressed.  I just mark a builder as "TEST" if
it proves to unstable (i.e., when GDB fails to compile on it because of
some problem with the builder itself).  I think it makes sense to mark
your builders as "MAIL", since they're relatively stable, from what you
said.

I'll have to contact you in private in order to give the password
necessary for connection to the BuildBot master.  I'll also apply your
patch now and enable the builders, so all it's left for you is to start
them on your side.

Thank you very much again for your contribution!
  

Hi Sergio,

> On Wednesday, September 19 2018, Rainer Orth wrote:
>
>> If the proposed Solaris 11 buildbots
>>
>> 	https://sourceware.org/ml/gdb/2018-09/msg00004.html
>
> Thanks for the offer, and for the patch!
>
> IMO, "the more the merrier".  So yeah, Solaris 11 buildslaves are
> absolutely welcome.

great.  It will save me the trouble of detecting post-factum when some
patch broke the Solaris build ;-)

>> * The buildslaves are configured to be compile-only at the moment: at
>>   -j4, a build takes ca. 15 minutes, while make check takes 1h 15 due to
>>   many timeouts (mostly in gdb.threads).  Until those are resolved, it's
>>   probably useless to run the tests.
>
> Right.  Until I implement a way to enable only a subset of tests from
> our testsuite, I agree that it's not a good idea to have builds taking
> that long to finish.

While I could perhaps increase the degree of parallelism, even full -j16
or -j48 builds on my regular build systems take about 5 minutes to
compile, but about half an hour to test, which is still way too long
compared to e.g. a Linux/x86_64 build/test cycle.

I suspect that those timeouts are due to a handful of root causes; once
those are fixed, it should be possible to enable the tests, too.

>> * I couldn't find proper documentation for at least two fields:
>>
>> ** arch in config.json (slaves), seems to be unused AFAICT
>>
>> ** tags in config.json (builders)
>
> Sorry about that.
>
> The "arch" field is indeed unused.  The initial plan was to use it to
> offer some filtering capabilities in the web interface, but that's been
> replaced by "tags".
>
> The "tags" filter is just a collection of tags that are used to
> categorize the builders.  If you go to
> <https://gdb-build.sergiodj.net/waterfall>, at the top of the page
> you'll see a "Tags:" header, with a bunch of tags.  You can use them to
> selectively display just the builders associated with a certain tag.
> For example, if you want to list the x86_64 builders, you'd go to
> <https://gdb-build.sergiodj.net/waterfall?tag=x86_64>.

Ah, I see.  Probably something to add to the wiki ;-)

> There are two "special" tags: "MAIL" and "TEST".  When a builder is
> marked as "MAIL", the BuildBot master will send e-mail notifications
> about it to gdb-testers/gdb-patches.  If it's marked as "TEST", then all
> e-mail notifications are supressed.  I just mark a builder as "TEST" if
> it proves to unstable (i.e., when GDB fails to compile on it because of
> some problem with the builder itself).  I think it makes sense to mark
> your builders as "MAIL", since they're relatively stable, from what you
> said.

That's the intent: the hosts (global zones in Solaris lingo) are among
my main (or only in case of the sparc box) development machines, so I'll
keep them up and running for that reason alone ;-)

The global zones are running current Solaris 11.5 Beta builds, and thus
will be upgraded/rebooted biweekly.  The kernel zones which host the
build slaves are on Solaris 11.4, with the intent of upgrading to the
latest SRU (support repository upgrade; collection of patches tested
together) once a month.  Apart from those planned downtimes (reboots in
the order of minutes), the systems have been rock solid.

> I'll have to contact you in private in order to give the password
> necessary for connection to the BuildBot master.  I'll also apply your
> patch now and enable the builders, so all it's left for you is to start
> them on your side.

That's has happened now, and both are registered with the master, though
they haven't run any builds yet.  I'll wait a bit until this has
completed successfully, then turn the buildslave start into a service so
they are automatically restarted on failure and rerun at boot.

Thanks for your help.

	Rainer
  

On Thursday, September 20 2018, Rainer Orth wrote:

> Hi Sergio,
>
>> On Wednesday, September 19 2018, Rainer Orth wrote:
>>
>>> If the proposed Solaris 11 buildbots
>>>
>>> 	https://sourceware.org/ml/gdb/2018-09/msg00004.html
>>
>> Thanks for the offer, and for the patch!
>>
>> IMO, "the more the merrier".  So yeah, Solaris 11 buildslaves are
>> absolutely welcome.
>
> great.  It will save me the trouble of detecting post-factum when some
> patch broke the Solaris build ;-)

Yeah, that's currently the main feature being used in our BuildBot: the
breakage emails.

>>> * The buildslaves are configured to be compile-only at the moment: at
>>>   -j4, a build takes ca. 15 minutes, while make check takes 1h 15 due to
>>>   many timeouts (mostly in gdb.threads).  Until those are resolved, it's
>>>   probably useless to run the tests.
>>
>> Right.  Until I implement a way to enable only a subset of tests from
>> our testsuite, I agree that it's not a good idea to have builds taking
>> that long to finish.
>
> While I could perhaps increase the degree of parallelism, even full -j16
> or -j48 builds on my regular build systems take about 5 minutes to
> compile, but about half an hour to test, which is still way too long
> compared to e.g. a Linux/x86_64 build/test cycle.
>
> I suspect that those timeouts are due to a handful of root causes; once
> those are fixed, it should be possible to enable the tests, too.

Right.

>>> * I couldn't find proper documentation for at least two fields:
>>>
>>> ** arch in config.json (slaves), seems to be unused AFAICT
>>>
>>> ** tags in config.json (builders)
>>
>> Sorry about that.
>>
>> The "arch" field is indeed unused.  The initial plan was to use it to
>> offer some filtering capabilities in the web interface, but that's been
>> replaced by "tags".
>>
>> The "tags" filter is just a collection of tags that are used to
>> categorize the builders.  If you go to
>> <https://gdb-build.sergiodj.net/waterfall>, at the top of the page
>> you'll see a "Tags:" header, with a bunch of tags.  You can use them to
>> selectively display just the builders associated with a certain tag.
>> For example, if you want to list the x86_64 builders, you'd go to
>> <https://gdb-build.sergiodj.net/waterfall?tag=x86_64>.
>
> Ah, I see.  Probably something to add to the wiki ;-)

Yeah, absolutely.  I'll do that.

>> There are two "special" tags: "MAIL" and "TEST".  When a builder is
>> marked as "MAIL", the BuildBot master will send e-mail notifications
>> about it to gdb-testers/gdb-patches.  If it's marked as "TEST", then all
>> e-mail notifications are supressed.  I just mark a builder as "TEST" if
>> it proves to unstable (i.e., when GDB fails to compile on it because of
>> some problem with the builder itself).  I think it makes sense to mark
>> your builders as "MAIL", since they're relatively stable, from what you
>> said.
>
> That's the intent: the hosts (global zones in Solaris lingo) are among
> my main (or only in case of the sparc box) development machines, so I'll
> keep them up and running for that reason alone ;-)
>
> The global zones are running current Solaris 11.5 Beta builds, and thus
> will be upgraded/rebooted biweekly.  The kernel zones which host the
> build slaves are on Solaris 11.4, with the intent of upgrading to the
> latest SRU (support repository upgrade; collection of patches tested
> together) once a month.  Apart from those planned downtimes (reboots in
> the order of minutes), the systems have been rock solid.

Cool.  Reboots are fine, as long as (as you said below) the buildslaves
reconnect automatically.

>> I'll have to contact you in private in order to give the password
>> necessary for connection to the BuildBot master.  I'll also apply your
>> patch now and enable the builders, so all it's left for you is to start
>> them on your side.
>
> That's has happened now, and both are registered with the master, though
> they haven't run any builds yet.  I'll wait a bit until this has
> completed successfully, then turn the buildslave start into a service so
> they are automatically restarted on failure and rerun at boot.

Thank you for contributing!
  

On Thursday, September 20 2018, I wrote:

> On Thursday, September 20 2018, Rainer Orth wrote:
>
>> Hi Sergio,
>>
>>> On Wednesday, September 19 2018, Rainer Orth wrote:
>>>
>>>> If the proposed Solaris 11 buildbots
>>>>
>>>> 	https://sourceware.org/ml/gdb/2018-09/msg00004.html
>>>
>>> Thanks for the offer, and for the patch!
>>>
>>> IMO, "the more the merrier".  So yeah, Solaris 11 buildslaves are
>>> absolutely welcome.
>>
>> great.  It will save me the trouble of detecting post-factum when some
>> patch broke the Solaris build ;-)
>
> Yeah, that's currently the main feature being used in our BuildBot: the
> breakage emails.

Hey Rainer,

So, there were a few problems in the master.cfg file which prevented the
builders to actually build GDB.  I fixed them, and they're now
processing the queue.  However, the builds are failing.  For example:

  https://gdb-build.sergiodj.net/builders/Solaris11-sparcv9-m64/builds/3/steps/compile%20gdb/logs/stdio

This seems to be the problem you were talking about, regarding
unittests/string_view-selftests.o.  You mentioned that this breakage
shouldn't happen if we don't use -D_GLIBCXX_DEBUG, right?  Can you
please take a look at this?

For now, I've disabled the email notifications for the Solaris builders.

Cheers,
  

Hi Sergio,

> So, there were a few problems in the master.cfg file which prevented the
> builders to actually build GDB.  I fixed them, and they're now
> processing the queue.  However, the builds are failing.  For example:
>
>   https://gdb-build.sergiodj.net/builders/Solaris11-sparcv9-m64/builds/3/steps/compile%20gdb/logs/stdio
>
> This seems to be the problem you were talking about, regarding
> unittests/string_view-selftests.o.  You mentioned that this breakage
> shouldn't happen if we don't use -D_GLIBCXX_DEBUG, right?  Can you
> please take a look at this?

until your --enable-unit-tests patch is in (when we can keep the default
options and just add --disable-unit-tests to the configure line), we'd
need to build with CFLAGS/CXXFLAGS=-g -O -D_GLIBCXX_DEBUG until PR
build/23676 is resolved.  That's what worked for me locally.

> For now, I've disabled the email notifications for the Solaris builders.

Good, thanks.  There's no point in spamming everyone with bogus errors
until the build works reliably.

	Rainer
  

On Thursday, September 20 2018, Rainer Orth wrote:

> Hi Sergio,
>
>> So, there were a few problems in the master.cfg file which prevented the
>> builders to actually build GDB.  I fixed them, and they're now
>> processing the queue.  However, the builds are failing.  For example:
>>
>>   https://gdb-build.sergiodj.net/builders/Solaris11-sparcv9-m64/builds/3/steps/compile%20gdb/logs/stdio
>>
>> This seems to be the problem you were talking about, regarding
>> unittests/string_view-selftests.o.  You mentioned that this breakage
>> shouldn't happen if we don't use -D_GLIBCXX_DEBUG, right?  Can you
>> please take a look at this?
>
> until your --enable-unit-tests patch is in (when we can keep the default
> options and just add --disable-unit-tests to the configure line), we'd
> need to build with CFLAGS/CXXFLAGS=-g -O -D_GLIBCXX_DEBUG until PR
> build/23676 is resolved.  That's what worked for me locally.

Ah, I thought that if we didn't use "-g -D_GLIBCXX_DEBUG", then the
build would be OK.

>> For now, I've disabled the email notifications for the Solaris builders.
>
> Good, thanks.  There's no point in spamming everyone with bogus errors
> until the build works reliably.

Fair enough.

Thanks,
  

Hi Sergio,

>>> There are two "special" tags: "MAIL" and "TEST".  When a builder is
>>> marked as "MAIL", the BuildBot master will send e-mail notifications
>>> about it to gdb-testers/gdb-patches.  If it's marked as "TEST", then all
>>> e-mail notifications are supressed.  I just mark a builder as "TEST" if
>>> it proves to unstable (i.e., when GDB fails to compile on it because of
>>> some problem with the builder itself).  I think it makes sense to mark
>>> your builders as "MAIL", since they're relatively stable, from what you
>>> said.
>>
>> That's the intent: the hosts (global zones in Solaris lingo) are among
>> my main (or only in case of the sparc box) development machines, so I'll
>> keep them up and running for that reason alone ;-)
>>
>> The global zones are running current Solaris 11.5 Beta builds, and thus
>> will be upgraded/rebooted biweekly.  The kernel zones which host the
>> build slaves are on Solaris 11.4, with the intent of upgrading to the
>> latest SRU (support repository upgrade; collection of patches tested
>> together) once a month.  Apart from those planned downtimes (reboots in
>> the order of minutes), the systems have been rock solid.
>
> Cool.  Reboots are fine, as long as (as you said below) the buildslaves
> reconnect automatically.

just FYI: I've now turned both buildslaves into Solaris SMF services,
and they survived reboots.  So I believe we're set on that front.

One question, though: the reboots (both host and kernel zones, which are
VMs effectively) took longer than the current missing_timeout of 5
minutes.  Would it be possible to double that or make it per-buildslave?

Thanks.

	Rainer
  

On Friday, September 21 2018, Rainer Orth wrote:

> Hi Sergio,
>
>>>> There are two "special" tags: "MAIL" and "TEST".  When a builder is
>>>> marked as "MAIL", the BuildBot master will send e-mail notifications
>>>> about it to gdb-testers/gdb-patches.  If it's marked as "TEST", then all
>>>> e-mail notifications are supressed.  I just mark a builder as "TEST" if
>>>> it proves to unstable (i.e., when GDB fails to compile on it because of
>>>> some problem with the builder itself).  I think it makes sense to mark
>>>> your builders as "MAIL", since they're relatively stable, from what you
>>>> said.
>>>
>>> That's the intent: the hosts (global zones in Solaris lingo) are among
>>> my main (or only in case of the sparc box) development machines, so I'll
>>> keep them up and running for that reason alone ;-)
>>>
>>> The global zones are running current Solaris 11.5 Beta builds, and thus
>>> will be upgraded/rebooted biweekly.  The kernel zones which host the
>>> build slaves are on Solaris 11.4, with the intent of upgrading to the
>>> latest SRU (support repository upgrade; collection of patches tested
>>> together) once a month.  Apart from those planned downtimes (reboots in
>>> the order of minutes), the systems have been rock solid.
>>
>> Cool.  Reboots are fine, as long as (as you said below) the buildslaves
>> reconnect automatically.
>
> just FYI: I've now turned both buildslaves into Solaris SMF services,
> and they survived reboots.  So I believe we're set on that front.

Great!

> One question, though: the reboots (both host and kernel zones, which are
> VMs effectively) took longer than the current missing_timeout of 5
> minutes.  Would it be possible to double that or make it per-buildslave?

Sure, I doubled the time to 10 minutes now.

Thanks,