[2/2] GDB test suite: Get core files on targets with systemd-coredump

  So far the test suite skips tests if they need system-generated core files
and the core files can not be found.  In particular this is usually the
case on systems with an active systemd-coredump service.  On such systems,
core files are not written into the local directory, but made accessible
via a command-line utitily "coredumpctl" instead.

This patch enables processing core files on such systems as well.  Note
that there are a few quirks:

* In my tests, after invoking a command that dumps core, it could happen
  that "coredumpctl" did not find the dump immediately afterwards.  After
  waiting a bit, the dump was found and could be accessed.  Thus the patch
  performs a single wait-and-retry in case of failure.

* There does not seem to be a way for a user to remove specific core dumps
  from the journal.  Thus it can happen that "coredumpctl" returns an old
  dump, and the test case continues with that instead of the new one.  It
  might be possible to improve the logic here, by considering the time
  stamps as well.  I leave that for a future patch.

* On the system I've tested it on, the bigcore.exp test case still failed
  because coredumpctl truncated the core file after 4G for some reason.

gdb/testsuite/ChangeLog:

	* lib/gdb.exp (find_core): Add handling for targets with
	systemd-coredump, using the coredumpctl command.
---
 gdb/testsuite/lib/gdb.exp | 13 +++++++++++++
 1 file changed, 13 insertions(+)
  

On 09/18/2017 07:41 PM, Andreas Arnez wrote:
> So far the test suite skips tests if they need system-generated core files
> and the core files can not be found.  In particular this is usually the
> case on systems with an active systemd-coredump service.  On such systems,
> core files are not written into the local directory, but made accessible
> via a command-line utitily "coredumpctl" instead.
> 
> This patch enables processing core files on such systems as well.  Note
> that there are a few quirks:
> 
> * In my tests, after invoking a command that dumps core, it could happen
>   that "coredumpctl" did not find the dump immediately afterwards.  After
>   waiting a bit, the dump was found and could be accessed.  Thus the patch
>   performs a single wait-and-retry in case of failure.
> 
> * There does not seem to be a way for a user to remove specific core dumps
>   from the journal.  Thus it can happen that "coredumpctl" returns an old
>   dump, and the test case continues with that instead of the new one.  It
>   might be possible to improve the logic here, by considering the time
>   stamps as well.  I leave that for a future patch.
> 
> * On the system I've tested it on, the bigcore.exp test case still failed
>   because coredumpctl truncated the core file after 4G for some reason.

I'm a bit unsure about whether this is the right approach,
expecially given the caveats above.  Also, this seems to mean that
running the testsuite on such a system clutters the system log on and on,
maybe even triggers dispatch of notifications to admins, etc.

I wonder whether there's a way to tell systemd-coredump to
let the core dumps be generated on the file system for the current
shell environment?  Like we try to run "ulimit -c unlimited".

Failing that, it may be better to instead make the testsuite skip
the tests gracefully, and display a big and visible warning
if systemd-coredump is detected as active.

I mean, you already have to tweak other things in the system in
order to be able to run the testsuite correctly.  For example,
you have to tweak /proc/sys/kernel/yama/ptrace_scope to make
attach tests work at all, for example.  systemd-coredump kind of
seems like more of the same.

Dunno, as I said, I'm unsure.

Thanks,
Pedro Alves
  

On Tue, Oct 17 2017, Pedro Alves wrote:

> On 09/18/2017 07:41 PM, Andreas Arnez wrote:
>> So far the test suite skips tests if they need system-generated core files
>> and the core files can not be found.  In particular this is usually the
>> case on systems with an active systemd-coredump service.  On such systems,
>> core files are not written into the local directory, but made accessible
>> via a command-line utitily "coredumpctl" instead.
>> 
>> This patch enables processing core files on such systems as well.  Note
>> that there are a few quirks:
>> 
>> * In my tests, after invoking a command that dumps core, it could happen
>>   that "coredumpctl" did not find the dump immediately afterwards.  After
>>   waiting a bit, the dump was found and could be accessed.  Thus the patch
>>   performs a single wait-and-retry in case of failure.
>> 
>> * There does not seem to be a way for a user to remove specific core dumps
>>   from the journal.  Thus it can happen that "coredumpctl" returns an old
>>   dump, and the test case continues with that instead of the new one.  It
>>   might be possible to improve the logic here, by considering the time
>>   stamps as well.  I leave that for a future patch.
>> 
>> * On the system I've tested it on, the bigcore.exp test case still failed
>>   because coredumpctl truncated the core file after 4G for some reason.
>
> I'm a bit unsure about whether this is the right approach,
> expecially given the caveats above.  Also, this seems to mean that
> running the testsuite on such a system clutters the system log on and on,
> maybe even triggers dispatch of notifications to admins, etc.

The caveats above are really bugs/design flaws of systemd-coredump.  If
nothing else, maybe this discussion helps addressing them.  Offering no
way to prevent system log cluttering could be viewed as another flaw,
see also below.

> I wonder whether there's a way to tell systemd-coredump to
> let the core dumps be generated on the file system for the current
> shell environment?  Like we try to run "ulimit -c unlimited".

I agree that there *should* be such a way -- but I haven't found any.
And such a mechanism should probably allow suppressing the log entries
as well.

> Failing that, it may be better to instead make the testsuite skip
> the tests gracefully, and display a big and visible warning
> if systemd-coredump is detected as active.

This might be the right trade-off if we expect test cases to be executed
only on systems that the user has full control over.  But I consider
this restriction too tight and would prefer a "best effort" approach
instead.  Maybe we should emit a warning *and* try our best to execute
the test?

> I mean, you already have to tweak other things in the system in
> order to be able to run the testsuite correctly.  For example,
> you have to tweak /proc/sys/kernel/yama/ptrace_scope to make
> attach tests work at all, for example.  systemd-coredump kind of
> seems like more of the same.

So should we document a sequence of admin commands that makes a system
debug-ready, or in particular ready for the GDB test suite?

But I'm not so sure about this.  IMHO a default mainstream Linux
installation should be suited for development- and debugging purposes
*without* any tweaking.  Also, if there are good reasons for a security
measure, we shouldn't rely on disabling it globally.

With respect to Yama's ptrace scope, the distributions seem to differ.
For instance, Fedora does not activate it by default
(https://fedoraproject.org/wiki/Security_Features_Matrix), while Ubuntu
does (https://wiki.ubuntu.com/Security/Features).  And I wonder whether
this feature couldn't be adjusted to be more debug-friendly either.

--
Andreas
  

On 10/17/2017 06:36 PM, Andreas Arnez wrote:
> On Tue, Oct 17 2017, Pedro Alves wrote:
> 
>> On 09/18/2017 07:41 PM, Andreas Arnez wrote:
>>> So far the test suite skips tests if they need system-generated core files
>>> and the core files can not be found.  In particular this is usually the
>>> case on systems with an active systemd-coredump service.  On such systems,
>>> core files are not written into the local directory, but made accessible
>>> via a command-line utitily "coredumpctl" instead.
>>>
>>> This patch enables processing core files on such systems as well.  Note
>>> that there are a few quirks:
>>>
>>> * In my tests, after invoking a command that dumps core, it could happen
>>>   that "coredumpctl" did not find the dump immediately afterwards.  After
>>>   waiting a bit, the dump was found and could be accessed.  Thus the patch
>>>   performs a single wait-and-retry in case of failure.
>>>
>>> * There does not seem to be a way for a user to remove specific core dumps
>>>   from the journal.  Thus it can happen that "coredumpctl" returns an old
>>>   dump, and the test case continues with that instead of the new one.  It
>>>   might be possible to improve the logic here, by considering the time
>>>   stamps as well.  I leave that for a future patch.
>>>
>>> * On the system I've tested it on, the bigcore.exp test case still failed
>>>   because coredumpctl truncated the core file after 4G for some reason.
>>
>> I'm a bit unsure about whether this is the right approach,
>> expecially given the caveats above.  Also, this seems to mean that
>> running the testsuite on such a system clutters the system log on and on,
>> maybe even triggers dispatch of notifications to admins, etc.
> 
> The caveats above are really bugs/design flaws of systemd-coredump.  If
> nothing else, maybe this discussion helps addressing them.  Offering no
> way to prevent system log cluttering could be viewed as another flaw,
> see also below.
> 
>> I wonder whether there's a way to tell systemd-coredump to
>> let the core dumps be generated on the file system for the current
>> shell environment?  Like we try to run "ulimit -c unlimited".
> 
> I agree that there *should* be such a way -- but I haven't found any.
> And such a mechanism should probably allow suppressing the log entries
> as well.
> 
>> Failing that, it may be better to instead make the testsuite skip
>> the tests gracefully, and display a big and visible warning
>> if systemd-coredump is detected as active.
> 
> This might be the right trade-off if we expect test cases to be executed
> only on systems that the user has full control over.  But I consider
> this restriction too tight and would prefer a "best effort" approach
> instead.  Maybe we should emit a warning *and* try our best to execute
> the test?

Not sure, really.  It seems like the "best effort" results in
racy tests, e.g., if "coredumpctl" returns an old dump, or
if coredumpctl decides to rate-limit core dump generation (which
according to the docs, it does).  It very much sounds like that
can lead to hard to diagnose problems and send GDB hackers tilting
at windmills.

> 
>> I mean, you already have to tweak other things in the system in
>> order to be able to run the testsuite correctly.  For example,
>> you have to tweak /proc/sys/kernel/yama/ptrace_scope to make
>> attach tests work at all, for example.  systemd-coredump kind of
>> seems like more of the same.
> 
> So should we document a sequence of admin commands that makes a system
> debug-ready, or in particular ready for the GDB test suite?

IMO, yes.  We already have something like that, but it's mixed with
the instructions for setting up builders:

  https://sourceware.org/gdb/wiki/BuildBot#Fedora-specific_instructions

(Note we already suggest disabling ABRT and tweaking 
kernel.core_pattern.)

It'd be great to move that info to some specific page about setting
up an environment for developing and testing GDB.  Also, some of
the command sequences there could move to scripts under gdb/contrib/,
IMHO.

> 
> But I'm not so sure about this.  IMHO a default mainstream Linux
> installation should be suited for development- and debugging purposes
> *without* any tweaking.  Also, if there are good reasons for a security
> measure, we shouldn't rely on disabling it globally.

I think that battle is lost.  Mainstream Linux installations are 
already very much not suited for development OOTB.  You have to install 
a bunch of development  packages that are not installed by default, 
before you can build  anything, including compiler, etc.  If you can
install packages, then you can also disable a few features that
really are not meant for development environments.  What we're missing
is a simple "one-click button" way to adapt an installation / user
environment for development.

> 
> With respect to Yama's ptrace scope, the distributions seem to differ.
> For instance, Fedora does not activate it by default
> (https://fedoraproject.org/wiki/Security_Features_Matrix), while Ubuntu
> does (https://wiki.ubuntu.com/Security/Features).  And I wonder whether
> this feature couldn't be adjusted to be more debug-friendly either.

The whole point of the feature is to prevent debugging, so I don't
see how, off hand.  Note that on Fedora nowadays, if you install GDB, 
then Yama's ptrace scope is relaxed via a rpm package dependency.
systemd-coredump could also be treated in the same way -- disable
it for users that have GDB or other development tools installed.
I don't actually know whether it's possible to disable systemd-coredump for a
single user, though it seems to me like an obviously desirable feature.

Thanks,
Pedro Alves
  

On 10/17/2017 07:08 PM, Pedro Alves wrote:

> Not sure, really.  It seems like the "best effort" results in
> racy tests, e.g., if "coredumpctl" returns an old dump, or
> if coredumpctl decides to rate-limit core dump generation (which
> according to the docs, it does).  It very much sounds like that
> can lead to hard to diagnose problems and send GDB hackers tilting
> at windmills.

I should add that I won't oppose this too strongly _if_ we make
the warning not easily lost in the noise, like e.g., maybe tacked
to the end of all test runs, near the final test results summary?

Thanks,
Pedro Alves
  

On 10/17/2017 07:14 PM, Pedro Alves wrote:
> On 10/17/2017 07:08 PM, Pedro Alves wrote:
> 
>> Not sure, really.  It seems like the "best effort" results in
>> racy tests, e.g., if "coredumpctl" returns an old dump, or
>> if coredumpctl decides to rate-limit core dump generation (which
>> according to the docs, it does).  It very much sounds like that
>> can lead to hard to diagnose problems and send GDB hackers tilting
>> at windmills.
> 
> I should add that I won't oppose this too strongly _if_ we make
> the warning not easily lost in the noise, like e.g., maybe tacked
> to the end of all test runs, near the final test results summary?

Another approach would be to not run the core tests by default,
unless you specify some 

make check \
 RUNTESTFLAGS="YES_I_KNOW_ALL_ABOUT_SYSTEMD_COREDUMP_RUN_THE_TESTS_ANYWAY_MKAY=1"
option.

(and then the testsuite would print a warning suggesting that.)

Thanks,
Pedro Alves
  

On Tue, Oct 17 2017, Pedro Alves wrote:

> On 10/17/2017 06:36 PM, Andreas Arnez wrote:

[...]

>> This might be the right trade-off if we expect test cases to be executed
>> only on systems that the user has full control over.  But I consider
>> this restriction too tight and would prefer a "best effort" approach
>> instead.  Maybe we should emit a warning *and* try our best to execute
>> the test?
>
> Not sure, really.  It seems like the "best effort" results in
> racy tests, e.g., if "coredumpctl" returns an old dump, or
> if coredumpctl decides to rate-limit core dump generation (which
> according to the docs, it does).  It very much sounds like that
> can lead to hard to diagnose problems and send GDB hackers tilting
> at windmills.

That might be.  However, the same problems may affect *any* coredumpctl
user, not just the GDB test suite.  And coredumpctl users are *our*
users, after all.  Maybe we should postpone GDB test suite support for
systemd-coredump until these problems are fixed.  But if all "informed
developers" just give up and disable systemd-coredump, I fear that they
will never be addressed.

>
>> 
>>> I mean, you already have to tweak other things in the system in
>>> order to be able to run the testsuite correctly.  For example,
>>> you have to tweak /proc/sys/kernel/yama/ptrace_scope to make
>>> attach tests work at all, for example.  systemd-coredump kind of
>>> seems like more of the same.
>> 
>> So should we document a sequence of admin commands that makes a system
>> debug-ready, or in particular ready for the GDB test suite?
>
> IMO, yes.  We already have something like that, but it's mixed with
> the instructions for setting up builders:
>
>   https://sourceware.org/gdb/wiki/BuildBot#Fedora-specific_instructions
>
> (Note we already suggest disabling ABRT and tweaking 
> kernel.core_pattern.)
>
> It'd be great to move that info to some specific page about setting
> up an environment for developing and testing GDB.  Also, some of
> the command sequences there could move to scripts under gdb/contrib/,
> IMHO.

Yeah, that would be good.

>
>> 
>> But I'm not so sure about this.  IMHO a default mainstream Linux
>> installation should be suited for development- and debugging purposes
>> *without* any tweaking.  Also, if there are good reasons for a security
>> measure, we shouldn't rely on disabling it globally.
>
> I think that battle is lost.

That surely sounds depressing...  I guess I'm late to the battlefield
then ;-)

> Mainstream Linux installations are already very much not suited for
> development OOTB.  You have to install a bunch of development packages
> that are not installed by default, before you can build anything,
> including compiler, etc.  If you can install packages, then you can
> also disable a few features that really are not meant for development
> environments.  What we're missing is a simple "one-click button" way
> to adapt an installation / user environment for development.

Let me just point out that I see a difference between installing
additional packages and disabling security measures.  Admins might be
easily convinced to do the former, but there will probably be more push
back on the latter.  A "one-click button" would not really help with
that.

And all this sounds as if developers were no longer seen as a target
group of a Fedora distribution, say.  On the other hand -- quote --:
"Fedora Workstation is a polished, easy to use operating system for
laptop and desktop computers, with a complete set of tools for
developers and makers of all kinds."

>
>> 
>> With respect to Yama's ptrace scope, the distributions seem to differ.
>> For instance, Fedora does not activate it by default
>> (https://fedoraproject.org/wiki/Security_Features_Matrix), while Ubuntu
>> does (https://wiki.ubuntu.com/Security/Features).  And I wonder whether
>> this feature couldn't be adjusted to be more debug-friendly either.
>
> The whole point of the feature is to prevent debugging, so I don't
> see how, off hand.

Well, I think the goal is to prevent visibility of sensitive data like
passwords and keys through ptrace -- which is a fair point.  But does
this really require disabling ptrace from "non-ancestor" processes
completely?  It just seems to me that the collateral damage to debug
capabilities was accepted too easily in this design.

[...]

Anyway, regarding GDB test suite support for systemd-coredump, I won't
push too hard.  While I have a slight preference towards "best effort",
I understand your concern with possible surprises.  So I'm fine with
dropping this patch.  Patch #1 in this series might still be useful, so
I'll send an updated version of it.

--
Andreas
  

On 10/18/2017 04:56 PM, Andreas Arnez wrote:
> On Tue, Oct 17 2017, Pedro Alves wrote:
> 
>> On 10/17/2017 06:36 PM, Andreas Arnez wrote:
> 
> [...]
> 
>>> This might be the right trade-off if we expect test cases to be executed
>>> only on systems that the user has full control over.  But I consider
>>> this restriction too tight and would prefer a "best effort" approach
>>> instead.  Maybe we should emit a warning *and* try our best to execute
>>> the test?
>>
>> Not sure, really.  It seems like the "best effort" results in
>> racy tests, e.g., if "coredumpctl" returns an old dump, or
>> if coredumpctl decides to rate-limit core dump generation (which
>> according to the docs, it does).  It very much sounds like that
>> can lead to hard to diagnose problems and send GDB hackers tilting
>> at windmills.
> 
> That might be.  However, the same problems may affect *any* coredumpctl
> user, not just the GDB test suite.  And coredumpctl users are *our*
> users, after all.  

Sure, but we're not interested in testing coredumpctl's ability to copy 
a core dump out of the log before that file is handed over to gdb?
We're testing GDB's ability to load a core dump file that the kernel produced,
doesn't matter to us really how many hoops the file went through before
it was fed to gdb.  I mean, we don't go and run filesystem-level stress-tests
because we copy files around in the filesystem -- we just assume the
filesystem is bug free, and let filesystem folks worry about such testing.

If coredumpctl Just Worked flawlessly for our testsuite's use
case (which is not like a regular user's use case), then I'd be all
for this.  My problem is really that it doesn't really work
all that well, and by not informing GDB developers that there's
(currently...) a better non-racy way to run
make-sure-you-can-load-kernel-core-dumps tests, then we're
not doing our follow GDB developers a real service.

> Maybe we should postpone GDB test suite support for
> systemd-coredump until these problems are fixed.  But if all "informed
> developers" just give up and disable systemd-coredump, I fear that they
> will never be addressed.

Judging from our rate of fixes for our own racy-tests, some of
which are caused by other-components issues (frequently glibc/kernel),
I don't think that exposing us to more sources of races will help with
that either.

>>> But I'm not so sure about this.  IMHO a default mainstream Linux
>>> installation should be suited for development- and debugging purposes
>>> *without* any tweaking.  Also, if there are good reasons for a security
>>> measure, we shouldn't rely on disabling it globally.
>>
>> I think that battle is lost.
> 
> That surely sounds depressing...  I guess I'm late to the battlefield
> then ;-)

My remark may have sounded fatalistic, but in reality nothing is ever
settled forever in software land.  All it really takes is someone
motivated to go and improve things, work with the relevant
communities, etc.  It's just that it's an uphill battle, where
there seems to be a lot more interest/investment in the community to
make things secure by default even if it makes it a little more
inconvenient for the system component developer. Balancing both a bit
better might be possible, but it requires commitment, involvement and
a vigilant eye over the whole stack.  History is showing that this
just won't come "for free" for us, unfortunately.

Let's also not forget that the needs of a GDB developer are not the same
as the needs of a developer that uses GDB to debug its own programs for
a given system.  GDB _is_ part of the system, and regression testing it involves
stressing other system components in ways that a normal developer
normally won't.  For example, running the testsuite could in theory well
end up producing hundreds of core dumps for a variation of tests in a few
seconds.   A regular developer doesn't need that.  Another example, ideally, we'd have
tests that make sure that GDB behaves correctly when Yama rejects an attach, when
it rejects spawning a child process, _in addition_ to testing actual run and
attach succeeding.  So ideally we'd have a way to control the Yama scope at run
time without affecting the whole system.  But that's something that you can't
do unless you're root, and running the testsuite with root permissions is
not something that I'd recommend (though maybe we could have an
optional "testsuite/gdb.root/" directory with such tests).

> 
>> Mainstream Linux installations are already very much not suited for
>> development OOTB.  You have to install a bunch of development packages
>> that are not installed by default, before you can build anything,
>> including compiler, etc.  If you can install packages, then you can
>> also disable a few features that really are not meant for development
>> environments.  What we're missing is a simple "one-click button" way
>> to adapt an installation / user environment for development.
> 
> Let me just point out that I see a difference between installing
> additional packages and disabling security measures.  Admins might be
> easily convinced to do the former, but there will probably be more push
> back on the latter.  A "one-click button" would not really help with
> that.
> 
> And all this sounds as if developers were no longer seen as a target
> group of a Fedora distribution, say.  On the other hand -- quote --:
> "Fedora Workstation is a polished, easy to use operating system for
> laptop and desktop computers, with a complete set of tools for
> developers and makers of all kinds."
> 
>>
>>>
>>> With respect to Yama's ptrace scope, the distributions seem to differ.
>>> For instance, Fedora does not activate it by default
>>> (https://fedoraproject.org/wiki/Security_Features_Matrix), while Ubuntu
>>> does (https://wiki.ubuntu.com/Security/Features).  And I wonder whether
>>> this feature couldn't be adjusted to be more debug-friendly either.
>>
>> The whole point of the feature is to prevent debugging, so I don't
>> see how, off hand.
> 
> Well, I think the goal is to prevent visibility of sensitive data like
> passwords and keys through ptrace -- which is a fair point.  But does
> this really require disabling ptrace from "non-ancestor" processes
> completely?  It just seems to me that the collateral damage to debug
> capabilities was accepted too easily in this design.
> 
> [...]
> 
> Anyway, regarding GDB test suite support for systemd-coredump, I won't
> push too hard.  While I have a slight preference towards "best effort",
> I understand your concern with possible surprises.  So I'm fine with
> dropping this patch.  

(Again, I don't mind this going in if we make it visibly warn/optional.)

> Patch #1 in this series might still be useful, so
> I'll send an updated version of it.

Yes, please.  Thanks!
  

On Thu, Oct 19 2017, Pedro Alves wrote:

[...]

> If coredumpctl Just Worked flawlessly for our testsuite's use
> case (which is not like a regular user's use case), then I'd be all
> for this.  My problem is really that it doesn't really work
> all that well, [...]

FWIW, to help the GDB test suite (and others) in the future, I've opened
a systemd feature request to support switching to "classical" core dump
behavior for certain programs:

  https://github.com/systemd/systemd/issues/7166

Anyone interested, feel free to subscribe.

--
Andreas
  

On 10/23/2017 02:41 PM, Andreas Arnez wrote:
> On Thu, Oct 19 2017, Pedro Alves wrote:
> 
> [...]
> 
>> If coredumpctl Just Worked flawlessly for our testsuite's use
>> case (which is not like a regular user's use case), then I'd be all
>> for this.  My problem is really that it doesn't really work
>> all that well, [...]
> 
> FWIW, to help the GDB test suite (and others) in the future, I've opened
> a systemd feature request to support switching to "classical" core dump
> behavior for certain programs:
> 
>   https://github.com/systemd/systemd/issues/7166
> 
> Anyone interested, feel free to subscribe.

Excellent, thanks much!  I've subscribed.

Pedro Alves