[RFC] Dynamic lock elision support

  Elided locks can have mixed overall performance in practice. That is, there is some non-trivial tuning a user might have to do to see the positive benefits. Additionally, when tuning the adaptive lock constants on PPC, my experimentation seems to correlate tuned values with both the number of hardware threads per core, and the behavior of the application.

My initial thought is elision should be disabled by default, with an environment variable to toggle both support, and potentially override tuning constants.

Paul
  

Hi,

Andi Kleen has sent some patches to add tunables for lock elision with
NPTL algorithms:

http://patchwork.sourceware.org/patch/4358/
http://patchwork.sourceware.org/patch/4355/
http://patchwork.sourceware.org/patch/4356/
http://patchwork.sourceware.org/patch/4359/
http://patchwork.sourceware.org/patch/4357/
http://patchwork.sourceware.org/patch/4361/
http://patchwork.sourceware.org/patch/4360/

But he has not yet ping or followed up with these.  I also would prefer
to add tunable to be as platform agnostic as possible, since we have
3 architectures that support LE (x86, ppc, zarch).  So I would suggest
you work on this options and/or adjust the patch on these.

Also, we may want to avoid a configure flag and enable LE support as
default for powerpc, but disable as default (enabled through env. vars
as Andi Kleen patchset).

On 18-08-2015 13:10, Paul E. Murphy wrote:
> Elided locks can have mixed overall performance in practice. That is, there is some non-trivial tuning a user might have to do to see the positive benefits. Additionally, when tuning the adaptive lock constants on PPC, my experimentation seems to correlate tuned values with both the number of hardware threads per core, and the behavior of the application.
> 
> My initial thought is elision should be disabled by default, with an environment variable to toggle both support, and potentially override tuning constants.
> 
> Paul
>
  

Was any consensus reached regarding adding environment variables for
tuneables? The only alternative I've seen floated is to use HWCAP bits to
point to a separate variant of libpthread. It seemed more undesirable.

I like Andi's approach, and am willing to rework and resubmit Andi's
patchset for the remaining archs if there is agreement with his approach.

I think we would all benefit from a runtime mechanism to transition
into TLE while we work to best utilize this new tool.


On 08/18/2015 11:46 AM, Adhemerval Zanella wrote:
> Hi,
>
> Andi Kleen has sent some patches to add tunables for lock elision with
> NPTL algorithms:
>
> http://patchwork.sourceware.org/patch/4358/
> http://patchwork.sourceware.org/patch/4355/
> http://patchwork.sourceware.org/patch/4356/
> http://patchwork.sourceware.org/patch/4359/
> http://patchwork.sourceware.org/patch/4357/
> http://patchwork.sourceware.org/patch/4361/
> http://patchwork.sourceware.org/patch/4360/
>
> But he has not yet ping or followed up with these.  I also would prefer
> to add tunable to be as platform agnostic as possible, since we have
> 3 architectures that support LE (x86, ppc, zarch).  So I would suggest
> you work on this options and/or adjust the patch on these.
>
> Also, we may want to avoid a configure flag and enable LE support as
> default for powerpc, but disable as default (enabled through env. vars
> as Andi Kleen patchset).
>
> On 18-08-2015 13:10, Paul E. Murphy wrote:
>> Elided locks can have mixed overall performance in practice. That is, there is some non-trivial tuning a user might have to do to see the positive benefits. Additionally, when tuning the adaptive lock constants on PPC, my experimentation seems to correlate tuned values with both the number of hardware threads per core, and the behavior of the application.
>>
>> My initial thought is elision should be disabled by default, with an environment variable to toggle both support, and potentially override tuning constants.
>>
>> Paul
>>
  

Siddhesh is working on some tunables for GLIBC [1] and on last cauldron
he has exposed his ideas, although the framework itself was not really
ready.

I also like the idea of exposing the LE tunables now through environment
variables.  I think we should now aim to define which will be parameters
that are platform neutral and later the arch-maintainer can work on each
parameter it will require for their platform.

[1] https://sourceware.org/glibc/wiki/TuningLibraryRuntimeBehavior

On 18-08-2015 18:58, Paul E. Murphy wrote:
> Was any consensus reached regarding adding environment variables for
> tuneables? The only alternative I've seen floated is to use HWCAP bits to
> point to a separate variant of libpthread. It seemed more undesirable.
> 
> I like Andi's approach, and am willing to rework and resubmit Andi's
> patchset for the remaining archs if there is agreement with his approach.
> 
> I think we would all benefit from a runtime mechanism to transition
> into TLE while we work to best utilize this new tool.
> 
> 
> On 08/18/2015 11:46 AM, Adhemerval Zanella wrote:
>> Hi,
>>
>> Andi Kleen has sent some patches to add tunables for lock elision with
>> NPTL algorithms:
>>
>> http://patchwork.sourceware.org/patch/4358/
>> http://patchwork.sourceware.org/patch/4355/
>> http://patchwork.sourceware.org/patch/4356/
>> http://patchwork.sourceware.org/patch/4359/
>> http://patchwork.sourceware.org/patch/4357/
>> http://patchwork.sourceware.org/patch/4361/
>> http://patchwork.sourceware.org/patch/4360/
>>
>> But he has not yet ping or followed up with these.  I also would prefer
>> to add tunable to be as platform agnostic as possible, since we have
>> 3 architectures that support LE (x86, ppc, zarch).  So I would suggest
>> you work on this options and/or adjust the patch on these.
>>
>> Also, we may want to avoid a configure flag and enable LE support as
>> default for powerpc, but disable as default (enabled through env. vars
>> as Andi Kleen patchset).
>>
>> On 18-08-2015 13:10, Paul E. Murphy wrote:
>>> Elided locks can have mixed overall performance in practice. That is, there is some non-trivial tuning a user might have to do to see the positive benefits. Additionally, when tuning the adaptive lock constants on PPC, my experimentation seems to correlate tuned values with both the number of hardware threads per core, and the behavior of the application.
>>>
>>> My initial thought is elision should be disabled by default, with an environment variable to toggle both support, and potentially override tuning constants.
>>>
>>> Paul
>>>
>
  

Adhemerval, thanks for pointing me to that wiki. I would find it helpful
to resummarize any shortcomings of [1] as a newcomer to the project. In
the spirit of cracking this nut, (to state the obvious) there are two
RFC's here:

1. Introduce a tuning framework
2. Introduce tuning bits for elision

I suspect there is agreement with the need for item 1, but not with how.

As Siddhesh notes, there are already quite a few env variables which alter
the behavior of libc and company. He does a good job of describing
the requirements of such a framework. I see no reason why it can't
be approached incrementally.

Adding to the requirements, would a common syntax for tunable environment
variables be desirable?

I think we should be starting to lay the framework. Converging the use of
the many disparate env variables used throughout glibc is a topic in itself.


[1] http://patchwork.sourceware.org/patch/4358/

On 08/19/2015 09:14 AM, Adhemerval Zanella wrote:
> Siddhesh is working on some tunables for GLIBC [1] and on last cauldron
> he has exposed his ideas, although the framework itself was not really
> ready.
>
> I also like the idea of exposing the LE tunables now through environment
> variables.  I think we should now aim to define which will be parameters
> that are platform neutral and later the arch-maintainer can work on each
> parameter it will require for their platform.
>
> [1] https://sourceware.org/glibc/wiki/TuningLibraryRuntimeBehavior
>
> On 18-08-2015 18:58, Paul E. Murphy wrote:
>> Was any consensus reached regarding adding environment variables for
>> tuneables? The only alternative I've seen floated is to use HWCAP bits to
>> point to a separate variant of libpthread. It seemed more undesirable.
>>
>> I like Andi's approach, and am willing to rework and resubmit Andi's
>> patchset for the remaining archs if there is agreement with his approach.
>>
>> I think we would all benefit from a runtime mechanism to transition
>> into TLE while we work to best utilize this new tool.
>>
>>
>> On 08/18/2015 11:46 AM, Adhemerval Zanella wrote:
>>> Hi,
>>>
>>> Andi Kleen has sent some patches to add tunables for lock elision with
>>> NPTL algorithms:
>>>
>>> http://patchwork.sourceware.org/patch/4358/
>>> http://patchwork.sourceware.org/patch/4355/
>>> http://patchwork.sourceware.org/patch/4356/
>>> http://patchwork.sourceware.org/patch/4359/
>>> http://patchwork.sourceware.org/patch/4357/
>>> http://patchwork.sourceware.org/patch/4361/
>>> http://patchwork.sourceware.org/patch/4360/
>>>
>>> But he has not yet ping or followed up with these.  I also would prefer
>>> to add tunable to be as platform agnostic as possible, since we have
>>> 3 architectures that support LE (x86, ppc, zarch).  So I would suggest
>>> you work on this options and/or adjust the patch on these.
>>>
>>> Also, we may want to avoid a configure flag and enable LE support as
>>> default for powerpc, but disable as default (enabled through env. vars
>>> as Andi Kleen patchset).
>>>
>>> On 18-08-2015 13:10, Paul E. Murphy wrote:
>>>> Elided locks can have mixed overall performance in practice. That is, there is some non-trivial tuning a user might have to do to see the positive benefits. Additionally, when tuning the adaptive lock constants on PPC, my experimentation seems to correlate tuned values with both the number of hardware threads per core, and the behavior of the application.
>>>>
>>>> My initial thought is elision should be disabled by default, with an environment variable to toggle both support, and potentially override tuning constants.
>>>>
>>>> Paul
>>>>
  

On Wed, Aug 19, 2015 at 11:24:39AM -0500, Paul E. Murphy wrote:
> As Siddhesh notes, there are already quite a few env variables which alter
> the behavior of libc and company. He does a good job of describing
> the requirements of such a framework. I see no reason why it can't
> be approached incrementally.

It will in fact be approached incrementally.  I have patches on the
siddhesh/tunables branch that I need to fix up following our
discussion at the GNU Tools Cauldron and post on the list.  I hope to
be able to do it before the end of the week.

> Adding to the requirements, would a common syntax for tunable environment
> variables be desirable?

The namespace requirements are frozen, but the open question currently
is how we read values for those tunable names.  My initial draft made
environment variables out of those names, but the consensus in the
room last week leaned towards an alternate expression, perhaps a
single environment variable with a string value that is parsed to get
all tunable values.  There could be other ideas, I'll think about it
once I get the initial framework in.

Siddhesh
  

On Tue, 2015-08-18 at 16:58 -0500, Paul E. Murphy wrote:
> Was any consensus reached regarding adding environment variables for
> tuneables? The only alternative I've seen floated is to use HWCAP bits to
> point to a separate variant of libpthread. It seemed more undesirable.
> 
> I like Andi's approach, and am willing to rework and resubmit Andi's
> patchset for the remaining archs if there is agreement with his approach.
> 
> I think we would all benefit from a runtime mechanism to transition
> into TLE while we work to best utilize this new tool.
> 
How do we move this forward and who needs to be brought into the
consensus?

> 
> On 08/18/2015 11:46 AM, Adhemerval Zanella wrote:
> > Hi,
> >
> > Andi Kleen has sent some patches to add tunables for lock elision with
> > NPTL algorithms:
> >
> > http://patchwork.sourceware.org/patch/4358/
> > http://patchwork.sourceware.org/patch/4355/
> > http://patchwork.sourceware.org/patch/4356/
> > http://patchwork.sourceware.org/patch/4359/
> > http://patchwork.sourceware.org/patch/4357/
> > http://patchwork.sourceware.org/patch/4361/
> > http://patchwork.sourceware.org/patch/4360/
> >
> > But he has not yet ping or followed up with these.  I also would prefer
> > to add tunable to be as platform agnostic as possible, since we have
> > 3 architectures that support LE (x86, ppc, zarch).  So I would suggest
> > you work on this options and/or adjust the patch on these.
> >
> > Also, we may want to avoid a configure flag and enable LE support as
> > default for powerpc, but disable as default (enabled through env. vars
> > as Andi Kleen patchset).
> >
> > On 18-08-2015 13:10, Paul E. Murphy wrote:
> >> Elided locks can have mixed overall performance in practice. That is, there is some non-trivial tuning a user might have to do to see the positive benefits. Additionally, when tuning the adaptive lock constants on PPC, my experimentation seems to correlate tuned values with both the number of hardware threads per core, and the behavior of the application.
> >>
> >> My initial thought is elision should be disabled by default, with an environment variable to toggle both support, and potentially override tuning constants.
> >>
> >> Paul
> >>
>
  

On Fri, Aug 28, 2015 at 04:27:57PM -0500, Steven Munroe wrote:
> On Tue, 2015-08-18 at 16:58 -0500, Paul E. Murphy wrote:
> > Was any consensus reached regarding adding environment variables for
> > tuneables? The only alternative I've seen floated is to use HWCAP bits to
> > point to a separate variant of libpthread. It seemed more undesirable.
> > 
> > I like Andi's approach, and am willing to rework and resubmit Andi's
> > patchset for the remaining archs if there is agreement with his approach.
> > 
> > I think we would all benefit from a runtime mechanism to transition
> > into TLE while we work to best utilize this new tool.
> > 
> How do we move this forward and who needs to be brought into the
> consensus?

Hi Steven,

I posted the patch a few times, but noone reviewed it. I guess it mainly
needs reviewers.

Yes tuning ability is very important for elision.

When the tuning was originally discussed Roland didn't like it. However
I later discussed it with him privately and I addressed all his
points (by adding a white list) for the environment tuning.
That was implemented in my latest patchkits.

There was unfortunately no consensus on a real run time tuning
capability for the program (other than turning it off by changing
the mutex type). I know some people do it by patching the mutex
structures though.

BTW I also have a hack to patch it through LD_PRELOAD, but yes
an official interface would be better.

-Andi
  

On 08/19/2015 01:41 PM, Siddhesh Poyarekar wrote:
> On Wed, Aug 19, 2015 at 11:24:39AM -0500, Paul E. Murphy wrote:
>> As Siddhesh notes, there are already quite a few env variables which alter
>> the behavior of libc and company. He does a good job of describing
>> the requirements of such a framework. I see no reason why it can't
>> be approached incrementally.
> 
> It will in fact be approached incrementally.  I have patches on the
> siddhesh/tunables branch that I need to fix up following our
> discussion at the GNU Tools Cauldron and post on the list.  I hope to
> be able to do it before the end of the week.
> 
>> Adding to the requirements, would a common syntax for tunable environment
>> variables be desirable?
> 
> The namespace requirements are frozen, but the open question currently
> is how we read values for those tunable names.  My initial draft made
> environment variables out of those names, but the consensus in the
> room last week leaned towards an alternate expression, perhaps a
> single environment variable with a string value that is parsed to get
> all tunable values.  There could be other ideas, I'll think about it
> once I get the initial framework in.

To be clear it was Roland McGrath who supported the position that we have
a single env var for all tunables. I heard no other opinions for this case.

I do not support that position. I posit we should have one tunable per env
var, and that we should follow namespace rules for those env vars e.g.
always start with GLIBC_*. There is precedent for this, the Gnome xdg-app
framework is already exporting XDG_* env vars in a sensible namespace, and
as a framework just above us, they already have to deal with this kind of
pseudo-API.

You can argue that unsetting all tunables is harder. It is.

Instead of:
unset GLIBC_TUNABLES

You have to:
env | while read env; do if [[ $env == GLIBC* ]]; then unset $env; fi; done;

But what's the more probable case? I would argue it's more probably that you
want to set or unset just one tunable. Having them all in one string is therefore
more complicated and requires special shell set and unset functions to assist.

If we start off on the right foot, and document that all GLIBC_* env vars need
to be unset if you want default behaviour, then people will listen, and they
will follow. Better yet we talk to Michael Kerrisk and get it documented in
the man pages early how to set or unset the tunables.

More people have to speak up about which interface they would like to see.

Cheers,
Carlos.
  

On 09/02/2015 12:33 PM, Carlos O'Donell wrote:
> On 08/19/2015 01:41 PM, Siddhesh Poyarekar wrote:
>> On Wed, Aug 19, 2015 at 11:24:39AM -0500, Paul E. Murphy wrote:
>>> As Siddhesh notes, there are already quite a few env variables which alter
>>> the behavior of libc and company. He does a good job of describing
>>> the requirements of such a framework. I see no reason why it can't
>>> be approached incrementally.
>>
>> It will in fact be approached incrementally.  I have patches on the
>> siddhesh/tunables branch that I need to fix up following our
>> discussion at the GNU Tools Cauldron and post on the list.  I hope to
>> be able to do it before the end of the week.
>>
>>> Adding to the requirements, would a common syntax for tunable environment
>>> variables be desirable?
>>
>> The namespace requirements are frozen, but the open question currently
>> is how we read values for those tunable names.  My initial draft made
>> environment variables out of those names, but the consensus in the
>> room last week leaned towards an alternate expression, perhaps a
>> single environment variable with a string value that is parsed to get
>> all tunable values.  There could be other ideas, I'll think about it
>> once I get the initial framework in.
> 
> To be clear it was Roland McGrath who supported the position that we have
> a single env var for all tunables. I heard no other opinions for this case.
> 
> I do not support that position. I posit we should have one tunable per env
> var, and that we should follow namespace rules for those env vars e.g.
> always start with GLIBC_*. There is precedent for this, the Gnome xdg-app
> framework is already exporting XDG_* env vars in a sensible namespace, and
> as a framework just above us, they already have to deal with this kind of
> pseudo-API.

I'd prefer multiple env vars. It may also give extra flexibility when parsing
them.

> You can argue that unsetting all tunables is harder. It is.
> 
> Instead of:
> unset GLIBC_TUNABLES
> 
> You have to:
> env | while read env; do if [[ $env == GLIBC* ]]; then unset $env; fi; done;
> 
> But what's the more probable case? I would argue it's more probably that you
> want to set or unset just one tunable. Having them all in one string is therefore
> more complicated and requires special shell set and unset functions to assist.
> 
> If we start off on the right foot, and document that all GLIBC_* env vars need
> to be unset if you want default behaviour, then people will listen, and they
> will follow. Better yet we talk to Michael Kerrisk and get it documented in
> the man pages early how to set or unset the tunables.
> 
> More people have to speak up about which interface they would like to see.
> 
> Cheers,
> Carlos.
>
  

On Wed, Sep 02, 2015 at 01:33:21PM -0400, Carlos O'Donell wrote:
> To be clear it was Roland McGrath who supported the position that we have
> a single env var for all tunables. I heard no other opinions for this case.
> 
> I do not support that position. I posit we should have one tunable per env
> var, and that we should follow namespace rules for those env vars e.g.
> always start with GLIBC_*. There is precedent for this, the Gnome xdg-app
> framework is already exporting XDG_* env vars in a sensible namespace, and
> as a framework just above us, they already have to deal with this kind of
> pseudo-API.

I am still ambivalent about this.  I don't see there being too many
tunables, so a single string should be fairly easy to parse and write.
That said, this is an advantage for the individual envvars too, since
we won't have too many environment variables to manage.  The single
environment variable has an additional advantage; the tunable list can
be initialized very early and will likely be cheaper than having to
traverse the environment for each environment variable.

Siddhesh
  

On Wed, Sep 2, 2015 at 12:33 PM, Carlos O'Donell <carlos@redhat.com> wrote:

>[...]
> But what's the more probable case? I would argue it's more probably that you
> want to set or unset just one tunable. Having them all in one string is therefore
> more complicated and requires special shell set and unset functions to assist.
>
> If we start off on the right foot, and document that all GLIBC_* env vars need
> to be unset if you want default behaviour, then people will listen, and they
> will follow. Better yet we talk to Michael Kerrisk and get it documented in
> the man pages early how to set or unset the tunables.
>
> More people have to speak up about which interface they would like to see.

I think one can make an objective case for going either way, but it
strikes me as
simpler in the long run to do separate environment variables.  Special syntax
always tends to seem plenty powerful in the beginning, then a couple years later
it's "who thought this made sense??", and then after that, someone comes
up with a use case for which the special syntax does not work well, and then
you're roped into lots of syntax-grafting hackery.

With multiple env vars, the fancy new tunable of 2020 can have its own
distinctive
syntax if necessary, no trying to fit into what seemed clever in 2015.

Stan
  

On 09/08/2015 06:32 PM, Stan Shebs wrote:
> On Wed, Sep 2, 2015 at 12:33 PM, Carlos O'Donell <carlos@redhat.com> wrote:
> 
>> [...]
>> But what's the more probable case? I would argue it's more probably that you
>> want to set or unset just one tunable. Having them all in one string is therefore
>> more complicated and requires special shell set and unset functions to assist.
>>
>> If we start off on the right foot, and document that all GLIBC_* env vars need
>> to be unset if you want default behaviour, then people will listen, and they
>> will follow. Better yet we talk to Michael Kerrisk and get it documented in
>> the man pages early how to set or unset the tunables.
>>
>> More people have to speak up about which interface they would like to see.
> 
> I think one can make an objective case for going either way, but it
> strikes me as
> simpler in the long run to do separate environment variables.  Special syntax
> always tends to seem plenty powerful in the beginning, then a couple years later
> it's "who thought this made sense??", and then after that, someone comes
> up with a use case for which the special syntax does not work well, and then
> you're roped into lots of syntax-grafting hackery.
> 
> With multiple env vars, the fancy new tunable of 2020 can have its own
> distinctive
> syntax if necessary, no trying to fit into what seemed clever in 2015.

Stan, Thanks for that insight. I had no considered it that far ahead,
but yes, we'd need some syntax to split the single env var, and if we
instead had one env var per variable it would be simpler for each to have
a syntax that suited the purpose.

Cheers,
Carlos.
  

----- Original Message -----
> I think one can make an objective case for going either way, but it
> strikes me as
> simpler in the long run to do separate environment variables.  Special syntax
> always tends to seem plenty powerful in the beginning, then a couple years
> later
> it's "who thought this made sense??", and then after that, someone comes
> up with a use case for which the special syntax does not work well, and then
> you're roped into lots of syntax-grafting hackery.
> 
> With multiple env vars, the fancy new tunable of 2020 can have its own
> distinctive
> syntax if necessary, no trying to fit into what seemed clever in 2015.

What kind of a fancy tunable do you foresee?  A struct that is filled in using
a formatted string value?  That should not matter to the framework at all since
the headache of parsing the string value and interpreting the actual value is
passed on to the setter function.

The tunable framework should only be concerned with string-based name-value
pairs.  Anything more complicated and I'd say we're trying to do too much. The
question then is whether those name-value pairs are individual environment
variables and their values or a single environment variable with a space
separated list of name-value pairs.

Siddhesh