[1/2] Add note on MALLOC_MMAP_* environment variables
Commit Message
The mallopt parameters manual does not mention the environment
variables that can be used to set these parameters at program startup.
Mention those environment variables for completeness.
* manual/memory.texi: Add environment variable alternatives to
setting mallopt parameters.
---
manual/memory.texi | 15 +++++++++++++++
1 file changed, 15 insertions(+)
Comments
These variables all end with "_", most likely the original intention is
that they are not official and may be removed or changed at any time.
By making them official, we lock in that ABI. Is that your intention?
(likewise your other patch).
On Monday 10 October 2016 11:03 PM, DJ Delorie wrote:
> These variables all end with "_", most likely the original intention is
> that they are not official and may be removed or changed at any time.
>
> By making them official, we lock in that ABI. Is that your intention?
>
> (likewise your other patch).
I don't see the point in not doing so because they're too widely in use
for us to take them back anyway.
Siddhesh
On Mon, Oct 10, 2016 at 7:33 PM, DJ Delorie <dj@redhat.com> wrote:
>
> These variables all end with "_", most likely the original intention is
> that they are not official and may be removed or changed at any time.
>
> By making them official, we lock in that ABI. Is that your intention?
I think the notion that because one does not document something, it's
not an official part of the ABI is at best highly dubious. Especially
when so many "official" parts of the ABI are also not documented.
The only way that documentation might be able to help in such
situations is where pieces are clearly and loudly documented right
from the beginning, in the official documentation, as "not supported,
may disappear at any moment in the future, use at your own risk", but
even then people are likely to ignore the documentation or be unaware
of it.
And in any case, these environment vars have long been documented in
various "unofficial" places including (since 2012)
http://man7.org/linux/man-pages/man3/mallopt.3.html
Cheers,
Michael
Siddhesh Poyarekar <siddhesh@sourceware.org> writes:
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_MMAP_MAX_} to the desired value.
startup.texi has e.g. "@cindex @code{TZ} environment variable".
Could add similar index entries for MALLOC_MMAP_MAX_ and others.
Inconsistently, @vindex is used for the PWD, LANGUAGE, and
COREFILE environment variables.
Ping!
On Monday 10 October 2016 10:56 PM, Siddhesh Poyarekar wrote:
> The mallopt parameters manual does not mention the environment
> variables that can be used to set these parameters at program startup.
> Mention those environment variables for completeness.
>
> * manual/memory.texi: Add environment variable alternatives to
> setting mallopt parameters.
> ---
> manual/memory.texi | 15 +++++++++++++++
> 1 file changed, 15 insertions(+)
>
> diff --git a/manual/memory.texi b/manual/memory.texi
> index 5383105..222f126 100644
> --- a/manual/memory.texi
> +++ b/manual/memory.texi
> @@ -1113,12 +1113,18 @@ choices for @var{param}, as defined in @file{malloc.h}, are:
> @item M_MMAP_MAX
> The maximum number of chunks to allocate with @code{mmap}. Setting this
> to zero disables all use of @code{mmap}.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_MMAP_MAX_} to the desired value.
> @item M_MMAP_THRESHOLD
> All chunks larger than this value are allocated outside the normal
> heap, using the @code{mmap} system call. This way it is guaranteed
> that the memory for these chunks can be returned to the system on
> @code{free}. Note that requests smaller than this threshold might still
> be allocated via @code{mmap}.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value.
> @comment TODO: @item M_MXFAST
> @item M_PERTURB
> If non-zero, memory blocks are filled with values depending on some
> @@ -1128,16 +1134,25 @@ use of uninitialized or freed heap memory. Note that this option does not
> guarantee that the freed block will have any specific values. It only
> guarantees that the content the block had before it was freed will be
> overwritten.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value.
> @item M_TOP_PAD
> This parameter determines the amount of extra memory to obtain from the
> system when a call to @code{sbrk} is required. It also specifies the
> number of bytes to retain when shrinking the heap by calling @code{sbrk}
> with a negative argument. This provides the necessary hysteresis in
> heap size such that excessive amounts of system calls can be avoided.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_TOP_PAD_} to the desired value.
> @item M_TRIM_THRESHOLD
> This is the minimum size (in bytes) of the top-most, releasable chunk
> that will cause @code{sbrk} to be called with a negative argument in
> order to return memory to the system.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
> @end table
>
> @end deftypefun
>
On 10/10/2016 01:26 PM, Siddhesh Poyarekar wrote:
> The mallopt parameters manual does not mention the environment
> variables that can be used to set these parameters at program startup.
> Mention those environment variables for completeness.
Please don't get upset with me :-)
I'm asking you to document the default for these env var tunables.
I think it's valid for users to want to know what value they would have
had if they _didn't_ set the environment variable.
Throwing our hands up in the air and saying "it's implementation defined"
isn't very useful IMO.
> * manual/memory.texi: Add environment variable alternatives to
> setting mallopt parameters.
> ---
> manual/memory.texi | 15 +++++++++++++++
> 1 file changed, 15 insertions(+)
>
> diff --git a/manual/memory.texi b/manual/memory.texi
> index 5383105..222f126 100644
> --- a/manual/memory.texi
> +++ b/manual/memory.texi
> @@ -1113,12 +1113,18 @@ choices for @var{param}, as defined in @file{malloc.h}, are:
> @item M_MMAP_MAX
> The maximum number of chunks to allocate with @code{mmap}. Setting this
> to zero disables all use of @code{mmap}.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_MMAP_MAX_} to the desired value.
What's the default?
> @item M_MMAP_THRESHOLD
> All chunks larger than this value are allocated outside the normal
> heap, using the @code{mmap} system call. This way it is guaranteed
> that the memory for these chunks can be returned to the system on
> @code{free}. Note that requests smaller than this threshold might still
> be allocated via @code{mmap}.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value.
What's the default?
> @comment TODO: @item M_MXFAST
> @item M_PERTURB
> If non-zero, memory blocks are filled with values depending on some
> @@ -1128,16 +1134,25 @@ use of uninitialized or freed heap memory. Note that this option does not
> guarantee that the freed block will have any specific values. It only
> guarantees that the content the block had before it was freed will be
> overwritten.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value.
What is the default?
> @item M_TOP_PAD
> This parameter determines the amount of extra memory to obtain from the
> system when a call to @code{sbrk} is required. It also specifies the
> number of bytes to retain when shrinking the heap by calling @code{sbrk}
> with a negative argument. This provides the necessary hysteresis in
> heap size such that excessive amounts of system calls can be avoided.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_TOP_PAD_} to the desired value.
What's the default?
> @item M_TRIM_THRESHOLD
> This is the minimum size (in bytes) of the top-most, releasable chunk
> that will cause @code{sbrk} to be called with a negative argument in
> order to return memory to the system.
> +
> +This parameter can also be set for the process at startup by setting the
> +environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
What's the default? Note: There is a limit even with dynamic trim threshold on.
This disables the dynamic trim threshold, and that's a very important thing to
mention.
> @end table
>
> @end deftypefun
>
On Monday 17 October 2016 09:43 PM, Carlos O'Donell wrote:
> On 10/10/2016 01:26 PM, Siddhesh Poyarekar wrote:
>> The mallopt parameters manual does not mention the environment
>> variables that can be used to set these parameters at program startup.
>> Mention those environment variables for completeness.
>
> Please don't get upset with me :-)
Grrr...
> I'm asking you to document the default for these env var tunables.
>
> I think it's valid for users to want to know what value they would have
> had if they _didn't_ set the environment variable.
>
> Throwing our hands up in the air and saying "it's implementation defined"
> isn't very useful IMO.
No I agree that is a good point. I'll work on it based on your comments
on the MALLOC_ARENA_* doc patch, i.e. first describe the GNU allocator
and then use that information to explain the envvars/mallopt params and
also add in defaults.
Siddhesh
On 10/11/2016 02:20 AM, Michael Kerrisk wrote:
> On Mon, Oct 10, 2016 at 7:33 PM, DJ Delorie <dj@redhat.com> wrote:
>>
>> These variables all end with "_", most likely the original intention is
>> that they are not official and may be removed or changed at any time.
>>
>> By making them official, we lock in that ABI. Is that your intention?
>
> I think the notion that because one does not document something, it's
> not an official part of the ABI is at best highly dubious. Especially
> when so many "official" parts of the ABI are also not documented.
>
> The only way that documentation might be able to help in such
> situations is where pieces are clearly and loudly documented right
> from the beginning, in the official documentation, as "not supported,
> may disappear at any moment in the future, use at your own risk", but
> even then people are likely to ignore the documentation or be unaware
> of it.
>
> And in any case, these environment vars have long been documented in
> various "unofficial" places including (since 2012)
> http://man7.org/linux/man-pages/man3/mallopt.3.html
The things that DJ is worried about are the arena ones
e.g. M_ARENA_MAX, and M_ARENA_TEST and their env vars.
In 2013 I brought up the discussion about ABI implications:
https://sourceware.org/ml/libc-alpha/2013-03/msg00376.html
Only Siddhesh and I commented, and we both agreed it was a forgone
conclusion that these were stable parts of the ABI/API.
In 2015 I documented them in the linux man pages project:
https://www.sourceware.org/ml/libc-alpha/2015-08/msg00991.html
~~~
Consensus among Siddhesh and myself was that they should be public,
and in fact they were already in the public header. Therefore there
may already be applications uses these constants and expecting them
to work. At best we could limit mallopt's acceptance of the options,
but that seems like a bad solution that could lead to unexpected
behaviour for user applications. A quick google search shows that
there are packages relying on these constants to tune the glibc
malloc implementation.
~~~
They are part of the ABI and public, and should be documented
in the manual as Siddhesh's patches do.
We should however be very very circumspect about adding more
of these mallopt tunables without first seeing that such tunables
are stable and long-term useful as generic tunables (via the tunables
interface). So go review Siddhesh's patches on tunables :-)
@@ -1113,12 +1113,18 @@ choices for @var{param}, as defined in @file{malloc.h}, are:
@item M_MMAP_MAX
The maximum number of chunks to allocate with @code{mmap}. Setting this
to zero disables all use of @code{mmap}.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_MMAP_MAX_} to the desired value.
@item M_MMAP_THRESHOLD
All chunks larger than this value are allocated outside the normal
heap, using the @code{mmap} system call. This way it is guaranteed
that the memory for these chunks can be returned to the system on
@code{free}. Note that requests smaller than this threshold might still
be allocated via @code{mmap}.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_MMAP_THRESHOLD_} to the desired value.
@comment TODO: @item M_MXFAST
@item M_PERTURB
If non-zero, memory blocks are filled with values depending on some
@@ -1128,16 +1134,25 @@ use of uninitialized or freed heap memory. Note that this option does not
guarantee that the freed block will have any specific values. It only
guarantees that the content the block had before it was freed will be
overwritten.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_MMAP_PERTURB_} to the desired value.
@item M_TOP_PAD
This parameter determines the amount of extra memory to obtain from the
system when a call to @code{sbrk} is required. It also specifies the
number of bytes to retain when shrinking the heap by calling @code{sbrk}
with a negative argument. This provides the necessary hysteresis in
heap size such that excessive amounts of system calls can be avoided.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_TOP_PAD_} to the desired value.
@item M_TRIM_THRESHOLD
This is the minimum size (in bytes) of the top-most, releasable chunk
that will cause @code{sbrk} to be called with a negative argument in
order to return memory to the system.
+
+This parameter can also be set for the process at startup by setting the
+environment variable @code{MALLOC_TRIM_THRESHOLD_} to the desired value.
@end table
@end deftypefun