[RFC] Add support for linux memfd_create syscall

  The memfd_create() syscall was released with linux-3.17. It's a linux-only
syscall and returns a shmem file-descriptor backed by anonymous memory
in a kernel-internal shmem mount.
---
 ChangeLog                             |  7 +++++++
 NEWS                                  |  2 ++
 sysdeps/unix/sysv/linux/Makefile      |  2 +-
 sysdeps/unix/sysv/linux/Versions      |  3 +++
 sysdeps/unix/sysv/linux/sys/memfd.h   | 32 ++++++++++++++++++++++++++++++++
 sysdeps/unix/sysv/linux/syscalls.list |  2 ++
 6 files changed, 47 insertions(+), 1 deletion(-)
 create mode 100644 sysdeps/unix/sysv/linux/sys/memfd.h
  

ping?

On Fri, Oct 17, 2014 at 11:21 AM, David Herrmann <dh.herrmann@gmail.com> wrote:
> The memfd_create() syscall was released with linux-3.17. It's a linux-only
> syscall and returns a shmem file-descriptor backed by anonymous memory
> in a kernel-internal shmem mount.
> ---
>  ChangeLog                             |  7 +++++++
>  NEWS                                  |  2 ++
>  sysdeps/unix/sysv/linux/Makefile      |  2 +-
>  sysdeps/unix/sysv/linux/Versions      |  3 +++
>  sysdeps/unix/sysv/linux/sys/memfd.h   | 32 ++++++++++++++++++++++++++++++++
>  sysdeps/unix/sysv/linux/syscalls.list |  2 ++
>  6 files changed, 47 insertions(+), 1 deletion(-)
>  create mode 100644 sysdeps/unix/sysv/linux/sys/memfd.h
>
> diff --git a/ChangeLog b/ChangeLog
> index 6c6d9d8..e11c875 100644
> --- a/ChangeLog
> +++ b/ChangeLog
> @@ -1,3 +1,10 @@
> +2014-10-17  David Herrmann  <dh.herrmann@gmail.com>
> +
> +       * sysdeps/unix/sysv/linux/Makefile: add sys/memfd.h
> +       * sysdeps/unix/sysv/linux/Versions: export memfd_create
> +       * sysdeps/unix/sysv/linux/syscalls.list: add entry for memfd_create
> +       * sysdeps/unix/sysv/linux/sys/memfd.h: new file
> +
>  2014-10-16  Roland McGrath  <roland@hack.frob.com>
>
>         * sysdeps/arm/soft-fp/sfp-machine.h: Filed moved ...
> diff --git a/NEWS b/NEWS
> index 987f306..b6af068 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -7,6 +7,8 @@ using `glibc' in the "product" field.
>
>  Version 2.21
>
> +* New Linux interfaces: memfd_create
> +
>  * The following bugs are resolved with this release:
>
>    6652, 12926, 14171, 17266, 17363, 17370, 17371, 17411, 17460.
> diff --git a/sysdeps/unix/sysv/linux/Makefile b/sysdeps/unix/sysv/linux/Makefile
> index 9ad6d22..850ef9b 100644
> --- a/sysdeps/unix/sysv/linux/Makefile
> +++ b/sysdeps/unix/sysv/linux/Makefile
> @@ -43,7 +43,7 @@ sysdep_headers += sys/mount.h sys/acct.h sys/sysctl.h \
>                   sys/timerfd.h sys/fanotify.h bits/eventfd.h bits/inotify.h \
>                   bits/signalfd.h bits/timerfd.h bits/epoll.h \
>                   bits/socket_type.h bits/syscall.h bits/sysctl.h \
> -                 bits/mman-linux.h
> +                 bits/mman-linux.h sys/memfd.h
>
>  tests += tst-clone tst-fanotify
>
> diff --git a/sysdeps/unix/sysv/linux/Versions b/sysdeps/unix/sysv/linux/Versions
> index 16bb281..15c04b5 100644
> --- a/sysdeps/unix/sysv/linux/Versions
> +++ b/sysdeps/unix/sysv/linux/Versions
> @@ -166,6 +166,9 @@ libc {
>    GLIBC_2.15 {
>      process_vm_readv; process_vm_writev;
>    }
> +  GLIBC_2.21 {
> +    memfd_create;
> +  }
>    GLIBC_PRIVATE {
>      # functions used in other libraries
>      __syscall_rt_sigqueueinfo;
> diff --git a/sysdeps/unix/sysv/linux/sys/memfd.h b/sysdeps/unix/sysv/linux/sys/memfd.h
> new file mode 100644
> index 0000000..3f10df4
> --- /dev/null
> +++ b/sysdeps/unix/sysv/linux/sys/memfd.h
> @@ -0,0 +1,32 @@
> +/* Copyright (C) 2010-2014 Free Software Foundation, Inc.
> +   This file is part of the GNU C Library.
> +
> +   The GNU C Library is free software; you can redistribute it and/or
> +   modify it under the terms of the GNU Lesser General Public
> +   License as published by the Free Software Foundation; either
> +   version 2.1 of the License, or (at your option) any later version.
> +
> +   The GNU C Library is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +   Lesser General Public License for more details.
> +
> +   You should have received a copy of the GNU Lesser General Public
> +   License along with the GNU C Library; if not, see
> +   <http://www.gnu.org/licenses/>.  */
> +
> +#ifndef        _SYS_MEMFD_H
> +#define        _SYS_MEMFD_H    1
> +
> +#include <stdint.h>
> +#include <linux/memfd.h>
> +
> +__BEGIN_DECLS
> +
> +/* Create and initialize an anonymous memory object.  */
> +extern int memfd_create (const char *__name, unsigned int __flags)
> +  __THROW;
> +
> +__END_DECLS
> +
> +#endif /* sys/memfd.h */
> diff --git a/sysdeps/unix/sysv/linux/syscalls.list b/sysdeps/unix/sysv/linux/syscalls.list
> index a0b8de8..f881f37 100644
> --- a/sysdeps/unix/sysv/linux/syscalls.list
> +++ b/sysdeps/unix/sysv/linux/syscalls.list
> @@ -123,3 +123,5 @@ setns               EXTRA   setns           i:ii    setns
>
>  process_vm_readv EXTRA process_vm_readv i:ipipii process_vm_readv
>  process_vm_writev EXTRA        process_vm_writev i:ipipii process_vm_writev
> +
> +memfd_create   EXTRA   memfd_create    i:si    memfd_create
> --
> 2.1.2
>
  

On 21 Nov 2014 16:43, David Herrmann wrote:
> ping?

since glibc doesn't use it and it's questionable whether the new func by itself 
is generally useful, this won't be considered until the general syscall wrapper 
debate is settled.
-mike
  

On Fri, 21 Nov 2014, Mike Frysinger wrote:

> On 21 Nov 2014 16:43, David Herrmann wrote:
> > ping?
> 
> since glibc doesn't use it and it's questionable whether the new func by 
> itself is generally useful, this won't be considered until the general 
> syscall wrapper debate is settled.

Specifically: please join in the discussion starting at 
<https://sourceware.org/ml/libc-alpha/2014-10/msg00541.html>.
  

On 10/17/2014 05:21 AM, David Herrmann wrote:
> The memfd_create() syscall was released with linux-3.17. It's a linux-only
> syscall and returns a shmem file-descriptor backed by anonymous memory
> in a kernel-internal shmem mount.

In general I'm trying hard to make this kind of patch easy to accept.

See the WIP consensus here:
https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers

In order to get your patch accepted you have answer some implicit
questions like "How do users use it?" "Why do users need it?"

From a high-level perspective your patch is missing:

- A test case if possible with non-root privs, or an xcheck test if root
  is required.
  e.g. ./sysdeps/unix/sysv/linux/tst-memfd_create.c

- Documentation for the manual covering the use of the function.
  e.g. ./manual/llio.texi, new section under low-level IO, and specify
  that it is Linux specific. Feel free to submit your own text to the
  linux-kernel man pages to get a new man page created.

Lastly you'll want to review:
https://sourceware.org/glibc/wiki/Contribution%20checklist

The present patch is mechanical in nature and doesn't require copyright
assignment, but adding the test case, and manual entry brings you into
legally significant territory. Do you have a copyright assignment with
the FSF for glibc?

Cheers,
Carlos.
  

Hi

On Fri, Nov 21, 2014 at 7:12 PM, Carlos O'Donell <carlos@redhat.com> wrote:
> On 10/17/2014 05:21 AM, David Herrmann wrote:
>> The memfd_create() syscall was released with linux-3.17. It's a linux-only
>> syscall and returns a shmem file-descriptor backed by anonymous memory
>> in a kernel-internal shmem mount.
>
> In general I'm trying hard to make this kind of patch easy to accept.
>
> See the WIP consensus here:
> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
>
> In order to get your patch accepted you have answer some implicit
> questions like "How do users use it?" "Why do users need it?"

Sure, will add that in v2.

> From a high-level perspective your patch is missing:
>
> - A test case if possible with non-root privs, or an xcheck test if root
>   is required.
>   e.g. ./sysdeps/unix/sysv/linux/tst-memfd_create.c

I already wrote extensive tests which are included in the kernel. I
will try to extract a sensible part and put it into glibc.

> - Documentation for the manual covering the use of the function.
>   e.g. ./manual/llio.texi, new section under low-level IO, and specify
>   that it is Linux specific. Feel free to submit your own text to the
>   linux-kernel man pages to get a new man page created.

My kernel patches already had man-pages included. I'm currently
talking with Michael Kerrisk to get them upstream. I will try to
include something for llio.texi in v2.

> Lastly you'll want to review:
> https://sourceware.org/glibc/wiki/Contribution%20checklist
>
> The present patch is mechanical in nature and doesn't require copyright
> assignment, but adding the test case, and manual entry brings you into
> legally significant territory. Do you have a copyright assignment with
> the FSF for glibc?

I remember someone telling me RedHat had a copyright agreement with
the FSF, but the link explicitly states you need it from the _author_.
I will send v2 as 2 patches, if you consider either a non-mechanical
contribution, we will have to delay it until my copyright agreement
with the FSF is done.

Thanks
David
  

On Fri, 21 Nov 2014, David Herrmann wrote:

> I remember someone telling me RedHat had a copyright agreement with
> the FSF, but the link explicitly states you need it from the _author_.
> I will send v2 as 2 patches, if you consider either a non-mechanical
> contribution, we will have to delay it until my copyright agreement
> with the FSF is done.

If there is a corporate assignment (as opposed to a corporate disclaimer) 
then that suffices - but in that case it helps to post patches from the 
relevant email address that makes it clear which assignment applies.  In 
this case I see:

ANY     Red Hat, Inc.   1999-10-21
Assigns past and future changes to FSF software (until
agreement is terminated with 30 days notice).

which looks sufficient.
  

On 11/21/2014 01:26 PM, David Herrmann wrote:
> Hi
> 
> On Fri, Nov 21, 2014 at 7:12 PM, Carlos O'Donell <carlos@redhat.com> wrote:
>> On 10/17/2014 05:21 AM, David Herrmann wrote:
>>> The memfd_create() syscall was released with linux-3.17. It's a linux-only
>>> syscall and returns a shmem file-descriptor backed by anonymous memory
>>> in a kernel-internal shmem mount.
>>
>> In general I'm trying hard to make this kind of patch easy to accept.
>>
>> See the WIP consensus here:
>> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
>>
>> In order to get your patch accepted you have answer some implicit
>> questions like "How do users use it?" "Why do users need it?"
> 
> Sure, will add that in v2.

Awesome. Thanks.

>> From a high-level perspective your patch is missing:
>>
>> - A test case if possible with non-root privs, or an xcheck test if root
>>   is required.
>>   e.g. ./sysdeps/unix/sysv/linux/tst-memfd_create.c
> 
> I already wrote extensive tests which are included in the kernel. I
> will try to extract a sensible part and put it into glibc.

Thanks again. Part of the purpose of the glibc tests is to make sure
the wrapper is working, and the kernel meets glibc's expectations.
The tests need not be as thorough as the kernel tests, but should
exercise the ways in which a user might call the function. Generally
I would like to see it's common use, common failures, and that's it.

>> - Documentation for the manual covering the use of the function.
>>   e.g. ./manual/llio.texi, new section under low-level IO, and specify
>>   that it is Linux specific. Feel free to submit your own text to the
>>   linux-kernel man pages to get a new man page created.
> 
> My kernel patches already had man-pages included. I'm currently
> talking with Michael Kerrisk to get them upstream. I will try to
> include something for llio.texi in v2.

Thank you. Anything would be better than nothing.
 
>> Lastly you'll want to review:
>> https://sourceware.org/glibc/wiki/Contribution%20checklist
>>
>> The present patch is mechanical in nature and doesn't require copyright
>> assignment, but adding the test case, and manual entry brings you into
>> legally significant territory. Do you have a copyright assignment with
>> the FSF for glibc?
> 
> I remember someone telling me RedHat had a copyright agreement with
> the FSF, but the link explicitly states you need it from the _author_.
> I will send v2 as 2 patches, if you consider either a non-mechanical
> contribution, we will have to delay it until my copyright agreement
> with the FSF is done.

If you work for Red Hat, then Red Hat's copyright assignment covers you.

If you don't work for Red Hat then you need to have a copyright assignment
signed by your employer, and I can help with that. Once that's signed
you can contribute all you want to glibc (following the rules for the
type of assignment you signed).

If you are an individual then you can sign an individual copyright
assignment, if your work can't claim copyright on your work e.g. did
it with your own laptop, on your own time, and distinctly different
from your day job.

If you have any questions here, please don't hesitate to ask.

Cheers,
Carlos.
  

On 11/21/2014 07:41 PM, Carlos O'Donell wrote:
>> My kernel patches already had man-pages included. I'm currently
>> talking with Michael Kerrisk to get them upstream. I will try to
>> include something for llio.texi in v2.
>
> Thank you. Anything would be better than nothing.

Specifically, for sealing support, we should provide a sequence of 
system calls by which senders and receivers can cooperate to make sure 
that the receiver cannot run into the denial-of-service issues usually 
associated with mapping files across trust boundaries (i.e., the sender 
prepares the magic descriptor, and the receiver checks that the 
preparation has been done properly).  If the kernel is not willing to 
make such a commitment, this functionality should go into a separate 
library, and glibc should probably not expose the sealing functionality.

(memfd_create is useful in its own right to create aliasable mappings, 
though.)
  

Hi David and Carlos,

@David: what's the state of the with respect to revising your patches for glibc?

On Fri, Nov 21, 2014 at 7:12 PM, Carlos O'Donell <carlos@redhat.com> wrote:
> On 10/17/2014 05:21 AM, David Herrmann wrote:
>> The memfd_create() syscall was released with linux-3.17. It's a linux-only
>> syscall and returns a shmem file-descriptor backed by anonymous memory
>> in a kernel-internal shmem mount.
>
> In general I'm trying hard to make this kind of patch easy to accept.
>
> See the WIP consensus here:
> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
>
> In order to get your patch accepted you have answer some implicit
> questions like "How do users use it?" "Why do users need it?"
>
> From a high-level perspective your patch is missing:
>
> - A test case if possible with non-root privs, or an xcheck test if root
>   is required.
>   e.g. ./sysdeps/unix/sysv/linux/tst-memfd_create.c
>
> - Documentation for the manual covering the use of the function.
>   e.g. ./manual/llio.texi, new section under low-level IO, and specify
>   that it is Linux specific. Feel free to submit your own text to the
>   linux-kernel man pages to get a new man page created.

@ Carlos: it *sounds* like you're saying that it's glibc policy that
an API must have tests and glibc manual documentation in order to get
admitted into the library. Is that really what you mean?

FWIW, I've recently been doing heavy editing of David's man pages
patches. They're now in a much better shape for release, but there's
still a number of points I want to resolve before releasing them. In
the meantime, they are sitting in a branch at
http://git.kernel.org/cgit/docs/man-pages/man-pages.git/log/?h=draft_memfd_create
and I have just sent out a mail (linux-man@vger.kernel.org, lkml)
requesting some review of the pages.

Thanks,

Michael
  

On 01/09/2015 09:15 AM, Michael Kerrisk wrote:
> Hi David and Carlos,
> 
> @David: what's the state of the with respect to revising your patches for glibc?
> 
> On Fri, Nov 21, 2014 at 7:12 PM, Carlos O'Donell <carlos@redhat.com> wrote:
>> On 10/17/2014 05:21 AM, David Herrmann wrote:
>>> The memfd_create() syscall was released with linux-3.17. It's a linux-only
>>> syscall and returns a shmem file-descriptor backed by anonymous memory
>>> in a kernel-internal shmem mount.
>>
>> In general I'm trying hard to make this kind of patch easy to accept.
>>
>> See the WIP consensus here:
>> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
>>
>> In order to get your patch accepted you have answer some implicit
>> questions like "How do users use it?" "Why do users need it?"
>>
>> From a high-level perspective your patch is missing:
>>
>> - A test case if possible with non-root privs, or an xcheck test if root
>>   is required.
>>   e.g. ./sysdeps/unix/sysv/linux/tst-memfd_create.c
>>
>> - Documentation for the manual covering the use of the function.
>>   e.g. ./manual/llio.texi, new section under low-level IO, and specify
>>   that it is Linux specific. Feel free to submit your own text to the
>>   linux-kernel man pages to get a new man page created.
> 
> @ Carlos: it *sounds* like you're saying that it's glibc policy that
> an API must have tests and glibc manual documentation in order to get
> admitted into the library. Is that really what you mean?

There is no policy, and nothing is set in stone. However, *I* want
a test written for this, and as a submitter you're looking for reviewer
consensus. Would anybody really argue that we don't need tests in this
day and age?

See step 6 in the contribution checklist:

"6. Testing"
"* Adding a test-case to the test suite may increase the likelihood 
   of acceptance of your patch. In some cases it may be necessary. "
https://sourceware.org/glibc/wiki/Contribution%20checklist#Testing

In this case I would really like to see a test case. It should not be
too hard to write and I can help with the glibc esoterica if required.

> FWIW, I've recently been doing heavy editing of David's man pages
> patches. They're now in a much better shape for release, but there's
> still a number of points I want to resolve before releasing them. In
> the meantime, they are sitting in a branch at
> http://git.kernel.org/cgit/docs/man-pages/man-pages.git/log/?h=draft_memfd_create
> and I have just sent out a mail (linux-man@vger.kernel.org, lkml)
> requesting some review of the pages.

I'm not going to look at those in the event that I want to help
draft the GFDL version of the glibc manual for the function.
Though I'd hope David and you contribute your text to glibc for
inclusion in some form in the manual.

My apologies if any of this sounds cranky, I left my morning
coffee downstairs :-)

Cheers,
Carlos.
  

Morning Carlos,

On 9 January 2015 at 15:40, Carlos O'Donell <carlos@redhat.com> wrote:
> On 01/09/2015 09:15 AM, Michael Kerrisk wrote:
>> Hi David and Carlos,
>>
>> @David: what's the state of the with respect to revising your patches for glibc?
>>
>> On Fri, Nov 21, 2014 at 7:12 PM, Carlos O'Donell <carlos@redhat.com> wrote:
>>> On 10/17/2014 05:21 AM, David Herrmann wrote:
>>>> The memfd_create() syscall was released with linux-3.17. It's a linux-only
>>>> syscall and returns a shmem file-descriptor backed by anonymous memory
>>>> in a kernel-internal shmem mount.
>>>
>>> In general I'm trying hard to make this kind of patch easy to accept.
>>>
>>> See the WIP consensus here:
>>> https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers
>>>
>>> In order to get your patch accepted you have answer some implicit
>>> questions like "How do users use it?" "Why do users need it?"
>>>
>>> From a high-level perspective your patch is missing:
>>>
>>> - A test case if possible with non-root privs, or an xcheck test if root
>>>   is required.
>>>   e.g. ./sysdeps/unix/sysv/linux/tst-memfd_create.c
>>>
>>> - Documentation for the manual covering the use of the function.
>>>   e.g. ./manual/llio.texi, new section under low-level IO, and specify
>>>   that it is Linux specific. Feel free to submit your own text to the
>>>   linux-kernel man pages to get a new man page created.
>>
>> @ Carlos: it *sounds* like you're saying that it's glibc policy that
>> an API must have tests and glibc manual documentation in order to get
>> admitted into the library. Is that really what you mean?
>
> There is no policy, and nothing is set in stone. However, *I* want
> a test written for this, and as a submitter you're looking for reviewer
> consensus. Would anybody really argue that we don't need tests in this
> day and age?

I certainly would not.

But, my question was a little tendentious. It seems the bar has been
rather lower for native glibc APIs in the past, with many of them not
getting documented. So one reading of your comment could have been:
we're imposing a higher bar on outsiders (i.e., APIs imported from the
Linux kernel). I don't imagine that's what you really mean, but when I
first read your comments I did a double take along those lines.

> See step 6 in the contribution checklist:
>
> "6. Testing"
> "* Adding a test-case to the test suite may increase the likelihood
>    of acceptance of your patch. In some cases it may be necessary. "
> https://sourceware.org/glibc/wiki/Contribution%20checklist#Testing
>
> In this case I would really like to see a test case. It should not be
> too hard to write and I can help with the glibc esoterica if required.

Sounds good.

>> FWIW, I've recently been doing heavy editing of David's man pages
>> patches. They're now in a much better shape for release, but there's
>> still a number of points I want to resolve before releasing them. In
>> the meantime, they are sitting in a branch at
>> http://git.kernel.org/cgit/docs/man-pages/man-pages.git/log/?h=draft_memfd_create
>> and I have just sent out a mail (linux-man@vger.kernel.org, lkml)
>> requesting some review of the pages.
>
> I'm not going to look at those in the event that I want to help
> draft the GFDL version of the glibc manual for the function.
> Though I'd hope David and you contribute your text to glibc for
> inclusion in some form in the manual.

I could do that, but is it possible to do this without a CLA (e.g.,
placing into public domain, or disclaiming copyright, or some such)?
Thing is, I'm not a fan of CLAs.

> My apologies if any of this sounds cranky, I left my morning
> coffee downstairs :-)

It's okay -- I hope you got your coffee by now ;-).

Cheers,

Michael
  

On 01/09/2015 10:26 AM, Michael Kerrisk (man-pages) wrote:
>>> @ Carlos: it *sounds* like you're saying that it's glibc policy that
>>> an API must have tests and glibc manual documentation in order to get
>>> admitted into the library. Is that really what you mean?
>>
>> There is no policy, and nothing is set in stone. However, *I* want
>> a test written for this, and as a submitter you're looking for reviewer
>> consensus. Would anybody really argue that we don't need tests in this
>> day and age?
> 
> I certainly would not.
> 
> But, my question was a little tendentious. It seems the bar has been
> rather lower for native glibc APIs in the past, with many of them not
> getting documented. So one reading of your comment could have been:
> we're imposing a higher bar on outsiders (i.e., APIs imported from the
> Linux kernel). I don't imagine that's what you really mean, but when I
> first read your comments I did a double take along those lines.

The bar is the same for everyone, it's in the contribution checklist
which must be followed by all developers. I did not intend my comments
to be interpreted as an adoption of double standards for the community.

Those that commit without updating both the manual and man pages are
doing a serious disservice to our users and other developers. If I 
had reviewed those patches I would have also requested they create
documentation.

If you'd like some testimony about the tenacity of my documentation
requests just ask Florian Weimer (Red Hat) about commit
585367266923156ac6fb789939a923641ba5aaf4 :-)

In the end all I can do is lead by example.
 
>> See step 6 in the contribution checklist:
>>
>> "6. Testing"
>> "* Adding a test-case to the test suite may increase the likelihood
>>    of acceptance of your patch. In some cases it may be necessary. "
>> https://sourceware.org/glibc/wiki/Contribution%20checklist#Testing
>>
>> In this case I would really like to see a test case. It should not be
>> too hard to write and I can help with the glibc esoterica if required.
> 
> Sounds good.

And to add to that:

"4.2 Documentation"
https://sourceware.org/glibc/wiki/Contribution%20checklist#Documentation

Which also links to the Linux man-pages project :)

>>> FWIW, I've recently been doing heavy editing of David's man pages
>>> patches. They're now in a much better shape for release, but there's
>>> still a number of points I want to resolve before releasing them. In
>>> the meantime, they are sitting in a branch at
>>> http://git.kernel.org/cgit/docs/man-pages/man-pages.git/log/?h=draft_memfd_create
>>> and I have just sent out a mail (linux-man@vger.kernel.org, lkml)
>>> requesting some review of the pages.
>>
>> I'm not going to look at those in the event that I want to help
>> draft the GFDL version of the glibc manual for the function.
>> Though I'd hope David and you contribute your text to glibc for
>> inclusion in some form in the manual.
> 
> I could do that, but is it possible to do this without a CLA (e.g.,
> placing into public domain, or disclaiming copyright, or some such)?
> Thing is, I'm not a fan of CLAs.

Please ask a lawyer.

You can disclaim your changes.

See:
https://sourceware.org/glibc/wiki/Contribution%20checklist#FSF_copyright_Assignment

The disclaimer request is here:
http://git.savannah.gnu.org/cgit/gnulib.git/plain/doc/Copyright/request-disclaim.changes

>> My apologies if any of this sounds cranky, I left my morning
>> coffee downstairs :-)
> 
> It's okay -- I hope you got your coffee by now ;-).

Coffee firmly in hand. I think I still sound cranky :-)

Cheers,
Carlos.
  

Hi Carlos,

On 9 January 2015 at 17:17, Carlos O'Donell <carlos@redhat.com> wrote:
> On 01/09/2015 10:26 AM, Michael Kerrisk (man-pages) wrote:
>>>> @ Carlos: it *sounds* like you're saying that it's glibc policy that
>>>> an API must have tests and glibc manual documentation in order to get
>>>> admitted into the library. Is that really what you mean?
>>>
>>> There is no policy, and nothing is set in stone. However, *I* want
>>> a test written for this, and as a submitter you're looking for reviewer
>>> consensus. Would anybody really argue that we don't need tests in this
>>> day and age?
>>
>> I certainly would not.
>>
>> But, my question was a little tendentious. It seems the bar has been
>> rather lower for native glibc APIs in the past, with many of them not
>> getting documented. So one reading of your comment could have been:
>> we're imposing a higher bar on outsiders (i.e., APIs imported from the
>> Linux kernel). I don't imagine that's what you really mean, but when I
>> first read your comments I did a double take along those lines.
>
> The bar is the same for everyone, it's in the contribution checklist
> which must be followed by all developers. I did not intend my comments
> to be interpreted as an adoption of double standards for the community.

Fair enough. Probably it's all a matter of the fact that it used to be
that nobody was doing what you do now, and that's why I noticed it.

> Those that commit without updating both the manual and man pages are
> doing a serious disservice to our users and other developers. If I
> had reviewed those patches I would have also requested they create
> documentation.
>
> If you'd like some testimony about the tenacity of my documentation
> requests just ask Florian Weimer (Red Hat) about commit
> 585367266923156ac6fb789939a923641ba5aaf4 :-)

Okay ;-).

> In the end all I can do is lead by example.

Yes, and you set a good example.

>>> See step 6 in the contribution checklist:
>>>
>>> "6. Testing"
>>> "* Adding a test-case to the test suite may increase the likelihood
>>>    of acceptance of your patch. In some cases it may be necessary. "
>>> https://sourceware.org/glibc/wiki/Contribution%20checklist#Testing
>>>
>>> In this case I would really like to see a test case. It should not be
>>> too hard to write and I can help with the glibc esoterica if required.
>>
>> Sounds good.
>
> And to add to that:
>
> "4.2 Documentation"
> https://sourceware.org/glibc/wiki/Contribution%20checklist#Documentation
>
> Which also links to the Linux man-pages project :)


Yes, thanks for that.

>>>> FWIW, I've recently been doing heavy editing of David's man pages
>>>> patches. They're now in a much better shape for release, but there's
>>>> still a number of points I want to resolve before releasing them. In
>>>> the meantime, they are sitting in a branch at
>>>> http://git.kernel.org/cgit/docs/man-pages/man-pages.git/log/?h=draft_memfd_create
>>>> and I have just sent out a mail (linux-man@vger.kernel.org, lkml)
>>>> requesting some review of the pages.
>>>
>>> I'm not going to look at those in the event that I want to help
>>> draft the GFDL version of the glibc manual for the function.
>>> Though I'd hope David and you contribute your text to glibc for
>>> inclusion in some form in the manual.
>>
>> I could do that, but is it possible to do this without a CLA (e.g.,
>> placing into public domain, or disclaiming copyright, or some such)?
>> Thing is, I'm not a fan of CLAs.
>
> Please ask a lawyer.
>
> You can disclaim your changes.
>
> See:
> https://sourceware.org/glibc/wiki/Contribution%20checklist#FSF_copyright_Assignment
>
> The disclaimer request is here:
> http://git.savannah.gnu.org/cgit/gnulib.git/plain/doc/Copyright/request-disclaim.changes

Okay. I'd be okay with doing that. If you want that for the
memfd_create() material, let me know.

But, to take the larger point a little further. For system calls such
as memfd_create(), presumably the most that that glibc is going to do
is supply a wrapper and a header file with some declarations and
constants. Does the glibc project then really want to get into the
business of maintaining documentation for another project's APIs?
Somehow, that seems a little... odd. (Yes, I know glibc is exposing
the APIs, but in the end, it has fairly limited control over them.)

Cheers,

Michael
  

On 01/14/2015 02:36 AM, Michael Kerrisk (man-pages) wrote:
>> You can disclaim your changes.
>>
>> See:
>> https://sourceware.org/glibc/wiki/Contribution%20checklist#FSF_copyright_Assignment
>>
>> The disclaimer request is here:
>> http://git.savannah.gnu.org/cgit/gnulib.git/plain/doc/Copyright/request-disclaim.changes
> 
> Okay. I'd be okay with doing that. If you want that for the
> memfd_create() material, let me know.
> 
> But, to take the larger point a little further. For system calls such
> as memfd_create(), presumably the most that that glibc is going to do
> is supply a wrapper and a header file with some declarations and
> constants. Does the glibc project then really want to get into the
> business of maintaining documentation for another project's APIs?

Yes, and no.

If glibc has wrappers for the functions it should document them.
There should be no hidden APIs that we export publicly that aren't
listed in the manual as being APIs we support.

Exactly what gets documented can be discussed at the time of the
inclusion of the API. For example the documentation might be only
the declaration of the API, safety information, a brief discussion,
and a reference to the permanent URL for the linux man-pages project
reference (we have one of those right? ;-)).

Does that clarify my position?

The POSIX docs, C standard docs, glibc manual, and linux man pages
all serve slightly different purposes.

I agree that duplicating documentation seems like a terrible waste
of resources, but if the audience and style changes we need to
adjust our documentation.

This kind of discussion is new territory, and we need to flesh out
exactly how we're going to collaborate on this kind of thing.

> Somehow, that seems a little... odd. (Yes, I know glibc is exposing
> the APIs, but in the end, it has fairly limited control over them.)

It does have limited control over the implementation, but it has
API and ABI guarantees to uphold, and those are serious issues for
applications and part of the reason we document things i.e. to give
our users a guarantee.

Cheers,
Carlos.
  

On Wed, 14 Jan 2015, Michael Kerrisk (man-pages) wrote:

> But, to take the larger point a little further. For system calls such
> as memfd_create(), presumably the most that that glibc is going to do
> is supply a wrapper and a header file with some declarations and
> constants. Does the glibc project then really want to get into the
> business of maintaining documentation for another project's APIs?

The first question is: does glibc want to get into the business of 
exporting interfaces for another project's APIs (and to what extent)?

What I suggested (and Carlos put at 
<https://sourceware.org/glibc/wiki/Consensus#WIP:_Kernel_syscalls_wrappers>) 
about adding syscall wrappers by default accords with historic practice, 
but there were objections at 
<https://sourceware.org/ml/libc-alpha/2014-10/msg00591.html>.  I don't 
believe we have reached consensus on when it should be considered useful 
to add an API of the form "call this Linux kernel syscall" (where the only 
glibc choices relate to such things as thread cancellation), as opposed to 
an OS-independent API (considered to be generally useful as part of the 
glibc API on all systems, even if any underlying kernel has chosen a 
different interface to similar functionality) that might initially be 
implemented only for systems using the Linux kernel.

(Given such consensus, it would then be very useful for someone to review 
Linux kernel syscalls - at least those that are architecture-independent - 
to identify cases that, according to such consensus, should have 
interfaces in glibc, and then to send patches implementing such 
interfaces.  Some syscalls already have bugs in Bugzilla requesting such 
interfaces: bug 6399 for gettid, bug 9712 for futex, bug 17252 for 
getrandom (a clear case for having an OS-independent interface, possibly 
in addition to a direct syscall interface), bug 17662 for renameat2.  But 
missing interfaces for one syscall does not of course block adding them 
for another, if that other syscall meets the relevant consensus.)

If an interface is exported, I think there must be a testcase - even if it 
does nothing useful on execution (and it may not be able to do anything 
useful on execution on old kernels, at least), it will at least make sure 
that calls to the function compile and link (we've had problems before 
with functions missing on some architectures).  See 
<https://sourceware.org/ml/libc-alpha/2013-07/msg00386.html> for a list of 
untested symbols (maybe a bit out of date, but I expect the vast bulk are 
still untested) - contributions of tests for existing untested symbols are 
welcome.