[RFC,0/3] O_TMPFILE and SHM_ANON for the Hurd

  This is an attempt to add O_TMPFILE (from Linux) and SHM_ANON (from FreeBSD)
to the Hurd.

The Hurd already has the very handly dir_mkfile API that lets you create
anonymous/unnamed files -- ones that can either be used as just tmp files,
given to other processes via fork or IPC, or linked into the filesystem to
become regular, named files.

This little gem is used by glibc internally to implement atomic creation/
replacement of some files (/etc/hostname and the like, symlinks, etc), as well
as for the tmpfile () function (which makes a tmp file under $TMPDIR), but is
not otherwise exposed to user programs through a Unix-level API; you have to
drop down to Mach/Hurd level if you want to use it.

But there is a Unix-level API for this functionality in Linux: the O_TMPFILE
flag to open () & co, where you pass the parent directory path w/o a basename
to open () along with O_TMPFILE, and that makes an anonymous file "in that
directory's filesystem". So I thought it would make sense to expose the Hurd's
dir_mkfile by supporting the same O_TMPFILE flag in our open ().

Then, Linux also has memfd_create (), which makes a temp file without touching
any fs at all. This API is widely used in the Wayland ecosystem as the
recommended way to create shared memory.  But such an approach would not work
as-is on the Hurd: in order for there to be an fd, there has to be a server
somewhere, servicing that fd. We can't just make an fd out of "pure memory" --
it may be an fd to /hurd/tmpfs, but that /hurd/tmpfs needs to exist and be
accessible. So being usable in an empty chroot is not going to happen anyway,
unless we start spawning our own instances of /hurd/tmpfs for each memfd, which
sounds like a terrible idea.

And so in that light, the FreeBSD alternative to memfd_create () -- namely
SHM_ANON -- sounds much more approachable to me, while being, well, a bit less
widely supported in the Wayland ecosystem than memfd, but still quite popular.
We already implement shm by creating files under /dev/shm/ (which should
normally be a tmpfs, but for some reason on my Debian GNU/Hurd box it does not
seem to be?), so it seems only natural to use the just-introduced O_TMPFILE to
create anonymous shm files there.

So that is my motivation for adding O_TMPFILE and SHM_ANON. That being said,
maybe we don't actually want these? tmpfile () already gets you most of the way
there, and the fact that it creates a file under /tmp/ and not /dev/shm/ is not
*that* important.

As for the implementation: basically all of the SHM_ANON implementation, except
the very definition, ended up in the generic / POSIX version of shm_open () and
__shm_get_name (), predicated on defined (SHM_ANON) && defined (O_TMPFILE).
This sounds problematic: while there is indeed nothing Hurd-specific about the
implementation, and any port that supports O_TMPFILE and wants to support
SHM_ANON could use these code paths, what if another port wants to implement
SHM_ANON differently? Should I make a separate copy of shm_open.c in
sysdeps/mach/hurd instead of modifying the generic version?

I've used the following simple program to test shm_open () with SHM_ANON:

#include <sys/mman.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>

int main() {
    int fd = shm_open(SHM_ANON, O_RDWR | O_CREAT, 0600);
    printf("Opened fd %d\n", fd);
    write(fd, "hello", 5);
    lseek(fd, SEEK_SET, 0);
    char buffer[6] = { 0 };
    read(fd, buffer, 5);
    printf("Read %s\n", buffer);
}

and the following to test O_TMPFILE:

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <error.h>
#include <errno.h>

int main() {
    int fd = open("/tmp", O_TMPFILE | O_RDWR, 0700);
    if (fd < 0)
        error(1, errno, "open(O_TMPFILE)");
    write(fd, "Hello", 5);
    int r = fchown(fd, -1, 24);
    if (r < 0)
        error(0, errno, "fchown");
    r = linkat(fd, "", AT_FDCWD, "/tmp/hello.txt", AT_EMPTY_PATH);
    if (r < 0)
        error(1, errno, "linkat");
}

After running the latter, /tmp/hello.txt should appear, owned by $(id -u):24,
(where 24 is the group 'cdrom' on my system, chosen arbitrarily among those
that my user is a member of), containing "hello". Importantly, it does so
atomically, i.e. these properties are immediately visible once it appears.

Sergey Bugaev (3):
  hurd: Consolidate file_name_lookup implementation
  hurd: Implement O_TMPFILE
  hurd: Implement SHM_ANON

 hurd/hurdlookup.c                 | 10 +----
 hurd/lookup-at.c                  | 69 ++++++++++++++++++++++++-------
 posix/shm-directory.c             | 25 +++++++++--
 rt/shm_open.c                     |  5 +++
 sysdeps/mach/hurd/bits/fcntl.h    |  5 +++
 sysdeps/mach/hurd/bits/mman_ext.h | 25 +++++++++++
 6 files changed, 112 insertions(+), 27 deletions(-)
 create mode 100644 sysdeps/mach/hurd/bits/mman_ext.h
  

Hello,

Sergey Bugaev, le lun. 12 déc. 2022 14:46:33 +0300, a ecrit:
> Then, Linux also has memfd_create (), which makes a temp file without touching
> any fs at all. This API is widely used in the Wayland ecosystem as the
> recommended way to create shared memory.  But such an approach would not work
> as-is on the Hurd: in order for there to be an fd, there has to be a server
> somewhere, servicing that fd.

Can't glibc itself serve it?

The drawback is that it'd only keep working while the creator is
running.

> We can't just make an fd out of "pure memory" -- it may be an fd to
> /hurd/tmpfs, but that /hurd/tmpfs needs to exist and be accessible. So
> being usable in an empty chroot is not going to happen anyway, unless
> we start spawning our own instances of /hurd/tmpfs for each memfd,
> which sounds like a terrible idea.

For a really-working memfd, it'd have to be so anyway.

> And so in that light, the FreeBSD alternative to memfd_create () -- namely
> SHM_ANON -- sounds much more approachable to me, while being, well, a bit less
> widely supported in the Wayland ecosystem than memfd, but still quite popular.
> We already implement shm by creating files under /dev/shm/

That has the benefit of factorizing the server part, indeed.

> As for the implementation: basically all of the SHM_ANON implementation, except
> the very definition, ended up in the generic / POSIX version of shm_open () and
> __shm_get_name (), predicated on defined (SHM_ANON) && defined (O_TMPFILE).
> This sounds problematic: while there is indeed nothing Hurd-specific about the
> implementation, and any port that supports O_TMPFILE and wants to support
> SHM_ANON could use these code paths, what if another port wants to implement
> SHM_ANON differently? Should I make a separate copy of shm_open.c in
> sysdeps/mach/hurd instead of modifying the generic version?

I would say that we can afford not foreseeing this, and wait for the
case to happen to see how to refactor things?

Thanks for this! We'll be able to commit things after the 2.37 release.
Samuel