Linux: Add <sys/direntries.h>
Commit Message
This header file provides the types struct direntry and struct
direntries, and the functions direntries_init, direntries_read,
and direntries_next. Using a separate header file (instead of
augmenting <dirent.h>) allows more straightforward type names
because identifier collisions are less of a problem (new code
can work around them).
The d_off member is not exposed via struct direntry because it is
difficult to use correctly (it refers to the *next* entry), and
it is also difficult to emulate with other interfaces.
2019-06-18 Florian Weimer <fweimer@redhat.com>
Linux: Add a directory stream iterator.
* manual/filesys.texi (Low-level Directory Access): Document
struct direntries, struct direntry, direntries_init,
direntries_read, and direntries_next.
* manual/examples/direntries.c: New file.
* include/sys/direntries.h: Likewise.
* sysdeps/unix/sysv/linux/Makefile
[$(subdir) == dirent] (sysdep_headers): Add sys/direntries.h.
[$(subdir) == dirent] (sysdep_routines): Add direntries_init,
direntries_read, direntries_next.
[$(subdir) == dirent] (tests): Add tst-direntries.
* sysdeps/unix/sysv/linux/direntries_init.c: New file.
* sysdeps/unix/sysv/linux/direntries_next.c: Likewise.
* sysdeps/unix/sysv/linux/direntries_read.c: Likewise.
* sysdeps/unix/sysv/linux/sys/direntries.h: Likewise.
* sysdeps/unix/sysv/linux/tst-direntries.c: Likewise.
* sysdeps/unix/sysv/linux/Versions (GLIBC_2.30): Export
direntries_init, direntries_read, direntries_next.
* sysdeps/unix/sysv/linux/aarch64/libc.abilist (GLIBC_2.30): Add
direntries_init, direntries_read, direntries_next.
* sysdeps/unix/sysv/linux/alpha/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/arm/libc.abilist (GLIBC_2.30): Likewise.
* sysdeps/unix/sysv/linux/csky/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/hppa/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/i386/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/ia64/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/m68k/coldfire/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/m68k/m680x0/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/microblaze/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/mips/mips32/fpu/libc.abilist
(GLIBC_2.30): Likewise.
* sysdeps/unix/sysv/linux/mips/mips32/nofpu/libc.abilist
(GLIBC_2.30): Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n32/libc.abilist
(GLIBC_2.30): Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/n64/libc.abilist
(GLIBC_2.30): Likewise.
* sysdeps/unix/sysv/linux/nios2/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/fpu/libc.abilist
(GLIBC_2.30): Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/nofpu/libc.abilist
(GLIBC_2.30): Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/be/libc.abilist
(GLIBC_2.30): Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/le/libc.abilist
(GLIBC_2.30): Likewise.
* sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/sh/libc.abilist (GLIBC_2.30): Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/x86_64/64/libc.abilist (GLIBC_2.30):
Likewise.
* sysdeps/unix/sysv/linux/x86_64/x32/libc.abilist (GLIBC_2.30):
Likewise.
Comments
On Jun 18 2019, Florian Weimer <fweimer@redhat.com> wrote:
> +@deftp {Data Type} {struct direntries}
> +@standards {GNU, sys/direntries.h}
> +
> +This type is used to store an iterator which is used to traverse a
> +buffer filled by the @code{getdents64} function.
Did you mean direntries_read?
Andreas.
* Andreas Schwab:
> On Jun 18 2019, Florian Weimer <fweimer@redhat.com> wrote:
>
>> +@deftp {Data Type} {struct direntries}
>> +@standards {GNU, sys/direntries.h}
>> +
>> +This type is used to store an iterator which is used to traverse a
>> +buffer filled by the @code{getdents64} function.
>
> Did you mean direntries_read?
No, the data originally comes from getdents64. Maybe this can be
presented better. But below I describe direntries_read in terms of
direntries_init and getdents64, so I think this is fine. But
suggestions welcome.
Thanks,
Florian
* Florian Weimer:
> This header file provides the types struct direntry and struct
> direntries, and the functions direntries_init, direntries_read,
> and direntries_next. Using a separate header file (instead of
> augmenting <dirent.h>) allows more straightforward type names
> because identifier collisions are less of a problem (new code
> can work around them).
>
> The d_off member is not exposed via struct direntry because it is
> difficult to use correctly (it refers to the *next* entry), and
> it is also difficult to emulate with other interfaces.
Any comments on this patch? I think it fits the addition of getdents64.
<https://sourceware.org/ml/libc-alpha/2019-06/msg00418.html>
Thanks,
Florian
On 08/07/2019 08:19, Florian Weimer wrote:
> * Florian Weimer:
>
>> This header file provides the types struct direntry and struct
>> direntries, and the functions direntries_init, direntries_read,
>> and direntries_next. Using a separate header file (instead of
>> augmenting <dirent.h>) allows more straightforward type names
>> because identifier collisions are less of a problem (new code
>> can work around them).
>>
>> The d_off member is not exposed via struct direntry because it is
>> difficult to use correctly (it refers to the *next* entry), and
>> it is also difficult to emulate with other interfaces.
>
> Any comments on this patch? I think it fits the addition of getdents64.
>
> <https://sourceware.org/ml/libc-alpha/2019-06/msg00418.html>
I really think these function should be part of the application
that wants to use getdents instead of glibc. Some of functionalities
are already provided by dirent functions and the API adds the extensible
layer which in turn adds even more indirection (some getdents usage
I see in the wild is exactly to avoid the multiple buffer copies).
The only advantage I see is it allows use provide a specific buffer
for getdents usage instead of the pre-defined one using in opendir.
However I would prefer to add a opendir_np which uses a user-provided
buffer instead (similar to the hack I did for posix_spawn closeall).
* Adhemerval Zanella:
> On 08/07/2019 08:19, Florian Weimer wrote:
>> * Florian Weimer:
>>
>>> This header file provides the types struct direntry and struct
>>> direntries, and the functions direntries_init, direntries_read,
>>> and direntries_next. Using a separate header file (instead of
>>> augmenting <dirent.h>) allows more straightforward type names
>>> because identifier collisions are less of a problem (new code
>>> can work around them).
>>>
>>> The d_off member is not exposed via struct direntry because it is
>>> difficult to use correctly (it refers to the *next* entry), and
>>> it is also difficult to emulate with other interfaces.
>>
>> Any comments on this patch? I think it fits the addition of getdents64.
>>
>> <https://sourceware.org/ml/libc-alpha/2019-06/msg00418.html>
>
> I really think these function should be part of the application
> that wants to use getdents instead of glibc. Some of functionalities
> are already provided by dirent functions and the API adds the extensible
> layer which in turn adds even more indirection (some getdents usage
> I see in the wild is exactly to avoid the multiple buffer copies).
How many of these application implementations get the alignment and
type-punning correct? I agree that this code should be easy to write,
but it is not so in ISO C.
I can remove the extension fields. We can probably use symbol
versioning if the kernel adds another dirent field, similarly to what it
did for d_type.
> The only advantage I see is it allows use provide a specific buffer
> for getdents usage instead of the pre-defined one using in opendir.
> However I would prefer to add a opendir_np which uses a user-provided
> buffer instead (similar to the hack I did for posix_spawn closeall).
To support 64-bit directory offsets on 32-bit architectures, readdir and
readdir64 must call malloc after all previous telldir call, to allocate
space for the 64-to-32 mapping. telldir cannot report failure and
therefore cannot call malloc. So extending opendir in this way looks
rather problematic.
Thanks,
Florian
On 08/07/2019 10:49, Florian Weimer wrote:
> * Adhemerval Zanella:
>
>> On 08/07/2019 08:19, Florian Weimer wrote:
>>> * Florian Weimer:
>>>
>>>> This header file provides the types struct direntry and struct
>>>> direntries, and the functions direntries_init, direntries_read,
>>>> and direntries_next. Using a separate header file (instead of
>>>> augmenting <dirent.h>) allows more straightforward type names
>>>> because identifier collisions are less of a problem (new code
>>>> can work around them).
>>>>
>>>> The d_off member is not exposed via struct direntry because it is
>>>> difficult to use correctly (it refers to the *next* entry), and
>>>> it is also difficult to emulate with other interfaces.
>>>
>>> Any comments on this patch? I think it fits the addition of getdents64.
>>>
>>> <https://sourceware.org/ml/libc-alpha/2019-06/msg00418.html>
>>
>> I really think these function should be part of the application
>> that wants to use getdents instead of glibc. Some of functionalities
>> are already provided by dirent functions and the API adds the extensible
>> layer which in turn adds even more indirection (some getdents usage
>> I see in the wild is exactly to avoid the multiple buffer copies).
>
> How many of these application implementations get the alignment and
> type-punning correct? I agree that this code should be easy to write,
> but it is not so in ISO C.
Probably not many, even now you see on debian sparc maillist programs
that raise alignment issue due sparc restrictions. But masking wrong C
usage is not intention of such interfaces.
>
> I can remove the extension fields. We can probably use symbol
> versioning if the kernel adds another dirent field, similarly to what it
> did for d_type.
>
>> The only advantage I see is it allows use provide a specific buffer
>> for getdents usage instead of the pre-defined one using in opendir.
>> However I would prefer to add a opendir_np which uses a user-provided
>> buffer instead (similar to the hack I did for posix_spawn closeall).
>
> To support 64-bit directory offsets on 32-bit architectures, readdir and
> readdir64 must call malloc after all previous telldir call, to allocate
> space for the 64-to-32 mapping. telldir cannot report failure and
> therefore cannot call malloc. So extending opendir in this way looks
> rather problematic.
I think you meant seekdir instead of telldir. Currently opendir allocate
a large buffer which does not trigger this issue, and we can limit the
minimum workable buffer size for such cases. Also for readddir{64} we do
have the filesystem contraints of the minimum and maximum filename size,
so we might have a estimate buffer size for some operations. I don't see
this as problematic when we document the expected usage of the buffer and
its size constraints.
* Adhemerval Zanella:
> Probably not many, even now you see on debian sparc maillist programs
> that raise alignment issue due sparc restrictions. But masking wrong C
> usage is not intention of such interfaces.
I think code that's difficult to implement in a portable fashion may
have a place in glibc, particularly if it is closely related to a glibc
or kernel interface.
>> To support 64-bit directory offsets on 32-bit architectures, readdir and
>> readdir64 must call malloc after all previous telldir call, to allocate
>> space for the 64-to-32 mapping. telldir cannot report failure and
>> therefore cannot call malloc. So extending opendir in this way looks
>> rather problematic.
>
> I think you meant seekdir instead of telldir.
No, I meant telldir. telldir needs space to store a new mapping if it
is called (which we do not know beforehand). We can preallocate a fixed
number of entries (one is quite enough), but if telldir has been called,
and *then* readdir is called, we need to make sure that we have space
for another mapping if telldir is called again.
This is the proper fix for bug 23960 and is what FreeBSD does.
Thanks,
Florian
On Tue, Jun 18, 2019 at 04:24:58PM +0200, Florian Weimer wrote:
> This header file provides the types struct direntry and struct
> direntries, and the functions direntries_init, direntries_read,
> and direntries_next. Using a separate header file (instead of
> augmenting <dirent.h>) allows more straightforward type names
> because identifier collisions are less of a problem (new code
> can work around them).
I don't understand the motivation for these interfaces vs using the
standard ones. It seems like adding them explicitly discourages
portable code and has few if any benefits.
Rich
On 08/07/2019 12:44, Florian Weimer wrote:
> * Adhemerval Zanella:
>
>> Probably not many, even now you see on debian sparc maillist programs
>> that raise alignment issue due sparc restrictions. But masking wrong C
>> usage is not intention of such interfaces.
>
> I think code that's difficult to implement in a portable fashion may
> have a place in glibc, particularly if it is closely related to a glibc
> or kernel interface.
But we do have dirent functions exactly to abstract it. If user wants to
optimize by using more direct function, such as getdents, its usage are
very specific and the auxiliary functions, as the one you intend to provide,
are just guesses of what user might want.
>
>>> To support 64-bit directory offsets on 32-bit architectures, readdir and
>>> readdir64 must call malloc after all previous telldir call, to allocate
>>> space for the 64-to-32 mapping. telldir cannot report failure and
>>> therefore cannot call malloc. So extending opendir in this way looks
>>> rather problematic.
>>
>> I think you meant seekdir instead of telldir.
>
> No, I meant telldir. telldir needs space to store a new mapping if it
> is called (which we do not know beforehand). We can preallocate a fixed
> number of entries (one is quite enough), but if telldir has been called,
> and *then* readdir is called, we need to make sure that we have space
> for another mapping if telldir is called again.
>
> This is the proper fix for bug 23960 and is what FreeBSD does.
Right, I see what you mean now. It was not clear you was referring to a
possible solution of the telldir/seekdir issues where there is an extra map
between the fs d_off and the expected result long type for telldir.
But even the FreeBSD solution is to assume the return value will hold
on 'long' return and allocate the new maps entries on a linked list
external to the buffer used for getdirentries.
* Rich Felker:
> On Tue, Jun 18, 2019 at 04:24:58PM +0200, Florian Weimer wrote:
>> This header file provides the types struct direntry and struct
>> direntries, and the functions direntries_init, direntries_read,
>> and direntries_next. Using a separate header file (instead of
>> augmenting <dirent.h>) allows more straightforward type names
>> because identifier collisions are less of a problem (new code
>> can work around them).
>
> I don't understand the motivation for these interfaces vs using the
> standard ones. It seems like adding them explicitly discourages
> portable code and has few if any benefits.
Using these interfaces together with getdents64 has the following
benefits:
* The combination is async-signal-safe (except on MIPS on older kernels,
which is hopefully fixable).
* There is no buffer bloat due to auto-sizing the DIR * buffer based on
preferred I/O sizes, as advertised by the file system.
* The combination does not suffer from the telldir problem (which
returns long on 32-bit architectures, which is incompatible with
struct dirent64 in the kernel). (Only seeking to the beginning is
supported.)
* Exposing the block read size means that the application knows when it
has to rewind after deleting enumerated files, to avoid skipping
entries or returning entries twice.
Whether these benefits are substantial enough to warrant the addition of
the new interfaces is of course subject to debate.
Thanks,
Florian
* Adhemerval Zanella:
> On 08/07/2019 12:44, Florian Weimer wrote:
>> * Adhemerval Zanella:
>>
>>> Probably not many, even now you see on debian sparc maillist programs
>>> that raise alignment issue due sparc restrictions. But masking wrong C
>>> usage is not intention of such interfaces.
>>
>> I think code that's difficult to implement in a portable fashion may
>> have a place in glibc, particularly if it is closely related to a glibc
>> or kernel interface.
>
> But we do have dirent functions exactly to abstract it. If user wants to
> optimize by using more direct function, such as getdents, its usage are
> very specific and the auxiliary functions, as the one you intend to provide,
> are just guesses of what user might want.
I think the interfaces cover this use case pretty well:
<https://sourceware.org/ml/libc-help/2019-07/msg00007.html>
>>>> To support 64-bit directory offsets on 32-bit architectures, readdir and
>>>> readdir64 must call malloc after all previous telldir call, to allocate
>>>> space for the 64-to-32 mapping. telldir cannot report failure and
>>>> therefore cannot call malloc. So extending opendir in this way looks
>>>> rather problematic.
>>>
>>> I think you meant seekdir instead of telldir.
>>
>> No, I meant telldir. telldir needs space to store a new mapping if it
>> is called (which we do not know beforehand). We can preallocate a fixed
>> number of entries (one is quite enough), but if telldir has been called,
>> and *then* readdir is called, we need to make sure that we have space
>> for another mapping if telldir is called again.
>>
>> This is the proper fix for bug 23960 and is what FreeBSD does.
>
> Right, I see what you mean now. It was not clear you was referring to a
> possible solution of the telldir/seekdir issues where there is an extra map
> between the fs d_off and the expected result long type for telldir.
>
> But even the FreeBSD solution is to assume the return value will hold
> on 'long' return and allocate the new maps entries on a linked list
> external to the buffer used for getdirentries.
Yes, I think there is no way aound memory allocation for this. We can
make it rare enough if we pre-allocate just one place, though. I think
it's still a reasonable solution for a bad fringe case.
But it inteferes with using readdir in an async-signal-safe (or at least
malloc-unsafe) context, which is something people actually do. If we
intend to tell people to move to safer interfaces (useful after vfork or
just even multi-threaded fork), I think we are under an obligation to
provide such interfaces.
Thanks,
Florian
On 15/07/2019 09:10, Florian Weimer wrote:
> * Adhemerval Zanella:
>
>> On 08/07/2019 12:44, Florian Weimer wrote:
>>> * Adhemerval Zanella:
>>>
>>>> Probably not many, even now you see on debian sparc maillist programs
>>>> that raise alignment issue due sparc restrictions. But masking wrong C
>>>> usage is not intention of such interfaces.
>>>
>>> I think code that's difficult to implement in a portable fashion may
>>> have a place in glibc, particularly if it is closely related to a glibc
>>> or kernel interface.
>>
>> But we do have dirent functions exactly to abstract it. If user wants to
>> optimize by using more direct function, such as getdents, its usage are
>> very specific and the auxiliary functions, as the one you intend to provide,
>> are just guesses of what user might want.
>
> I think the interfaces cover this use case pretty well:
>
> <https://sourceware.org/ml/libc-help/2019-07/msg00007.html>
>
>>>>> To support 64-bit directory offsets on 32-bit architectures, readdir and
>>>>> readdir64 must call malloc after all previous telldir call, to allocate
>>>>> space for the 64-to-32 mapping. telldir cannot report failure and
>>>>> therefore cannot call malloc. So extending opendir in this way looks
>>>>> rather problematic.
>>>>
>>>> I think you meant seekdir instead of telldir.
>>>
>>> No, I meant telldir. telldir needs space to store a new mapping if it
>>> is called (which we do not know beforehand). We can preallocate a fixed
>>> number of entries (one is quite enough), but if telldir has been called,
>>> and *then* readdir is called, we need to make sure that we have space
>>> for another mapping if telldir is called again.
>>>
>>> This is the proper fix for bug 23960 and is what FreeBSD does.
>>
>> Right, I see what you mean now. It was not clear you was referring to a
>> possible solution of the telldir/seekdir issues where there is an extra map
>> between the fs d_off and the expected result long type for telldir.
>>
>> But even the FreeBSD solution is to assume the return value will hold
>> on 'long' return and allocate the new maps entries on a linked list
>> external to the buffer used for getdirentries.
>
> Yes, I think there is no way aound memory allocation for this. We can
> make it rare enough if we pre-allocate just one place, though. I think
> it's still a reasonable solution for a bad fringe case.
I think pre-allocating the d_off table entry is a 'pessimization' which
falls in the idea of pay for not what want. Due current telldir/seekdir
breakage for some filesystem, I would say it is not extensively used or usage
is limited that the issue does not hit very often. Also, since d_off for some
file system is a hashed 'cookie', even 64-bit call does requires this
(although I am not sure how is frequency the hash algorithm used set the
higher bit).
>
> But it inteferes with using readdir in an async-signal-safe (or at least
> malloc-unsafe) context, which is something people actually do. If we
> intend to tell people to move to safer interfaces (useful after vfork or
> just even multi-threaded fork), I think we are under an obligation to
> provide such interfaces.
To summarize, opendir async-signal-safe issue are:
1. malloc call on opendir to allocate the initial buffer;
2. (mips64 only for kernel prior 3.10) __getdents calls malloc prior
getdents syscall (due initial opendir buffers malloc will always
be called);
3. readddir uses lock associate to DIR to provide thread-safey;
4. telldir/seekdir might require associated data-structure allocation
to handle the return value limitation (not implemented).
For 1. we can either provide an alternative to allow an user-provided
buffer or use mmap directly. The user provided buffer also allows help
the buffer bloat due the preferred large block sizes from file system.
For 2. we can get back to a VLA and avoid malloc altogether, the default
buffer used will trigger at most 32kb allocation and usage is pretty
limited in a specific scenario.
For 3. we might provide an 'unlocked' version same as we do for stdio.
I am not sure how often opendir is used in a multithread environment
with concurrent accesses.
For 4. it is not an issue *now* for glibc, since it does not really
handle the d_off value correctly. Once we fixed it, most likely by
using the similar strategy FreeBSD does, it will need to both a lock
and a malloc call. I am not sure if we would be able to provide
async-signal-safe version with current constraint of using 'long' as
indication of position.
I give you that opendir interfaces has some issues and it is not really
meant to be used on async scenarios or even in the limbo state after
vfork/fork in multithread scenarios. However, the main issue I see it why
developers are relying on such interfaces in theses scenarios and which are
the issues they are trying to solve.
Both from previous iterations about this topic and the recent libc-help,
it seems what we should provide instead is a safe way to spawn an process
while closing any lingering open file descriptor that caller can't guarantee
that has been opened with CLOCEXEC.
So I prefer if we integrate your 'sys/direntries.h' on my proposed
posix_spawn closefrom code instead of provide another GNU extension that
mimics an already POSIX interface to solve a very specific issue.
* Adhemerval Zanella:
>> Yes, I think there is no way aound memory allocation for this. We can
>> make it rare enough if we pre-allocate just one place, though. I think
>> it's still a reasonable solution for a bad fringe case.
>
> I think pre-allocating the d_off table entry is a 'pessimization' which
> falls in the idea of pay for not what want. Due current telldir/seekdir
> breakage for some filesystem, I would say it is not extensively used or usage
> is limited that the issue does not hit very often. Also, since d_off for some
> file system is a hashed 'cookie', even 64-bit call does requires this
> (although I am not sure how is frequency the hash algorithm used set the
> higher bit).
We only need to preallocate the space for one mapping entry (number to
64-bit offset), so that telldir doesn't have to allocate anything and
therefore cannot fail. If telldir is not called, we can overwrite the
entry at the next readdir call, without to allocate anything. Only if
telldir is actually called, the next readdir call needs to allocate
something.
>> But it inteferes with using readdir in an async-signal-safe (or at least
>> malloc-unsafe) context, which is something people actually do. If we
>> intend to tell people to move to safer interfaces (useful after vfork or
>> just even multi-threaded fork), I think we are under an obligation to
>> provide such interfaces.
>
> To summarize, opendir async-signal-safe issue are:
>
> 1. malloc call on opendir to allocate the initial buffer;
And the DIR struct itself, which is not a problem inside glibc, but is
externally, it would add some ABI which does not exist today.
> 2. (mips64 only for kernel prior 3.10) __getdents calls malloc prior
> getdents syscall (due initial opendir buffers malloc will always
> be called);
>
> 3. readddir uses lock associate to DIR to provide thread-safey;
>
> 4. telldir/seekdir might require associated data-structure allocation
> to handle the return value limitation (not implemented).
>
> For 1. we can either provide an alternative to allow an user-provided
> buffer or use mmap directly. The user provided buffer also allows help
> the buffer bloat due the preferred large block sizes from file system.
>
> For 2. we can get back to a VLA and avoid malloc altogether, the default
> buffer used will trigger at most 32kb allocation and usage is pretty
> limited in a specific scenario.
That would still intefere with the posix_spawn code, I think.
> For 3. we might provide an 'unlocked' version same as we do for stdio.
> I am not sure how often opendir is used in a multithread environment
> with concurrent accesses.
I don't think this lock is a problem because it's specific to one DIR *.
It would only affect the async-signal-safe context if the DIR * was
allocated on the outside and reused from the async-signal-safe context.
Thanks,
Florian
* Adhemerval Zanella:
> 2. (mips64 only for kernel prior 3.10) __getdents calls malloc prior
> getdents syscall (due initial opendir buffers malloc will always
> be called);
> For 2. we can get back to a VLA and avoid malloc altogether, the default
> buffer used will trigger at most 32kb allocation and usage is pretty
> limited in a specific scenario.
I don't know the MIPS ABI. If it doesn't require alignment for the
char d_name[256] member, surely we can do the conversion inline, just
by calling memmove on the name? d_ino, d_off, d_reclen already have the
right size.
The present code already assumes that the d_type member is always
present at the end of the record, so there is always enough space, I
think.
Thanks,
Florian
On 15/07/2019 10:34, Florian Weimer wrote:
> * Adhemerval Zanella:
>
>>> Yes, I think there is no way aound memory allocation for this. We can
>>> make it rare enough if we pre-allocate just one place, though. I think
>>> it's still a reasonable solution for a bad fringe case.
>>
>> I think pre-allocating the d_off table entry is a 'pessimization' which
>> falls in the idea of pay for not what want. Due current telldir/seekdir
>> breakage for some filesystem, I would say it is not extensively used or usage
>> is limited that the issue does not hit very often. Also, since d_off for some
>> file system is a hashed 'cookie', even 64-bit call does requires this
>> (although I am not sure how is frequency the hash algorithm used set the
>> higher bit).
>
> We only need to preallocate the space for one mapping entry (number to
> 64-bit offset), so that telldir doesn't have to allocate anything and
> therefore cannot fail. If telldir is not called, we can overwrite the
> entry at the next readdir call, without to allocate anything. Only if
> telldir is actually called, the next readdir call needs to allocate
> something.
Although POSIX states that, for 'loc' used with seekdir that were not
obtained from telldir subsequent, calls to readdir() are unspecified, I
think we should also avoid calling lseek for invalid positions. This is
what FreeBSD does by allocating a mapping for each call to telldir and
thus checking on seekdir if the value is valid (it only need to check for
values that overflow 'long' though).
>
>>> But it inteferes with using readdir in an async-signal-safe (or at least
>>> malloc-unsafe) context, which is something people actually do. If we
>>> intend to tell people to move to safer interfaces (useful after vfork or
>>> just even multi-threaded fork), I think we are under an obligation to
>>> provide such interfaces.
>>
>> To summarize, opendir async-signal-safe issue are:
>>
>> 1. malloc call on opendir to allocate the initial buffer;
>
> And the DIR struct itself, which is not a problem inside glibc, but is
> externally, it would add some ABI which does not exist today.
That the malloc was referring in fact, at __alloc_dir.
>
>> 2. (mips64 only for kernel prior 3.10) __getdents calls malloc prior
>> getdents syscall (due initial opendir buffers malloc will always
>> be called);
>>
>> 3. readddir uses lock associate to DIR to provide thread-safey;
>>
>> 4. telldir/seekdir might require associated data-structure allocation
>> to handle the return value limitation (not implemented).
>>
>> For 1. we can either provide an alternative to allow an user-provided
>> buffer or use mmap directly. The user provided buffer also allows help
>> the buffer bloat due the preferred large block sizes from file system.
>>
>> For 2. we can get back to a VLA and avoid malloc altogether, the default
>> buffer used will trigger at most 32kb allocation and usage is pretty
>> limited in a specific scenario.
>
> That would still intefere with the posix_spawn code, I think.
We will need to provision a large stack size for mips64 (most likely by
defining it on arch-specific internal header).
>
>> For 3. we might provide an 'unlocked' version same as we do for stdio.
>> I am not sure how often opendir is used in a multithread environment
>> with concurrent accesses.
>
> I don't think this lock is a problem because it's specific to one DIR *.
> It would only affect the async-signal-safe context if the DIR * was
> allocated on the outside and reused from the async-signal-safe context.
Indeed it should not prevent if opendir is called within the context of
the signal handler.
@@ -20,7 +20,8 @@ Major new features:
twalk function, but it passes an additional caller-supplied argument
to the callback function.
-* On Linux, the getdents64, gettid, and tgkill functions have been added.
+* On Linux, the getdents64, direntries_init, direntries_read,
+ direntries_next, gettid, and tgkill functions have been added.
* Minguo (Republic of China) calendar support has been added as an
alternative calendar for the following locales: zh_TW, cmn_TW, hak_TW,
new file mode 100644
@@ -0,0 +1,38 @@
+/* Wrapper for <sys/direntries.h>.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+/* The direntries parser is available on Linux only for now. */
+
+#ifndef _LIBC_DIRENTRIES_H
+#define _LIBC_DIRENTRIES_H
+
+#include_next <sys/direntries.h>
+
+#ifndef _ISOMAC
+extern __typeof__ (direntries_init) __direntries_init;
+libc_hidden_proto (__direntries_init)
+extern __typeof__ (direntries_read) __direntries_read;
+libc_hidden_proto (__direntries_read)
+extern __typeof__ (direntries_next) __direntries_next;
+libc_hidden_proto (__direntries_next)
+
+/* Members of struct direntries. */
+# define direntries_buffer_begin __glibc_internal_1
+# define direntries_buffer_end __glibc_internal_2
+#endif /* !_ISOMAC */
+#endif /* _LIBC_DIRENTRIES_H */
new file mode 100644
@@ -0,0 +1,42 @@
+/* Implement directory iteration using <sys/direntries.h>.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, see <http://www.gnu.org/licenses/>.
+*/
+
+#include <stdbool.h>
+#include <sys/direntries.h>
+
+int
+foreach_directory_entry (int fd,
+ int (*callback) (const struct direntry *, void *),
+ void *closure)
+{
+ while (true)
+ {
+ char buffer[4096];
+ struct direntries entries;
+ ssize_t ret = direntries_read (fd, &entries, buffer, sizeof (buffer));
+ if (ret <= 0)
+ return ret;
+
+ struct direntry entry;
+ while (direntries_next (&entries, &entry) == 0)
+ {
+ int callback_result = callback (&entry, closure);
+ if (callback_result != 0)
+ return callback_result;
+ }
+ }
+}
@@ -864,6 +864,88 @@ characters), so a buffer size of at least 1024 is recommended.
This function is specific to Linux.
@end deftypefun
+@deftp {Data Type} {struct direntries}
+@standards {GNU, sys/direntries.h}
+
+This type is used to store an iterator which is used to traverse a
+buffer filled by the @code{getdents64} function.
+@end deftp
+
+@deftp {Data Type} {struct direntry}
+@standards {GNU, sys/direntries.h}
+
+This type provides access to one directory entry. It is similar to
+@code{struct dirent} defined in @file{dirent.h}. The
+@code{direntries_next} function described below uses this type to
+provide information to its caller.
+
+The following members are available to applications.
+
+@table @code
+@item d_name
+The name of the directory entry. Note that unlike
+@code{struct dirent}, this is a pointer to a null-terminated string.
+
+@item d_ino
+The inode number of the directory entry.
+
+@item d_type
+The file type. This is one of the @code{DT_} constants described for
+@code{struct direntry}. It is @code{DT_UNKNOWN} if the information is
+not available. @xref{Directory Entries}.
+
+@item d_flags
+Flags indicating extensions. Currently always zero.
+@end table
+@end deftp
+
+@deftypefun void direntries_init (struct direntries *@var{iterator}, void *@var{buffer}, size_t @var{length})
+@standards{Linux, sys/direntries.h}
+@safety{@prelim{}@mtsafe{}@assafe{}@acsafe{}}
+This function initializes @code{*@var{iterator}} for traversing
+@var{length} bytes of directory data at @var{buffer}. The buffer must
+have been filled with the @code{getdents64} function, and @var{length}
+should correspond to the return value of a successful call to
+@code{getdents64} for that buffer.
+
+This function is specific to Linux.
+@end deftypefun
+
+@deftypefun int direntries_read (int @var{fd}, struct direntries *@var{iterator}, void *@var{buffer}, size_t @var{length})
+@standards{Linux, sys/direntries.h}
+@safety{@prelim{}@mtsafe{}@assafe{}@acsafe{}}
+This function combines reading directory data from the file descriptor
+@var{fd} using the @code{getdents64} function and initialization of the
+iterator using the @code{direntries_init} function. On success, the
+@code{direntries_read} function returns either zero (if there are no
+more directory entries available in @var{fd}), or a positive value (if
+more entries are availabe). On failure, it returns @code{-1} and sets
+@code{errno} accordingly, and @code{*@var{iterator}} is not
+initialized.
+
+The buffer size considerations for @code{getdents64} also apply to
+this function.
+
+This function is specific to Linux.
+@end deftypefun
+
+@deftypefun int direntries_next (struct direntries *@var{iterator}, struct direntry *@var{entry})
+@standards{Linux, sys/direntries.h}
+@safety{@prelim{}@mtsafe{}@assafe{}@acsafe{}}
+This function copies the next directory entry (if available) from the
+iterator to @code{*@var{entry}}, and updates @code{*@var{iterator}}
+accordingly. On success, it returns zero, and on failure, it returns
+@code{-1} and sets @code{errno} to @code{ENOENT}.
+
+This function is specific to Linux.
+@end deftypefun
+
+The example below shows how implement callback-based traversal of one
+directory using @code{direntries_read} and @code{direntries_next}.
+
+@smallexample
+@include direntries.c.texi
+@end smallexample
@node Working with Directory Trees
@section Working with Directory Trees
@@ -187,8 +187,10 @@ endif
inhibit-glue = yes
ifeq ($(subdir),dirent)
-sysdep_routines += getdirentries getdirentries64
-tests += tst-getdents64
+sysdep_headers += sys/direntries.h
+sysdep_routines += getdirentries getdirentries64 \
+ direntries_init direntries_read direntries_next
+tests += tst-getdents64 tst-direntries
tests-internal += tst-readdir64-compat
endif
@@ -176,6 +176,7 @@ libc {
}
GLIBC_2.30 {
getdents64; gettid; tgkill;
+ direntries_init; direntries_read; direntries_next;
}
GLIBC_PRIVATE {
# functions used in other libraries
@@ -2141,6 +2141,9 @@ GLIBC_2.28 thrd_yield F
GLIBC_2.29 getcpu F
GLIBC_2.29 posix_spawn_file_actions_addchdir_np F
GLIBC_2.29 posix_spawn_file_actions_addfchdir_np F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2216,6 +2216,9 @@ GLIBC_2.30 __nldbl_vwarn F
GLIBC_2.30 __nldbl_vwarnx F
GLIBC_2.30 __nldbl_warn F
GLIBC_2.30 __nldbl_warnx F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -126,6 +126,9 @@ GLIBC_2.28 thrd_yield F
GLIBC_2.29 getcpu F
GLIBC_2.29 posix_spawn_file_actions_addchdir_np F
GLIBC_2.29 posix_spawn_file_actions_addfchdir_np F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2085,6 +2085,9 @@ GLIBC_2.29 xdrstdio_create F
GLIBC_2.29 xencrypt F
GLIBC_2.29 xprt_register F
GLIBC_2.29 xprt_unregister F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
new file mode 100644
@@ -0,0 +1,34 @@
+/* Initialization of directory iterators.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <sys/direntries.h>
+#include <stdio.h>
+
+void
+__direntries_init (struct direntries *iterator, void *buffer, size_t length)
+{
+ /* Cheap security check if the caller accidentally passed an error
+ result from getdirentries to this function. */
+ if ((ssize_t) length < 0)
+ __fortify_fail ("invalid direntries_init call");
+
+ iterator->direntries_buffer_begin = buffer;
+ iterator->direntries_buffer_end = (char *) buffer + length;
+}
+libc_hidden_def (__direntries_init)
+strong_alias (__direntries_init, direntries_init)
new file mode 100644
@@ -0,0 +1,51 @@
+/* Advancing directory iterators.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <sys/direntries.h>
+
+#include <dirent.h>
+#include <errno.h>
+#include <string.h>
+
+int
+__direntries_next (struct direntries *iterator, struct direntry *result)
+{
+ if (iterator->direntries_buffer_begin == iterator->direntries_buffer_end)
+ {
+ __set_errno (ENOENT);
+ return -1;
+ }
+
+ char *begin = iterator->direntries_buffer_begin;
+
+ /* The caller may have supplied an unaligned buffer. Make an
+ aligned copy of the entry, excluding its name. */
+ struct dirent64 entry;
+ memcpy (&entry, begin, offsetof (struct dirent64, d_name));
+
+ /* The name is not copied. It points into the existing buffer. */
+ result->d_name = begin + offsetof (struct dirent64, d_name);
+ result->d_ino = entry.d_ino;
+ result->d_type = entry.d_type;
+ result->d_flags = 0;
+
+ iterator->direntries_buffer_begin = begin + entry.d_reclen;
+ return 0;
+}
+libc_hidden_def (__direntries_next)
+strong_alias (__direntries_next, direntries_next)
new file mode 100644
@@ -0,0 +1,34 @@
+/* Buffer reading for directory iterators.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <sys/direntries.h>
+
+#include <dirent.h>
+
+ssize_t
+__direntries_read (int fd, struct direntries *iterator,
+ void *buffer, size_t length)
+{
+ ssize_t ret = __getdents64 (fd, buffer, length);
+ if (ret < 0)
+ return ret;
+ __direntries_init (iterator, buffer, ret);
+ return ret;
+}
+libc_hidden_def (__direntries_read)
+strong_alias (__direntries_read, direntries_read)
@@ -2037,6 +2037,9 @@ GLIBC_2.3.4 setipv4sourcefilter F
GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2203,6 +2203,9 @@ GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 vm86 F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2069,6 +2069,9 @@ GLIBC_2.3.4 setipv4sourcefilter F
GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -127,6 +127,9 @@ GLIBC_2.28 thrd_yield F
GLIBC_2.29 getcpu F
GLIBC_2.29 posix_spawn_file_actions_addchdir_np F
GLIBC_2.29 posix_spawn_file_actions_addfchdir_np F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2146,6 +2146,9 @@ GLIBC_2.3.4 setipv4sourcefilter F
GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2133,6 +2133,9 @@ GLIBC_2.28 thrd_yield F
GLIBC_2.29 getcpu F
GLIBC_2.29 posix_spawn_file_actions_addchdir_np F
GLIBC_2.29 posix_spawn_file_actions_addfchdir_np F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2120,6 +2120,9 @@ GLIBC_2.3.4 setipv4sourcefilter F
GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2118,6 +2118,9 @@ GLIBC_2.3.4 setipv4sourcefilter F
GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2126,6 +2126,9 @@ GLIBC_2.3.4 setipv4sourcefilter F
GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2120,6 +2120,9 @@ GLIBC_2.3.4 setipv4sourcefilter F
GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2174,6 +2174,9 @@ GLIBC_2.28 thrd_yield F
GLIBC_2.29 getcpu F
GLIBC_2.29 posix_spawn_file_actions_addchdir_np F
GLIBC_2.29 posix_spawn_file_actions_addfchdir_np F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2176,6 +2176,9 @@ GLIBC_2.30 __nldbl_vwarn F
GLIBC_2.30 __nldbl_vwarnx F
GLIBC_2.30 __nldbl_warn F
GLIBC_2.30 __nldbl_warnx F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2209,6 +2209,9 @@ GLIBC_2.30 __nldbl_vwarn F
GLIBC_2.30 __nldbl_vwarnx F
GLIBC_2.30 __nldbl_warn F
GLIBC_2.30 __nldbl_warnx F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2039,6 +2039,9 @@ GLIBC_2.30 __nldbl_vwarn F
GLIBC_2.30 __nldbl_vwarnx F
GLIBC_2.30 __nldbl_warn F
GLIBC_2.30 __nldbl_warnx F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2243,6 +2243,9 @@ GLIBC_2.30 __nldbl_vwarn F
GLIBC_2.30 __nldbl_vwarnx F
GLIBC_2.30 __nldbl_warn F
GLIBC_2.30 __nldbl_warnx F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2103,6 +2103,9 @@ GLIBC_2.28 thrd_yield F
GLIBC_2.29 getcpu F
GLIBC_2.29 posix_spawn_file_actions_addchdir_np F
GLIBC_2.29 posix_spawn_file_actions_addfchdir_np F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2171,6 +2171,9 @@ GLIBC_2.30 __nldbl_vwarn F
GLIBC_2.30 __nldbl_vwarnx F
GLIBC_2.30 __nldbl_warn F
GLIBC_2.30 __nldbl_warnx F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2075,6 +2075,9 @@ GLIBC_2.30 __nldbl_vwarn F
GLIBC_2.30 __nldbl_vwarnx F
GLIBC_2.30 __nldbl_warn F
GLIBC_2.30 __nldbl_warnx F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2041,6 +2041,9 @@ GLIBC_2.3.4 setipv4sourcefilter F
GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2165,6 +2165,9 @@ GLIBC_2.30 __nldbl_vwarn F
GLIBC_2.30 __nldbl_vwarnx F
GLIBC_2.30 __nldbl_warn F
GLIBC_2.30 __nldbl_warnx F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2092,6 +2092,9 @@ GLIBC_2.3.4 setipv4sourcefilter F
GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
new file mode 100644
@@ -0,0 +1,93 @@
+/* Parsing directory streams. Linux version.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#ifndef _SYS_DIRENTRIES_H
+#define _SYS_DIRENTRIES_H
+
+#include <features.h>
+#include <sys/types.h>
+
+/* One directory entry. See struct dirent in <dirent.h>. The main
+ difference is that d_name is a pointer here, and not an array, as
+ in struct dirent. */
+struct direntry
+{
+ /* Name of the directory entry. The string is part of the buffer
+ passed to direntries_init or direntries_read below. */
+ const char *d_name;
+
+ /* Inode number of this directory entry. */
+ __ino64_t d_ino;
+
+ /* Type of the entry. Can be DT_UNKNOWN if unknown. */
+ unsigned char d_type;
+
+ /* Currently always zero. */
+ unsigned char d_flags;
+
+ /* Room for further extension. Do not use. */
+ union
+ {
+ __int64_t __glibc__reserved_1[3];
+ void *__glibc__reserved_2[3];
+ } __glibc__reserved_3;
+};
+
+/* Initialized by direntries_init or direntires_read below and used by
+ direntry_next. */
+struct direntries
+{
+ /* Internal information. Do not use. */
+ void *__glibc_internal_1;
+ void *__glibc_internal_2;
+
+ /* Room for further extension. Do not use. */
+ union
+ {
+ __int64_t __glibc__reserved_1[4];
+ void *__glibc__reserved_2[4];
+ } __glibc__reserved_3;
+};
+
+__BEGIN_DECLS
+
+/* Initialize *ITERATOR based on BUFFER and LENGTH. BUFFER must have
+ been filled by getdents, and length must be a non-negative return
+ value from this function. Afterwards, individual entries can be
+ retrieved using direntries_next. */
+void direntries_init (struct direntries *__iterator,
+ void *__buffer, size_t __length)
+ __THROW __nonnull ((1, 2));
+
+/* Read directory entries from FD and write them to LENGTH bytes at
+ BUFFER and initialize *ITERATOR so that individual entries can be
+ retrieved using direntries_next. Returns -1 on error, 0 on end of
+ stream, and a positive value in case of more data. */
+__ssize_t direntries_read (int __fd, struct direntries *__iterator,
+ void *__buffer, size_t __length)
+ __THROW __nonnull ((2, 3)) __wur;
+
+/* If there is a next entry, copy the next directory entry in ITERATOR
+ to *ENTRY, update *ITERATOR, and returns 0. If there is no next
+ entry, return -1 and set errno to ENOENT. */
+int direntries_next (struct direntries *__iterator, struct direntry *__entry)
+ __THROW __nonnull ((1, 2)) __wur;
+
+__END_DECLS
+
+#endif /* _SYS_DIRENTRIES_H */
new file mode 100644
@@ -0,0 +1,120 @@
+/* Test for directory iterators.
+ Copyright (C) 2019 Free Software Foundation, Inc.
+ This file is part of the GNU C Library.
+
+ The GNU C Library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ The GNU C Library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with the GNU C Library; if not, see
+ <http://www.gnu.org/licenses/>. */
+
+#include <dirent.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <support/check.h>
+#include <support/support.h>
+#include <support/test-driver.h>
+#include <support/xunistd.h>
+#include <sys/direntries.h>
+#include <unistd.h>
+
+static void
+test_one (bool do_read)
+{
+ /* The test compares the iteration order with readdir64. */
+ DIR *reference = opendir (".");
+ TEST_VERIFY_EXIT (reference != NULL);
+
+ int fd = xopen (".", O_RDONLY | O_DIRECTORY, 0);
+ TEST_VERIFY (fd >= 0);
+
+ /* Check that we need to fill the buffer multiple times. */
+ int read_count = 0;
+
+ while (true)
+ {
+ char buffer[1024];
+ struct direntries entries;
+ ssize_t ret;
+ if (do_read)
+ {
+ ret = direntries_read (fd, &entries, buffer, sizeof (buffer));
+ if (ret < 0)
+ FAIL_EXIT1 ("direntries_read: %m");
+ }
+ else
+ {
+ ret = getdents64 (fd, buffer, sizeof (buffer));
+ if (ret < 0)
+ FAIL_EXIT1 ("getdents64: %m");
+ if (ret > 0)
+ direntries_init (&entries, buffer, ret);
+ }
+ if (ret == 0)
+ break;
+ ++read_count;
+ if (test_verbose > 0)
+ {
+ char *s = support_quote_blob (buffer, ret);
+ printf ("info: data: \"%s\"\n", s);
+ free (s);
+ }
+
+ struct direntry entry;
+ errno = 0;
+ while (direntries_next (&entries, &entry) == 0)
+ {
+ errno = 0;
+ struct dirent64 *refentry = readdir64 (reference);
+ if (refentry == NULL && errno == 0)
+ FAIL_EXIT1 ("readdir64 failed too early, at: %s",
+ entry.d_name);
+ else if (refentry == NULL)
+ FAIL_EXIT1 ("readdir64: %m");
+
+ TEST_COMPARE_STRING (entry.d_name, refentry->d_name);
+ TEST_COMPARE (entry.d_ino, refentry->d_ino);
+ TEST_COMPARE (entry.d_type, refentry->d_type);
+ TEST_COMPARE (entry.d_flags, 0);
+
+ errno = 0;
+ }
+ TEST_COMPARE (errno, ENOENT);
+ }
+
+ /* We expect to have reached the end of the stream. */
+ errno = 0;
+ TEST_VERIFY (readdir64 (reference) == NULL);
+ TEST_COMPARE (errno, 0);
+
+ /* We should have read more than one block. */
+ TEST_VERIFY (read_count > 0);
+
+ xclose (fd);
+ closedir (reference);
+}
+
+static int
+do_test (void)
+{
+ puts ("info: test using getdents64");
+ test_one (false);
+
+ puts ("info: test using direntries_read");
+ test_one (true);
+
+ return 0;
+}
+
+#include <support/test-driver.c>
@@ -2050,6 +2050,9 @@ GLIBC_2.3.4 setipv4sourcefilter F
GLIBC_2.3.4 setsourcefilter F
GLIBC_2.3.4 xdr_quad_t F
GLIBC_2.3.4 xdr_u_quad_t F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F
@@ -2149,6 +2149,9 @@ GLIBC_2.28 thrd_yield F
GLIBC_2.29 getcpu F
GLIBC_2.29 posix_spawn_file_actions_addchdir_np F
GLIBC_2.29 posix_spawn_file_actions_addfchdir_np F
+GLIBC_2.30 direntries_init F
+GLIBC_2.30 direntries_next F
+GLIBC_2.30 direntries_read F
GLIBC_2.30 getdents64 F
GLIBC_2.30 gettid F
GLIBC_2.30 tgkill F