[RFC] Add envvar IO_BUFMODE to forcibly set buffering mode
Commit Message
Hello all,
When a program interacts via pipes with a program that doesn't call
fflush(stdout) well, they may not communicate data through the pipes
at a proper time.
Let me give an example.
test.c:
// compile with gcc -o test test.c
#include <unistd.h>
#include <stdio.h>
int main() {
for(int i = 0; i < 5; i++) {
printf("test");
usleep(500000); // sleep 0.5 sec
putchar('\n');
usleep(500000); // sleep 0.5 sec
}
return 0;
}
When I execute this program via tty, I would get "test\n" every 1
second. If I execute the program via "./test > test.log" and read the
output by "tail -f test.log", otherwise, I will get the entire output
after 5 seconds.
Let me give another example.
test.py:
import subprocess
import sys
p = subprocess.Popen(['./test'], stdout=subprocess.PIPE)
while True:
tmp = p.stdout.read(1)
if not tmp:
break
print(tmp.decode(), end='')
sys.stdout.flush()
The python code tries to interact with './test', it would get the
entire output at a time after 5 seconds, however.
I thought it's too hard and tricky to solve this problem. Some of
the solutions are: to modify the source code, to open a pty to get the
output through it, or to use LD_PRELOAD to hook printf or some
functions.
Therefore, I propose a new environment variable IO_BUFMODE to
forcibly set the buffering mode in these cases. To force "fully
buffering mode", I can easily set IO_BUFMODE=0. To force "line
buffering mode", I set IO_BUFMODE=1. And to force "no buffered mode",
I set IO_BUFMODE=2. The constants are from _IOFBF, _IOLBF, _IONBF used
in setvbuf().
With IO_BUFMODE support, it's not needed to write a program that
opens a pty to interact. Also, it is not required to use LD_PRELOAD to
hook functions to set "no buffered mode" (because no buffered mode is
impossible even with pty.)
Thank you.
Comments
Such a feature would need documentation and tests. Also, as it breaks
standard semantics, it needs to be disabled for setuid programs (see
sysdeps/generic/unsecvars.h etc.).
On 3/22/2017 1:14 PM, Sunyeop Lee wrote:
> I thought it's too hard and tricky to solve this problem. Some of
> the solutions are: to modify the source code, to open a pty to get the
> output through it, or to use LD_PRELOAD to hook printf or some
> functions.
> Therefore, I propose a new environment variable IO_BUFMODE to
> forcibly set the buffering mode in these cases. To force "fully
> buffering mode", I can easily set IO_BUFMODE=0. To force "line
> buffering mode", I set IO_BUFMODE=1. And to force "no buffered mode",
> I set IO_BUFMODE=2. The constants are from _IOFBF, _IOLBF, _IONBF used
> in setvbuf().
A few concerns with this: it applies the change to every opened file,
which I doubt is what you want; it will make writes to files be
line-buffered at all times, which is likely not what you want. Also,
I'm not sure calling getenv() for every file open is good for
performance. And I'm a bit dubious about your just copying in the
constants _IOFBF and friends.
Here we have a similar issue in one application and found that solving
it with LD_PRELOAD was not that onerous. We wanted to spawn
subprocesses and force console stdout to be line buffered, even if the
whole process group was having its output passed into a pipe.
Accordingly, the main process does an fstat() on stdout to get its
dev/ino information and puts that into $STDIO_LBF_DEV_INO,
then sets $LD_PRELOAD to point to a trivial program that does this:
static void __attribute((constructor)) set_linebuf(void)
{
// Get "dev" and "ino" from the environment.
unsigned long dev, ino;
const char* env = getenv("STDIO_LBF_DEV_INO");
if (env == NULL || sscanf(env, "%lu:%lu", &dev, &ino) != 2)
return;
// Set line buffering if stdout is pointing to the specified dev/ino.
struct stat s;
if (fstat(STDOUT_FILENO, &s) == 0 && s.st_ino == ino && s.st_dev == dev)
setlinebuf(stdout);
}
@@ -1,3 +1,9 @@
+2017-03-22 Sunyeop Lee <sunyeop97@gmail.com>
+
+ * libio/filedoallocate.c (_IO_file_doallocate): Add environment variable
+ IO_BUFMODE and relevant constants (_IOFBF, _IOLBF, _IONBF) to forcibly
+ set buffering mode
+
2017-03-20 Mike Frysinger <vapier@gentoo.org>
[BZ #21275]
@@ -61,6 +61,12 @@
#include <stdlib.h>
#include <unistd.h>
+
+#define _IOFBF 0 /* Fully buffered. */
+#define _IOLBF 1 /* Line buffered. */
+#define _IONBF 2 /* No buffering. */
+
+
/* Return the result of isatty, without changing errno. */
static int
local_isatty (int fd)
@@ -77,7 +83,8 @@ int
_IO_file_doallocate (_IO_FILE *fp)
{
_IO_size_t size;
- char *p;
+ char *p, *temp;
+ int io_bufmode;
struct stat64 st;
size = _IO_BUFSIZ;
@@ -98,6 +105,24 @@ _IO_file_doallocate (_IO_FILE *fp)
size = st.st_blksize;
#endif
}
+ if ((temp = getenv ("IO_BUFMODE")))
+ {
+ io_bufmode = atoi (temp);
+ switch (io_bufmode)
+ {
+ case _IOFBF:
+ fp->_flags &= ~(_IO_LINE_BUF|_IO_UNBUFFERED);
+ break;
+ case _IOLBF:
+ fp->_flags &= ~_IO_UNBUFFERED;
+ fp->_flags |= _IO_LINE_BUF;
+ break;
+ case _IONBF:
+ fp->_flags &= ~_IO_LINE_BUF;
+ fp->_flags |= _IO_UNBUFFERED;
+ break;
+ }
+ }
p = malloc (size);
if (__glibc_unlikely (p == NULL))
return EOF;