[v2] libio: Ignore setbuf for open_memstream [BZ #34019]

  From: Xiang Gao <gaoxiang@kylinos.cn>

open_memstream and open_wmemstream manage an internal growable buffer.
The default setbuf hook can reset that buffer, breaking the assumptions
used by the string stream overflow paths.

Install setbuf hooks that leave the internal buffer unchanged, and add
regression test cases for the narrow and wide cases, based on the
reproducer in BZ #34019.

Checked on x86_64 with no regression in the libio tests.

Reported-by: Rocket Ma <marocketbd@gmail.com>
Signed-off-by: Xiang Gao <gaoxiang@kylinos.cn>

---

Changes in v2: Update the prototypes in libioP.h to using FILE *fp
spacing consistently.
---
 libio/Makefile         |  1 +
 libio/libioP.h         | 12 ++++++----
 libio/memstream.c      |  9 +++++++
 libio/tst-memstream5.c | 54 ++++++++++++++++++++++++++++++++++++++++++
 libio/vtables.c        |  6 +++--
 libio/wmemstream.c     | 10 ++++++++
 6 files changed, 86 insertions(+), 6 deletions(-)
 create mode 100644 libio/tst-memstream5.c
  

On 28/04/26 05:14, Gao Xiang wrote:
> From: Xiang Gao <gaoxiang@kylinos.cn>
> 
> open_memstream and open_wmemstream manage an internal growable buffer.
> The default setbuf hook can reset that buffer, breaking the assumptions
> used by the string stream overflow paths.
> 
> Install setbuf hooks that leave the internal buffer unchanged, and add
> regression test cases for the narrow and wide cases, based on the
> reproducer in BZ #34019.
> 
> Checked on x86_64 with no regression in the libio tests.
> 
> Reported-by: Rocket Ma <marocketbd@gmail.com>
> Signed-off-by: Xiang Gao <gaoxiang@kylinos.cn>


Wouldn't this change the behavior or setbuf (NULL, ..) and setbuf (..., 0)
to not be _IO_UNBUFFERED? 

I think it does not seems to matter though, since the resulting buffer operated 
by open_memstream can only be accessible after a fflush/fclose.  The
setbuf/setvbuf would only be a way to optimize the stdio buffer flush to
final one, which I am also not fully sure it would matter here.

> 
> ---
> 
> Changes in v2: Update the prototypes in libioP.h to using FILE *fp
> spacing consistently.
> ---
>  libio/Makefile         |  1 +
>  libio/libioP.h         | 12 ++++++----
>  libio/memstream.c      |  9 +++++++
>  libio/tst-memstream5.c | 54 ++++++++++++++++++++++++++++++++++++++++++
>  libio/vtables.c        |  6 +++--
>  libio/wmemstream.c     | 10 ++++++++
>  6 files changed, 86 insertions(+), 6 deletions(-)
>  create mode 100644 libio/tst-memstream5.c
> 
> diff --git a/libio/Makefile b/libio/Makefile
> index 93656466df..584fcdb14d 100644
> --- a/libio/Makefile
> +++ b/libio/Makefile
> @@ -120,6 +120,7 @@ tests = \
>    tst-memstream2 \
>    tst-memstream3 \
>    tst-memstream4 \
> +  tst-memstream5 \
>    tst-mmap-eofsync \
>    tst-mmap-fflushsync \
>    tst-mmap-offend \
> diff --git a/libio/libioP.h b/libio/libioP.h
> index 1485d22619..17c0b6e76d 100644
> --- a/libio/libioP.h
> +++ b/libio/libioP.h
> @@ -738,10 +738,14 @@ extern size_t __IO_obstack_xsputn (FILE *fp, const void *data, size_t n)
>    attribute_hidden;
>  
>  /* Jumptable functions for open_{w}memstream.  */
> -extern int _IO_mem_sync (FILE* fp) __THROW attribute_hidden;
> -extern void _IO_mem_finish (FILE* fp, int) __THROW attribute_hidden;
> -extern int _IO_wmem_sync (FILE* fp) __THROW attribute_hidden;
> -extern void _IO_wmem_finish (FILE* fp, int) __THROW attribute_hidden;
> +extern int _IO_mem_sync (FILE *fp) __THROW attribute_hidden;
> +extern void _IO_mem_finish (FILE *fp, int) __THROW attribute_hidden;
> +extern FILE *_IO_mem_setbuf (FILE *fp, char *buf, ssize_t size)
> +  __THROW attribute_hidden;
> +extern int _IO_wmem_sync (FILE *fp) __THROW attribute_hidden;
> +extern void _IO_wmem_finish (FILE *fp, int) __THROW attribute_hidden;
> +extern FILE *_IO_wmem_setbuf (FILE *fp, char *buf, ssize_t size)
> +  __THROW attribute_hidden;

These changes does not seem required, I am failing to see what it is
doing here.

>  
>  /* Other strfile functions */
>  struct _IO_strfile_;
> diff --git a/libio/memstream.c b/libio/memstream.c
> index 0456adb92f..a5f909cf64 100644
> --- a/libio/memstream.c
> +++ b/libio/memstream.c
> @@ -112,3 +112,12 @@ _IO_mem_finish (FILE *fp, int dummy)
>  
>    _IO_str_finish (fp, 0);
>  }
> +
> +FILE *
> +_IO_mem_setbuf (FILE *fp, char *p, ssize_t len)
> +{
> +  /* memstream manage a growable buffer internally.  */
> +  (void) p;
> +  (void) len;
> +  return fp;

No need to use these cast to avoid compiler warnings here.

> +}
> diff --git a/libio/tst-memstream5.c b/libio/tst-memstream5.c
> new file mode 100644
> index 0000000000..e9d05bb1df
> --- /dev/null
> +++ b/libio/tst-memstream5.c
> @@ -0,0 +1,54 @@
> +/* Test for open_memstream BZ #34019.
> +   Copyright (C) 2026 Free Software Foundation, Inc.
> +   This file is part of the GNU C Library.
> +
> +   The GNU C Library is free software; you can redistribute it and/or
> +   modify it under the terms of the GNU Lesser General Public
> +   License as published by the Free Software Foundation; either
> +   version 2.1 of the License, or (at your option) any later version.
> +
> +   The GNU C Library is distributed in the hope that it will be useful,
> +   but WITHOUT ANY WARRANTY; without even the implied warranty of
> +   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +   Lesser General Public License for more details.
> +
> +   You should have received a copy of the GNU Lesser General Public
> +   License along with the GNU C Library; if not, see
> +   <https://www.gnu.org/licenses/>.  */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <wchar.h>
> +#include <support/check.h>
> +
> +static int
> +do_test (void)
> +{
> +  /* Narrow: setbuf must not replace the internal growable buffer.  */
> +  char *buf = NULL;
> +  size_t len = 0;
> +  FILE *fp = open_memstream (&buf, &len);
> +  setbuf (fp, NULL);
> +  TEST_COMPARE (fputc ('A', fp), 'A');
> +  TEST_COMPARE (fclose (fp), 0);
> +  TEST_COMPARE (len, 1);
> +  TEST_COMPARE_STRING (buf, "A");
> +  free (buf);
> +
> +  /* Wide: same crash via _IO_wstr_overflow.   */
> +  wchar_t *wbuf = NULL;
> +  size_t wlen = 0;
> +  FILE *wfp = open_wmemstream (&wbuf, &wlen);
> +  TEST_VERIFY_EXIT (wfp != NULL);
> +  setbuf (wfp, NULL);
> +  TEST_COMPARE (fputwc (L'A', wfp), L'A');
> +  TEST_COMPARE (fclose (wfp), 0);
> +  TEST_COMPARE (wlen, 1);
> +  TEST_VERIFY (wbuf != NULL);
> +  TEST_VERIFY (wbuf[0] == L'A' && wbuf[1] == L'\0');
> +  free (wbuf);

Could you add setvbuf tests along with different modes (_IONBF, etc.)?

> +
> +  return 0;
> +}
> +
> +#include <support/test-driver.c>
> diff --git a/libio/vtables.c b/libio/vtables.c
> index 00d9d25b5e..ca9f1b2dc4 100644
> --- a/libio/vtables.c
> +++ b/libio/vtables.c
> @@ -77,9 +77,11 @@
>  # pragma weak _IO_cookie_write
>  
>  # pragma weak _IO_mem_finish
> +# pragma weak _IO_mem_setbuf
>  # pragma weak _IO_mem_sync
>  
>  # pragma weak _IO_wmem_finish
> +# pragma weak _IO_wmem_setbuf
>  # pragma weak _IO_wmem_sync
>  
>  # pragma weak __printf_buffer_as_file_overflow
> @@ -334,7 +336,7 @@ const struct _IO_jump_t __io_vtables[] attribute_relro =
>      JUMP_INIT (xsgetn, _IO_default_xsgetn),
>      JUMP_INIT (seekoff, _IO_str_seekoff),
>      JUMP_INIT (seekpos, _IO_default_seekpos),
> -    JUMP_INIT (setbuf, _IO_default_setbuf),
> +    JUMP_INIT (setbuf, _IO_mem_setbuf),
>      JUMP_INIT (sync, _IO_mem_sync),
>      JUMP_INIT (doallocate, _IO_default_doallocate),
>      JUMP_INIT (read, _IO_default_read),
> @@ -357,7 +359,7 @@ const struct _IO_jump_t __io_vtables[] attribute_relro =
>      JUMP_INIT (xsgetn, _IO_wdefault_xsgetn),
>      JUMP_INIT (seekoff, _IO_wstr_seekoff),
>      JUMP_INIT (seekpos, _IO_default_seekpos),
> -    JUMP_INIT (setbuf, _IO_default_setbuf),
> +    JUMP_INIT (setbuf, _IO_wmem_setbuf),
>      JUMP_INIT (sync, _IO_wmem_sync),
>      JUMP_INIT (doallocate, _IO_wdefault_doallocate),
>      JUMP_INIT (read, _IO_default_read),
> diff --git a/libio/wmemstream.c b/libio/wmemstream.c
> index d0c639be70..8f37a6efb8 100644
> --- a/libio/wmemstream.c
> +++ b/libio/wmemstream.c
> @@ -117,3 +117,13 @@ _IO_wmem_finish (FILE *fp, int dummy)
>  
>    _IO_wstr_finish (fp, 0);
>  }
> +
> +
> +FILE *
> +_IO_wmem_setbuf (FILE *fp, char *p, ssize_t len)
> +{
> +  /* wmemstreams manage a growable buffer internally.  */
> +  (void) p;
> +  (void) len;
> +  return fp;
> +}
  

> Wouldn't this change the behavior or setbuf (NULL, ..) and setbuf (..., 0)
> to not be _IO_UNBUFFERED?
>
> I think it does not seems to matter though, since the resulting buffer operated
> by open_memstream can only be accessible after a fflush/fclose.  The
> setbuf/setvbuf would only be a way to optimize the stdio buffer flush to
> final one, which I am also not fully sure it would matter here.
>

Referencing POSIX standard[1], fflush and fclose are explicitly
mentioned when describing `bufp` and `sizep`, which may indicate that
user shall always flush the stream. But the standard does not mention
the behavior of setbuf. I'm also not sure if we should allow user to
setbuf.

[1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/open_memstream.html
  

On 01/05/26 13:24, Rocket Ma wrote:
>> Wouldn't this change the behavior or setbuf (NULL, ..) and setbuf (..., 0)
>> to not be _IO_UNBUFFERED?
>>
>> I think it does not seems to matter though, since the resulting buffer operated
>> by open_memstream can only be accessible after a fflush/fclose.  The
>> setbuf/setvbuf would only be a way to optimize the stdio buffer flush to
>> final one, which I am also not fully sure it would matter here.
>>
> 
> Referencing POSIX standard[1], fflush and fclose are explicitly
> mentioned when describing `bufp` and `sizep`, which may indicate that
> user shall always flush the stream. But the standard does not mention
> the behavior of setbuf. I'm also not sure if we should allow user to
> setbuf.
> 
> [1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/open_memstream.html

I think ignoring set{v}buf makes sense here, we can make it orthogonal to other
stream by the current POSIX description makes invisible to user anyway.
  

On 2026/5/4 20:04, Adhemerval Zanella Netto wrote:
> 
> 
> On 01/05/26 13:24, Rocket Ma wrote:
>>> Wouldn't this change the behavior or setbuf (NULL, ..) and setbuf (..., 0)
>>> to not be _IO_UNBUFFERED?
>>>
>>> I think it does not seems to matter though, since the resulting buffer operated
>>> by open_memstream can only be accessible after a fflush/fclose.  The
>>> setbuf/setvbuf would only be a way to optimize the stdio buffer flush to
>>> final one, which I am also not fully sure it would matter here.
>>>
>>
>> Referencing POSIX standard[1], fflush and fclose are explicitly
>> mentioned when describing `bufp` and `sizep`, which may indicate that
>> user shall always flush the stream. But the standard does not mention
>> the behavior of setbuf. I'm also not sure if we should allow user to
>> setbuf.
>>
>> [1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/open_memstream.html
> 
> I think ignoring set{v}buf makes sense here, we can make it orthogonal to other
> stream by the current POSIX description makes invisible to user anyway.

Thanks, that makes sense.

I will treat setbuf/setvbuf as orthogonal to the memory-stream result 
buffer.The backing buffer of open_memstream/open_wmemstream is the 
growable result object itself, not discardable. Therefore the generic 
buffering operation is not appropriate here: it must not reset the 
get/put areas.

While adding setvbuf coverage, I confirmed that the current hook change 
covers the setvbuf paths that reach _IO_SETBUF, but I also found one 
related case:

setvbuf(fp, NULL, _IOFBF, 0)

This can bypass the setbuf hook and go through doallocate instead. In 
that path default wide buffering code may still disturb the wide result 
buffer. I will send a separate patch(v3) for wmemstream doallocate path.
  

On 04/05/26 13:54, Alex Gao wrote:
> On 2026/5/4 20:04, Adhemerval Zanella Netto wrote:
>>
>>
>> On 01/05/26 13:24, Rocket Ma wrote:
>>>> Wouldn't this change the behavior or setbuf (NULL, ..) and setbuf (..., 0)
>>>> to not be _IO_UNBUFFERED?
>>>>
>>>> I think it does not seems to matter though, since the resulting buffer operated
>>>> by open_memstream can only be accessible after a fflush/fclose.  The
>>>> setbuf/setvbuf would only be a way to optimize the stdio buffer flush to
>>>> final one, which I am also not fully sure it would matter here.
>>>>
>>>
>>> Referencing POSIX standard[1], fflush and fclose are explicitly
>>> mentioned when describing `bufp` and `sizep`, which may indicate that
>>> user shall always flush the stream. But the standard does not mention
>>> the behavior of setbuf. I'm also not sure if we should allow user to
>>> setbuf.
>>>
>>> [1]: https://pubs.opengroup.org/onlinepubs/9799919799/functions/open_memstream.html
>>
>> I think ignoring set{v}buf makes sense here, we can make it orthogonal to other
>> stream by the current POSIX description makes invisible to user anyway.
> 
> Thanks, that makes sense.
> 
> I will treat setbuf/setvbuf as orthogonal to the memory-stream result buffer.The backing buffer of open_memstream/open_wmemstream is the growable result object itself, not discardable. Therefore the generic buffering operation is not appropriate here: it must not reset the get/put areas.
> 
> While adding setvbuf coverage, I confirmed that the current hook change covers the setvbuf paths that reach _IO_SETBUF, but I also found one related case:
> 
> setvbuf(fp, NULL, _IOFBF, 0)
> 
> This can bypass the setbuf hook and go through doallocate instead. In that path default wide buffering code may still disturb the wide result buffer. I will send a separate patch(v3) for wmemstream doallocate path.
Alright, sounds reasonable. Thanks for working on this.