[v2,2/2] malloc: make malloc fail with requests larger than PTRDIFF_MAX

  Changes from previous version:

  - Fix wording on NEWS entry.
  - Added static assert to check for PTRDIFF_MAX == SIZE_MAX / 2.
  - Remove padsize check, requested size after aligning and padding
    are checked against PTRDIFF_MAX.  This does not guarantee that
    sysmalloc will eventually issue an request larger than PTRDIFF_MAX.
    One option to enforce it is to either add the PTRDIFF_MAX check on
    mmap or create an internal mmap implementation to enforce it.

--

As discussed previously on libc-alpha [1], this patch follows up the idea
and add both the __attribute_alloc_size__ on malloc functions (malloc,
calloc, realloc, reallocarray, valloc, pvalloc, memaling, and
posix_memalign) and limit maximum requested allocation size to up
PTRDIFF_MAX (taking in consideration internal padding and alignment).

This aligns glibc with gcc expected size defined by default warning
-Walloc-size-larger-than value which warns for allocation larger than
PTRDIFF_MAX.  It also aligns with gcc expectation regarding libc and
expected size, such as described in PR#67999 [2] and previously discussed
ISO C11 issues [3] on libc-alpha.

From the RFC thread [4] and previous discussion, it seems that consensus
is only to limit such requested size for malloc functions, not system
allocation one (mmap, sbrk, etc.).

The implementation changes checked_request2size to check for both overflow
and maximum object size up to PTRDIFF_MAX. No additional checks are done
on sysmalloc, so it can still issues mmap with values larger than
PTRDIFF_T depending of requested size.

Checked on x86_64-linux-gnu and i686-linux-gnu.

	[BZ #23741]
	* malloc/hooks.c (malloc_check, realloc_check): Use
	__builtin_add_overflow on overflow check and adapt to
	checked_request2size change.
	* malloc/malloc.c (__libc_malloc, __libc_realloc, _mid_memalign,
	__libc_pvalloc, __libc_calloc, _int_memalign): Limit maximum
	allocation size to PTRDIFF_MAX.
	(REQUEST_OUT_OF_RANGE): Remove macro.
	(checked_request2size): Change to inline function, use
	__builtin_add_overflow for overflow check and limit maximum requested
	size to PTRDIFF_MAX.
	(__libc_malloc, __libc_realloc, _int_malloc, _int_memalign): Limit
	maximum allocation size to PTRDIFF_MAX.
	(_mid_memalign): Rely on _int_memalign call for overflow check.
	(__libc_pvalloc): Use __builtin_add_overflow on overflow check.
	(__libc_calloc): Use __builtin_mul_overflow for overflow check and
	limit maximum requested size to PTRDIFF_MAX.
	* malloc/malloc.h (malloc, calloc, realloc, reallocarray, memalign,
	valloc, pvalloc): Add __attribute_alloc_size__.
	* stdlib/stdlib.h (malloc, realloc, reallocarray, valloc,
	posix_memalign): Likewise.
	* malloc/tst-malloc-too-large.c (do_test): Add check for allocation
	larger than PTRDIFF_MAX.
	* malloc/tst-memalign.c (do_test): Disable -Walloc-size-larger-than=
	around tests of malloc with negative sizes.
	* malloc/tst-posix_memalign.c (do_test): Likewise.
	* malloc/tst-pvalloc.c (do_test): Likewise.
	* malloc/tst-valloc.c (do_test): Likewise.
	* malloc/tst-reallocarray.c (do_test): Replace call to reallocarray
	with resulting size allocation larger than PTRDIFF_MAX with
	reallocarray_nowarn.
	(reallocarray_nowarn): New function.
	* NEWS: Mention the malloc function semantic change.

[1] https://sourceware.org/ml/libc-alpha/2018-11/msg00223.html
[2] https://gcc.gnu.org/bugzilla//show_bug.cgi?id=67999
[3] https://sourceware.org/ml/libc-alpha/2011-12/msg00066.html
[4] https://sourceware.org/ml/libc-alpha/2018-11/msg00224.html
---
 ChangeLog                     |  34 ++++++++++
 NEWS                          |   6 ++
 malloc/hooks.c                |  18 +++--
 malloc/malloc.c               | 121 ++++++++++++++++------------------
 malloc/malloc.h               |  17 +++--
 malloc/tst-malloc-too-large.c |  10 +++
 malloc/tst-memalign.c         |  10 +++
 malloc/tst-posix_memalign.c   |  10 +++
 malloc/tst-pvalloc.c          |  10 +++
 malloc/tst-reallocarray.c     |  27 ++++++--
 malloc/tst-valloc.c           |  10 +++
 stdlib/stdlib.h               |  13 ++--
 12 files changed, 198 insertions(+), 88 deletions(-)

[v2,2/2] malloc: make malloc fail with requests larger than PTRDIFF_MAX

Commit Message

Comments

Patch