[v5,0/8] malloc hooks removal

  This patchset removes the malloc hooks __malloc_hook, __free_hook,
__realloc_hook and __memalign_hook from the API and leaves compatibility
symbols so that existing applications can continue to link to them.  The
reading and execution of the hooks has been moved to a DSO
libc_malloc_debug.so, which can be preloaded for applications that need
it.  By default these hooks no longer have any effect in the library.

Further, debugging features such as MALLOC_CHECK_, mcheck() and mtrace
have been weaned away from these hooks and also moved to
libc_malloc_debug.so.  With this change, these features are only enabled
when libc_malloc_debug.so is preloaded using LD_PRELOAD.

Finally, the __morecore, __morecore_after_hook and __default_morecore
hooks have also been moved to compat symbols and removed from the API.
Existing applications will continue to link to them but they won't have
any effect on malloc behaviour.

Changes from v4:
- Patchset has a different approach, starting with moving out hooks
  first to restrict all major malloc.c changes to the first patch
- Renamed libmalloc_compathooks.so to libc_malloc_debug.so
- Moved all debugging features into libc_malloc_debug.so
- Made more documentation updates
- Simplified __malloc_initialized variable use
- Removed debugging tests on static variables since that is no longer
  supported

Changes from v3:
- Remove source file dependencies
- Commit mcheck tests

Changes from v2:
- Move hooks dependencies to malloc.o{,sS}

Changes from v1:

- Added makefile dependencies for the new hooks files
- Fixed memset call in calloc debugging hooks
- Added the tr_break deprecation patch and mcheck test patch to this
  series

Siddhesh Poyarekar (8):
  Move malloc hooks into a compat DSO
  mcheck: Wean away from malloc hooks
  Simplify __malloc_initialized
  mtrace: Wean away from malloc hooks
  glibc.malloc.check: Wean away from malloc hooks
  Remove malloc hooks
  Remove __after_morecore_hook
  Remove __morecore and __default_morecore

 Makeconfig                                 |   2 +-
 NEWS                                       |  21 ++
 Rules                                      |   9 +-
 catgets/Makefile                           |   4 +-
 elf/Makefile                               |  18 +-
 elf/tst-leaks1-static.c                    |   1 -
 iconvdata/Makefile                         |   3 +-
 include/malloc.h                           |   6 -
 include/mcheck.h                           |   4 -
 include/stdlib.h                           |   3 -
 intl/tst-gettext.sh                        |   1 +
 libio/Makefile                             |  12 +-
 localedata/Makefile                        |   3 +-
 malloc/Makefile                            |  44 ++-
 malloc/Versions                            |   9 +
 malloc/arena.c                             |  42 +--
 malloc/hooks.c                             |  84 ++++-
 malloc/malloc-check.c                      |  86 +++--
 malloc/{malloc-hooks.h => malloc-check.h}  |  22 +-
 malloc/malloc-debug.c                      | 346 ++++++++++++++++++
 malloc/malloc-internal.h                   |   5 +
 malloc/malloc.c                            | 135 ++-----
 malloc/malloc.h                            |  27 --
 malloc/mcheck-impl.c                       | 406 +++++++++++++++++++++
 malloc/mcheck.c                            | 396 ++------------------
 malloc/morecore.c                          |  15 +-
 malloc/mtrace-impl.c                       | 224 ++++++++++++
 malloc/mtrace.c                            | 313 +---------------
 malloc/tst-compathooks-off.c               | 145 ++++++++
 malloc/tst-compathooks-on.c                |   2 +
 malloc/tst-malloc-usable-static-tunables.c |   1 -
 malloc/tst-malloc-usable-static.c          |   1 -
 malloc/tst-mtrace.sh                       |   1 +
 manual/memory.texi                         | 207 +----------
 manual/tunables.texi                       |   4 +-
 misc/Makefile                              |   6 +-
 nptl/Makefile                              |   3 +-
 posix/Makefile                             |  40 +-
 resolv/Makefile                            |   9 +-
 stdio-common/Makefile                      |  15 +-
 sysdeps/pthread/Makefile                   |   3 +-
 41 files changed, 1502 insertions(+), 1176 deletions(-)
 delete mode 100644 elf/tst-leaks1-static.c
 rename malloc/{malloc-hooks.h => malloc-check.h} (55%)
 create mode 100644 malloc/malloc-debug.c
 create mode 100644 malloc/mcheck-impl.c
 create mode 100644 malloc/mtrace-impl.c
 create mode 100644 malloc/tst-compathooks-off.c
 create mode 100644 malloc/tst-compathooks-on.c
 delete mode 100644 malloc/tst-malloc-usable-static-tunables.c
 delete mode 100644 malloc/tst-malloc-usable-static.c
  

32-bit builds broke...

In file included from mtrace.c:22,
                 from malloc-debug.c:84:
mtrace-impl.c: In function 'tr_where':
mtrace-impl.c:66:37: error: format '%lx' expects argument of type 'long unsigned int', but argument 5 has type 'ptrdiff_t' {aka 'int'} [-Werror=format=]
   66 |               sprintf (buf, "(%s%c%lx)", info->dli_sname, sign, offset);
      |                                   ~~^                           ~~~~~~
      |                                     |                           |
      |                                     long unsigned int           ptrdiff_t {aka int}
      |                                   %x
cc1: all warnings being treated as errors
  

On 7/7/21 1:15 AM, DJ Delorie wrote:
> 
> 32-bit builds broke...
> 
> In file included from mtrace.c:22,
>                   from malloc-debug.c:84:
> mtrace-impl.c: In function 'tr_where':
> mtrace-impl.c:66:37: error: format '%lx' expects argument of type 'long unsigned int', but argument 5 has type 'ptrdiff_t' {aka 'int'} [-Werror=format=]
>     66 |               sprintf (buf, "(%s%c%lx)", info->dli_sname, sign, offset);
>        |                                   ~~^                           ~~~~~~
>        |                                     |                           |
>        |                                     long unsigned int           ptrdiff_t {aka int}
>        |                                   %x
> cc1: all warnings being treated as errors
> 

Ugh, thanks for checking.  I'll fix this locally and also test builds on 
a couple of different architectures this time.

Siddhesh