Message ID | 20210719024134.155398-1-siddhesh@sourceware.org |
---|---|
Headers |
Return-Path: <libc-alpha-bounces+patchwork=sourceware.org@sourceware.org> X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 44B993861C68 for <patchwork@sourceware.org>; Mon, 19 Jul 2021 02:42:19 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 44B993861C68 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1626662539; bh=tdnWOnnBplq8bb5yxHwX7fN91fKyPCCn48Fveasazu4=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=CWi0sOsh6w2X3tbwea6xZI5kxZug5nhHcgPeoovYyvk7Z8FaZHhOVZuXzPigOF6c3 4fQoa2ut+zxDZZHIYiFMIWXiFW/yK6ccwfDNBKTZXPfBMp/8XxojOxqIe1GGK4l7bL 0tKMuFYNnn1NqHcX9R2d62a1qcn9nFwewOmwqvdo= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from dog.birch.relay.mailchannels.net (dog.birch.relay.mailchannels.net [23.83.209.48]) by sourceware.org (Postfix) with ESMTPS id B7D123857814 for <libc-alpha@sourceware.org>; Mon, 19 Jul 2021 02:41:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B7D123857814 X-Sender-Id: dreamhost|x-authsender|siddhesh@gotplt.org Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 7260C92269E; Mon, 19 Jul 2021 02:41:54 +0000 (UTC) Received: from pdx1-sub0-mail-a83.g.dreamhost.com (100-96-133-125.trex.outbound.svc.cluster.local [100.96.133.125]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 5EADA9227AA; Mon, 19 Jul 2021 02:41:52 +0000 (UTC) X-Sender-Id: dreamhost|x-authsender|siddhesh@gotplt.org Received: from pdx1-sub0-mail-a83.g.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384) by 100.96.133.125 (trex/6.3.3); Mon, 19 Jul 2021 02:41:54 +0000 X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|siddhesh@gotplt.org X-MailChannels-Auth-Id: dreamhost X-Imminent-Turn: 158214030f9a6a16_1626662514333_1227702564 X-MC-Loop-Signature: 1626662514333:1197379579 X-MC-Ingress-Time: 1626662514333 Received: from pdx1-sub0-mail-a83.g.dreamhost.com (localhost [127.0.0.1]) by pdx1-sub0-mail-a83.g.dreamhost.com (Postfix) with ESMTP id 1E7617F17F; Sun, 18 Jul 2021 19:41:52 -0700 (PDT) Received: from rhbox.intra.reserved-bit.com (unknown [1.186.101.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: siddhesh@gotplt.org) by pdx1-sub0-mail-a83.g.dreamhost.com (Postfix) with ESMTPSA id A54A77E5F6; Sun, 18 Jul 2021 19:41:49 -0700 (PDT) X-DH-BACKEND: pdx1-sub0-mail-a83 To: libc-alpha@sourceware.org Subject: [PATCH v9 00/10] malloc hooks removal Date: Mon, 19 Jul 2021 08:11:24 +0530 Message-Id: <20210719024134.155398-1-siddhesh@sourceware.org> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-3486.0 required=5.0 tests=BAYES_00, JMQ_SPF_NEUTRAL, KAM_DMARC_NONE, KAM_DMARC_STATUS, RCVD_IN_BARRACUDACENTRAL, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NEUTRAL, TXREP, URIBL_BLACK autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list <libc-alpha.sourceware.org> List-Unsubscribe: <https://sourceware.org/mailman/options/libc-alpha>, <mailto:libc-alpha-request@sourceware.org?subject=unsubscribe> List-Archive: <https://sourceware.org/pipermail/libc-alpha/> List-Post: <mailto:libc-alpha@sourceware.org> List-Help: <mailto:libc-alpha-request@sourceware.org?subject=help> List-Subscribe: <https://sourceware.org/mailman/listinfo/libc-alpha>, <mailto:libc-alpha-request@sourceware.org?subject=subscribe> From: Siddhesh Poyarekar via Libc-alpha <libc-alpha@sourceware.org> Reply-To: Siddhesh Poyarekar <siddhesh@sourceware.org> Cc: fweimer@redhat.com Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" <libc-alpha-bounces+patchwork=sourceware.org@sourceware.org> |
Series |
malloc hooks removal
|
|
Message
Siddhesh Poyarekar
July 19, 2021, 2:41 a.m. UTC
This patchset removes the malloc hooks __malloc_hook, __free_hook, __realloc_hook and __memalign_hook from the API and leaves compatibility symbols so that existing applications can continue to link to them. The reading and execution of the hooks has been moved to a DSO libc_malloc_debug.so, which can be preloaded for applications that need it. By default these hooks no longer have any effect in the library. Further, debugging features such as MALLOC_CHECK_, mcheck() and mtrace have been weaned away from these hooks and also moved to libc_malloc_debug.so. With this change, these features are only enabled when libc_malloc_debug.so is preloaded using LD_PRELOAD. Finally, the __morecore, __morecore_after_hook and __default_morecore hooks have also been moved to compat symbols and removed from the API. Existing applications will continue to link to them but they won't have any effect on malloc behaviour. Testing: The patchset has gone through a full build and test on x86_64, i686, s390x, ppc64le, armv7l and aarch64 through a scratch build on Fedora rawhide. I also installed a test package on x86_64 and rebooted to make sure the system boots to shell. I did a build-many-glibcs run with the patchset and it ran clean on all targets. Changes from v8: - Make hook-dependent tests conditional on GLIBC_2.23 instead of GLIBC_2.24 - Interpose reallocarray. Changes from v7: - Made mcheck tests conditional on GLIBC_2.24 - Added SHLIB_COMPAT guard around __malloc_initialize_hook usage in libc_malloc_debug.so - Fixed hurd build failure. - Add another patch to fix malloc_usable_size to mcheck Changes from v6: - Moved malloc-check into libc_malloc_debug.so. Tweaked malloc.c to allow building twice, once inside libc.so and next in libc_malloc_debug.so - Included morecore.c into malloc.c - Moved hook initialization too into libc_malloc_debug.so. - Interposed more functions in libc_malloc_debug.so - Versioned all symbols exported from libc_malloc_debug.so and finalized them so that the library cannot be linked against Changes from v5: - Rebased on latest master - Fixed realloc-mcheck - Removed residual mention of libmalloc_compathooks - Removed LD_PRELOAD from elf/tst-setuid Changes from v4: - Patchset has a different approach, starting with moving out hooks first to restrict all major malloc.c changes to the first patch - Renamed libmalloc_compathooks.so to libc_malloc_debug.so - Moved all debugging features into libc_malloc_debug.so - Made more documentation updates - Simplified __malloc_initialized variable use - Removed debugging tests on static variables since that is no longer supported Changes from v3: - Remove source file dependencies - Commit mcheck tests Changes from v2: - Move hooks dependencies to malloc.o{,sS} Changes from v1: - Added makefile dependencies for the new hooks files - Fixed memset call in calloc debugging hooks - Added the tr_break deprecation patch and mcheck test patch to this series Siddhesh Poyarekar (10): Make mcheck tests conditional on GLIBC_2.23 or earlier Remove __after_morecore_hook Remove __morecore and __default_morecore Move malloc hooks into a compat DSO mcheck: Wean away from malloc hooks [BZ #23489] Simplify __malloc_initialized mtrace: Wean away from malloc hooks glibc.malloc.check: Wean away from malloc hooks Remove malloc hooks [BZ #23328] mcheck Fix malloc_usable_size [BZ #22057] NEWS | 21 + Rules | 9 +- catgets/Makefile | 4 +- elf/Makefile | 15 +- elf/tst-leaks1-static.c | 1 - iconvdata/Makefile | 3 +- include/malloc.h | 6 - include/mcheck.h | 4 - include/stdlib.h | 3 - intl/tst-gettext.sh | 1 + libio/Makefile | 12 +- localedata/Makefile | 3 +- malloc/Makefile | 58 +- malloc/Versions | 44 ++ malloc/arena.c | 48 +- malloc/hooks.c | 71 ++- malloc/malloc-check.c | 63 ++- malloc/malloc-debug.c | 535 ++++++++++++++++++ malloc/malloc-hooks.h | 24 - malloc/malloc.c | 163 ++---- malloc/malloc.h | 27 - malloc/mcheck-impl.c | 412 ++++++++++++++ malloc/mcheck.c | 397 +------------ malloc/morecore.c | 34 +- malloc/mtrace-impl.c | 226 ++++++++ malloc/mtrace.c | 313 +--------- malloc/tst-compathooks-off.c | 145 +++++ malloc/tst-compathooks-on.c | 2 + malloc/tst-malloc-usable-static-tunables.c | 1 - malloc/tst-malloc-usable-static.c | 1 - malloc/tst-mtrace.sh | 1 + manual/memory.texi | 207 +------ manual/tunables.texi | 4 +- misc/Makefile | 6 +- nptl/Makefile | 3 +- posix/Makefile | 48 +- resolv/Makefile | 9 +- shlib-versions | 3 + stdio-common/Makefile | 15 +- sysdeps/aarch64/Makefile | 3 + sysdeps/generic/libc_malloc_debug.abilist | 0 sysdeps/generic/localplt.data | 1 - sysdeps/mach/hurd/Makefile | 1 + .../mach/hurd/i386/libc_malloc_debug.abilist | 26 + sysdeps/mach/hurd/i386/localplt.data | 1 - sysdeps/pthread/Makefile | 3 +- .../linux/aarch64/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/aarch64/localplt.data | 1 - .../linux/alpha/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/alpha/localplt.data | 1 - .../sysv/linux/arc/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/arc/localplt.data | 1 - .../linux/arm/be/libc_malloc_debug.abilist | 26 + .../linux/arm/le/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/arm/localplt.data | 1 - .../sysv/linux/csky/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/csky/localplt.data | 1 - .../sysv/linux/hppa/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/hppa/localplt.data | 1 - sysdeps/unix/sysv/linux/hppa/shlib-versions | 2 + .../sysv/linux/i386/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/i386/localplt.data | 1 - .../sysv/linux/ia64/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/ia64/localplt.data | 1 - sysdeps/unix/sysv/linux/ia64/shlib-versions | 2 + .../m68k/coldfire/libc_malloc_debug.abilist | 26 + .../sysv/linux/m68k/coldfire/localplt.data | 1 - .../m68k/m680x0/libc_malloc_debug.abilist | 26 + .../unix/sysv/linux/m68k/m680x0/localplt.data | 1 - .../microblaze/be/libc_malloc_debug.abilist | 26 + .../microblaze/le/libc_malloc_debug.abilist | 26 + .../unix/sysv/linux/microblaze/localplt.data | 1 - .../mips/mips32/fpu/libc_malloc_debug.abilist | 26 + .../mips32/nofpu/libc_malloc_debug.abilist | 26 + .../mips/mips64/n32/libc_malloc_debug.abilist | 26 + .../mips/mips64/n64/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/mips/shlib-versions | 2 + .../linux/nios2/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/nios2/localplt.data | 1 - .../powerpc32/fpu/libc_malloc_debug.abilist | 26 + .../linux/powerpc/powerpc32/fpu/localplt.data | 1 - .../powerpc32/nofpu/libc_malloc_debug.abilist | 26 + .../powerpc/powerpc32/nofpu/localplt.data | 1 - .../powerpc64/be/libc_malloc_debug.abilist | 26 + .../powerpc64/le/libc_malloc_debug.abilist | 26 + .../linux/powerpc/powerpc64/localplt.data | 1 - sysdeps/unix/sysv/linux/riscv/localplt.data | 1 - .../riscv/rv32/libc_malloc_debug.abilist | 26 + .../riscv/rv64/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/s390/localplt.data | 1 - .../s390/s390-32/libc_malloc_debug.abilist | 26 + .../s390/s390-64/libc_malloc_debug.abilist | 26 + .../linux/sh/be/libc_malloc_debug.abilist | 26 + .../linux/sh/le/libc_malloc_debug.abilist | 26 + sysdeps/unix/sysv/linux/sh/localplt.data | 1 - sysdeps/unix/sysv/linux/sh/shlib-versions | 1 + .../sparc/sparc32/libc_malloc_debug.abilist | 26 + .../sysv/linux/sparc/sparc32/localplt.data | 1 - .../sparc/sparc64/libc_malloc_debug.abilist | 26 + .../sysv/linux/sparc/sparc64/localplt.data | 1 - .../sysv/linux/sparc/sparc64/shlib-versions | 1 + .../linux/x86_64/64/libc_malloc_debug.abilist | 26 + .../x86_64/x32/libc_malloc_debug.abilist | 26 + sysdeps/x86_64/localplt.data | 1 - 104 files changed, 2571 insertions(+), 1267 deletions(-) delete mode 100644 elf/tst-leaks1-static.c create mode 100644 malloc/malloc-debug.c delete mode 100644 malloc/malloc-hooks.h create mode 100644 malloc/mcheck-impl.c create mode 100644 malloc/mtrace-impl.c create mode 100644 malloc/tst-compathooks-off.c create mode 100644 malloc/tst-compathooks-on.c delete mode 100644 malloc/tst-malloc-usable-static-tunables.c delete mode 100644 malloc/tst-malloc-usable-static.c create mode 100644 sysdeps/generic/libc_malloc_debug.abilist create mode 100644 sysdeps/mach/hurd/i386/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/aarch64/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/alpha/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/arc/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/arm/be/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/arm/le/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/csky/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/hppa/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/i386/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/ia64/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/m68k/coldfire/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/m68k/m680x0/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/microblaze/be/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/microblaze/le/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/mips/mips32/fpu/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/mips/mips32/nofpu/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/mips/mips64/n32/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/mips/mips64/n64/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/nios2/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/powerpc/powerpc32/fpu/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/powerpc/powerpc32/nofpu/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/powerpc/powerpc64/be/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/powerpc/powerpc64/le/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/riscv/rv32/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/riscv/rv64/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/s390/s390-32/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/s390/s390-64/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/sh/be/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/sh/le/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/sparc/sparc32/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/sparc/sparc64/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/x86_64/64/libc_malloc_debug.abilist create mode 100644 sysdeps/unix/sysv/linux/x86_64/x32/libc_malloc_debug.abilist
Comments
This series break old Emacs binaries (e.g. emacs-24.3-23.el7.x86_64): $ bash testrun.sh /usr/bin/emacs Fatal error 11: Segmentation fault Backtrace: /usr/bin/emacs[0x4f8653] /usr/bin/emacs[0x4ddc81] /usr/bin/emacs[0x4f70ee] /usr/bin/emacs[0x4f7273] ./libc.so.6(+0x3cd80)[0x7f88c1a14d80] ./libc.so.6(+0x90c21)[0x7f88c1a68c21] ./libc.so.6(+0x94011)[0x7f88c1a6c011] ./libc.so.6(realloc+0x291)[0x7f88c1a6cea1] /usr/bin/emacs[0x536ef2] /usr/bin/emacs(re_compile_pattern+0x9cc)[0x53074c] /usr/bin/emacs[0x5242c7] /usr/bin/emacs[0x524d69] /usr/bin/emacs[0x526259] /usr/bin/emacs[0x5264bc] /usr/bin/emacs[0x551b87] /usr/bin/emacs[0x586c0b] /usr/bin/emacs[0x55171f] /usr/bin/emacs[0x551a2b] /usr/bin/emacs[0x586c0b] /usr/bin/emacs[0x55171f] /usr/bin/emacs[0x551a2b] /usr/bin/emacs[0x586c0b] /usr/bin/emacs[0x55171f] /usr/bin/emacs[0x551a2b] /usr/bin/emacs[0x551dcf] /usr/bin/emacs[0x51243c] /usr/bin/emacs[0x551b6e] /usr/bin/emacs[0x586c0b] /usr/bin/emacs[0x55171f] /usr/bin/emacs[0x551a2b] /usr/bin/emacs[0x586c0b] /usr/bin/emacs[0x55171f] /usr/bin/emacs[0x551a2b] /usr/bin/emacs[0x586c0b] /usr/bin/emacs[0x551a2b] /usr/bin/emacs[0x586c0b] /usr/bin/emacs[0x550b3d] /usr/bin/emacs[0x550f02] /usr/bin/emacs[0x5544ad] /usr/bin/emacs[0x55002a] /usr/bin/emacs[0x4de186] ... Segmentation fault It is customary to call malloc_set_state from __malloc_initialize_hook. If we no longer call __malloc_initialize_hook, we probably should change malloc_set_state to provide a better error message (via __libc_fatal). Thanks, Florian
On 7/19/21 4:37 PM, Florian Weimer via Libc-alpha wrote: > This series break old Emacs binaries (e.g. emacs-24.3-23.el7.x86_64): > > $ bash testrun.sh /usr/bin/emacs Do they work if you run with LD_PRELOAD=malloc/libc_malloc_debug.so? Siddhesh
* Siddhesh Poyarekar: > On 7/19/21 4:37 PM, Florian Weimer via Libc-alpha wrote: >> This series break old Emacs binaries (e.g. emacs-24.3-23.el7.x86_64): >> $ bash testrun.sh /usr/bin/emacs > > Do they work if you run with LD_PRELOAD=malloc/libc_malloc_debug.so? Basic testing suggests it works. Thanks, Florian
On 7/19/21 4:41 PM, Florian Weimer wrote: > * Siddhesh Poyarekar: > >> On 7/19/21 4:37 PM, Florian Weimer via Libc-alpha wrote: >>> This series break old Emacs binaries (e.g. emacs-24.3-23.el7.x86_64): >>> $ bash testrun.sh /usr/bin/emacs >> >> Do they work if you run with LD_PRELOAD=malloc/libc_malloc_debug.so? > > Basic testing suggests it works. The thing is, set_malloc_state will just not be called without the preload, so it doesn't matter what it returns. What we need to make a more graceful exit is to detect the existence of a non-NULL __malloc_initialize_hook within libc.so and abort. I'm not sure if it's worth the effort. Siddhesh
* Siddhesh Poyarekar: > On 7/19/21 4:41 PM, Florian Weimer wrote: >> * Siddhesh Poyarekar: >> >>> On 7/19/21 4:37 PM, Florian Weimer via Libc-alpha wrote: >>>> This series break old Emacs binaries (e.g. emacs-24.3-23.el7.x86_64): >>>> $ bash testrun.sh /usr/bin/emacs >>> >>> Do they work if you run with LD_PRELOAD=malloc/libc_malloc_debug.so? >> Basic testing suggests it works. > > The thing is, set_malloc_state will just not be called without the > preload, so it doesn't matter what it returns. What we need to make a > more graceful exit is to detect the existence of a non-NULL > __malloc_initialize_hook within libc.so and abort. I'm not sure if > it's worth the effort. We could remove both symbols from libc.so.6. That would result in error that can be found with search engines. Thanks, Florian
On 7/19/21 4:50 PM, Florian Weimer via Libc-alpha wrote: >> The thing is, set_malloc_state will just not be called without the >> preload, so it doesn't matter what it returns. What we need to make a >> more graceful exit is to detect the existence of a non-NULL >> __malloc_initialize_hook within libc.so and abort. I'm not sure if >> it's worth the effort. > > We could remove both symbols from libc.so.6. That would result in error > that can be found with search engines. __malloc_initialize_hook is defined in the executable so we don't control that, but we could remove malloc_set_state (and maybe also malloc_get_state to be consistent?) and keep it only in libc_malloc_debug.so. May I do it on top of this set though? I'd like to avoid respinning this set if I could. Siddhesh