Message ID | 20240315122423.1963285-1-hjl.tools@gmail.com |
---|---|
Headers |
Return-Path: <binutils-bounces+patchwork=sourceware.org@sourceware.org> X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8D66438582B1 for <patchwork@sourceware.org>; Fri, 15 Mar 2024 12:26:58 +0000 (GMT) X-Original-To: binutils@sourceware.org Delivered-To: binutils@sourceware.org Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by sourceware.org (Postfix) with ESMTPS id 7F9603857C52 for <binutils@sourceware.org>; Fri, 15 Mar 2024 12:24:28 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 7F9603857C52 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 7F9603857C52 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2607:f8b0:4864:20::62d ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710505476; cv=none; b=AVDGTAjidDuCk/V608NlSvErjWlG/nM6kLIuu3bMf/RUjCkFBMzIVL3vTK4C1X6F1Whg1bC2vlzTSG1S/64mvZD7ekKrLIzTb2C1a+U6DqerGSziqHSDaOSn2DfmyeJ6D4CQ/tmSOtQ+KDq5jnaJ+NN/9RCyGju86kl1ljJ3AjU= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1710505476; c=relaxed/simple; bh=iDfgIkeU1HrLhAls4ZeIl2dSp85O4WaW5MMF4KDxexM=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=VWzEyD/qXUh4uS40pw32CEXvtQldRZQmPKyFjwaBQb8my1TmF4ugBkrdMiB6w1J/1IuXN6Uf0Edl6I4Uq2C6lmlHPXzj+2vOavfNZQlCp7BG0Y2w3kulm/piGzD/FKmaQCOqLyNuxfzwXXEgi2htTuTTlhZTXU9fGjEfsV2HtfU= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1dddbe47ac1so15868425ad.1 for <binutils@sourceware.org>; Fri, 15 Mar 2024 05:24:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710505467; x=1711110267; darn=sourceware.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=+BS8ydFgUfsIlKx3QmtBb0D3PNiun0njvPwpi/X9LKM=; b=MOqXqRYZwF30hCbjC/K+w2Mrcz9ENthZTGDgXOzw7lIjq35wI2Uzf9G87hxVkrxfG2 IiOT/q/VYOvfAAWAuvNWQnF3STawVparjcKSjzz0dE2ErqGkpbuCMay9rF8YjgY59O4i FcxAHr6dF/Zn77AVQQLSK005fee9MNeu0quCpZjsL3BtqRwz131PNu40paS7WFDWS7yy A/uPBgmLBZG0vguU+hmZXA1ywUQZcBwTwvfjGllwDuBigX/uZl7a1DAmRW3Scf5/pPpG 1nrH/szhj+1P9oQSAwvIsrf0A+DgS4+uwwQ5TE5DzDK3Ur0xTLdOBKltHQELr9+6Rd4Q Hp8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710505467; x=1711110267; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+BS8ydFgUfsIlKx3QmtBb0D3PNiun0njvPwpi/X9LKM=; b=FokCN87aODuEI8ZE8RYl0fpBb+MGwF+y/CHAfgEvopw+8K7Emg9aX+zUKvWJqVGQYN 6yQlsZjbUF0Va6LE/JSzsA6gRHazkwm69Qz/m6OcqMGlY5D/fbGXT+y6/LcclSHI7HK1 CJf6phBuZn7kFsbXKJCbXZQSVer2sl3gJSIajRCBBZy44xp/+HXMcgWZTf6Yp3xrI0cK 1eWmjAF3snqa+k6+T66t7ytRe7gk+ZcMaoJodNaVg2FSMlKBSZN0gYP1HO/Q79WFWv3k Vw12xe1svMHdPld/a3vV5nHLj6DFVGm65LVxbvBRnrzKKfA6nzUdqBkpEW6P6VYBVOUi +pog== X-Gm-Message-State: AOJu0YxaPowIvjNZGcuNDg+e1hbPfzS2Z6850kHwn6MjiusbEiRlaC2C NbmJylQZHPimhqtOJDwS6b96fKkgFSz9FA1eiZ/jnuk/MFF/ECCE X-Google-Smtp-Source: AGHT+IGBCpKcA1q1PRHnpKDd7FzN+BYYxiYgxQo28T++66Rac06kugPV7P8pti6oXK2FX0vbeuOtJQ== X-Received: by 2002:a17:902:a518:b0:1dc:b323:6f97 with SMTP id s24-20020a170902a51800b001dcb3236f97mr10038474plq.27.1710505467517; Fri, 15 Mar 2024 05:24:27 -0700 (PDT) Received: from gnu-cfl-3.localdomain ([172.58.89.72]) by smtp.gmail.com with ESMTPSA id u9-20020a170902bf4900b001deeac592absm1842105pls.180.2024.03.15.05.24.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Mar 2024 05:24:24 -0700 (PDT) Received: from gnu-cfl-3.. (localhost [IPv6:::1]) by gnu-cfl-3.localdomain (Postfix) with ESMTP id 7A91E740150; Fri, 15 Mar 2024 05:24:23 -0700 (PDT) From: "H.J. Lu" <hjl.tools@gmail.com> To: binutils@sourceware.org Cc: goldstein.w.n@gmail.com, sam@gentoo.org, amodra@gmail.com Subject: [PATCH v11 0/6] elf: Use mmap to map in section contents Date: Fri, 15 Mar 2024 05:24:17 -0700 Message-ID: <20240315122423.1963285-1-hjl.tools@gmail.com> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-3012.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, RCVD_IN_SBL_CSS, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: binutils@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Binutils mailing list <binutils.sourceware.org> List-Unsubscribe: <https://sourceware.org/mailman/options/binutils>, <mailto:binutils-request@sourceware.org?subject=unsubscribe> List-Archive: <https://sourceware.org/pipermail/binutils/> List-Post: <mailto:binutils@sourceware.org> List-Help: <mailto:binutils-request@sourceware.org?subject=help> List-Subscribe: <https://sourceware.org/mailman/listinfo/binutils>, <mailto:binutils-request@sourceware.org?subject=subscribe> Errors-To: binutils-bounces+patchwork=sourceware.org@sourceware.org |
Series |
elf: Use mmap to map in section contents
|
|
Message
H.J. Lu
March 15, 2024, 12:24 p.m. UTC
Changes in v11: 1. Update _bfd_mmap_read_temporary to allocate buffer as needed. 2. Don't allocate buffer when _bfd_mmap_read_temporary is called. Changes in v10: 1. Malloc a 4 * page size buffer for external relocations for the final link to avoid mmap/munmap on small relocation sections. 2. Malloc a buffer for input section contents which are smaller than 4 * page size or can't be mapped for the final link. Changes in v9: 1. Use MAP_FAILED for mmap failure. Changes in v8: 1. Rebase against master branch. 2. Add _bfd_elf_link_mmap_section_contents and _bfd_elf_link_munmap_section_contents. Changes in v7: 1. Don't add the --keep-memory linker option. Changes in v6: 1. Add the --keep-memory linker option and always cache symbol and relocation tables for --keep-memory. 2. Always keep symbol table and relocation info for eh_frame to speedup the Rust binary build by ~300x: https://sourceware.org/bugzilla/show_bug.cgi?id=31466 Changes in v5: 1. Drop 2 patches which have been merged onto master branch. 2. Rename _bfd_elf_mmap_section to _bfd_elf_mmap_section_contents. 3. Rename _bfd_mmap_readonly_tracked, _bfd_mmap_readonly_untracked, _bfd_munmap_readonly_untracked, _bfd_mmap_read_untracked to _bfd_mmap_readonly_persistent, _bfd_mmap_readonly_temporary, _bfd_munmap_readonly_temporary and _bfd_mmap_read_temporary. 4. Drop the setup_group change. 5. Fix a typo. 6. Update comments. Changes in v4: 1. Change don't cache symbol nor relocation tables with mmap to opt-in. Changes in v3: 1. Fix non-mmap build. 2. Change the argument name of bfd_mmap_local from flags to prot since its values are PROT_XXX. Changes in v2: 1. Don't hard-code BFD_JUMP_TABLE_COPY in bfd so that elf-bfd.h can be included in libbfd.c. 2. Change the --with-mmap default to true. 3. Check USE_MMAP instead of HAVE_MMAP. 4. Remove the asize parameter to _bfd_mmap_readonly_tracked. 5. Add contents_addr and contents_size to bfd_elf_section_data. 6. Rename _bfd_link_keep_memory to _bfd_elf_link_keep_memory. --- We can use mmap to map in ELF section contents, instead of copying them into memory by hand. We don't need to cache symbol nor relocation tables if they are mapped in. Data to link the 3.5GB clang executable in LLVM 17 debug build on Linux/x86-64 with 32GB RAM is: stdio mmap improvement user 86.73 87.02 -0.3% system 9.55 9.21 3.6% total 100.40 97.66 0.7% maximum set(GB) 17.34 13.14 24% page faults 4047667 3042877 25% and data to link the 275M cc1plus executable in GCC 14 stage 1 build is: user 5.41 5.44 -0.5% system 0.80 0.76 5% total 6.25 6.26 -0.2% maximum set(MB) 1323 968 27% page faults 323451 236371 27% Data shows that these won't improve the single copy linker performance. But they improve the overall system performance when linker is used by reducing linker memory usage and page faults. They allow more parallel linker jobs on LLVM debug build. Here is a quote from Noah Goldstein: "on a large project they are an extremely large speedup". H.J. Lu (6): elf: Use mmap to map in read-only sections elf: Add _bfd_elf_m[un]map_section_contents elf: Use mmap to map in symbol and relocation tables elf: Don't cache symbol nor relocation tables with mmap elf: Always keep symbol table and relocation info for eh_frame elf: Add _bfd_elf_link_m[un]map_section_contents bfd/bfd-in2.h | 24 ++++- bfd/bfd.c | 17 +++ bfd/bfdwin.c | 8 +- bfd/cache.c | 7 +- bfd/compress.c | 2 +- bfd/elf-bfd.h | 27 +++++ bfd/elf-eh-frame.c | 4 +- bfd/elf-sframe.c | 4 +- bfd/elf.c | 246 ++++++++++++++++++++++++++++++++++-------- bfd/elf32-i386.c | 8 +- bfd/elf64-x86-64.c | 12 +-- bfd/elfcode.h | 7 +- bfd/elflink.c | 213 ++++++++++++++++++++++++------------ bfd/elfxx-target.h | 6 +- bfd/elfxx-x86.c | 8 +- bfd/elfxx-x86.h | 1 + bfd/libbfd-in.h | 33 +++++- bfd/libbfd.c | 262 ++++++++++++++++++++++++++++++++++++++++++++- bfd/libbfd.h | 33 +++++- bfd/linker.c | 35 ------ bfd/lynx-core.c | 2 +- bfd/opncls.c | 21 ++++ bfd/section.c | 9 +- bfd/sysdep.h | 4 + 24 files changed, 798 insertions(+), 195 deletions(-)