From patchwork Fri Sep 20 12:19:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dimitri John Ledkov X-Patchwork-Id: 57467 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A64AB3858C3A for ; Fri, 20 Sep 2024 12:21:02 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by sourceware.org (Postfix) with ESMTPS id 340D03858D35 for ; Fri, 20 Sep 2024 12:19:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 340D03858D35 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=surgut.co.uk Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=surgut.co.uk ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 340D03858D35 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::42e ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1726834764; cv=none; b=CV0khEFkPoUOtT/vbIfL6T+oBuOK8u4yMV8ABS2aBWJ88XMZYaGzq8KkemC/MGZj8MyBBZGLHdE+w7S9eevlvmiS/k0AFeHQ9qSE4nEqfhXkK/4Rd5285ioAdUvbxN/8H7gDlGFzIFfzbPRhnx/gPAqPKmaosaRtQQT5X+Uey6o= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1726834764; c=relaxed/simple; bh=TQwmHcePVkPFmiAO6XdgIKacQz7xRYNAysYmUBZdjIY=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=ZNmmONhYDXSFJLJRreO9dwNPG+gOafvO1NuqymlmprsNHKrj5ouENtywIle0hoRnD0Quwa7ir09mjjjoHor5cmdU5prWLZ623oBThgEdjEo6zsLbqu1pPrXJrwtIGQnLtpnMqO1Sk3uc3DIsAR/rg3XS1ZGztw4pSND3AnqPuX8= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-374c8cef906so1431857f8f.2 for ; Fri, 20 Sep 2024 05:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=surgut.co.uk; s=google; t=1726834760; x=1727439560; darn=gcc.gnu.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=0xcMGUIYf1DSvSccEcD7qrLm8LrABfitJbtDz1bxIl0=; b=LAwxGERyEriJfOxsMKgSmnAPnhprbZPUWF3HrzpakoyWkhlbtk9Z6wQsGH/2Zrz0rs HX6xPQ7kzQl3AsGIKBnDl6FdQiIErrWdC3z5YjCAJAYEOKBC2dtWWFZUxufDy6TKKrg7 NPrBfTnVRZa13FxS20YtNv+OW8rJ9hP2zq29Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726834760; x=1727439560; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0xcMGUIYf1DSvSccEcD7qrLm8LrABfitJbtDz1bxIl0=; b=Ks6JzkwiQyu6qIq2BXhmgCY2Qud1e1J2rOZaJCMkuQfuAfWckvNQ/Pc8wL3CipnFTk nH5cMh77mAikrO3nEVwHvr1jG2rqV2hc/YAtmlzeXqsp2fwfKOmni/QtJmve4F5YwjsM PBkeCZy83lhgXjSQiV8gikI4eU647I0n0Lq83npqODxaU8gncqRZ4ohFruzbZrhBory7 iuWz79bhG3RXavdDISHAo5iqAqsHv607XShDw904aewtK3ZucCL4hGOQo1W1vCD7tbc0 baIR8xNDfjS6d4XxBmAylmEyLje9Ynt0bV19hfB7h/PhCHWkgX9bfntMcwcgdTbJ46P+ 1T9Q== X-Gm-Message-State: AOJu0YyDcboGfBPiVOOt6aqpY+f5oFZt9fGELSQhKczZ1IBw8n220ycA +u2NFTNd01dnoAkRK7mgXTba1aEX4P/s7aWK+bx8Yte+2zX8yjNQ3wpBwB2MaigLrzZKMkCOHEr 66G08DjXL X-Google-Smtp-Source: AGHT+IGtr7Upl8K3Gp22thQGuZeFhK7/iA8psXeA4yJLmZjJR6a+UnQ+NxB+s5xVkF7FoqxHkZwMfw== X-Received: by 2002:adf:f48f:0:b0:374:c8e5:d56a with SMTP id ffacd0b85a97d-37a431953admr1493322f8f.48.1726834760263; Fri, 20 Sep 2024 05:19:20 -0700 (PDT) Received: from chainguard.. ([2a01:4b00:85fd:d700:d4a1:48d7:e6c4:e84b]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-378e73f97cbsm17356250f8f.50.2024.09.20.05.19.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Sep 2024 05:19:19 -0700 (PDT) From: Dimitri John Ledkov To: gcc-patches@gcc.gnu.org Cc: Dimitri John Ledkov Subject: [PATCH 0/2] Enable supplementing built-in specs without command-line arguments Date: Fri, 20 Sep 2024 13:19:17 +0100 Message-ID: <20240920121919.65112-1-dimitri.ledkov@surgut.co.uk> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, KAM_COUK, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~patchwork=sourceware.org@gcc.gnu.org Today the following logic is in-place for loading spec files: * Discover /specs ** If available, load it ** Else generate builtin specs (equivalent of -dumpspecs) ** `%include`, `%include_noerr`, `%rename` not allowed * If `-specs=` passed, load them in order ** `%include`, `%include_noerr`, `%rename` not allowed This asymmetry results in very difficult choices, and experience when one tries to modify default spec files, and does not want to leak them via build-systems, or wants to ensure they are always in-effect. It leads to distributors rebuilding toolchains just to update a trivial single `--with-specs` configuration time option, as performing `-dumpspecs`, patching that, and shipping alternative .specs like that is cumbersome and error-prone. Also, having more flexible specs that can be loaded by default, would help producing context-aware build-flags whilst ensuring working builds in other contexts. For example consider an ELF Package Metadata notes spec file ``` *link: + --package-metadata={\"type\":\"rpm\",\"name\":\"%:getenv(RPM_PACKAGE_NAME \",\"version\":\"%:getenv(RPM_PACKAGE_VERSION -%:getenv(RPM_PACKAGE_RELEASE \",\"architecture\":\"%:getenv(RPM_ARCH \",\"osCpe\":\"@OSCPE@\"})))) ``` If specified in CFLAGS to build systems as `-specs=redhat-package-notes`, it can leak from build environment into the package-config files and extensions build systems. Then later when end-users build code outside of RPM build environment would lead to errors. See this bug report, that states that a different implementation of similar flags got enabled as default LDFLAGS, they leaked into another language compiled extensions ecosystems, resulting in build-failures or incorrect-context-unaware metadata. See [RHB#2043092](https://bugzilla.redhat.com/show_bug.cgi?id=2043092) Wouldn't it be nice if one could instead specify in the /specs ``` %include_noerr ``` With expectation that default built-in specs are still loaded as usual, and only then the `specs` file processed, with ability to use `%include_noerr`, and dynamically inject build context aware specs. For example Dak/Koji/OBS/Launchpad can then generate highly context aware build flags, to set -march, build notes, hardening options to truly take effect above all, despite all the build-systems in their way. But even simple things like '-g0' to actually not even generate debug symbols when not needed, or inverse force generating them to then split them into debug symbols. A simple change like this would universally speed up archive test rebuilds. Whilst ELF package notes are used in this example, other useful things can be injected like these. For example, distribution rebuilds for various -march levels, OpenSSF hardening guide flags, custom toolchain plugins, etc. Anecdotally, many distributions rebuild GCC multiple times just to change default mtune/march alone or set a single "--with-specs=". Such rebuilds could be avoided and made cheaper to implement by shipping multiple packages with `specs.overlay` that does just that. Further more such behavior would actually match the documentation of the [Spec files] and all existing guides on how to do SPECS. I.e. "use %rename, redefine your command, or use + to append to existing one" and so on. [Spec files]: https://gcc.gnu.org/onlinedocs/gcc/Spec-Files.html The current practice of prohibiting '%include_noerr' in /specs, and requiring it to be a fully flat one (output of -dumpspecs) is very restrictive to the point of not being used at all. I would like to propose a couple of design choices to address above limitations, either with or without breaking existing behaviour, and also update documentation to specify all of this to prevent pit falls. Just before loading user specified `-specs=`, attempt to lookup `/specs.overlay`, and load it with "%include" commands allowed and pretend it was user specified one, as the first "-specs=". Given file is an existing internal file api, that has been driven to the ground of being empty, and really to modify (whilst maintaining all features compatibility with -dumpspecs equivalent). Modify the current load behavior to this: 1) Always load built-in generated specs 2) Then load /specs if found, relax permissions and allow it to use "%include" commands, potentially move it closer to when user specs are loaded. Impact of behaviour change is small but not zero. By default specs does not exist, thus default behaviour remains the same. All existing /specs that are full equivalents of `-dumpspecs` will continue to result in identical configuration. Incomplete `-dumpspecs` which either didn't explicitly delete spec_name by setting it to empty, may potentially have different behavior if the default '-dumpspecs' sets them to something. If this proposal is accepted, there is no need to create a brand-new file api "specs.overlay". But behavior of /specs will be widely different across GCC releases, which may cause confusion if this changes makes "specs" files popular again. Today in the [Spec files] documentation it is not mentioned that when creating a custom Spec file, and testing it with `-specs` command, will result in widely different behavior if moved to /specs location. As right now it is not mentioned that '%include" commands are blocked, and that the built-in specs are not loaded, meaning '%rename' and '+' append functionality effectively cannot be used. But only want to update documentation for all supported gcc series, once there is consensus on any of the above proposals. [Spec files]: https://gcc.gnu.org/onlinedocs/gcc/Spec-Files.html [Clang config files] passed on commandline, work identical to the automatically discovered ones. On commandline they are passed with `--config`. The configuration files language is more simple one. It is a flat text file of comments, or literal command-line options appended. There is one `@file` directive to include another file. Automatic configuration file discovery uses compiler name and target arch to discover and load a matching configuration file. For example `x86_64-pc-linux-gnu-clang-g++`, tries to load `x86_64-pc-linux-gnu-clang++.cfg`, or `x86_64-pc-linux-gnu-clang-g++.cfg`, or `clang++.cfg` [Clang config files]: https://clang.llvm.org/docs/UsersManual.html#configuration-files In addition to toolchain internal locations (typically under `/usr/lib`), system-wide locations are also typically searched by Clang, i.e. `/etc/clang-18`. An equivalent `/etc/gcc` spec files discovery location does not exist today, but also not sure if it is needed. Distributions can use symlinks, and dynamically generated spec files with relevant includes as needed using their usual means to manage such things (update-alternatives, dpkg-reconfigure, etc). Equivalent to `--config` in GCC Spec file terms would be a `.cfg` file that automatically injects all strings into `*self_spec: +` lines, and converts `@file` includes into `#include_noerr` commands. Shockingly enough I did set /specs to ``` *self_spec: + %{!O:%{!O1:%{!O2:%{!O3:%{!O0:%{!Os:%{!0fast:%{!0g:%{!0z:-O2}}}}}}}}} -fhardened %{!Wall:%{!Wno-all:-Wall}} %{!Wformat=0:-Wformat=2} %{!Wconversion:%{!Wno-conversion:-Wconversion}} %{!Wimplicit-fallthrough=0:-Wimplicit-fallthrough=3} -Wl,--as-needed,-O1,--sort-common,-z,noexecstack,-z,relro,-z,now -fno-delete-null-pointer-checks -fno-strict-overflow -fno-strict-aliasing -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -mDO_NOT_COPY_AND_PASTE_INTO_SPECS ``` and that actually manages to compile a huge number of C/C++ software with common build systems, despite removing linking command and thus removing `-lgcc_s` from the link command and actually miss-building lots of things. (Intentionally added invalid arg to prevent others making the same mistake with a blind copy & paste). Dimitri John Ledkov (2): specs: load specs.overlay with all commands enabled specs: always generate built-in specs, before reading "specs" file gcc/gcc.cc | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)