From patchwork Fri Jul 7 11:42:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dodji Seketeli X-Patchwork-Id: 72290 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8C19539012B0 for ; Fri, 7 Jul 2023 12:11:07 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 8C19539012B0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1688731867; bh=69HsQ1HQz9VyD+oRWTp2++pwfKWmyNyeRvZpSm2i1W8=; h=To:Cc:Subject:References:Date:In-Reply-To:List-Id: List-Unsubscribe:List-Archive:List-Help:List-Subscribe:From: Reply-To:From; b=n7my02keYfA1TWjXqkITeGBuxD43yXRWbMtUgxzhdd2mFSNSXk3wDu58kbzYuUDwv qGopCLJ6pBCC1ZVhgKgP1gzpZdb+C8Afyw6QcKJ/DsqVeKNbXGGcttmYywg3OvLZ6r RomP8ITmwK7eX3lypo5Lj4Is5UGyETZFAs9TnCao= X-Original-To: libabigail@sourceware.org Delivered-To: libabigail@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 25C003852AF5 for ; Fri, 7 Jul 2023 11:42:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 25C003852AF5 Received: from mail-ua1-f70.google.com (mail-ua1-f70.google.com [209.85.222.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-262-iIx13PTeMbiKzCZhrhb5DQ-1; Fri, 07 Jul 2023 07:42:07 -0400 X-MC-Unique: iIx13PTeMbiKzCZhrhb5DQ-1 Received: by mail-ua1-f70.google.com with SMTP id a1e0cc1a2514c-794c8621562so336231241.1 for ; Fri, 07 Jul 2023 04:42:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688730127; x=1691322127; h=content-transfer-encoding:mime-version:user-agent:message-id :in-reply-to:date:references:organization:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NthsoGuQ67BzT9X2toX3yf0+gCtRJe41mdasHhHiC/g=; b=QPowJvwR0cxT5c75HQu5Pc+SFjm3pGw1JXjAwz4zD9JyLf51CI7Spd6+CKFgLuoxj+ Ac+RHBqDJC0p4fEC0YJNHMrUDrWuhsa7IApU+WegVP9SWuC7Sj3EuSXEwnF8F+63TbBL 0Cnp2EexDv7Su+ym3R0ewR79RXY+4soePuX/S1OPmRZg7bdos23y4umu6qFdWmnfVXTa BHVSTHOJ3RSZckOPJZ8kkU6qY31rDkb3z2pqOAoZmlLvOsNS6ezhign6N+wMh1QgKMap APrCf9Ph9TO+4sKDrF3ZauSJCSFhSNVBDIRa3jVpglRAzto8pULpKaqzO15eO8DtVLUD ePbw== X-Gm-Message-State: ABy/qLbBDn22mMy5wCIhP5b8XfnvHayuClIvZIIrWmJkTTQrcFcxQYlW PK7xtB7jDsiA0Gqm0E7rjQ5lIVwaGX0R7/YylUXdsPgdkyt/bkop29VACowBC8TKKmU5maWa4a2 5I2A8fJP4VspFnOJaZesx X-Received: by 2002:a67:ff91:0:b0:443:8eab:c664 with SMTP id v17-20020a67ff91000000b004438eabc664mr2772181vsq.13.1688730126799; Fri, 07 Jul 2023 04:42:06 -0700 (PDT) X-Google-Smtp-Source: APBJJlFNkzkZmnQCF6Rz1yADBoRGK9//TMjAdj6zQOVUgB2+M5SdxzJCWFd4IkqRGm0XmLxs9wZKvQ== X-Received: by 2002:a67:ff91:0:b0:443:8eab:c664 with SMTP id v17-20020a67ff91000000b004438eabc664mr2772170vsq.13.1688730126318; Fri, 07 Jul 2023 04:42:06 -0700 (PDT) Received: from localhost ([88.120.130.27]) by smtp.gmail.com with ESMTPSA id v15-20020ae9e30f000000b00767b0c35c15sm731198qkf.91.2023.07.07.04.42.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Jul 2023 04:42:05 -0700 (PDT) Received: by localhost (Postfix, from userid 1000) id 5ED63B4FFD; Fri, 7 Jul 2023 13:42:04 +0200 (CEST) To: Dodji Seketeli Cc: libabigail@sourceware.org Subject: [PATCH 2/4, applied] corpus,tools-utils: Support loading a corpus, its deps & other binaries Organization: Red Hat / France References: <87edlk0x3p.fsf@redhat.com> X-Operating-System: CentOS Stream release 9 X-URL: http://www.redhat.com Date: Fri, 07 Jul 2023 13:42:04 +0200 In-Reply-To: <87edlk0x3p.fsf@redhat.com> (Dodji Seketeli's message of "Fri, 07 Jul 2023 13:39:22 +0200") Message-ID: <875y6w0wz7.fsf@redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , X-Patchwork-Original-From: Dodji Seketeli via Libabigail From: Dodji Seketeli Reply-To: Dodji Seketeli Errors-To: libabigail-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libabigail" Hello, In preparation of implementing the support for "Handling split libraries", this patch implements the functions tools_utils::{get_comma_separated_args_of_option, find_file_under_dirs, add_binaries_into_corpus_group, add_dependencies_into_corpus_group, stick_corpus_and_binaries_into_corpus_group, stick_corpus_and_dependencies_into_corpus_group, get_dependencies}. Given an ABI corpus, these functions enable adding arbitrary binaries and dependencies found in a set of directories to form a corpus group. * include/abg-corpus.h (corpus_group::has_corpus): Declare new member function. * include/abg-tools-utils.h (get_comma_separated_args_of_option) (find_file_under_dirs, get_dependencies) (add_binaries_into_corpus_group) (add_dependencies_into_corpus_group) (stick_corpus_and_binaries_into_corpus_group) (stick_corpus_and_dependencies_into_corpus_group): Declare new functions. * src/abg-corpus.cc (corpus_group::priv::corpora_path): Add new data member. (corpus_group::add_corpus): Do not add a corpus that was already added to the group. Update the set of paths of added corpora so that we can detect if a corpus has already been added. * src/abg-tools-utils.cc (find_file_under_dir): If the file found is a symbolic link, return it. Otherwise if the symbolic link is not the file we were looking for, then skip it, rather than following it, in case it's a directory. (get_comma_separated_args_of_option, find_file_under_dirs) (get_dependencies, add_binaries_into_corpus_group) (add_dependencies_into_corpus_group) (stick_corpus_and_binaries_into_corpus_group) (stick_corpus_and_dependencies_into_corpus_group): New functions. Signed-off-by: Dodji Seketeli Applied to master. --- include/abg-corpus.h | 2 + include/abg-tools-utils.h | 35 ++++++ src/abg-corpus.cc | 21 ++++ src/abg-tools-utils.cc | 257 ++++++++++++++++++++++++++++++++++++-- 4 files changed, 306 insertions(+), 9 deletions(-) diff --git a/include/abg-corpus.h b/include/abg-corpus.h index cbc7ec24..6de251de 100644 --- a/include/abg-corpus.h +++ b/include/abg-corpus.h @@ -366,6 +366,8 @@ public: void add_corpus(const corpus_sptr&); + bool has_corpus(const string&); + const corpora_type& get_corpora() const; diff --git a/include/abg-tools-utils.h b/include/abg-tools-utils.h index 930557b5..7f931c5d 100644 --- a/include/abg-tools-utils.h +++ b/include/abg-tools-utils.h @@ -71,6 +71,9 @@ bool sorted_strings_common_prefix(vector&, string&); string get_library_version_string(); string get_abixml_version_string(); bool execute_command_and_get_output(const string&, vector&); +void get_comma_separated_args_of_option(const string& input_str, + const string& option, + vector& arguments); bool get_dsos_provided_by_rpm(const string& rpm_path, set& provided_dsos); string trim_white_space(const string&); @@ -125,6 +128,38 @@ find_file_under_dir(const string& root_dir, const string& file_path_to_look_for, string& result); +bool +find_file_under_dirs(const vector& root_dirs, + const string& file_path_to_look_for, + string& result); + +bool +get_dependencies(const corpus&, const vector&, set&); + +void +add_binaries_into_corpus_group(const fe_iface_sptr& reader, + const vector& binaries, + const vector& deps_dirs, + corpus_group& group); + +void +add_dependencies_into_corpus_group(const fe_iface_sptr& reader, + const corpus& korpus, + const vector& deps_dirs, + corpus_group& group); + +corpus_group_sptr +stick_corpus_and_binaries_into_corpus_group(const fe_iface_sptr& reader, + const corpus_sptr& korpus, + const vector& binaries, + const vector& deps_dirs); + +corpus_group_sptr +stick_corpus_and_dependencies_into_corpus_group(const fe_iface_sptr& reader, + const corpus_sptr& korpus, + const vector& deps_dirs); + + class temp_file; /// Convenience typedef for a shared_ptr to @ref temp_file. diff --git a/src/abg-corpus.cc b/src/abg-corpus.cc index fa308151..7cf2d11d 100644 --- a/src/abg-corpus.cc +++ b/src/abg-corpus.cc @@ -13,6 +13,7 @@ #include #include #include +#include #include "abg-internal.h" @@ -1654,6 +1655,7 @@ operator&=(corpus::origin &l, corpus::origin r) /// Type of the private data of @ref corpus_group struct corpus_group::priv { + std::set corpora_paths; corpora_type corpora; istring_function_decl_ptr_map_type fns_map; vector fns; @@ -1743,6 +1745,10 @@ corpus_group::add_corpus(const corpus_sptr& corp) if (!corp) return; + if (!corp->get_path().empty() + && has_corpus(corp->get_path())) + return; + // Ensure the new architecture name matches the current one. string cur_arch = get_architecture_name(), corp_arch = corp->get_architecture_name(); @@ -1758,6 +1764,7 @@ corpus_group::add_corpus(const corpus_sptr& corp) priv_->corpora.push_back(corp); corp->set_group(this); + priv_->corpora_paths.insert(corp->get_path()); /// Add the unreferenced function and variable symbols of this /// corpus to the unreferenced symbols of the current corpus group. @@ -1765,6 +1772,20 @@ corpus_group::add_corpus(const corpus_sptr& corp) priv_->add_unref_var_symbols(get_unreferenced_variable_symbols()); } +/// Test if a corpus of a given path has been added to the group. +/// +/// @param path the path to the corpus to consider. +/// +/// @return true iff a corpus with path @p path is already present in +/// the groupâ‹… +bool +corpus_group::has_corpus(const string& path) +{ + if (priv_->corpora_paths.find(path) != priv_->corpora_paths.end()) + return true; + return false; +} + /// Getter of the vector of corpora held by the current @ref /// corpus_group. /// diff --git a/src/abg-tools-utils.cc b/src/abg-tools-utils.cc index 4f6fd998..5287c99e 100644 --- a/src/abg-tools-utils.cc +++ b/src/abg-tools-utils.cc @@ -1235,6 +1235,34 @@ execute_command_and_get_output(const string& cmd, vector& lines) return true; } +/// Get a vector of arguments from a string containing a +/// comma-separated list of those arguments. +/// +/// @param input_str the input string containing the comma-separated +/// list of arguments The input string has the form +/// "option=arg1,arg2,arg3,arg4". +/// +/// @param option if the content of the input string @p input_str is +/// "option=arg1,arg2,arg3", then this parameter should be "option". +/// +/// @param arguments this is set by the fonction the the arguments +/// that were a comma-separated list of arguments on the right hand +/// side of the '=' sign in the string @p input_str. +void +get_comma_separated_args_of_option(const string& input_str, + const string& option, + vector& arguments) +{ + string s = input_str; + + string_suffix(s, option, s); + if (string_begins_with(s, "\"")) + s = s.substr(1); + if (string_ends_with(s, "\"")) + s = s.substr(0, s.size() - 1); + split_string(s, ",", arguments); +} + /// Get the SONAMEs of the DSOs advertised as being "provided" by a /// given RPM. That set can be considered as being the set of /// "public" DSOs of the RPM. @@ -2363,22 +2391,233 @@ find_file_under_dir(const string& root_dir, FTSENT *entry; while ((entry = fts_read(file_hierarchy))) { + if (entry_of_file_with_name(entry, file_path_to_look_for, r)) + { + result = entry->fts_path; + return true; + } // Skip descendents of symbolic links. if (entry->fts_info == FTS_SL || entry->fts_info == FTS_SLNONE) { fts_set(file_hierarchy, entry, FTS_SKIP); continue; } - if (entry_of_file_with_name(entry, file_path_to_look_for, r)) - { - result = entry->fts_path; - return true; - } } fts_close(file_hierarchy); return false; } + +/// Find a given file possibly under a set of directories and return +/// its absolute path. +/// +/// @param root_dirs the vector of root directories under which to +/// look for. +/// +/// @param file_path_to_look_for the file to look for under the +/// directory @p root_dir. +/// +/// @param result the resulting path to @p file_path_to_look_for. +/// This is set iff the file has been found. +bool +find_file_under_dirs(const vector& root_dirs, + const string& file_path_to_look_for, + string& result) +{ + if (root_dirs.empty()) + return find_file_under_dir(".", file_path_to_look_for, result); + + for (const auto& root_dir : root_dirs) + if (find_file_under_dir(root_dir, file_path_to_look_for, result)) + return true; + + return false; +} + +/// Get the dependencies of an ABI corpus, which are found in a set of +/// directories. Note that the dependencies are listed as properties +/// of the ABI corpus. +/// +/// If the corpus has a dependency that is not found under any of the +/// given directories, then the dependency is ignored and not +/// returned. +/// +/// @param korpus the ABI corpus to consider. +/// +/// @param deps_dirs the list of directories where to look for the +/// dependencies. +/// +/// @param dependencies output parameter that is set the dependencies +/// of the corpus denoted by @p korpus which are found in the +/// directories @p deps_dirs. This is set iff the function returns +/// true. +/// +/// @return true iff some dependencies of the corpus @p korpus were +/// found in directories @p deps_dirs. +bool +get_dependencies(const corpus& korpus, + const vector& deps_dirs, + set& dependencies) +{ + const vector& set_of_needed = korpus.get_needed(); + if (set_of_needed.empty()) + return false; + + bool found_at_least_one_dependency =false; + for (const auto& n :set_of_needed) + { + string dependency; + if (dependencies.find(n) == dependencies.end() + && find_file_under_dirs(deps_dirs, n, dependency)) + { + dependencies.insert(dependency); + found_at_least_one_dependency = true; + } + } + + return found_at_least_one_dependency; +} + +/// For each binary of a vector of binaries, if the binary is present +/// in at least one of the directories listed in a given vector, +/// construct a corpus and add it to a corpus group. +/// +/// @param reader the reader used to read the binaries into an ABI corpus. +/// +/// @param binaries the vector of binaries to read and add to a corpus +/// group. +/// +/// @param deps_dirs the vector of directories where to look for the +/// binaries in @p binaries. +/// +/// @param group the corpus group to add the corpus. +void +add_binaries_into_corpus_group(const fe_iface_sptr& reader, + const vector& binaries, + const vector& deps_dirs, + corpus_group& group) +{ + vector bins; + + for (const auto& b : binaries) + { + string bin; + if (find_file_under_dirs(deps_dirs, b, bin)) + bins.push_back(bin); + } + + for (const auto& b : bins) + { + if (group.has_corpus(b)) + continue; + + reader->initialize(b); + fe_iface::status stat = fe_iface::STATUS_UNKNOWN; + corpus_sptr c = reader->read_corpus(stat); + if (c && (stat & fe_iface::STATUS_OK)) + group.add_corpus(c); + } +} + +/// For each dependency of a given corpus, if it is present in at +/// least one of the directories listed in a given vector, construct a +/// corpus and add it to a corpus group. +/// +/// @param reader the reader used to read the binaries into an ABI corpus. +/// +/// @param korpus the corpus to consider. +/// +/// @param deps_dirs the vector of directories where to look for the +/// dependencies of @p korpus. +/// +/// @param group the corpus group to add the corpus. +void +add_dependencies_into_corpus_group(const fe_iface_sptr& reader, + const corpus& korpus, + const vector& deps_dirs, + corpus_group& group) + +{ + set deps; + if (!get_dependencies(korpus, deps_dirs, deps)) + return; + + for (const auto& dep: deps) + { + if (group.has_corpus(dep)) + continue; + + reader->initialize(dep); + fe_iface::status stat = fe_iface::STATUS_UNKNOWN; + corpus_sptr c = reader->read_corpus(stat); + if (c && (stat & fe_iface::STATUS_OK)) + { + group.add_corpus(c); + add_dependencies_into_corpus_group(reader, *c, deps_dirs, group); + } + } +} + +/// Create a corpus group made of a given korpus and a set of binaries +/// found in a set of directories. +/// +/// @param reader the reader to use to read the binaries. +/// +/// @param korpus the ABI corpus to add to the corpus group. +/// +/// @param binaries the set of binaries to add to the corpus group, if +/// they are present one of the directories denoted by the vector @p +/// deps_dirs. +/// +/// @param bins_dirs the directories where the binaries listed in @p +/// binaries are to be found. +/// +/// @return a corpus group made of @p korpus and the binaries listed +/// in @p binaries and found in at least one of the directories found +/// in @p bins_dirs. +corpus_group_sptr +stick_corpus_and_binaries_into_corpus_group(const fe_iface_sptr& reader, + const corpus_sptr& korpus, + const vector& binaries, + const vector& bins_dirs) +{ + corpus_group_sptr result (new corpus_group(korpus->get_environment(), + korpus->get_path())); + result->add_corpus(korpus); + + add_binaries_into_corpus_group(reader, binaries, bins_dirs, *result); + + return result; +} + +/// Create a corpus group made of a given korpus and the subset of its +/// dependencies that can be found found in a set of directories. +/// +/// @param reader the reader to use to read the binaries. +/// +/// @param korpus the ABI corpus to add to the corpus group along with +/// its dependencies that can be found in a subset of directories. +/// +/// @param deps_dirs the directories where the dependencies of the ABI +/// corpus denoted by @p korpus binaries are to be found. +/// +/// @return a corpus group made of @p korpus and the subset of its +/// dependencies found in at least one of the directories denoted by +/// @p deps_dirs. +corpus_group_sptr +stick_corpus_and_dependencies_into_corpus_group(const fe_iface_sptr& reader, + const corpus_sptr& korpus, + const vector& deps_dirs) +{ + corpus_group_sptr result (new corpus_group(korpus->get_environment(), + korpus->get_path())); + result->add_corpus(korpus); + + add_dependencies_into_corpus_group(reader, *korpus, deps_dirs, *result); + + return result; +} + /// If we were given suppression specification files or kabi whitelist /// files, this function parses those, come up with suppression /// specifications as a result, and set them to the read context. @@ -2399,10 +2638,10 @@ find_file_under_dir(const string& root_dir, /// /// @param opts the options to consider. static void -load_generate_apply_suppressions(elf_based_reader &rdr, - vector& suppr_paths, - vector& kabi_whitelist_paths, - suppressions_type& supprs) +load_generate_apply_suppressions(elf_based_reader& rdr, + vector& suppr_paths, + vector& kabi_whitelist_paths, + suppressions_type& supprs) { if (supprs.empty()) {