From patchwork Tue Jul 16 14:55:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dodji Seketeli X-Patchwork-Id: 94005 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D5A27386C592 for ; Tue, 16 Jul 2024 14:57:02 +0000 (GMT) X-Original-To: libabigail@sourceware.org Delivered-To: libabigail@sourceware.org Received: from relay3-d.mail.gandi.net (relay3-d.mail.gandi.net [217.70.183.195]) by sourceware.org (Postfix) with ESMTPS id BDD41386C5A4 for ; Tue, 16 Jul 2024 14:55:44 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org BDD41386C5A4 Authentication-Results: sourceware.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=seketeli.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org BDD41386C5A4 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=217.70.183.195 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1721141755; cv=none; b=SrnNl8NU1CBCg0oCzPOhNyhWeKNNDYZ6et5aLrZWlM9rEehCBEUHORJ11MiLz9uJNCntFFHQI9RuplEvDwdF4DpTTQOnGzC53hXdj2rIhOLRZS8I8JubJQInjRhhXwEFtPN5BkGLZEec7JGsEwUhvsJDOS5bP0LGM03HF/HvTZw= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1721141755; c=relaxed/simple; bh=oCArIB4+pqPm2uCPDua6nZTllXvP20YJGY2XmKDOVZU=; h=From:To:Subject:Date:Message-ID:MIME-Version; b=M6bKZAxZ8gnuVqrpYQlaD8x4E4tIdAZyVJehf13Brf+hozs0R/KkVc9kFb5fA8RYiKiWc0/PQWjLe1YNBRBKQ0S8am/t7Lw+7PdnQBWj+K85QPGFJkBkJbgJ5AO11Vd/eW6ZGqOYP7XuCTi/azTv23jEm4QvylNvaBRuiiruhNc= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail.gandi.net (Postfix) with ESMTPSA id C3ED060006; Tue, 16 Jul 2024 14:55:43 +0000 (UTC) Received: by localhost (Postfix, from userid 1000) id 8C9CBC1B7B6C; Tue, 16 Jul 2024 16:55:41 +0200 (CEST) From: dodji@redhat.com To: libabigail@sourceware.org Cc: dodji@redhat.com Subject: [PATCH 17/17] configure: Support the optional 'big-tests' sub-directory Date: Tue, 16 Jul 2024 16:55:28 +0200 Message-ID: <20240716145541.473065-18-dodji@redhat.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240716145541.473065-1-dodji@redhat.com> References: <20240716145541.473065-1-dodji@redhat.com> MIME-Version: 1.0 X-GND-Sasl: dodj@seketeli.org X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, GIT_PATCH_0, HEADER_FROM_DIFFERENT_DOMAINS, JMQ_SPF_NEUTRAL, KAM_ASCII_DIVIDERS, KAM_DMARC_STATUS, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libabigail-bounces~patchwork=sourceware.org@sourceware.org From: Dodji Seketeli If the 'big-tests' sub-directory (or a symbolic link to it) is present then this patch configures it. By default, the patch defines a new git sub-module into the 'big-tests' sub-directory. That git sub-module is the repository git://sourceware.org/git/libabigail-tests. Users can now add the --recurse-submodules option to git clone when they clone git://sourceware.org/git/libabigail, so that it clones git big-tests submodule as well. The patch adds a new 'big-tests' target (to the top-most Makefile.am) which runs "make check" in the big-tests sub-directory. The patch also adds a new 'full-check' target that runs targets check, check-self-compare and big-tests. * Makefile.am: Add big-tests,clean-big-tests and full-check targets. * configure.ac: Add an --enable-big-tests option. For now, this option is mandatory to handle the optional 'big-tests' support. If the --enable-big-tests option is provided and if the 'big-tests' sub-directory is present, configure the big-tests/ sub-package. * .gitmodules: New sub-module configuration to get git://sourceware.org/git/libabigail-tests and stick into ... * big-tests: ... this directory as a sub-module. * CONTRIBUTING: Update this to explain how to get the Big Tests going. * tests/Makefile.am: Update the test summary for the check-self-compare target. Signed-off-by: Dodji Seketeli --- .gitmodules | 3 + CONTRIBUTING | 159 ++++++++++++++++++++++++++++------------------ Makefile.am | 9 ++- big-tests | 1 + configure.ac | 32 ++++++++++ tests/Makefile.am | 2 +- 6 files changed, 142 insertions(+), 64 deletions(-) create mode 100644 .gitmodules create mode 160000 big-tests diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 00000000..b8e6a0d5 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "big-tests"] + path = big-tests + url = git://sourceware.org/git/libabigail-tests.git diff --git a/CONTRIBUTING b/CONTRIBUTING index cbdc9843..9f653f94 100644 --- a/CONTRIBUTING +++ b/CONTRIBUTING @@ -125,54 +125,100 @@ the libabigail *library* directly (as opposed to forking libabigail command line tools) will be verified. How tests are organized ------------------------ - -There are two kinds of regression tests. Those that use the -libabigail *library* directly, and those that spawn one of the -libabigail command line tools. - -Generally, both are usually made of a loop that churns through a set of input -binaries to compare. Once the comparison is done, the resulting -report is compared against a reference report that is provided. - -Test executables have names that starts with 'runtest*'. For instance, -under /tests/ you can find tests named -runtestdiffdwarf, runtestabidiff, etc... - -If a test executable is named -/tests/runtestdiffdwarf, then its source code is -tests/test-diff-dwarf.cc. Similarly, the source code of the test -/tests/runtestabidiff is tests/test-abidiff.cc. - -The data provided for each test (for instance the input binaries to -compare and the reference report that should result from the -comparison) is to be found under tests/data. So data for the test -runtestdiffdwarf is to be found under tests/data/test-diff-dwarf. -Data for the test runtestabidiff is to be found under -tests/data/test-abidiff.cc. - -So adding your own tests usually just amounts to adding the right input -into the right sub-directory of tests/data/. To do so, -look at several tests/test-*.cc to see which one you'd like to add -some input binaries to be compared in. - -Then once you know which tests/test-*.cc you'd like to extend, and if -you added your input binaries and reference reports (maybe other -things too) to the right sub-director of tests/data/, you just need to -extend the array of input binaries/reference reports that the test -walks to perform the comparisons. It's generally a global variable -before the main() function of the test. In test-diff-dwarf.cc, for -instance, the variable name is "in_out_specs". You just have to add a -new entry to that array; that new entry contains the paths to your new -input binaries and reference reports. Just read the code in there and -use your brains. It should be straight forward. - -Ah, also, if you added new files for the tests, then the build system -needs to be told that those files have to be added to the distribution -tarball when we do "make dist" (or make distcheck). To do so, please -make sure to add your new test input files to the -tests/data/Makefile.am file, in the EXTRA_DIST variable. Look at how -things are organized in that file, and please do things similarly. +======================= + +There are two kinds of tests in libabigail: + + * Regression tests. + + These are in the tests/ sub-directory of the package, hosted + in the libabigail Git repository on sourceware. They are + meant to run quickly and not be too big. They are also meant + to end up in the released libabigail tarball. + + There are two kinds of regression tests. Those that use the + libabigail *library* directly, and those that spawn one of the + libabigail command line tools. + + Generally, both are usually made of a loop that churns through a set of input + binaries to compare. Once the comparison is done, the resulting + report is compared against a reference report that is provided. + + Test executables have names that starts with 'runtest*'. For instance, + under /tests/ you can find tests named + runtestdiffdwarf, runtestabidiff, etc... + + If a test executable is named + /tests/runtestdiffdwarf, then its source code is + tests/test-diff-dwarf.cc. Similarly, the source code of the test + /tests/runtestabidiff is tests/test-abidiff.cc. + + The data provided for each test (for instance the input binaries to + compare and the reference report that should result from the + comparison) is to be found under tests/data. So data for the test + runtestdiffdwarf is to be found under tests/data/test-diff-dwarf. + Data for the test runtestabidiff is to be found under + tests/data/test-abidiff.cc. + + So adding your own tests usually just amounts to adding the right input + into the right sub-directory of tests/data/. To do so, + look at several tests/test-*.cc to see which one you'd like to add + some input binaries to be compared in. + + Then once you know which tests/test-*.cc you'd like to extend, and if + you added your input binaries and reference reports (maybe other + things too) to the right sub-director of tests/data/, you just need to + extend the array of input binaries/reference reports that the test + walks to perform the comparisons. It's generally a global variable + before the main() function of the test. In test-diff-dwarf.cc, for + instance, the variable name is "in_out_specs". You just have to add a + new entry to that array; that new entry contains the paths to your new + input binaries and reference reports. Just read the code in there and + use your brains. It should be straight forward. + + Ah, also, if you added new files for the tests, then the build system + needs to be told that those files have to be added to the distribution + tarball when we do "make dist" (or make distcheck). To do so, please + make sure to add your new test input files to the + tests/data/Makefile.am file, in the EXTRA_DIST variable. Look at how + things are organized in that file, and please do things similarly. + + * Integration tests AKA Big Tests. + + These are bigger and slower tests that would be too cumbersome + to be shipped in the libabigail tarball. They are in a + separate Git repository which is at + git://sourceware.org/git/libabigail-tests. + + That repository is meant to be checked out as a sub-module in + a libabigail tree. We affectionately call these tests "Big + Tests". + + Suppose you've checked out a libabigail tree by doing: + + $ git clone git://sourceware.org/git/libabigail + + To get the integration tests do this: + + $ cd libabigail/big-tests + $ git submodule init + $ git submodule update + + The Git repository at + git://sourceware.org/git/libabigail-tests is now checked-out + as a sub-module in the "big-tests" sub-directory of the + libabigail tree. + + To configure libabigail with the support of the Big Tests, do + this: + + $ cd + $ /path/to/libabigail/configure --enable-big-tests + + To launch the Big Tests, do this in the : + + $ make big-tests + fedabipkgdiff testing --------------------- @@ -291,20 +337,9 @@ the commit log message for you. Modifying the website ===================== -The source code of the website of libabigail is stored in CVS (sigh, -yeah, that is so old school). You can check out that web source code -by doing: - - CVS_RSH=ssh cvs -z9 -d :ext:user@sourceware.org:/cvs/libabigail/ co htdocs - -where 'user' is your username on the sourceware system. -Alternatively, you can check out the the web source code anonymously, -if you don't have any user account on the sourceware system by doing: - - export CVSROOT=:pserver:anoncvs@cygwin.com:/cvs/libabigail - cvs login - (the CVS anonymous password to use is "anoncvs") - cvs checkout htdocs +The source code of the website of libabigail is stored in Git. You +can check out that web source code by doing: + git co git://sourceware.org/git/libabigail-htdocs.git Happy Hacking! diff --git a/Makefile.am b/Makefile.am index a0310368..72e6d1fa 100644 --- a/Makefile.am +++ b/Makefile.am @@ -30,7 +30,7 @@ libabigail.pc.in # Don't make make repeat. AM_MAKEFLAGS = --no-print-directory -.PHONY: doc +.PHONY: doc big-tests doc: html-doc man info @@ -94,3 +94,10 @@ distcheck-fast: check-self-compare: $(MAKE) -C tests check-self-compare +big-tests: + test -d big-tests && $(MAKE) -C big-tests check + +clean-big-tests: + test -d big-tests && $(MAKE) -C big-tests clean + +full-check: check check-self-compare big-tests diff --git a/big-tests b/big-tests new file mode 160000 index 00000000..8447ddfd --- /dev/null +++ b/big-tests @@ -0,0 +1 @@ +Subproject commit 8447ddfd40ee4d3dbb7b3ba8e5fae809a89334a5 diff --git a/configure.ac b/configure.ac index 5fd77821..9c8e5511 100644 --- a/configure.ac +++ b/configure.ac @@ -231,6 +231,14 @@ AC_ARG_ENABLE(btf, [disable support of btf files)]), ENABLE_BTF=$enableval, ENABLE_BTF=auto) + +ENABLE_BIG_TESTS=no +AC_ARG_ENABLE(big-tests, + AS_HELP_STRING([--enable-big-tests=yes|no], + [enable the support of libabigail big tests (in the big-tests subdir)]), + ENABLE_BIG_TESTS=$enableval, + ENABLE_BIG_TESTS=no) + dnl ************************************************* dnl check for dependencies dnl ************************************************* @@ -457,6 +465,29 @@ dnl Test if various functions and structs are present. fi fi +if test x$ENABLE_BIG_TESTS = xyes; then + AC_MSG_NOTICE([looking for $srcdir/big-tests ...]); + if test -d $srcdir/big-tests -o -L $srcdir/big-tests; then + AC_CONFIG_COMMANDS([create-big-tests-build-dir-and-configure], + [AC_MSG_NOTICE([Found $srcdir/big-tests!]) + cd ac_abs_top_srcdir/big-tests + autoreconf -i + cd $ac_abs_top_builddir + test -d big-tests || mkdir big-tests + cd big-tests + CTF_OPTS= + BTF_OPTS= + if test x$CTF_ENABLED = xyes; then CTF_OPTS="--enable-ctf"; fi; + if test x$BTF_ENABLED = xyes; then BTF_OPTS="--enable-btf"; fi; + AC_MSG_NOTICE([configuring $srcdir/big-tests --with-libabigail-tools-dir=$ac_abs_top_builddir/tools --with-libabigail-headers-dir=$ac_abs_top_srcdir/include $CTF_OPTS $BTF_OPTS]) + $ac_abs_top_srcdir/big-tests/configure --with-libabigail-tools-dir=$ac_abs_top_builddir/tools --with-libabigail-headers-dir=$ac_abs_top_srcdir/include $CTF_OPTS $BTF_OPTS], + [CTF_ENABLED=$ENABLE_CTF + BTF_ENABLED=$ENABLE_BTF]) + else + AC_MSG_ERROR([No sub-directory big-tests found. Please checkout libabigail-big-tests under a sub-directory named 'big-tests']) + fi +fi + dnl Check for dependency: libxml LIBXML2_VERSION=2.6.22 PKG_CHECK_MODULES(XML, libxml-2.0 >= $LIBXML2_VERSION) @@ -1267,6 +1298,7 @@ AC_MSG_NOTICE([ Enable python 3 : ${ENABLE_PYTHON3} Enable CTF front-end : ${ENABLE_CTF} Enable BTF front-end : ${ENABLE_BTF} + Enable Big Libabigail tests : ${ENABLE_BIG_TESTS} Enable running tests under Valgrind : ${enable_valgrind} Enable build with -fsanitize=address : ${ENABLE_ASAN} Enable build with -fsanitize=memory : ${ENABLE_MSAN} diff --git a/tests/Makefile.am b/tests/Makefile.am index 141c8768..8204bdc5 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -274,5 +274,5 @@ check-valgrind-helgrind-recursive: $(MAKE) check-valgrind-helgrind VALGRIND_FLAGS="${RECURSIVE_VALGRIND_FLAGS}" check-self-compare: - $(MAKE) check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes + $(MAKE) check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes AM_TESTSUITE_SUMMARY_HEADER=' for runtestslowselfcompare.sh'