From patchwork Wed Dec 22 10:49:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Schwinge X-Patchwork-Id: 49172 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 93D613858429 for ; Wed, 22 Dec 2021 10:50:13 +0000 (GMT) X-Original-To: libabigail@sourceware.org Delivered-To: libabigail@sourceware.org Received: from smtprelay01.ispgateway.de (smtprelay01.ispgateway.de [80.67.18.13]) by sourceware.org (Postfix) with ESMTPS id 0735D3858C2C for ; Wed, 22 Dec 2021 10:50:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 0735D3858C2C Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=schwinge.name Received: from [192.94.31.2] (helo=dem-tschwing-1.ger.mentorg.com) by smtprelay01.ispgateway.de with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mzzCN-0006Ek-In; Wed, 22 Dec 2021 11:50:19 +0100 Received: (nullmailer pid 3998443 invoked by uid 1000); Wed, 22 Dec 2021 10:50:04 -0000 From: Thomas Schwinge To: libabigail@sourceware.org Subject: [PATCH] Promote 'tests/runtestslowselfcompare.sh' to 'tests/runtestselfcompare.sh' Date: Wed, 22 Dec 2021 11:49:36 +0100 Message-Id: <20211222104936.3998331-1-thomas@codesourcery.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <87tuf1ynf3.fsf@euler.schwinge.homeip.net> References: <87tuf1ynf3.fsf@euler.schwinge.homeip.net> MIME-Version: 1.0 X-Df-Sender: b3V0Z29pbmdAc2Nod2luZ2UubmFtZQ== X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, GIT_PATCH_0, HEADER_FROM_DIFFERENT_DOMAINS, KAM_DMARC_STATUS, LIKELY_SPAM_BODY, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: libabigail@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list of the Libabigail project List-Unsubscribe: , List-Archive: List-Help: List-Subscribe: , Cc: Dodji Seketeli , Thomas Schwinge Errors-To: libabigail-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libabigail" Per commit cac59a176a0c0d6d6c693cb1cfb475517ec33e97 "Bug 26769 - Fix missing types in abixml output": | * tests/runtestslowselfcompare.sh.in: New test that compares | libabigail.so against its own ABIXML representation. I consider this to be a pretty important test case -- "eat our own dog food". Thus, I find this a bit unfortunate: | * tests/Makefile.am: Add the new test runtestslowselfcompare.sh to | source distribution. This test is too slow to be run during the | course of 'make check'. It takes more than 5 minutes on my slow | box here. Rather, it can be run using 'make check-self-compare'. | I plan to run this before releases now. ..., that is, that 'tests/runtestslowselfcompare.sh' isn't run during standard 'make check'. On my eight years old Dell Precision M4700, I see: $ \time make check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes 20.19user 0.64system 0:20.83elapsed 100%CPU (0avgtext+0avgdata 970468maxresident)k 20.25user 0.51system 0:20.83elapsed 99%CPU (0avgtext+0avgdata 969984maxresident)k 20.47user 0.53system 0:20.99elapsed 100%CPU (0avgtext+0avgdata 970016maxresident)k So, ~21 s. All the other test cases, running in parallel (just '-j5'): $ \time make check -j5 # with default 'ENABLE_SLOW_TEST=no' 364.42user 31.21system 1:02.64elapsed 631%CPU (0avgtext+0avgdata 605568maxresident)k 359.50user 31.18system 0:59.43elapsed 657%CPU (0avgtext+0avgdata 605720maxresident)k 359.72user 30.87system 0:59.44elapsed 657%CPU (0avgtext+0avgdata 605292maxresident)k So, ~61 s. Additionally running 'tests/runtestslowselfcompare.sh': $ \time make check -j5 ENABLE_SLOW_TEST=yes 389.44user 30.95system 1:06.35elapsed 633%CPU (0avgtext+0avgdata 971036maxresident)k 387.47user 30.78system 1:05.42elapsed 639%CPU (0avgtext+0avgdata 971000maxresident)k 388.99user 32.30system 1:04.94elapsed 648%CPU (0avgtext+0avgdata 970356maxresident)k So, ~66 s, and thus 'tests/runtestslowselfcompare.sh' makes the 'make check -j5' take just ~5 s longer -- acceptable, in my opinion. Per later commit b56e5aeb409b43fefc01e0397346b66d83e28030 "CONTRIBUTING: Update instructions about regression tests", it was noted that... | This is an important regression test. The | problem is that it can takes twice as much time as make distcheck. So | we've put it into its own separate target. Given the "5 minutes" number from above, this comment means that a 'make distcheck' (or rather 'make distcheck-fast', I suppose?) would run ~2.5 min? I've got the following numbers: $ \time make distcheck-fast -j5 # with default 'ENABLE_SLOW_TEST=no' 935.67user 72.00system 4:58.90elapsed 337%CPU (0avgtext+0avgdata 986144maxresident)k 946.23user 68.60system 4:59.07elapsed 339%CPU (0avgtext+0avgdata 984372maxresident)k 935.33user 67.69system 5:01.18elapsed 333%CPU (0avgtext+0avgdata 985388maxresident)k So, ~5 min, and thus for me, 'tests/runtestslowselfcompare.sh' alone takes just 1/15 the time of that, not "twice as much". Additionally enabling 'tests/runtestslowselfcompare.sh' here: $ \time make distcheck-fast -j5 ENABLE_SLOW_TEST=yes 965.52user 67.06system 5:04.04elapsed 339%CPU (0avgtext+0avgdata 984760maxresident)k 981.54user 67.57system 4:57.65elapsed 352%CPU (0avgtext+0avgdata 985836maxresident)k 972.32user 68.58system 5:09.81elapsed 335%CPU (0avgtext+0avgdata 985224maxresident)k ..., again 'tests/runtestslowselfcompare.sh' makes that just take ~5 s longer -- acceptable, in my opinion. * tests/runtestslowselfcompare.sh.in: Rename to... * tests/runtestselfcompare.sh.in: ... this, and assume 'ENABLE_SLOW_TEST' set. * tests/Makefile.am: Adjust. (check-self-compare): Remove target. * Makefile.am: Likewise. * configure.ac: Adjust. * CONTRIBUTING: Likewise. --- CONTRIBUTING | 18 ++++++++---------- Makefile.am | 9 +-------- configure.ac | 4 ++-- tests/Makefile.am | 14 ++++---------- ...fcompare.sh.in => runtestselfcompare.sh.in} | 7 +------ 5 files changed, 16 insertions(+), 36 deletions(-) rename tests/{runtestslowselfcompare.sh.in => runtestselfcompare.sh.in} (61%) diff --git a/CONTRIBUTING b/CONTRIBUTING index 5ab7bcce..588f87dd 100644 --- a/CONTRIBUTING +++ b/CONTRIBUTING @@ -58,6 +58,9 @@ tests in parallel to make whole thing go faster by doing: make -jN -lN check +distcheck +--------- + If you want to test the fabrication of the distribution tarball (this is important, because that is how we do to actually release the tarball of the project that you can download from the internet) then @@ -85,19 +88,14 @@ Please make sure you always launch "make distcheck-fast" before sending a patch, so that you are sure that we can always build a tarball after your patch is applied to the source tree. -A complementary regression checking target is "check-self-compare". -You invoke it by doing "make check-self-compare". That target -analyzes the ABI of the libabigail.so shared object, serializes it +tests/runtestselfcompare.sh +--------------------------- + +This one analyzes the ABI of the libabigail.so shared object, serializes it into the ABIXML format and then compares the ABI internal representation gathered from the libabigail.so binary against the one gathered from the ABIXML format. The two should be equal if -everything goes right. This is an important regression test. The -problem is that it can take a considerable amount of time. -So we've put it into its own separate target. - -So, to be complete the regression checking command to run against your -patch should be: "make check-self-compare distcheck-fast -j16", if you have -a machine with a 16 threads processors, for instance. +everything goes right. This is an important regression test. Launching regression tests in Valgrind -------------------------------------- diff --git a/Makefile.am b/Makefile.am index 65a1b121..99353e82 100644 --- a/Makefile.am +++ b/Makefile.am @@ -80,17 +80,10 @@ tarball-and-tag-and-upload: tarball-and-all tarball-and-all: tag-and-all $(MAKE) tarball -tag-and-all: distcheck check-self-compare +tag-and-all: distcheck $(MAKE) tag-release-only # the default for GZIP_ENV is --best, which is pretty slow for check runs # distcheck-fast therefore compresses with --fast instead distcheck-fast: $(MAKE) distcheck GZIP_ENV="--fast" - -# This makes us compare libabigail.so against its own ABIXML -# representation. It's super slow (more than 5 minutes on one of my -# old boxes) so I do this only before releases. -check-self-compare: - $(MAKE) -C tests check-self-compare - diff --git a/configure.ac b/configure.ac index 29130175..1172a9fe 100644 --- a/configure.ac +++ b/configure.ac @@ -969,8 +969,8 @@ libabigail.pc dnl Some test scripts are generated by autofoo. AC_CONFIG_FILES([tests/runtestcanonicalizetypes.sh], [chmod +x tests/runtestcanonicalizetypes.sh]) - AC_CONFIG_FILES([tests/runtestslowselfcompare.sh], - [chmod +x tests/runtestslowselfcompare.sh]) +AC_CONFIG_FILES([tests/runtestselfcompare.sh], + [chmod +x tests/runtestselfcompare.sh]) AC_CONFIG_FILES([tests/mockfedabipkgdiff], [chmod +x tests/mockfedabipkgdiff]) AC_CONFIG_FILES([tests/runtestfedabipkgdiff.py], diff --git a/tests/Makefile.am b/tests/Makefile.am index 17e2d4b4..27ecb623 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -12,11 +12,9 @@ FEDABIPKGDIFF_TEST += runtestfedabipkgdiff.py endif endif -RUN_TEST_SLOW_SELF_COMPARE=runtestslowselfcompare.sh - # rather expensive tests (keep in this order), > 1s runtime TESTS= \ -$(RUN_TEST_SLOW_SELF_COMPARE) \ +runtestselfcompare.sh \ runtestdiffsuppr \ runtesttypesstability \ runtestdiffpkg \ @@ -60,7 +58,7 @@ endif EXTRA_DIST = \ runtestcanonicalizetypes.sh.in \ runtestfedabipkgdiff.py.in \ -runtestslowselfcompare.sh.in \ +runtestselfcompare.sh.in \ mockfedabipkgdiff.in \ test-valgrind-suppressions.supp @@ -183,8 +181,8 @@ testdiff2_LDADD=$(top_builddir)/src/libabigail.la printdifftree_SOURCES = print-diff-tree.cc printdifftree_LDADD = $(top_builddir)/src/libabigail.la -runtestslowselfcompare_sh_SOURCES = -runtestslowselfcompare.sh$(EXEEXT): +runtestselfcompare_sh_SOURCES = +runtestselfcompare.sh$(EXEEXT): runtestcanonicalizetypes_sh_SOURCES = runtestcanonicalizetypes.sh$(EXEEXT): @@ -242,7 +240,3 @@ check-valgrind-memcheck-recursive: check-valgrind-helgrind-recursive: $(MAKE) check-valgrind-helgrind VALGRIND_FLAGS="${RECURSIVE_VALGRIND_FLAGS}" - -check-self-compare: - $(MAKE) check TESTS=runtestslowselfcompare.sh ENABLE_SLOW_TEST=yes - diff --git a/tests/runtestslowselfcompare.sh.in b/tests/runtestselfcompare.sh.in similarity index 61% rename from tests/runtestslowselfcompare.sh.in rename to tests/runtestselfcompare.sh.in index 3d0ef123..a6f16009 100644 --- a/tests/runtestslowselfcompare.sh.in +++ b/tests/runtestselfcompare.sh.in @@ -5,9 +5,4 @@ abidw=@top_builddir@/tools/abidw objdir=@top_builddir@/src/.libs -echo "ENABLE_SLOW_TEST="$ENABLE_SLOW_TEST - -if test x$ENABLE_SLOW_TEST != x; then - `$abidw --abidiff $objdir/libabigail.so` -fi - +`$abidw --abidiff $objdir/libabigail.so`