From patchwork Thu Jun 1 10:14:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70442 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 7613E3858421 for ; Thu, 1 Jun 2023 10:14:57 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7613E3858421 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1685614497; bh=ABAxl/C+/MNtdCbTS0bwlbNioRgr6UcljoCgrE0uO4c=; h=To:Cc:Subject:Date:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From:Reply-To:From; b=oPzcz1sOoyOEllrZnUDAgYMeCd19LioqahyYqfTZrgTtipTqbkknxeq00HTtiIjom jBVaXjNQRcXJ2KpxeqAHSCbb1QAsdW5h9X+uznPkK/FiNzjhWnyhZCLDJoIzskEiAz QMN+dySvyYy0s5lvmZP5dUKttpIXmJWMtmEidXMg= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) by sourceware.org (Postfix) with ESMTPS id 8B877385840D for ; Thu, 1 Jun 2023 10:14:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 8B877385840D Received: by mail-lf1-x133.google.com with SMTP id 2adb3069b0e04-4f60924944aso63744e87.0 for ; Thu, 01 Jun 2023 03:14:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685614468; x=1688206468; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ABAxl/C+/MNtdCbTS0bwlbNioRgr6UcljoCgrE0uO4c=; b=Ahudc2uhTqwsnG5ej2XRt83+EKqTo99asaoe3nMLdqD+rhOHgJx8k2RYMaL2j1YenQ +dt4ep6tMYcwLmMnIA3YJPfmCC8XyC6JNFIj7EYlxsz461h9+bfRRvjv+fbY1LfbdoD4 L39NYuzBYDi9589AwJKL0A+xxPkVmiIlGyY3bUdCRmyGneNXzNBdJK8Kc4rphD+oyXPj RHQyKd7Iy+CTgJ+yjiyi8Mg5L4Tw1NnQGa6yep7k7LtNzeTacPbj2avgF9tE40z8GG8T Nlif+dxsFugS79KCB6n7ruP6k1BGyLmVdpbx9r1nkeVY3F0d+y89V5tFFnQ3p8foF5Rr 0iww== X-Gm-Message-State: AC+VfDzg/N2OXnuI2GdVrlcqGeLbsz2fKuBL8ZvTPIlb9UBwWUdMaRnT Mv2RqwBmzNJ+cgpGKE0bKNqPyl4XEpofkIOrMehJqds= X-Google-Smtp-Source: ACHHUZ5jasdVHgxEYFAxxxv5QwxxWP1scG0ssUus6+n20P4SNMv531TPaL7EuOc43M70PP+qS7aXgw== X-Received: by 2002:a05:6512:247:b0:4e9:c792:c950 with SMTP id b7-20020a056512024700b004e9c792c950mr3121014lfo.1.1685614468422; Thu, 01 Jun 2023 03:14:28 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id v1-20020ac25581000000b004f39bb71ed2sm1029223lfg.138.2023.06.01.03.14.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Jun 2023 03:14:27 -0700 (PDT) To: libc-alpha@sourceware.org Cc: Carlos O'Donell , DJ Delorie , Maxim Kuvyrkov Subject: [PATCH] Format test results closer to what DejaGnu does Date: Thu, 1 Jun 2023 10:14:24 +0000 Message-Id: <20230601101424.1848817-1-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Libc-alpha From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org Sender: "Libc-alpha" The years of dealing with Binutils, GCC and GDB test results made the community create good tools for comparison and analysis of DejaGnu test results. This change allows to use those tools for Glibc's test results as well. The motivation for this change is Linaro's pre-commit testers, which use a modified version of GCC's validate_failures.py to create test xfail lists with baseline failures and known flaky tests. See below links for an example xfails file (only one link is supposed to work at any given time): - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail/*view*/ - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/ Specifacally, this patch changes format of glibc's .sum files from ... FAIL: elf/test1 PASS: string/test2 ... to ... === glibc tests === Running elf ... FAIL: elf/test1 Running string ... PASS: string/test2 . And output of "make check" from ... FAIL: elf/test1 ... to ... === glibc failures === FAIL: elf/test1 === Summary of results === 1 FAIL 1 PASS . Signed-off-by: Maxim Kuvyrkov --- Makefile | 12 ++++++++---- scripts/merge-test-results.sh | 3 +++ 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index 224c792185..779938f3b7 100644 --- a/Makefile +++ b/Makefile @@ -564,11 +564,15 @@ $(objpfx)check-wrapper-headers.out: scripts/check-wrapper-headers.py $(headers) --generated $(common-generated) > $@; $(evaluate-test) endif # $(headers) +# Print test summary for tests in $1 .sum file; +# $2 is optional test identifier. +# Fail if there are unexpected failures in the test results. define summarize-tests -@grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true -@echo "Summary of test results$2:" -@sed 's/:.*//' < $(objpfx)$1 | sort | uniq -c -@! grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' $(objpfx)$1 +@echo " === glibc failures ===" +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true +@echo " === Summary of results$2 ===" +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' endef # The intention here is to do ONE install of our build into the diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh index e4dcc2520a..8f9d81f6eb 100755 --- a/scripts/merge-test-results.sh +++ b/scripts/merge-test-results.sh @@ -50,7 +50,10 @@ case $type in -t) subdir_file_name=$1 shift + echo " === glibc tests ===" for d in "$@"; do + echo + echo "Running $d ..." if [ -f "$objpfx$d/$subdir_file_name" ]; then cat "$objpfx$d/$subdir_file_name" else