From patchwork Wed Nov 1 10:28:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 78871 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id CB38838582BE for ; Wed, 1 Nov 2023 10:29:00 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-lf1-x12c.google.com (mail-lf1-x12c.google.com [IPv6:2a00:1450:4864:20::12c]) by sourceware.org (Postfix) with ESMTPS id 234D63858C00 for ; Wed, 1 Nov 2023 10:28:36 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 234D63858C00 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=linaro.org Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=linaro.org ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 234D63858C00 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::12c ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698834526; cv=none; b=sQdLBP0Q8qc37aSHMd0zV9ooLAQyBOszMkixkfZxpDelVUm7ZnADjuxRLQoCWGvEZD7D5cRHpqeAVVX3ldts07VzTFvLrFD5aaV/aTbZ7+76X8jB0FKPh6TFzSl66Y54gCQkp2R5zGBN5mhm8UrQ9DZqgg01tiC+P4nrapLrpGw= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1698834526; c=relaxed/simple; bh=DLz1XLXudrdhTI1pmIwHDNHmOYcm+2llMf8f1oEahFQ=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=drJHoNiWgM5JmUA61NQmDrdccO17kM+Vi3Y90f3xcIaZ1iyMwEkTg+T5AQaaPrOTYmv6KmLF00lTSXCW7hTNzxxUCCueZN5TKUy/MB4Z+Uk3xCE8/hHo66zo3CfE/QXesvzjyVkj6GdfhpnJuwBlFOn6L9tXyDj52HMMeFT1dUk= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lf1-x12c.google.com with SMTP id 2adb3069b0e04-507f63fcbf6so2230719e87.1 for ; Wed, 01 Nov 2023 03:28:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1698834514; x=1699439314; darn=sourceware.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HR6Y/gQA2x6lN25nILPzoSYcKFQvOsy8sazIc700DqU=; b=jigb/30Scb8QFI/7aznrmyGIzsFccM/hRALJ3Bn/1H+OUj8pe0bWoBKUxVheAMCk7y xV5M1ZCssjSgaV6xbkOxn0N2l3OCeARos8lj/c6TltaaXRCS3aOhlHm/+G0jdLwS5Z3c Tbhx6aOrji+xXveeUZRMxHKTj99aYP0/8tKVOtANn5p+rYjnjDlK/UtjrF8vyFANpCEx rdBmbb7u+jsomSnoux2RT4rJnQvewiC74D5BL7qSKNtchU5DoPQJOnYQ+F2PKLDWe5PC 1Q13p/qQ//V6pEJQ0MKOt8vhykrcH4Mfh3CGPPNhUi+j2EeUGK+/EmnJRj2Lxa7hA7v5 eC7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698834514; x=1699439314; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HR6Y/gQA2x6lN25nILPzoSYcKFQvOsy8sazIc700DqU=; b=BOn1KZuiFsntpOad8/vukJ9AhTnN5LN27hzUXB8qTmAU2WtHPGboyCWQXKdk6X1grk LQXbvH55upBCa+ZcVYmti9KL7uRM/gv1AxIhWPpmWUNuxClLcUNLXOf7cQYZVCtxXmKy pGTIHJF6q4tiBKn/hl38VS71coUJ4yTjz3DPe7JLgOKTHvZmVaKk44UmZ8n4kAjGSxuB u9uu8uTAGnGZXJXuKJiyFQOlg7/4EKHtHtitcc5Npz9GVkhEI1zkUH+z3nypTMu4Qkbs iHoGU6R+V9M07zEeHnied8xFcKu+fQlE6HEKvkbCwkFFm5EtlZQK6CRW8ZcbM8f8s+LP tdGg== X-Gm-Message-State: AOJu0YxMMYpKa0RZo5ZMBtz5c2mNJsnRnN8UoWbQMjXMsrXDcBhhU7tT sbiL10DACXgHyj58H7ao7G3YlxqI0/q9etldphQm X-Google-Smtp-Source: AGHT+IHWi4fTKZWiphyz+s8mWFl6aw959OVEqbSzhzThKgzkWiIpnG9+5qZpMsb9uuDJWRPiIL4O6w== X-Received: by 2002:a05:6512:1242:b0:507:9d5c:62e3 with SMTP id fb2-20020a056512124200b005079d5c62e3mr11736334lfb.5.1698834513851; Wed, 01 Nov 2023 03:28:33 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id w13-20020a0565120b0d00b00502ae8db086sm181031lfu.19.2023.11.01.03.28.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 03:28:32 -0700 (PDT) From: Maxim Kuvyrkov To: libc-alpha@sourceware.org Cc: Maxim Kuvyrkov , Florian Weimer Subject: [PATCH v2] Format test results closer to what DejaGnu does Date: Wed, 1 Nov 2023 10:28:28 +0000 Message-Id: <20231101102828.3207231-1-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230601101424.1848817-1-maxim.kuvyrkov@linaro.org> References: <20230601101424.1848817-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+patchwork=sourceware.org@sourceware.org The only change from v1 is removal of "=== glibc failures ===" line. The years of dealing with Binutils, GCC and GDB test results made the community create good tools for comparison and analysis of DejaGnu test results. This change allows to use those tools for Glibc's test results as well. The motivation for this change is Linaro's pre-commit testers, which use a modified version of GCC's validate_failures.py to create test xfail lists with baseline failures and known flaky tests. See below links for an example xfails file (only one link is supposed to work at any given time): - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/artifacts.precommit/sumfiles/xfails.xfail/*view*/ - https://ci.linaro.org/job/tcwg_glibc_check--master-arm-build/lastSuccessfulBuild/artifact/artifacts/sumfiles/xfails.xfail/*view*/ Specifacally, this patch changes format of glibc's .sum files from ... FAIL: elf/test1 PASS: string/test2 ... to ... === glibc tests === Running elf ... FAIL: elf/test1 Running string ... PASS: string/test2 . And output of "make check" from ... FAIL: elf/test1 ... to ... FAIL: elf/test1 === Summary of results === 1 FAIL 1 PASS . Signed-off-by: Maxim Kuvyrkov Reviewed-by: Adhemerval Zanella --- Makefile | 11 +++++++---- scripts/merge-test-results.sh | 3 +++ 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/Makefile b/Makefile index b938721166..a4f3378e21 100644 --- a/Makefile +++ b/Makefile @@ -577,11 +577,14 @@ $(objpfx)lint-makefiles.out: scripts/lint-makefiles.sh $(SHELL) $< "$(PYTHON)" `pwd` > $@ ; \ $(evaluate-test) +# Print test summary for tests in $1 .sum file; +# $2 is optional test identifier. +# Fail if there are unexpected failures in the test results. define summarize-tests -@grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true -@echo "Summary of test results$2:" -@sed 's/:.*//' < $(objpfx)$1 | sort | uniq -c -@! grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' $(objpfx)$1 +@grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -v '^(PASS|XFAIL):' || true +@echo " === Summary of results$2 ===" +@sed -e '/:.*/!d' -e 's/:.*//' < $(objpfx)$1 | sort | uniq -c +@! grep -E '^[A-Z]+:' $(objpfx)$1 | grep -E -q -v '^(X?PASS|XFAIL|UNSUPPORTED):' endef # The intention here is to do ONE install of our build into the diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh index e4dcc2520a..8f9d81f6eb 100755 --- a/scripts/merge-test-results.sh +++ b/scripts/merge-test-results.sh @@ -50,7 +50,10 @@ case $type in -t) subdir_file_name=$1 shift + echo " === glibc tests ===" for d in "$@"; do + echo + echo "Running $d ..." if [ -f "$objpfx$d/$subdir_file_name" ]; then cat "$objpfx$d/$subdir_file_name" else