From patchwork Fri Jun 2 15:20:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70517 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 2D82C3856611 for ; Fri, 2 Jun 2023 15:21:47 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2D82C3856611 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719307; bh=muTDMilMZzhrMEa98ml7GPANVoRgx7x3w3gZEa4UzVk=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=CLrjZ7Ep7tIpslGhgaogsLxTo55l9gdBf9sG8Ic/KKdNt6Bd+DpiCyx3zvHmfCN5A D6GplQSjTFK/oT74fabCskPEGgiHzNNnSFdjgCEl6DqalJtHMOa4NGppk9Y3L7/pIE 3yJz9jfeoNl/W6vPEc4jqJ3LgUbiV03mYO+uepK4= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x22e.google.com (mail-lj1-x22e.google.com [IPv6:2a00:1450:4864:20::22e]) by sourceware.org (Postfix) with ESMTPS id EC4F6385772A for ; Fri, 2 Jun 2023 15:21:16 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EC4F6385772A Received: by mail-lj1-x22e.google.com with SMTP id 38308e7fff4ca-2b072753301so2343851fa.1 for ; Fri, 02 Jun 2023 08:21:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719275; x=1688311275; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=muTDMilMZzhrMEa98ml7GPANVoRgx7x3w3gZEa4UzVk=; b=U1tuhtfGmIvbj6idJPTXdkSmZOypKTQDUvv2tuwUyQ/JINUKAWBW3edDV35JzyIaC7 LuZMH+jlEyoRl/Kv7FR3pjyjCdiOdX5PBVe/u2prIERSl8ckmt+VlBw36b6a8iJhwn0W /qJ77k9T7fxzqVxvckj0R5vOQ4PzFS2qa/VtUDThGsPiVm1q+p/CxG6aTRsaPUacDkDQ 1O9LO6AVg/ew44jcO5P2ekGHYILWGjKCSp1CofYOYisr6EzeuGX26A2V38bWr4xtVlCv H5hBk+9ZBODvOE4yI0KdKl3Mws3YzafL0X836+Q5JJRrOI0Mr1ggY2rD7qQeoZka9ax7 LNjg== X-Gm-Message-State: AC+VfDxlGxlsvDaT7K9urwGyn4DQmJbtkFV1s8hjLcO5arBrWJCP9wlh 1grOKOSKQF416aGX6JZMePNAih3J3dGiBwZleQx9 X-Google-Smtp-Source: ACHHUZ4c2Juh7wX7niJWYNuoxDs4fLJ3QcPIdN0mFeDIjZJz3+ubkRh1gOAsrFV6M3B9uiNRYS1B0w== X-Received: by 2002:a2e:a4ac:0:b0:2b1:a69e:6a8e with SMTP id g12-20020a2ea4ac000000b002b1a69e6a8emr2215637ljm.3.1685719274840; Fri, 02 Jun 2023 08:21:14 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:13 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 01/12] [contrib] validate_failures.py: Avoid testsuite aliasing Date: Fri, 2 Jun 2023 15:20:41 +0000 Message-Id: <20230602152052.1874860-2-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" This patch adds tracking of current testsuite "tool" and "exp" to the processing of .sum files. This avoids aliasing between tests from different testsuites with same name+description. E.g., this is necessary for testsuite/c-c++-common, which is ran for both gcc and g++ "tools". This patch changes manifest format from ... FAIL: gcc_test FAIL: g++_test ... to ... === gcc tests === Running gcc/foo.exp ... FAIL: gcc_test === gcc Summary == === g++ tests === Running g++/bar.exp ... FAIL: g++_test === g++ Summary == . The new format uses same formatting as DejaGnu's .sum files to specify which "tool" and "exp" the test belongs to. --- .../testsuite-management/validate_failures.py | 137 +++++++++++++++--- 1 file changed, 115 insertions(+), 22 deletions(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 43d9d50af8d..94ba2e58b51 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -64,6 +64,16 @@ import sys _VALID_TEST_RESULTS = [ 'FAIL', 'UNRESOLVED', 'XPASS', 'ERROR' ] _VALID_TEST_RESULTS_REX = re.compile("%s" % "|".join(_VALID_TEST_RESULTS)) +# Formats of .sum file sections +_TOOL_LINE_FORMAT = '\t\t=== %s tests ===\n' +_EXP_LINE_FORMAT = '\nRunning %s ...\n' +_SUMMARY_LINE_FORMAT = '\n\t\t=== %s Summary ===\n' + +# ... and their compiled regexs. +_TOOL_LINE_REX = re.compile('^\t\t=== (.*) tests ===\n') +_EXP_LINE_REX = re.compile('^Running (.*\.exp) \.\.\.\n') +_SUMMARY_LINE_REX = re.compile('^\t\t=== (.*) Summary ===\n') + # Subdirectory of srcdir in which to find the manifest file. _MANIFEST_SUBDIR = 'contrib/testsuite-management' @@ -111,9 +121,11 @@ class TestResult(object): ordinal: Monotonically increasing integer. It is used to keep results for one .exp file sorted by the order the tests were run. + tool: Top-level testsuite name (aka "tool" in DejaGnu parlance) of the test. + exp: Name of .exp testsuite file. """ - def __init__(self, summary_line, ordinal=-1): + def __init__(self, summary_line, ordinal, tool, exp): try: (self.attrs, summary_line) = SplitAttributesFromSummaryLine(summary_line) try: @@ -125,6 +137,12 @@ class TestResult(object): print('Failed to parse summary line: "%s"' % summary_line) raise self.ordinal = ordinal + if tool == None or exp == None: + # .sum file seem to be broken. There was no "tool" and/or "exp" + # lines preceding this result. + raise + self.tool = tool + self.exp = exp except ValueError: Error('Cannot parse summary line "%s"' % summary_line) @@ -133,14 +151,27 @@ class TestResult(object): self.state, summary_line, self)) def __lt__(self, other): - return (self.name < other.name or - (self.name == other.name and self.ordinal < other.ordinal)) + if (self.tool != other.tool): + return self.tool < other.tool + if (self.exp != other.exp): + return self.exp < other.exp + if (self.name != other.name): + return self.name < other.name + return self.ordinal < other.ordinal def __hash__(self): - return hash(self.state) ^ hash(self.name) ^ hash(self.description) - + return (hash(self.state) ^ hash(self.tool) ^ hash(self.exp) + ^ hash(self.name) ^ hash(self.description)) + + # Note that we don't include "attrs" in this comparison. This means that + # result entries "FAIL: test" and "flaky | FAIL: test" are considered + # the same. Therefore the ResultSet will preserve only the first occurence. + # In practice this means that flaky entries should preceed expected fails + # entries. def __eq__(self, other): return (self.state == other.state and + self.tool == other.tool and + self.exp == other.exp and self.name == other.name and self.description == other.description) @@ -174,6 +205,43 @@ class TestResult(object): return now > expiration_date +class ResultSet(set): + """Describes a set of DejaGNU test results. + This set can be read in from .sum files or emitted as a manifest. + + Attributes: + current_tool: Name of the current top-level DejaGnu testsuite. + current_exp: Name of the current .exp testsuite file. + """ + + def __init__(self): + super().__init__() + self.ResetToolExp() + + def ResetToolExp(self): + self.current_tool = None + self.current_exp = None + + def MakeTestResult(self, summary_line, ordinal=-1): + return TestResult(summary_line, ordinal, + self.current_tool, self.current_exp) + + def Print(self, outfile=sys.stdout): + current_tool = None + current_exp = None + + for result in sorted(self): + if current_tool != result.tool: + current_tool = result.tool + outfile.write(_TOOL_LINE_FORMAT % current_tool) + if current_exp != result.exp: + current_exp = result.exp + outfile.write(_EXP_LINE_FORMAT % current_exp) + outfile.write('%s\n' % result) + + outfile.write(_SUMMARY_LINE_FORMAT % 'Results') + + def GetMakefileValue(makefile_name, value_name): if os.path.exists(makefile_name): makefile = open(makefile_name, encoding='latin-1', mode='r') @@ -216,6 +284,21 @@ def IsInterestingResult(line): return bool(_VALID_TEST_RESULTS_REX.match(line)) +def IsToolLine(line): + """Return True if line mentions the tool (in DejaGnu terms) for the following tests.""" + return bool(_TOOL_LINE_REX.match(line)) + + +def IsExpLine(line): + """Return True if line mentions the .exp file for the following tests.""" + return bool(_EXP_LINE_REX.match(line)) + + +def IsSummaryLine(line): + """Return True if line starts .sum footer.""" + return bool(_SUMMARY_LINE_REX.match(line)) + + def IsInclude(line): """Return True if line is an include of another file.""" return line.startswith("@include ") @@ -244,18 +327,24 @@ def ParseManifestWorker(result_set, manifest_path): if _OPTIONS.verbosity >= 1: print('Parsing manifest file %s.' % manifest_path) manifest_file = open(manifest_path, encoding='latin-1', mode='r') - for line in manifest_file: - line = line.strip() + for orig_line in manifest_file: + line = orig_line.strip() if line == "": pass elif IsComment(line): pass elif IsNegativeResult(line): - result_set.remove(TestResult(GetNegativeResult(line))) + result_set.remove(result_set.MakeTestResult(GetNegativeResult(line))) elif IsInclude(line): ParseManifestWorker(result_set, GetIncludeFile(line, manifest_path)) elif IsInterestingResult(line): - result_set.add(TestResult(line)) + result_set.add(result_set.MakeTestResult(line)) + elif IsExpLine(orig_line): + result_set.current_exp = _EXP_LINE_REX.match(orig_line).groups()[0] + elif IsToolLine(orig_line): + result_set.current_tool = _TOOL_LINE_REX.match(orig_line).groups()[0] + elif IsSummaryLine(orig_line): + result_set.ResetToolExp() else: Error('Unrecognized line in manifest file: %s' % line) manifest_file.close() @@ -263,21 +352,21 @@ def ParseManifestWorker(result_set, manifest_path): def ParseManifest(manifest_path): """Create a set of TestResult instances from the given manifest file.""" - result_set = set() + result_set = ResultSet() ParseManifestWorker(result_set, manifest_path) return result_set def ParseSummary(sum_fname): """Create a set of TestResult instances from the given summary file.""" - result_set = set() + result_set = ResultSet() # ordinal is used when sorting the results so that tests within each # .exp file are kept sorted. ordinal=0 sum_file = open(sum_fname, encoding='latin-1', mode='r') for line in sum_file: if IsInterestingResult(line): - result = TestResult(line, ordinal) + result = result_set.MakeTestResult(line, ordinal) ordinal += 1 if result.HasExpired(): # Tests that have expired are not added to the set of expected @@ -286,6 +375,13 @@ def ParseSummary(sum_fname): print('WARNING: Expected failure "%s" has expired.' % line.strip()) continue result_set.add(result) + elif IsExpLine(line): + result_set.current_exp = _EXP_LINE_REX.match(line).groups()[0] + elif IsToolLine(line): + result_set.current_tool = _TOOL_LINE_REX.match(line).groups()[0] + result_set.current_exp = None + elif IsSummaryLine(line): + result_set.ResetToolExp() sum_file.close() return result_set @@ -301,7 +397,7 @@ def GetManifest(manifest_path): if os.path.exists(manifest_path): return ParseManifest(manifest_path) else: - return set() + return ResultSet() def CollectSumFiles(builddir): @@ -318,7 +414,7 @@ def CollectSumFiles(builddir): def GetResults(sum_files): """Collect all the test results from the given .sum files.""" - build_results = set() + build_results = ResultSet() for sum_fname in sum_files: print('\t%s' % sum_fname) build_results |= ParseSummary(sum_fname) @@ -332,7 +428,7 @@ def CompareResults(manifest, actual): """ # Collect all the actual results not present in the manifest. # Results in this set will be reported as errors. - actual_vs_manifest = set() + actual_vs_manifest = ResultSet() for actual_result in actual: if actual_result not in manifest: actual_vs_manifest.add(actual_result) @@ -341,7 +437,7 @@ def CompareResults(manifest, actual): # in the actual results. # Results in this set will be reported as warnings (since # they are expected failures that are not failing anymore). - manifest_vs_actual = set() + manifest_vs_actual = ResultSet() for expected_result in manifest: # Ignore tests marked flaky. if 'flaky' in expected_result.attrs: @@ -390,9 +486,7 @@ def GetBuildData(): def PrintSummary(msg, summary): print('\n\n%s' % msg) - for result in sorted(summary): - print(result) - + summary.Print() def GetSumFiles(results, build_dir): if not results: @@ -452,9 +546,8 @@ def ProduceManifest(): sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.build_dir) actual = GetResults(sum_files) manifest_file = open(manifest_path, encoding='latin-1', mode='w') - for result in sorted(actual): - print(result) - manifest_file.write('%s\n' % result) + actual.Print(manifest_file) + actual.Print() manifest_file.close() return True From patchwork Fri Jun 2 15:20:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70521 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 535BC384F1D3 for ; Fri, 2 Jun 2023 15:22:48 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 535BC384F1D3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719368; bh=N8Y+TY/KH8yB3KaXgX/oBowRbFs/NbLqwPU1kxXlxUc=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=ACwr569vCqGlcuHcHJ6fm8Cd44eI3ObgJx1fcnXIzbnORgOE/4qGqtZ95bpMX17KL rr3zZS9XDvwQYci6Lk5Im9V8BvwMFpy64+hSMFT2tSCDu+JnXZ9QMYZcnbyIw9nRCl U9CSMFO3OzE9z8x4xHH2+eVCzDmzwvjJqcmvNTYw= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x22e.google.com (mail-lj1-x22e.google.com [IPv6:2a00:1450:4864:20::22e]) by sourceware.org (Postfix) with ESMTPS id 420093857714 for ; Fri, 2 Jun 2023 15:21:19 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 420093857714 Received: by mail-lj1-x22e.google.com with SMTP id 38308e7fff4ca-2b1a4be0ba1so2785011fa.0 for ; Fri, 02 Jun 2023 08:21:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719277; x=1688311277; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N8Y+TY/KH8yB3KaXgX/oBowRbFs/NbLqwPU1kxXlxUc=; b=N6nXBFihbl67nHbd/NsMYwc7EooIkxggah+6W3ZY+dzDkLU38uic9UqNphGYOZ+OzV GuW/qV4kempOdq4hmJgJkeL+5ubBV7BNe4GVbqNmPrxE4215Jh6HSVipXmNHGjY9cx7r nIjUrZJ0LHr78pPdZunDKXQp2Pm2k3MknDwt3pjMv4hXqYl95zLuOY+XzjSbHG75N1Ny cFNT4Szc/ilxttV/KY4BumU8moHtzKNu8BCSk6XhWx5iyMILVWGFm2upgD84JqJPqMtf 204kinWe7A45WfBYlWbFBOHv/EaZH8CUiE1yvY6sbEflRYk/f0ngad1BNaLuf3BIL4St aueA== X-Gm-Message-State: AC+VfDxRQq0qIrxWCzqrp2W8mjw73DMqYLMUOQyH7KXWd0xJ6bNWQp/I 9UXAJsyLX6FGh4oNz8lmCedNR+9oKiatmBkO1BlY54k= X-Google-Smtp-Source: ACHHUZ62ZSv07MGAkH/QtxdJ+VI36dNGUmHd0XtL5LpC2k0NpYlE0Xyl0OmqER5O3Rwez4Cs9yem9w== X-Received: by 2002:a05:651c:b99:b0:2b1:acd4:162c with SMTP id bg25-20020a05651c0b9900b002b1acd4162cmr1142079ljb.4.1685719277253; Fri, 02 Jun 2023 08:21:17 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:15 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 02/12] [contrib] validate_failures.py: Support expiry attributes in manifests Date: Fri, 2 Jun 2023 15:20:42 +0000 Message-Id: <20230602152052.1874860-3-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" --- contrib/testsuite-management/validate_failures.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 94ba2e58b51..7351ba120b7 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -338,7 +338,13 @@ def ParseManifestWorker(result_set, manifest_path): elif IsInclude(line): ParseManifestWorker(result_set, GetIncludeFile(line, manifest_path)) elif IsInterestingResult(line): - result_set.add(result_set.MakeTestResult(line)) + result = result_set.MakeTestResult(line) + if result.HasExpired(): + # Ignore expired manifest entries. + if _OPTIONS.verbosity >= 4: + print('WARNING: Expected failure "%s" has expired.' % line.strip()) + continue + result_set.add(result) elif IsExpLine(orig_line): result_set.current_exp = _EXP_LINE_REX.match(orig_line).groups()[0] elif IsToolLine(orig_line): @@ -369,6 +375,8 @@ def ParseSummary(sum_fname): result = result_set.MakeTestResult(line, ordinal) ordinal += 1 if result.HasExpired(): + # ??? What is the use-case for this? How "expiry" annotations are + # ??? supposed to be added to .sum results? # Tests that have expired are not added to the set of expected # results. If they are still present in the set of actual results, # they will cause an error to be reported. From patchwork Fri Jun 2 15:20:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70520 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 7268E38515C9 for ; Fri, 2 Jun 2023 15:22:33 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7268E38515C9 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719353; bh=ixW3JZDLCjI2AmfDvTRWK2EjdWamQ9DMDYZmrg/mE/Q=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=iYS4LZ8LbNnKOP59P4XX/GEVDQ2trWIgv6uikFxmfMIlSUuScEPV/xpX0dys231Vl bshu+NHAtoS/+NMsqophtxyFMvQzIfqFvQdxTIffJvrpJWppEKwVbSn50bmHiWHFwt 2qrhS91de4HcXVL1+7kD8Fb/FdcfQ26O2JDvQDU4= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) by sourceware.org (Postfix) with ESMTPS id 65AAA3857719 for ; Fri, 2 Jun 2023 15:21:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 65AAA3857719 Received: by mail-lj1-x229.google.com with SMTP id 38308e7fff4ca-2b12834a65bso1785341fa.1 for ; Fri, 02 Jun 2023 08:21:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719280; x=1688311280; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ixW3JZDLCjI2AmfDvTRWK2EjdWamQ9DMDYZmrg/mE/Q=; b=Ngk+WFbuNF1knNPrzeI9E+qRrgt2Vk9vv0n/neVhHVcQqpGJPmgS00Qn3YBu0Tx7jM jIjhDSxHNXHoQYf1BWwCN4Gc8Ow0/23IXqPJnD/jWE0/UV84ehUAoFmJ5vLAWU2e65e1 PMb877qBvYKC2f7QHv5fXZ6V7/oKRWiYzhQCdbJc7Nsg+zSbOZSes/CuhPgW/VR1fejF 7PynXBoLKmiL2k5L8vS98O2NGCQnwBZtoIJsFeb85YdsGR8dqpgIsJKx+OeE8M2b4KeB xD7Czw6Ju5siG799MZ2A5SaD2/korvygBqO3g38JiT4oPHiutn9m5uhm/jYdlJNksQFJ WqDA== X-Gm-Message-State: AC+VfDwp79k5/8IXlRpBtNP0uVhh/PYfg7FN0b6Mv4FXmLQxYfj+Mt5c pMbR7KnPGhLDvS0Szo6v/um26/cE2FPtM3xS5mGkYCA= X-Google-Smtp-Source: ACHHUZ4lqbyGQtFKrU5vLoxbW6eiMEs9jgLXCtwiYULbv3MbzpCbPyCEP4SyTSJ0l9yJVco9uHUdZA== X-Received: by 2002:a05:651c:2ce:b0:2b0:361f:80e1 with SMTP id f14-20020a05651c02ce00b002b0361f80e1mr5570940ljo.4.1685719280219; Fri, 02 Jun 2023 08:21:20 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:19 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 03/12] [contrib] validate_failures.py: Read in manifest when comparing build dirs Date: Fri, 2 Jun 2023 15:20:43 +0000 Message-Id: <20230602152052.1874860-4-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" This allows comparison of two build directories with a manifest listing known flaky tests on the side. --- contrib/testsuite-management/validate_failures.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 7351ba120b7..4733dd89dcb 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -420,9 +420,10 @@ def CollectSumFiles(builddir): return sum_files -def GetResults(sum_files): +def GetResults(sum_files, build_results = None): """Collect all the test results from the given .sum files.""" - build_results = ResultSet() + if build_results == None: + build_results = ResultSet() for sum_fname in sum_files: print('\t%s' % sum_fname) build_results |= ParseSummary(sum_fname) @@ -567,8 +568,15 @@ def CompareBuilds(): sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.build_dir) actual = GetResults(sum_files) + clean = ResultSet() + + if _OPTIONS.manifest: + manifest_path = GetManifestPath(srcdir, target, True) + print('Manifest: %s' % manifest_path) + clean = GetManifest(manifest_path) + clean_sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.clean_build) - clean = GetResults(clean_sum_files) + clean = GetResults(clean_sum_files, clean) return PerformComparison(clean, actual, _OPTIONS.ignore_missing_failures) From patchwork Fri Jun 2 15:20:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70518 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 7A6223856600 for ; Fri, 2 Jun 2023 15:21:54 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7A6223856600 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719314; bh=vOMV6dKNLkWxbMOXtMGHhFaHhab9ak8qmS1731nXkkg=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=AbF2I7ZNPH66bXVhGoPq8vAj0BBMRwod7TmCuH/3sbawLL08Tniy0DONgPdWR9+dv vHBFg0ltsfiyDSbdg+dVYFGUhw84E9Ou8Vi+bzKPrERAnmxeH6THmR9UVO9RrW0xuI 3t1Ch9i4OOcRds4XfUjIs5QTrC6ERsLFAwQqXGS0= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c]) by sourceware.org (Postfix) with ESMTPS id 6BF4C3857711 for ; Fri, 2 Jun 2023 15:21:24 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6BF4C3857711 Received: by mail-lj1-x22c.google.com with SMTP id 38308e7fff4ca-2b035e94072so2729541fa.1 for ; Fri, 02 Jun 2023 08:21:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719282; x=1688311282; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vOMV6dKNLkWxbMOXtMGHhFaHhab9ak8qmS1731nXkkg=; b=i3wBIpa3bl1kQZ8+2FyMM84tZRD2r7zHS4oSSHOi6PaMhrLHu7yZ2nMERyEfigP1SX 4GQQ1zDbV58v+dVaJ8ijG8DozMTydIB3uIgqKeyQtzQqYJguAUlqCRDTMm9wa5CTB5/h Ib1+tQEjFInFr2VpT6hG6+/MqHrylOBO9L+mJjvujJ1RGMZzJXPNDqzNkTdYg/10Qsep dY4GIqkadNmF+UVMKjKdNLQAIpPG4RDCKXRZUcTFzN+CXsCgXQkGcirfKeVrQLSZCTSF t866Dr5HcLUwINnWi/vcQrOiNXHa1vs9yWq2xIDiHLXjQHHC1yScLk80tC0Df69/67qv VXMw== X-Gm-Message-State: AC+VfDza+wAPHpnIjHz1iB3fQtgCMXOOT9F2vciaIn3KxBr0+TOp1Xzq atu0kaLzJdEECg/1iTdEZ8GJlgZ9cpmCOO7xbkgSt4Q= X-Google-Smtp-Source: ACHHUZ5nmdngJmdOzORDjurEv2SlWrTBkxuHWm3d8CdwoiRZtVE5/PcFCqd2HkCtVcPgr8SHjqvjpg== X-Received: by 2002:a2e:b55a:0:b0:2ad:9d06:8c6e with SMTP id a26-20020a2eb55a000000b002ad9d068c6emr4508655ljn.0.1685719282258; Fri, 02 Jun 2023 08:21:22 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:20 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 04/12] [contrib] validate_failures.py: Simplify GetManifestPath() Date: Fri, 2 Jun 2023 15:20:44 +0000 Message-Id: <20230602152052.1874860-5-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" ... and don't require a valid build directory when no data from it is necessary. --- contrib/testsuite-management/validate_failures.py | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 4733dd89dcb..1bd09e0c20c 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -457,7 +457,7 @@ def CompareResults(manifest, actual): return actual_vs_manifest, manifest_vs_actual -def GetManifestPath(srcdir, target, user_provided_must_exist): +def GetManifestPath(user_provided_must_exist): """Return the full path to the manifest file.""" manifest_path = _OPTIONS.manifest if manifest_path: @@ -465,6 +465,7 @@ def GetManifestPath(srcdir, target, user_provided_must_exist): Error('Manifest does not exist: %s' % manifest_path) return manifest_path else: + (srcdir, target) = GetBuildData() if not srcdir: Error('Could not determine the location of GCC\'s source tree. ' 'The Makefile does not contain a definition for "srcdir".') @@ -530,8 +531,7 @@ def PerformComparison(expected, actual, ignore_missing_failures): def CheckExpectedResults(): - srcdir, target = GetBuildData() - manifest_path = GetManifestPath(srcdir, target, True) + manifest_path = GetManifestPath(True) print('Manifest: %s' % manifest_path) manifest = GetManifest(manifest_path) sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.build_dir) @@ -545,8 +545,7 @@ def CheckExpectedResults(): def ProduceManifest(): - (srcdir, target) = GetBuildData() - manifest_path = GetManifestPath(srcdir, target, False) + manifest_path = GetManifestPath(False) print('Manifest: %s' % manifest_path) if os.path.exists(manifest_path) and not _OPTIONS.force: Error('Manifest file %s already exists.\nUse --force to overwrite.' % @@ -563,15 +562,13 @@ def ProduceManifest(): def CompareBuilds(): - (srcdir, target) = GetBuildData() - sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.build_dir) actual = GetResults(sum_files) clean = ResultSet() if _OPTIONS.manifest: - manifest_path = GetManifestPath(srcdir, target, True) + manifest_path = GetManifestPath(True) print('Manifest: %s' % manifest_path) clean = GetManifest(manifest_path) From patchwork Fri Jun 2 15:20:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70523 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D18A7385624C for ; Fri, 2 Jun 2023 15:23:31 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D18A7385624C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719411; bh=PgHWyQGQe8J/X0vEjTsRjanH8gSTVOI0LdUsVAzAAc8=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=tcwsADDygEY5uiubK7pRR8UqmRcwzaSzi7u+31FZdLsT8RIG/bitbneH0WycBDD7g wWiFCvvr8k9XscINT2bdXBi7apIQYg798+MtoukVvgNQc6tcjICZ+StKPsP0IkJkZK YHu6rF5vvctyOq6myKd8VgzhkhJzyFU9gfEuV24Q= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x234.google.com (mail-lj1-x234.google.com [IPv6:2a00:1450:4864:20::234]) by sourceware.org (Postfix) with ESMTPS id D87D7385589C for ; Fri, 2 Jun 2023 15:21:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D87D7385589C Received: by mail-lj1-x234.google.com with SMTP id 38308e7fff4ca-2b072753301so2345021fa.1 for ; Fri, 02 Jun 2023 08:21:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719304; x=1688311304; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PgHWyQGQe8J/X0vEjTsRjanH8gSTVOI0LdUsVAzAAc8=; b=TbXu18LWqgEwAK1vQP40LvpHlIyJj7weXVCRPZYbz3zD8IW8/YHJjpjYz/FnVQsF0A AiMqnrjKGuPeZJWwLIMjwZEGTSnO/SRpDvdkYIXAr5Z+iM4nPQ7yn4uvrYtaYAy3rMEg Ly1FVHS7OWI2BoLxyvzjN6IlrFnwsddc42zt0gWVZD0OSTq90KR8KHTJZVOchQtZloSz B+s1HjgJmKGfAtEPnOWXEbTVMl+NfLkNfTMQTYOKF55igVUMbHYZ7Kh+e/C8qiMRUVEq vwNOiXQyzo5ouc6D7BefmK2xSCCV4bPB6LhEvCUxzIKpq42QueqsbCkvdDLfeXfze5XS /bBQ== X-Gm-Message-State: AC+VfDwIFHl0wMHm36FwBdY4XRJTMvoWfyocOQG2M3jwlUZWFjl9DXTv 84bKHvt7v99BYlseepgiv5IxAtWrW+gNTZSvS6ysfV4= X-Google-Smtp-Source: ACHHUZ49A7PmEz2h2bE3Yt0m8tJZkrasM+Tkjo6AqboWj/MN3OX5FpSoj1YBlSczy9uX/e4vEq71EQ== X-Received: by 2002:a2e:a4ac:0:b0:2b1:a69e:6a8e with SMTP id g12-20020a2ea4ac000000b002b1a69e6a8emr2216523ljm.3.1685719303894; Fri, 02 Jun 2023 08:21:43 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:22 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 05/12] [contrib] validate_failures.py: Add more verbosity levels Date: Fri, 2 Jun 2023 15:20:45 +0000 Message-Id: <20230602152052.1874860-6-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" ... to control validate_failures.py output --- .../testsuite-management/validate_failures.py | 82 +++++++++++-------- 1 file changed, 46 insertions(+), 36 deletions(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 1bd09e0c20c..26ea1d6f53b 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -324,7 +324,7 @@ def GetNegativeResult(line): def ParseManifestWorker(result_set, manifest_path): """Read manifest_path, adding the contents to result_set.""" - if _OPTIONS.verbosity >= 1: + if _OPTIONS.verbosity >= 5: print('Parsing manifest file %s.' % manifest_path) manifest_file = open(manifest_path, encoding='latin-1', mode='r') for orig_line in manifest_file: @@ -380,7 +380,8 @@ def ParseSummary(sum_fname): # Tests that have expired are not added to the set of expected # results. If they are still present in the set of actual results, # they will cause an error to be reported. - print('WARNING: Expected failure "%s" has expired.' % line.strip()) + if _OPTIONS.verbosity >= 4: + print('WARNING: Expected failure "%s" has expired.' % line.strip()) continue result_set.add(result) elif IsExpLine(line): @@ -425,7 +426,8 @@ def GetResults(sum_files, build_results = None): if build_results == None: build_results = ResultSet() for sum_fname in sum_files: - print('\t%s' % sum_fname) + if _OPTIONS.verbosity >= 3: + print('\t%s' % sum_fname) build_results |= ParseSummary(sum_fname) return build_results @@ -489,42 +491,46 @@ def GetBuildData(): return None, None srcdir = GetMakefileValue('%s/Makefile' % _OPTIONS.build_dir, 'srcdir =') target = GetMakefileValue('%s/Makefile' % _OPTIONS.build_dir, 'target_alias=') - print('Source directory: %s' % srcdir) - print('Build target: %s' % target) + if _OPTIONS.verbosity >= 3: + print('Source directory: %s' % srcdir) + print('Build target: %s' % target) return srcdir, target -def PrintSummary(msg, summary): - print('\n\n%s' % msg) +def PrintSummary(summary): summary.Print() def GetSumFiles(results, build_dir): if not results: - print('Getting actual results from build directory %s' % build_dir) + if _OPTIONS.verbosity >= 3: + print('Getting actual results from build directory %s' % build_dir) sum_files = CollectSumFiles(build_dir) else: - print('Getting actual results from user-provided results') + if _OPTIONS.verbosity >= 3: + print('Getting actual results from user-provided results') sum_files = results.split() return sum_files -def PerformComparison(expected, actual, ignore_missing_failures): +def PerformComparison(expected, actual): actual_vs_expected, expected_vs_actual = CompareResults(expected, actual) tests_ok = True if len(actual_vs_expected) > 0: - PrintSummary('Unexpected results in this build (new failures)', - actual_vs_expected) + if _OPTIONS.verbosity >= 3: + print('\n\nUnexpected results in this build (new failures)') + if _OPTIONS.verbosity >= 1: + PrintSummary(actual_vs_expected) tests_ok = False - if not ignore_missing_failures and len(expected_vs_actual) > 0: - PrintSummary('Expected results not present in this build (fixed tests)' - '\n\nNOTE: This is not a failure. It just means that these ' - 'tests were expected\nto fail, but either they worked in ' - 'this configuration or they were not\npresent at all.\n', - expected_vs_actual) + if _OPTIONS.verbosity >= 2 and len(expected_vs_actual) > 0: + print('\n\nExpected results not present in this build (fixed tests)' + '\n\nNOTE: This is not a failure. It just means that these ' + 'tests were expected\nto fail, but either they worked in ' + 'this configuration or they were not\npresent at all.\n') + PrintSummary(expected_vs_actual) - if tests_ok: + if tests_ok and _OPTIONS.verbosity >= 3: print('\nSUCCESS: No unexpected failures.') return tests_ok @@ -532,21 +538,25 @@ def PerformComparison(expected, actual, ignore_missing_failures): def CheckExpectedResults(): manifest_path = GetManifestPath(True) - print('Manifest: %s' % manifest_path) + if _OPTIONS.verbosity >= 3: + print('Manifest: %s' % manifest_path) manifest = GetManifest(manifest_path) sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.build_dir) actual = GetResults(sum_files) - if _OPTIONS.verbosity >= 1: - PrintSummary('Tests expected to fail', manifest) - PrintSummary('\nActual test results', actual) + if _OPTIONS.verbosity >= 5: + print('\n\nTests expected to fail') + PrintSummary(manifest) + print('\n\nActual test results') + PrintSummary(actual) - return PerformComparison(manifest, actual, _OPTIONS.ignore_missing_failures) + return PerformComparison(manifest, actual) def ProduceManifest(): manifest_path = GetManifestPath(False) - print('Manifest: %s' % manifest_path) + if _OPTIONS.verbosity >= 3: + print('Manifest: %s' % manifest_path) if os.path.exists(manifest_path) and not _OPTIONS.force: Error('Manifest file %s already exists.\nUse --force to overwrite.' % manifest_path) @@ -569,13 +579,14 @@ def CompareBuilds(): if _OPTIONS.manifest: manifest_path = GetManifestPath(True) - print('Manifest: %s' % manifest_path) + if _OPTIONS.verbosity >= 3: + print('Manifest: %s' % manifest_path) clean = GetManifest(manifest_path) clean_sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.clean_build) clean = GetResults(clean_sum_files, clean) - return PerformComparison(clean, actual, _OPTIONS.ignore_missing_failures) + return PerformComparison(clean, actual) def Main(argv): @@ -597,14 +608,6 @@ def Main(argv): default=False, help='When used with --produce_manifest, ' 'it will overwrite an existing manifest file ' '(default = False)') - parser.add_option('--ignore_missing_failures', action='store_true', - dest='ignore_missing_failures', default=False, - help='When a failure is expected in the manifest but ' - 'it is not found in the actual results, the script ' - 'produces a note alerting to this fact. This means ' - 'that the expected failure has been fixed, or ' - 'it did not run, or it may simply be flaky ' - '(default = False)') parser.add_option('--manifest', action='store', type='string', dest='manifest', default=None, help='Name of the manifest file to use (default = ' @@ -621,7 +624,14 @@ def Main(argv): 'starting with FAIL, XPASS or UNRESOLVED (default = ' '.sum files collected from the build directory).') parser.add_option('--verbosity', action='store', dest='verbosity', - type='int', default=0, help='Verbosity level (default = 0)') + type='int', default=3, help='Verbosity level ' + '(default = 3). Level 0: only error output, this is ' + 'useful in scripting when only the exit code is used. ' + 'Level 1: output unexpected failures. ' + 'Level 2: output unexpected passes. ' + 'Level 3: output helpful information. ' + 'Level 4: output notification on expired entries. ' + 'Level 5: output debug information.') global _OPTIONS (_OPTIONS, _) = parser.parse_args(argv[1:]) From patchwork Fri Jun 2 15:20:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70522 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A896D385414F for ; Fri, 2 Jun 2023 15:23:02 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A896D385414F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719382; bh=MHExmFQlO7OiX+5ARLxny2T+Rwu+tq4GkyRo+LhevqE=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=OPJJmQp5VtyrfUpl6sKV77/VEsOUyumcCwo6ndM1H1ksIeRH/f0BWs9VxAWiHw5+t 70QU4fmteDD4xc8TAMaWD7BncdfZtm4/tuYiuyzJbFT9IOJCJLfhMo1x5I1uYOtqq6 /7cxksMGbQABPY8ZxvZFkuhRVYdLld9L+JCqGhO8= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x235.google.com (mail-lj1-x235.google.com [IPv6:2a00:1450:4864:20::235]) by sourceware.org (Postfix) with ESMTPS id 332663856618 for ; Fri, 2 Jun 2023 15:21:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 332663856618 Received: by mail-lj1-x235.google.com with SMTP id 38308e7fff4ca-2b072753301so2345111fa.1 for ; Fri, 02 Jun 2023 08:21:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719306; x=1688311306; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MHExmFQlO7OiX+5ARLxny2T+Rwu+tq4GkyRo+LhevqE=; b=ftZgGzD5JgfQNqvjQisinvuLrRWxnEzptwUdwlndJFZoip0pRuvzNWxsGA1xnoXMuD pERC7umld2GegLv4+sSS0/DPrRpOMDfrG4zKCPy4/NirI+PkGFZ9RdlKOAepn/vnVVpy 9tedL1E/CaDQ7vMG8yB0ZF9abJLjMh3VRhugZyLRLOiE6f3FHWsuX5ZlLFd3aNW8Irxs yoGY+45hAVHy8ATwMGkfDx3ciimDAbxIQIye1vyk86yBmDDnjfd2uhQzVCb7a9zwOziM gF04VyZtYyt7T+RqwHPZp/kadVJNFXvYeJ9mOmj/9jP5T1a8u9jhfI4ptdNKWZh495Cu 05bA== X-Gm-Message-State: AC+VfDx2bawexYXjZ6UcMYki6tg73+GcNMXzGQLfE5OxiMPs4CWMlvK9 F6S4drgCZwrvQIxMpkdr4YHxq/mIgZ/mN3wqSfWywBI= X-Google-Smtp-Source: ACHHUZ66pTKF9ojGZ4PmJHwIvd/c8vxiwO35b+jwG8sEoNRQWTi9jXMWDSMzSv/9CtZEtIcGXTha4w== X-Received: by 2002:a2e:b55a:0:b0:2b1:5a32:7bef with SMTP id a26-20020a2eb55a000000b002b15a327befmr2723291ljn.5.1685719305902; Fri, 02 Jun 2023 08:21:45 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:44 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 06/12] [contrib] validate_failures.py: Be more stringent in parsing result lines Date: Fri, 2 Jun 2023 15:20:46 +0000 Message-Id: <20230602152052.1874860-7-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" Before this patch we would identify malformed line "UNRESOLVEDTest run by tcwg-buildslave on Mon Aug 23 10:17:50 2021" as an interesting result, only to fail in TestResult:__init__ due to missing ":" after UNRESOLVED. This patch makes all places that parse result lines use a single compiled regex. --- contrib/testsuite-management/validate_failures.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 26ea1d6f53b..f2d7b099d78 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -60,9 +60,10 @@ import os import re import sys -# Handled test results. _VALID_TEST_RESULTS = [ 'FAIL', 'UNRESOLVED', 'XPASS', 'ERROR' ] -_VALID_TEST_RESULTS_REX = re.compile("%s" % "|".join(_VALID_TEST_RESULTS)) +# : X-Patchwork-Id: 70524 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8D82D38449DC for ; Fri, 2 Jun 2023 15:23:46 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 8D82D38449DC DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719426; bh=VEsqlXJY5bHMUCoK4K4uTgPE58ye0ANw73IxP0tRZ0o=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=uoRdK1f3CAwHtizxT3Kh7Lbk5ujmqLTjM5ogIJ1BWmhLhS5iQPLLpQJVzISC2A4cZ omWgnH21MEVJVzj3cEsecw5I70yiFRPuRt//Cn9wWjvjM5FH5O7UUcFZ73AR+IXGhx JKcaoUP0YiLwYOnDKnKL/hbYhq0a0O7l9pMNOHDU= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) by sourceware.org (Postfix) with ESMTPS id CECA33856604 for ; Fri, 2 Jun 2023 15:21:49 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org CECA33856604 Received: by mail-lf1-x12e.google.com with SMTP id 2adb3069b0e04-4f13c41c957so179902e87.1 for ; Fri, 02 Jun 2023 08:21:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719308; x=1688311308; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VEsqlXJY5bHMUCoK4K4uTgPE58ye0ANw73IxP0tRZ0o=; b=j6Gg531HUzIM461DLiE5iqsjsF/zEJ4VoUathmS/OGfZar8nfGqjhl8AUtHSKShfDT mul0jQK2LMyCAxqRnnfC8ckbnUsT5AenNDgBSnxoM59m4f0qpKci5cRNysajgLF95nck RT4TIo5ZDa9G52k10wTpMUTy0FGMXKfCdpekbs5SVQAIImrhaU1Ay1k/neOYqrRB3CKN HoECOIWaW7RUrnzDMP8bMHapgJp+JnjhfS2Ka3asGcLrnTjWkv+KxcHJEJTcqCN83HbT 3VBM7cCJpDUOKQSjIQAsXZrVgifzNw1S1HG7l7by9Qy/G9lzDMDdniW2e0D91KD4OGvc CY1g== X-Gm-Message-State: AC+VfDwdCNT5sfppFA88qvLlr83nt+oFC1xS8IjYDMUczT0JyxbDehKU /yhuZMPGWywFBcrPBMGlH/rdoWvNv68kGhANotnlG9g= X-Google-Smtp-Source: ACHHUZ6R9JpVvD2rt33Qxsr0EgGSzfpr0x7pTW7H7E5if9/PSC8p0PNQ6KUrQztQAjamJk8JiLG73Q== X-Received: by 2002:a05:651c:1251:b0:2ac:767c:fba0 with SMTP id h17-20020a05651c125100b002ac767cfba0mr4407767ljh.2.1685719307893; Fri, 02 Jun 2023 08:21:47 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:46 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 07/12] [contrib] validate_failures.py: Use exit code "2" to indicate regression Date: Fri, 2 Jun 2023 15:20:47 +0000 Message-Id: <20230602152052.1874860-8-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" ... in the results. Python exits with code "1" on exceptions and internal errors, which we use to detect failure to parse results. --- contrib/testsuite-management/validate_failures.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index f2d7b099d78..c4b9fc377ce 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -645,7 +645,7 @@ def Main(argv): if retval: return 0 else: - return 1 + return 2 if __name__ == '__main__': From patchwork Fri Jun 2 15:20:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70527 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4EA653874842 for ; Fri, 2 Jun 2023 15:24:44 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 4EA653874842 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719484; bh=Pliar+4onTE6ABkdjBDjqTCiPEw8Xpesa7S8TGuSmJo=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=rmND/8ReydedchGlgHguoM8+FGOK836YSh2I+mfJFW6dQ7cWbx1Wv8g1IYC3jJHNU eM9yA0lPMWkXjCtKzOSH2oogVcFkZ9UPdr34rbvfkG8IKGL6IkarzhjgO1RySaRKJc m4rIcaqvTb1NFwE+2SFlrbpT+UBWkMcRAawaAjfM= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x235.google.com (mail-lj1-x235.google.com [IPv6:2a00:1450:4864:20::235]) by sourceware.org (Postfix) with ESMTPS id 43CDC385559E for ; Fri, 2 Jun 2023 15:21:52 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 43CDC385559E Received: by mail-lj1-x235.google.com with SMTP id 38308e7fff4ca-2b1b5d1bbf1so569331fa.1 for ; Fri, 02 Jun 2023 08:21:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719310; x=1688311310; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Pliar+4onTE6ABkdjBDjqTCiPEw8Xpesa7S8TGuSmJo=; b=L8Jvh7dTJVI7eKVYMqZ+276cVMyzSpIeoDHn1nch3g8fmjqW2Jo7pqKXUNSs4Kc2tI zurnrGUnfMtGCwQoCX7rH9IGewxuw4fBxvoqZEi+x1DzU8MY/mI1jlCT5rDs88jQKVPE 7eQgeAbqPT6ti2lO7EGCykbR5PilqTM76RN5irWFX08H8R0mYdUZaxkxMsrn3pL2/5XZ Iux5PMkW3OGyA09FY2XJXcp3II9te8yvIXd/hHVw5Y2UAEKy3/Lny7r2WVzvzQRNgiAo O68Hpa85UoJS/DJDdFg+8dA6zaHTRORcFBqXZnmcd6m+kBmu4insW5U8AXGDbjJnmNRp /3gQ== X-Gm-Message-State: AC+VfDwgCW4oSxFFllYkz50truflMyGhWOlrGvCQyfIzf4vY0whtBugJ d52z4CZvmm27WmSV0th7zKa0tq3CL5TKDfb0BGelGrc= X-Google-Smtp-Source: ACHHUZ5pOY4Wh09lry66+jUZxb2FrINI0wVTUHvmkwtO0FwCeFW5ibuJe6wH6tibPT5dFQ45SSFvyA== X-Received: by 2002:a05:651c:118b:b0:2af:1c95:c4a8 with SMTP id w11-20020a05651c118b00b002af1c95c4a8mr4005915ljo.5.1685719310207; Fri, 02 Jun 2023 08:21:50 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:49 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Christophe Lyon Subject: [PATCH 08/12] [contrib] validate_failures.py: Support "$tool:" prefix in exp names Date: Fri, 2 Jun 2023 15:20:48 +0000 Message-Id: <20230602152052.1874860-9-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" From: Christophe Lyon This makes it easier to extract the $tool:$exp pair when iterating over failures/flaky tests, which, in turn, simplifies re-running testsuite parts that have unexpected failures or passes. --- contrib/testsuite-management/validate_failures.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index c4b9fc377ce..6dcdcf5c69b 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -67,12 +67,14 @@ _VALID_TEST_RESULTS_REX = re.compile('(%s):\s*(\S+)\s*(.*)' # Formats of .sum file sections _TOOL_LINE_FORMAT = '\t\t=== %s tests ===\n' -_EXP_LINE_FORMAT = '\nRunning %s ...\n' +_EXP_LINE_FORMAT = '\nRunning %s:%s ...\n' _SUMMARY_LINE_FORMAT = '\n\t\t=== %s Summary ===\n' # ... and their compiled regexs. _TOOL_LINE_REX = re.compile('^\t\t=== (.*) tests ===\n') -_EXP_LINE_REX = re.compile('^Running (.*\.exp) \.\.\.\n') +# Match .exp file name, optionally prefixed by a "tool:" name and a +# path ending with "testsuite/" +_EXP_LINE_REX = re.compile('^Running (?:.*:)?(.*) \.\.\.\n') _SUMMARY_LINE_REX = re.compile('^\t\t=== (.*) Summary ===\n') # Subdirectory of srcdir in which to find the manifest file. @@ -236,7 +238,7 @@ class ResultSet(set): outfile.write(_TOOL_LINE_FORMAT % current_tool) if current_exp != result.exp: current_exp = result.exp - outfile.write(_EXP_LINE_FORMAT % current_exp) + outfile.write(_EXP_LINE_FORMAT % (current_tool, current_exp)) outfile.write('%s\n' % result) outfile.write(_SUMMARY_LINE_FORMAT % 'Results') From patchwork Fri Jun 2 15:20:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70525 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4A55D38715B3 for ; Fri, 2 Jun 2023 15:24:00 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 4A55D38715B3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719440; bh=kEHVymmzGyeT2fcBEqoousHeo1UQdMKPg3UXNuyj9NA=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=S8KMsCFwytvgU4bf4zUNQgKXULMiHnQ7lVXGYmEKno6HKu8+bCuMxmqbfu71YE2X4 2EruchwK5ciguynk1sBa/LwrWeyBfww/Q+koQQBpdrEBaFQz1VEf50HGNvGD3v5Imo 7eFCEIpLBB1eov5BbVUdWSrORws8ucyEDBjV+x3Y= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com [IPv6:2a00:1450:4864:20::22a]) by sourceware.org (Postfix) with ESMTPS id DBD143856975 for ; Fri, 2 Jun 2023 15:21:53 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org DBD143856975 Received: by mail-lj1-x22a.google.com with SMTP id 38308e7fff4ca-2b1b1dd208dso1167601fa.0 for ; Fri, 02 Jun 2023 08:21:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719312; x=1688311312; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kEHVymmzGyeT2fcBEqoousHeo1UQdMKPg3UXNuyj9NA=; b=etptt3oYxvRrTJHxKCLqFNaLgvaUqIV7RUCjvlOHPS9wq7zMe/Hw4AbEXtOV7q05oK FiAAGjwo451YQgqGgVfELZxzHpzK/AtRfbFQF/F4+TrcQvQMpYbl++C58rs5zt37R3Ue 6JfCvltXd0xWMXboSjxGopjDp6P86tQL9mNO7pQg1wQvLmiql9/TCunbOgu38ymsnYal K6y+x/O5dCcslI48uzXz4NJGSgz1r44cdY0fel2eebkias4+lTNGOwGCOrcp7fVVQwGD AaTuCwMe4DLaP+Jo3tg6/Csa7C4HGsT5v5KlV2em5fHys73tPcZa80EwprrZakUSY7Qe Ampg== X-Gm-Message-State: AC+VfDxNLudPLuB3mJxDfHcuFyvfMbedH5QVGZHAd3Xt2pgRMbqMLUKe hmLeVRrmkK+lqB0k+KfG9/lWTnpBhyeqkhxxnFiyR+M= X-Google-Smtp-Source: ACHHUZ662rPnvB210sD/vkW4Q8uwhzvzY+/VsApsObHcvNanDCA5AE7QCLLhk+eE7Ao8zguK08w9Ww== X-Received: by 2002:a05:651c:118b:b0:2af:1c95:c4a8 with SMTP id w11-20020a05651c118b00b002af1c95c4a8mr4005943ljo.5.1685719311838; Fri, 02 Jun 2023 08:21:51 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:51 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Thiago Bauermann Subject: [PATCH 09/12] [contrib] validate_failures.py: Improve error output Date: Fri, 2 Jun 2023 15:20:49 +0000 Message-Id: <20230602152052.1874860-10-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" From: Thiago Bauermann - Print message in case of broken sum file error. - Print error messages to stderr. The script's stdout is, usually, redirected to a file, and error messages shouldn't go there. --- contrib/testsuite-management/validate_failures.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 6dcdcf5c69b..1919935cf53 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -136,12 +136,15 @@ class TestResult(object): self.name, self.description) = _VALID_TEST_RESULTS_REX.match(summary_line).groups() except: - print('Failed to parse summary line: "%s"' % summary_line) + print('Failed to parse summary line: "%s"' % summary_line, + file=sys.stderr) raise self.ordinal = ordinal if tool == None or exp == None: # .sum file seem to be broken. There was no "tool" and/or "exp" # lines preceding this result. + print(f'.sum file seems to be broken: tool="{tool}", exp="{exp}", summary_line="{summary_line}"', + file=sys.stderr) raise self.tool = tool self.exp = exp From patchwork Fri Jun 2 15:20:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70519 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 39B4438532E8 for ; Fri, 2 Jun 2023 15:22:27 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 39B4438532E8 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719347; bh=xQFTZoYDemdqdSTQllOBrlgnGkqEbAZxOG6pYMRIkM4=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=An7xgoS7JpYK0Xt7b+9cPJILI/bBBt44GlkOY2kd1K7LqZmH1Ga4SPBFXFaz5IYIQ XK5zAt8Ui4G2C6t3ECPFAr05yzO9y7W+D07Ku1O7Jq4Hs5TLJtY+wPWEPSGZciAiTG 6j0Q4/HytIm3zQJkDF1XNdpAGHeumU+Cv70tK4ns= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) by sourceware.org (Postfix) with ESMTPS id D9D9A385C6C8 for ; Fri, 2 Jun 2023 15:21:55 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D9D9A385C6C8 Received: by mail-lf1-x130.google.com with SMTP id 2adb3069b0e04-4f3af42459bso226852e87.0 for ; Fri, 02 Jun 2023 08:21:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719314; x=1688311314; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xQFTZoYDemdqdSTQllOBrlgnGkqEbAZxOG6pYMRIkM4=; b=MmcVLskkVdpoJKFbyZHRtPMlSpIW/BxJX4jYJzmb5Tos25rY4vc7mMJ1OqB0PC/ySt nGjNbMnwzQZs+f011lIC3iCZjyxHoYC27WXRW52HMplFFi/sb97y0kaKHZemwyogGTjH r5OsZUvhzz2Jb4CJIjqyJ2InwiN0TbxcjXoItjcYxZZvx/jSkVreEY9ge9vBEgU1OvCv L3BUrhkTS35uCdfaT1+Yflei6nGZZcVqQ7yz4fIqPShBcgg/+O32iqccTHPfLCnI/5JW 0f3woP2fb8REzsKtYUrJDYH89ty88UYMHcu/5jGz6kk//rRfyGEww3nCLG04cwO1OAvU 1qGA== X-Gm-Message-State: AC+VfDwrcH+YSG5wC8MRjKfEl6wr/Mwuw6xjmy337JXFPYeQAuXHcCYq NcG+u4pz+B5My/R625dO7SRt528wGK0kQmdGkrrqoPY= X-Google-Smtp-Source: ACHHUZ6eNINpANSOqyWKeev1JCZoSJQyMhdzbrYfVdMOs/RP66cPwHwHJdI+vz86gLkQ1Hgvtqu8qA== X-Received: by 2002:a2e:b55a:0:b0:2b1:5a32:7bef with SMTP id a26-20020a2eb55a000000b002b15a327befmr2723472ljn.5.1685719313852; Fri, 02 Jun 2023 08:21:53 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:52 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 10/12] [contrib] validate_failures.py: Add new option --invert_match Date: Fri, 2 Jun 2023 15:20:50 +0000 Message-Id: <20230602152052.1874860-11-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" This option is used to detect flaky tests that FAILed in the clean build (or manifest), but PASSed in the current build (or manifest). The option inverts output logic similar to what "-v/--invert-match" does for grep. --- .../testsuite-management/validate_failures.py | 34 +++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 1919935cf53..6eb1acd473f 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -217,11 +217,17 @@ class ResultSet(set): Attributes: current_tool: Name of the current top-level DejaGnu testsuite. current_exp: Name of the current .exp testsuite file. + testsuites: A set of (tool, exp) tuples representing encountered testsuites. """ def __init__(self): super().__init__() self.ResetToolExp() + self.testsuites=set() + + def update(self, other): + super().update(other) + self.testsuites.update(other.testsuites) def ResetToolExp(self): self.current_tool = None @@ -246,6 +252,10 @@ class ResultSet(set): outfile.write(_SUMMARY_LINE_FORMAT % 'Results') + # Check if testsuite of expected_result is present in current results. + # This is used to compare partial test results against a full manifest. + def HasTestsuite(self, expected_result): + return (expected_result.tool, expected_result.exp) in self.testsuites def GetMakefileValue(makefile_name, value_name): if os.path.exists(makefile_name): @@ -391,6 +401,8 @@ def ParseSummary(sum_fname): result_set.add(result) elif IsExpLine(line): result_set.current_exp = _EXP_LINE_REX.match(line).groups()[0] + result_set.testsuites.add((result_set.current_tool, + result_set.current_exp)) elif IsToolLine(line): result_set.current_tool = _TOOL_LINE_REX.match(line).groups()[0] result_set.current_exp = None @@ -433,7 +445,7 @@ def GetResults(sum_files, build_results = None): for sum_fname in sum_files: if _OPTIONS.verbosity >= 3: print('\t%s' % sum_fname) - build_results |= ParseSummary(sum_fname) + build_results.update(ParseSummary(sum_fname)) return build_results @@ -458,7 +470,11 @@ def CompareResults(manifest, actual): # Ignore tests marked flaky. if 'flaky' in expected_result.attrs: continue - if expected_result not in actual: + # We try to support comparing partial results vs full manifest + # (e.g., manifest has failures for gcc, g++, gfortran, but we ran only + # g++ testsuite). To achieve this we record encountered testsuites in + # actual.testsuites set, and then we check it here using HasTestsuite(). + if expected_result not in actual and actual.HasTestsuite(expected_result): manifest_vs_actual.add(expected_result) return actual_vs_manifest, manifest_vs_actual @@ -520,6 +536,13 @@ def GetSumFiles(results, build_dir): def PerformComparison(expected, actual): actual_vs_expected, expected_vs_actual = CompareResults(expected, actual) + if _OPTIONS.inverse_match: + # Switch results if inverse comparison is requested. + # This is useful in detecting flaky tests that FAILed in expected set, + # but PASSed in actual set. + actual_vs_expected, expected_vs_actual \ + = expected_vs_actual, actual_vs_expected + tests_ok = True if len(actual_vs_expected) > 0: if _OPTIONS.verbosity >= 3: @@ -613,6 +636,13 @@ def Main(argv): default=False, help='When used with --produce_manifest, ' 'it will overwrite an existing manifest file ' '(default = False)') + parser.add_option('--inverse_match', action='store_true', + dest='inverse_match', default=False, + help='Inverse result sets in comparison. ' + 'Output unexpected passes as unexpected failures and ' + 'unexpected failures as unexpected passes. ' + 'This is used to catch FAIL->PASS flaky tests. ' + '(default = False)') parser.add_option('--manifest', action='store', type='string', dest='manifest', default=None, help='Name of the manifest file to use (default = ' From patchwork Fri Jun 2 15:20:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70526 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 1B9F5387484B for ; Fri, 2 Jun 2023 15:24:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 1B9F5387484B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719470; bh=o76IB/OTveZVnoVGKVgDE2mlJy03CERz8XhJ4cyUhd4=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=hN8pf+DRWhfUjOJmiATEwmqi5V8JV1LDyfYBOG3zHhZnUE+mQTv/vZoKg6gdgSiTn z46m7vDnaGafdVZgNGrQsptTulDISrRX7aD2p+HeZDcJx9ZF8n5ujFZL9rXRVuJgdf 3JPlRDnKLAl/lFv01khtqXDG1MRW3PNQlljPSJy8= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x235.google.com (mail-lj1-x235.google.com [IPv6:2a00:1450:4864:20::235]) by sourceware.org (Postfix) with ESMTPS id F2E36385C6CE for ; Fri, 2 Jun 2023 15:21:57 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org F2E36385C6CE Received: by mail-lj1-x235.google.com with SMTP id 38308e7fff4ca-2b1a4be0ba1so2786601fa.0 for ; Fri, 02 Jun 2023 08:21:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719315; x=1688311315; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o76IB/OTveZVnoVGKVgDE2mlJy03CERz8XhJ4cyUhd4=; b=IoX5HW7apBUxmitOB+wB2hYmO3zIGcfbZCevXXEyRmi5tKrHKuo3p2lWXttDC0kLdw 3zDoHTv2sO3vl8sAyrxkXq4cLKi2nFk00yu62BpLjwJhthhAL9REQ1ZZLXawYi1L/ilO 5IZfdbL+Aj/j/qTZpco8cxKNJMZdR3LLlgg0J0331L/YxVggYsPTk6eoA3epDNGXFBZv Nbeltpvt7zSbBFvKkh41JGdAlrTN8VEdUZBotWik6BlYRl+qfxpVRp7m1oYHXG9WFaDt 3pN/jPE7yhcwh8EtPPGT6xzXt3FeuWpBpDyVz1Xjh3HtM254ldraOXg5bgbgGteZgEOG 6eYA== X-Gm-Message-State: AC+VfDyZqx0d+2mKe6l+pG9AkqfX1ebCTvbj7WJfjE3TqpJOedncMc1K KBEgmpS2fwQTT7L9Y+/7uIWkz8CofApx5P7+4vK7fAU= X-Google-Smtp-Source: ACHHUZ5yA+KwGa+6G8r8mTmUZi6TG1Ykl9MrUHhK77vzsx5PsyBvNv33ujNkHm4E1E6wNFbqV4Xsqw== X-Received: by 2002:a05:651c:1251:b0:2ac:767c:fba0 with SMTP id h17-20020a05651c125100b002ac767cfba0mr4407992ljh.2.1685719315752; Fri, 02 Jun 2023 08:21:55 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:54 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 11/12] [contrib] validate_failures.py: Add "--expiry_date YYYYMMDD" option Date: Fri, 2 Jun 2023 15:20:51 +0000 Message-Id: <20230602152052.1874860-12-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" This option sets "today" date to compare expiration entries against. Setting expiration date into the future allows re-detection of flaky tests and creating fresh entries for them before the current flaky entries expire. --- .../testsuite-management/validate_failures.py | 24 +++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 6eb1acd473f..a77aabe0bdd 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -206,8 +206,7 @@ class TestResult(object): # Return True if the expiration date of this result has passed. expiration_date = self.ExpirationDate() if expiration_date: - now = datetime.date.today() - return now > expiration_date + return _OPTIONS.expiry_today_date > expiration_date class ResultSet(set): @@ -636,6 +635,11 @@ def Main(argv): default=False, help='When used with --produce_manifest, ' 'it will overwrite an existing manifest file ' '(default = False)') + parser.add_option('--expiry_date', action='store', + dest='expiry_today_date', default=None, + help='Use provided date YYYYMMDD to decide whether ' + 'manifest entries with expiry settings have expired ' + 'or not. (default = Use today date)') parser.add_option('--inverse_match', action='store_true', dest='inverse_match', default=False, help='Inverse result sets in comparison. ' @@ -670,6 +674,22 @@ def Main(argv): global _OPTIONS (_OPTIONS, _) = parser.parse_args(argv[1:]) + # Set "today" date to compare expiration entries against. + # Setting expiration date into the future allows re-detection of flaky + # tests and creating fresh entries for them before the current flaky entries + # expire. + if _OPTIONS.expiry_today_date: + today_date = re.search(r'(\d\d\d\d)(\d\d)(\d\d)', + _OPTIONS.expiry_today_date) + if not today_date: + Error('Invalid --expiry_today_date format "%s". Must be of the form ' + '"expire=YYYYMMDD"' % _OPTIONS.expiry_today_date) + _OPTIONS.expiry_today_date=datetime.date(int(today_date.group(1)), + int(today_date.group(2)), + int(today_date.group(3))) + else: + _OPTIONS.expiry_today_date = datetime.date.today() + if _OPTIONS.produce_manifest: retval = ProduceManifest() elif _OPTIONS.clean_build: From patchwork Fri Jun 2 15:20:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70528 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id BD3FD3839BEA for ; Fri, 2 Jun 2023 15:25:41 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org BD3FD3839BEA DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719541; bh=HMtHJ9HJBcLm6iuRM6ms9W94+BEXFAZM7hdAwPKceHs=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=qi0mRc4ewqqkS3WC6rwvhHscubUVLTBvtNNkMXMQnc9M5GbcK4p+h8RZBd7MhImtJ TmR6VUYP7uLqtDc5E3PJIts7OqAqw/UgKEImOgJ0rE1wwaUf9BfR/K5CK8SyGlSqL2 5EH97Udmqi1+iGGVuCUzJneobGOPPKFMu73lcvlo= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129]) by sourceware.org (Postfix) with ESMTPS id 68BFE385C6E4 for ; Fri, 2 Jun 2023 15:22:00 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 68BFE385C6E4 Received: by mail-lf1-x129.google.com with SMTP id 2adb3069b0e04-4f4453b607eso369031e87.1 for ; Fri, 02 Jun 2023 08:22:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719318; x=1688311318; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HMtHJ9HJBcLm6iuRM6ms9W94+BEXFAZM7hdAwPKceHs=; b=bDAjt9RPuPNY6kPR5eg8ZVKyF0IAqkaC4zCU+6YZbhp1nq88zC65/bIJ3nQt5t+zg2 Bynlput3u5tttwgboRFSA3bn75sVDfzWdW/r9rE4Jk6IjyQgirli4nCbCAx2ztt+dWsN xpdfnEVP/KKOVPnAa7XwQast7Jendg5Cj1+uhZdAI/0ToAIs+y29q2nRv6emIzD1aJ6m vVbcrrtZTECg+dPcF5iG9+PHX+1Sue8z6YIer6C7tGqiPgIIwIVeOBV9Uw1EXYsnRgmk Utgvy7IgXDgLqxXJaqMxVz3N/4AycUN1WXFk7j/U705x2oB/iUT03PS6pHrAfdlcpxU2 Bjcg== X-Gm-Message-State: AC+VfDwPtG4flT/6uhpmfKaBtFXzifA167donf5zdXLBLRtNvW6HaFiB JoamDdEWKJ4ueQuhn5wJooTL44VfkmrwdfURc9cjTHY= X-Google-Smtp-Source: ACHHUZ4yWKqYE2XgXfY0tzJCtTdV1aoTQw9tXqqYt9y252kpKWwiVqOL4J1qxyZXVgJgky7XDG4ECA== X-Received: by 2002:a05:651c:118b:b0:2af:1c95:c4a8 with SMTP id w11-20020a05651c118b00b002af1c95c4a8mr4006123ljo.5.1685719318193; Fri, 02 Jun 2023 08:21:58 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:56 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 12/12] [contrib] validate_failures.py: Ignore stray filesystem paths in results Date: Fri, 2 Jun 2023 15:20:52 +0000 Message-Id: <20230602152052.1874860-13-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" This patch simplifies comparison of results that have filesystem paths. E.g., (assuming different values of ): Running /home/user/gcc-N/gcc/testsuite/gcc.target/aarch64/sve/acle/aarch64-sve-acle-asm.exp ... ERROR: tcl error sourcing /home/user/gcc-N/gcc/testsuite/gcc.target/aarch64/sve/acle/aarch64-sve-acle-asm.exp. We add "--srcpath ", option, and set it by default to "[^ ]+/testsuite/", which works well for all components of the GNU Toolchain. We then remove substrings matching from paths of .exp files and from occasional "ERROR:" results. --- contrib/testsuite-management/validate_failures.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index a77aabe0bdd..4dfd9cda4e2 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -135,6 +135,9 @@ class TestResult(object): (self.state, self.name, self.description) = _VALID_TEST_RESULTS_REX.match(summary_line).groups() + if _OPTIONS.srcpath_regex and _OPTIONS.srcpath_regex != '': + self.description = re.sub(_OPTIONS.srcpath_regex, '', + self.description) except: print('Failed to parse summary line: "%s"' % summary_line, file=sys.stderr) @@ -361,6 +364,9 @@ def ParseManifestWorker(result_set, manifest_path): result_set.add(result) elif IsExpLine(orig_line): result_set.current_exp = _EXP_LINE_REX.match(orig_line).groups()[0] + if _OPTIONS.srcpath_regex and _OPTIONS.srcpath_regex != '': + result_set.current_exp = re.sub(_OPTIONS.srcpath_regex, '', + result_set.current_exp) elif IsToolLine(orig_line): result_set.current_tool = _TOOL_LINE_REX.match(orig_line).groups()[0] elif IsSummaryLine(orig_line): @@ -400,6 +406,9 @@ def ParseSummary(sum_fname): result_set.add(result) elif IsExpLine(line): result_set.current_exp = _EXP_LINE_REX.match(line).groups()[0] + if _OPTIONS.srcpath_regex and _OPTIONS.srcpath_regex != '': + result_set.current_exp = re.sub(_OPTIONS.srcpath_regex, '', + result_set.current_exp) result_set.testsuites.add((result_set.current_tool, result_set.current_exp)) elif IsToolLine(line): @@ -640,6 +649,12 @@ def Main(argv): help='Use provided date YYYYMMDD to decide whether ' 'manifest entries with expiry settings have expired ' 'or not. (default = Use today date)') + parser.add_option('--srcpath', action='store', type='string', + dest='srcpath_regex', default='[^ ]+/testsuite/', + help='Remove provided path (can be a regex) from ' + 'the result entries. This is useful to remove ' + 'occasional filesystem path from the results. ' + '(default = "[^ ]+/testsuite/")') parser.add_option('--inverse_match', action='store_true', dest='inverse_match', default=False, help='Inverse result sets in comparison. '