From patchwork Fri Jun 2 15:20:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70523 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D18A7385624C for ; Fri, 2 Jun 2023 15:23:31 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D18A7385624C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719411; bh=PgHWyQGQe8J/X0vEjTsRjanH8gSTVOI0LdUsVAzAAc8=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=tcwsADDygEY5uiubK7pRR8UqmRcwzaSzi7u+31FZdLsT8RIG/bitbneH0WycBDD7g wWiFCvvr8k9XscINT2bdXBi7apIQYg798+MtoukVvgNQc6tcjICZ+StKPsP0IkJkZK YHu6rF5vvctyOq6myKd8VgzhkhJzyFU9gfEuV24Q= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x234.google.com (mail-lj1-x234.google.com [IPv6:2a00:1450:4864:20::234]) by sourceware.org (Postfix) with ESMTPS id D87D7385589C for ; Fri, 2 Jun 2023 15:21:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D87D7385589C Received: by mail-lj1-x234.google.com with SMTP id 38308e7fff4ca-2b072753301so2345021fa.1 for ; Fri, 02 Jun 2023 08:21:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719304; x=1688311304; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PgHWyQGQe8J/X0vEjTsRjanH8gSTVOI0LdUsVAzAAc8=; b=TbXu18LWqgEwAK1vQP40LvpHlIyJj7weXVCRPZYbz3zD8IW8/YHJjpjYz/FnVQsF0A AiMqnrjKGuPeZJWwLIMjwZEGTSnO/SRpDvdkYIXAr5Z+iM4nPQ7yn4uvrYtaYAy3rMEg Ly1FVHS7OWI2BoLxyvzjN6IlrFnwsddc42zt0gWVZD0OSTq90KR8KHTJZVOchQtZloSz B+s1HjgJmKGfAtEPnOWXEbTVMl+NfLkNfTMQTYOKF55igVUMbHYZ7Kh+e/C8qiMRUVEq vwNOiXQyzo5ouc6D7BefmK2xSCCV4bPB6LhEvCUxzIKpq42QueqsbCkvdDLfeXfze5XS /bBQ== X-Gm-Message-State: AC+VfDwIFHl0wMHm36FwBdY4XRJTMvoWfyocOQG2M3jwlUZWFjl9DXTv 84bKHvt7v99BYlseepgiv5IxAtWrW+gNTZSvS6ysfV4= X-Google-Smtp-Source: ACHHUZ49A7PmEz2h2bE3Yt0m8tJZkrasM+Tkjo6AqboWj/MN3OX5FpSoj1YBlSczy9uX/e4vEq71EQ== X-Received: by 2002:a2e:a4ac:0:b0:2b1:a69e:6a8e with SMTP id g12-20020a2ea4ac000000b002b1a69e6a8emr2216523ljm.3.1685719303894; Fri, 02 Jun 2023 08:21:43 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:22 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 05/12] [contrib] validate_failures.py: Add more verbosity levels Date: Fri, 2 Jun 2023 15:20:45 +0000 Message-Id: <20230602152052.1874860-6-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" ... to control validate_failures.py output --- .../testsuite-management/validate_failures.py | 82 +++++++++++-------- 1 file changed, 46 insertions(+), 36 deletions(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 1bd09e0c20c..26ea1d6f53b 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -324,7 +324,7 @@ def GetNegativeResult(line): def ParseManifestWorker(result_set, manifest_path): """Read manifest_path, adding the contents to result_set.""" - if _OPTIONS.verbosity >= 1: + if _OPTIONS.verbosity >= 5: print('Parsing manifest file %s.' % manifest_path) manifest_file = open(manifest_path, encoding='latin-1', mode='r') for orig_line in manifest_file: @@ -380,7 +380,8 @@ def ParseSummary(sum_fname): # Tests that have expired are not added to the set of expected # results. If they are still present in the set of actual results, # they will cause an error to be reported. - print('WARNING: Expected failure "%s" has expired.' % line.strip()) + if _OPTIONS.verbosity >= 4: + print('WARNING: Expected failure "%s" has expired.' % line.strip()) continue result_set.add(result) elif IsExpLine(line): @@ -425,7 +426,8 @@ def GetResults(sum_files, build_results = None): if build_results == None: build_results = ResultSet() for sum_fname in sum_files: - print('\t%s' % sum_fname) + if _OPTIONS.verbosity >= 3: + print('\t%s' % sum_fname) build_results |= ParseSummary(sum_fname) return build_results @@ -489,42 +491,46 @@ def GetBuildData(): return None, None srcdir = GetMakefileValue('%s/Makefile' % _OPTIONS.build_dir, 'srcdir =') target = GetMakefileValue('%s/Makefile' % _OPTIONS.build_dir, 'target_alias=') - print('Source directory: %s' % srcdir) - print('Build target: %s' % target) + if _OPTIONS.verbosity >= 3: + print('Source directory: %s' % srcdir) + print('Build target: %s' % target) return srcdir, target -def PrintSummary(msg, summary): - print('\n\n%s' % msg) +def PrintSummary(summary): summary.Print() def GetSumFiles(results, build_dir): if not results: - print('Getting actual results from build directory %s' % build_dir) + if _OPTIONS.verbosity >= 3: + print('Getting actual results from build directory %s' % build_dir) sum_files = CollectSumFiles(build_dir) else: - print('Getting actual results from user-provided results') + if _OPTIONS.verbosity >= 3: + print('Getting actual results from user-provided results') sum_files = results.split() return sum_files -def PerformComparison(expected, actual, ignore_missing_failures): +def PerformComparison(expected, actual): actual_vs_expected, expected_vs_actual = CompareResults(expected, actual) tests_ok = True if len(actual_vs_expected) > 0: - PrintSummary('Unexpected results in this build (new failures)', - actual_vs_expected) + if _OPTIONS.verbosity >= 3: + print('\n\nUnexpected results in this build (new failures)') + if _OPTIONS.verbosity >= 1: + PrintSummary(actual_vs_expected) tests_ok = False - if not ignore_missing_failures and len(expected_vs_actual) > 0: - PrintSummary('Expected results not present in this build (fixed tests)' - '\n\nNOTE: This is not a failure. It just means that these ' - 'tests were expected\nto fail, but either they worked in ' - 'this configuration or they were not\npresent at all.\n', - expected_vs_actual) + if _OPTIONS.verbosity >= 2 and len(expected_vs_actual) > 0: + print('\n\nExpected results not present in this build (fixed tests)' + '\n\nNOTE: This is not a failure. It just means that these ' + 'tests were expected\nto fail, but either they worked in ' + 'this configuration or they were not\npresent at all.\n') + PrintSummary(expected_vs_actual) - if tests_ok: + if tests_ok and _OPTIONS.verbosity >= 3: print('\nSUCCESS: No unexpected failures.') return tests_ok @@ -532,21 +538,25 @@ def PerformComparison(expected, actual, ignore_missing_failures): def CheckExpectedResults(): manifest_path = GetManifestPath(True) - print('Manifest: %s' % manifest_path) + if _OPTIONS.verbosity >= 3: + print('Manifest: %s' % manifest_path) manifest = GetManifest(manifest_path) sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.build_dir) actual = GetResults(sum_files) - if _OPTIONS.verbosity >= 1: - PrintSummary('Tests expected to fail', manifest) - PrintSummary('\nActual test results', actual) + if _OPTIONS.verbosity >= 5: + print('\n\nTests expected to fail') + PrintSummary(manifest) + print('\n\nActual test results') + PrintSummary(actual) - return PerformComparison(manifest, actual, _OPTIONS.ignore_missing_failures) + return PerformComparison(manifest, actual) def ProduceManifest(): manifest_path = GetManifestPath(False) - print('Manifest: %s' % manifest_path) + if _OPTIONS.verbosity >= 3: + print('Manifest: %s' % manifest_path) if os.path.exists(manifest_path) and not _OPTIONS.force: Error('Manifest file %s already exists.\nUse --force to overwrite.' % manifest_path) @@ -569,13 +579,14 @@ def CompareBuilds(): if _OPTIONS.manifest: manifest_path = GetManifestPath(True) - print('Manifest: %s' % manifest_path) + if _OPTIONS.verbosity >= 3: + print('Manifest: %s' % manifest_path) clean = GetManifest(manifest_path) clean_sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.clean_build) clean = GetResults(clean_sum_files, clean) - return PerformComparison(clean, actual, _OPTIONS.ignore_missing_failures) + return PerformComparison(clean, actual) def Main(argv): @@ -597,14 +608,6 @@ def Main(argv): default=False, help='When used with --produce_manifest, ' 'it will overwrite an existing manifest file ' '(default = False)') - parser.add_option('--ignore_missing_failures', action='store_true', - dest='ignore_missing_failures', default=False, - help='When a failure is expected in the manifest but ' - 'it is not found in the actual results, the script ' - 'produces a note alerting to this fact. This means ' - 'that the expected failure has been fixed, or ' - 'it did not run, or it may simply be flaky ' - '(default = False)') parser.add_option('--manifest', action='store', type='string', dest='manifest', default=None, help='Name of the manifest file to use (default = ' @@ -621,7 +624,14 @@ def Main(argv): 'starting with FAIL, XPASS or UNRESOLVED (default = ' '.sum files collected from the build directory).') parser.add_option('--verbosity', action='store', dest='verbosity', - type='int', default=0, help='Verbosity level (default = 0)') + type='int', default=3, help='Verbosity level ' + '(default = 3). Level 0: only error output, this is ' + 'useful in scripting when only the exit code is used. ' + 'Level 1: output unexpected failures. ' + 'Level 2: output unexpected passes. ' + 'Level 3: output helpful information. ' + 'Level 4: output notification on expired entries. ' + 'Level 5: output debug information.') global _OPTIONS (_OPTIONS, _) = parser.parse_args(argv[1:])