From patchwork Fri Jun 2 15:20:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxim Kuvyrkov X-Patchwork-Id: 70520 Return-Path: X-Original-To: patchwork@sourceware.org Delivered-To: patchwork@sourceware.org Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 7268E38515C9 for ; Fri, 2 Jun 2023 15:22:33 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 7268E38515C9 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1685719353; bh=ixW3JZDLCjI2AmfDvTRWK2EjdWamQ9DMDYZmrg/mE/Q=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=iYS4LZ8LbNnKOP59P4XX/GEVDQ2trWIgv6uikFxmfMIlSUuScEPV/xpX0dys231Vl bshu+NHAtoS/+NMsqophtxyFMvQzIfqFvQdxTIffJvrpJWppEKwVbSn50bmHiWHFwt 2qrhS91de4HcXVL1+7kD8Fb/FdcfQ26O2JDvQDU4= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) by sourceware.org (Postfix) with ESMTPS id 65AAA3857719 for ; Fri, 2 Jun 2023 15:21:22 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 65AAA3857719 Received: by mail-lj1-x229.google.com with SMTP id 38308e7fff4ca-2b12834a65bso1785341fa.1 for ; Fri, 02 Jun 2023 08:21:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685719280; x=1688311280; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ixW3JZDLCjI2AmfDvTRWK2EjdWamQ9DMDYZmrg/mE/Q=; b=Ngk+WFbuNF1knNPrzeI9E+qRrgt2Vk9vv0n/neVhHVcQqpGJPmgS00Qn3YBu0Tx7jM jIjhDSxHNXHoQYf1BWwCN4Gc8Ow0/23IXqPJnD/jWE0/UV84ehUAoFmJ5vLAWU2e65e1 PMb877qBvYKC2f7QHv5fXZ6V7/oKRWiYzhQCdbJc7Nsg+zSbOZSes/CuhPgW/VR1fejF 7PynXBoLKmiL2k5L8vS98O2NGCQnwBZtoIJsFeb85YdsGR8dqpgIsJKx+OeE8M2b4KeB xD7Czw6Ju5siG799MZ2A5SaD2/korvygBqO3g38JiT4oPHiutn9m5uhm/jYdlJNksQFJ WqDA== X-Gm-Message-State: AC+VfDwp79k5/8IXlRpBtNP0uVhh/PYfg7FN0b6Mv4FXmLQxYfj+Mt5c pMbR7KnPGhLDvS0Szo6v/um26/cE2FPtM3xS5mGkYCA= X-Google-Smtp-Source: ACHHUZ4lqbyGQtFKrU5vLoxbW6eiMEs9jgLXCtwiYULbv3MbzpCbPyCEP4SyTSJ0l9yJVco9uHUdZA== X-Received: by 2002:a05:651c:2ce:b0:2b0:361f:80e1 with SMTP id f14-20020a05651c02ce00b002b0361f80e1mr5570940ljo.4.1685719280219; Fri, 02 Jun 2023 08:21:20 -0700 (PDT) Received: from localhost.localdomain (static.225.72.216.95.clients.your-server.de. [95.216.72.225]) by smtp.gmail.com with ESMTPSA id a18-20020a2eb172000000b002a8ac166e55sm255122ljm.44.2023.06.02.08.21.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Jun 2023 08:21:19 -0700 (PDT) To: gcc-patches@gcc.gnu.org Cc: Diego Novillo , Doug Evans , Maxim Kuvyrkov Subject: [PATCH 03/12] [contrib] validate_failures.py: Read in manifest when comparing build dirs Date: Fri, 2 Jun 2023 15:20:43 +0000 Message-Id: <20230602152052.1874860-4-maxim.kuvyrkov@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> References: <20230602152052.1874860-1-maxim.kuvyrkov@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Maxim Kuvyrkov via Gcc-patches From: Maxim Kuvyrkov Reply-To: Maxim Kuvyrkov Errors-To: gcc-patches-bounces+patchwork=sourceware.org@gcc.gnu.org Sender: "Gcc-patches" This allows comparison of two build directories with a manifest listing known flaky tests on the side. --- contrib/testsuite-management/validate_failures.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index 7351ba120b7..4733dd89dcb 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -420,9 +420,10 @@ def CollectSumFiles(builddir): return sum_files -def GetResults(sum_files): +def GetResults(sum_files, build_results = None): """Collect all the test results from the given .sum files.""" - build_results = ResultSet() + if build_results == None: + build_results = ResultSet() for sum_fname in sum_files: print('\t%s' % sum_fname) build_results |= ParseSummary(sum_fname) @@ -567,8 +568,15 @@ def CompareBuilds(): sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.build_dir) actual = GetResults(sum_files) + clean = ResultSet() + + if _OPTIONS.manifest: + manifest_path = GetManifestPath(srcdir, target, True) + print('Manifest: %s' % manifest_path) + clean = GetManifest(manifest_path) + clean_sum_files = GetSumFiles(_OPTIONS.results, _OPTIONS.clean_build) - clean = GetResults(clean_sum_files) + clean = GetResults(clean_sum_files, clean) return PerformComparison(clean, actual, _OPTIONS.ignore_missing_failures)