From patchwork Thu Mar 6 23:23:49 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joseph Myers X-Patchwork-Id: 7 Return-Path: X-Original-To: siddhesh@wilcox.dreamhost.com Delivered-To: siddhesh@wilcox.dreamhost.com Received: from homiemail-mx22.g.dreamhost.com (caibbdcaabij.dreamhost.com [208.113.200.189]) by wilcox.dreamhost.com (Postfix) with ESMTP id 99A42360123 for ; Thu, 6 Mar 2014 15:24:02 -0800 (PST) Received: by homiemail-mx22.g.dreamhost.com (Postfix, from userid 14307373) id 48CDF4C7A166; Thu, 6 Mar 2014 15:24:02 -0800 (PST) X-Original-To: glibc@patchwork.siddhesh.in Delivered-To: x14307373@homiemail-mx22.g.dreamhost.com Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by homiemail-mx22.g.dreamhost.com (Postfix) with ESMTPS id 20BEA4C7A167 for ; Thu, 6 Mar 2014 15:24:02 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:date:from:to:subject:message-id:mime-version :content-type; q=dns; s=default; b=UiUmsVBPl/AxD/M9Z6SsYrhTSJDSI URAmfpBe5kGKuvS6AYF1mxrA3N19ZF+KBEtLxFF8VNQH3SfayIe5Xqxl17oWw3qj rXpv6VC4oQeRV417avn+wT3op85RqrJ56GIToWNZF0zrZqu3ToOsYabqJPUKflEh RRUvepDUl013xg= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:date:from:to:subject:message-id:mime-version :content-type; s=default; bh=yjHsbHQiR6ZPFXZlxxcgnVhj8/Q=; b=pTw lMhmpmsT+ATEdKnPSkKgw93fttgg4t+9Q6UdYaUV738VvpLmA6Mfruj4vzhjq4sN uSs/ZMs+KOj5reRm+kyVldJp2aWVMP0+ALMdmkgmSL/GUIEVKL3Qp9H4Tgc+gW1m N51sZT1AVCErh57wrNLfbOd2V3eWP1CnjgXdg7gM= Received: (qmail 11194 invoked by alias); 6 Mar 2014 23:24:00 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 11185 invoked by uid 89); 6 Mar 2014 23:23:59 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.0 required=5.0 tests=AWL, BAYES_00 autolearn=ham version=3.3.2 X-HELO: relay1.mentorg.com Date: Thu, 6 Mar 2014 23:23:49 +0000 From: "Joseph S. Myers" To: , Carlos O'Donell Subject: Generate overall summary of test results Message-ID: MIME-Version: 1.0 X-DH-Original-To: glibc@patchwork.siddhesh.in This patch, an updated version of now proposed for inclusion in glibc, makes testsuite runs generate an overall summary of test results. A new script merge-test-results.sh deals both with collecting results within a directory to a file with all the results from that directory, and collecting the results from subdirectories into a single overall file (there's not much in common between the two modes of operation of the script, but it seemed silly to have two separate scripts for this). Within a directory, missing results produce UNRESOLVED lines; at top level, missing results for a whole directory produce an ERROR line (since toplevel can't identify what the specific missing tests are in this case). Note that this does not change the rules for when "make" considers there has been an error, or terminates, so unexpected failures will still cause make to terminate, or, with -k, mean the commands for "tests" don't get run because of failure of a dependency. Tested x86_64, including that the summary does in fact reflect all the tests with .test-result files. (Changes from previous version: an error message is adjusted as suggested by Brooks; the merge script is updated to reflect that .test-result files now contain a second line with the original exit status, which shouldn't end up in the .sum file, rather than just the main test result; the makefile computations of the list of tests whose results need merging are adjusted to reflect that tests producing results in subdirectories now get the .test-result file and test name properly reflecting those subdirectories.) 2014-03-06 Joseph Myers * scripts/merge-test-results.sh: New file. * Makefile (tests-special-notdir): New variable. (tests): Run merge-test-results.sh. (xtests): Likewise. * Rules (tests-special-notdir): New variable. (xtests-special-notdir): Likewise. (tests): Run merge-test-results.sh (xtests): Likewise. diff --git a/Makefile b/Makefile index aee88e1..8214dda 100644 --- a/Makefile +++ b/Makefile @@ -316,7 +316,18 @@ $(objpfx)begin-end-check.out: scripts/begin-end-check.pl $(evaluate-test) endif +tests-special-notdir = $(patsubst $(objpfx)%, %, $(tests-special)) tests: $(tests-special) + $(..)scripts/merge-test-results.sh -s $(objpfx) "" \ + $(sort $(tests-special-notdir:.out=)) \ + > $(objpfx)subdir-tests.sum + $(..)scripts/merge-test-results.sh -t $(objpfx) subdir-tests.sum \ + $(sort $(subdirs) .) \ + > $(objpfx)tests.sum +xtests: + $(..)scripts/merge-test-results.sh -t $(objpfx) subdir-xtests.sum \ + $(sort $(subdirs)) \ + > $(objpfx)xtests.sum # The realclean target is just like distclean for the parent, but we want # the subdirs to know the difference in case they care. diff --git a/Rules b/Rules index d37ae52..feb304d 100644 --- a/Rules +++ b/Rules @@ -102,6 +102,17 @@ tests: $(tests:%=$(objpfx)%.out) $(tests-special) xtests: tests $(xtests:%=$(objpfx)%.out) $(xtests-special) endif +tests-special-notdir = $(patsubst $(objpfx)%, %, $(tests-special)) +xtests-special-notdir = $(patsubst $(objpfx)%, %, $(xtests-special)) +tests: + $(..)scripts/merge-test-results.sh -s $(objpfx) $(subdir) \ + $(sort $(tests) $(tests-special-notdir:.out=)) \ + > $(objpfx)subdir-tests.sum +xtests: + $(..)scripts/merge-test-results.sh -s $(objpfx) $(subdir) \ + $(sort $(xtests) $(xtests-special-notdir:.out=)) \ + > $(objpfx)subdir-xtests.sum + ifeq ($(build-programs),yes) binaries-all-notests = $(others) $(sysdep-others) binaries-all-tests = $(tests) $(xtests) $(test-srcs) diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh new file mode 100755 index 0000000..6255e24 --- /dev/null +++ b/scripts/merge-test-results.sh @@ -0,0 +1,61 @@ +#! /bin/sh +# Merge test results of individual tests or subdirectories. +# Copyright (C) 2014 Free Software Foundation, Inc. +# This file is part of the GNU C Library. + +# The GNU C Library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. + +# The GNU C Library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. + +# You should have received a copy of the GNU Lesser General Public +# License along with the GNU C Library; if not, see +# . + +# usage: merge-test-results.sh -s objpfx subdir test-name... +# (subdirectory tests; empty subdir at top level), or +# merge-test-results.sh -t objpfx subdir-file-name subdir... +# (top-level merge) + +set -e + +type=$1 +objpfx=$2 +shift 2 + +case $type in + -s) + subdir=$1 + shift + subdir=${subdir:+$subdir/} + for t in "$@"; do + if [ -s "$objpfx$t.test-result" ]; then + head -n1 "$objpfx$t.test-result" + else + echo "UNRESOLVED: $subdir$t" + fi + done + ;; + + -t) + subdir_file_name=$1 + shift + for d in "$@"; do + if [ -f "$objpfx$d/$subdir_file_name" ]; then + cat "$objpfx$d/$subdir_file_name" + else + echo "ERROR: test results for $d directory missing" + fi + done + ;; + + *) + echo "unknown type $type" >&2 + exit 1 + ;; +esac