Update dg-extract-results.* from gcc
Commit Message
Hi Tom,
getting back to making gcc's mail-report.log target work...
>> Rainer> * One could also copy over gcc's contrib/test_summary, used by the
>> Rainer> toplevel make mail-report.log to provide a nice summary of test
>> Rainer> results. However, this is currently hampered by the fact that for
>> Rainer> parallel make check the gdb.sum and gdb.log files are left in
>> Rainer> outputs/*/*/gdb.{sum,log} after dg-extract-results.sh has run
>> Rainer> instead
>> Rainer> of moving them to *.sep like gcc's gcc/Makefile.in does, so
>> Rainer> mail-report.log lists every failure twice.
>>
>> I don't understand the "*.sep" comment - would you mind spelling it out?
>
> The test_summary scripts works by searching for *.sum and *.log files in
> the whole tree (given that those live at different levels in the build
> tree and cannot easily be found with a glob pattern).
>
> Currently, once dg-extract-results.sh has summarized the individual
> gdb.sum and gdb.log files in outputs, we have both the individual
> per-subdir files in place and the summarized one in gdb/testsuite. When
> test_summary runs, it find all of of those and lists every non-PASS
> result twice in its output, which isn't particularly helpful.
>
> To avoid this, the gcc testsuite moves the subdir .sum/.log files for
> parallelized testsuites to .sum.sep/.log.sep before passing them to
> dg-extract-results.sh. This way, we get one summary per testsuite
> (e.g. gcc or g++, or gdb in the case at hand), and test_summary won't
> pick them up twice.
>
>> Anyway, if this script is useful to you, it's fine with me if you want
>> to find a way to make it work. I think the outputs/** stuff can be
>> moved around or messed with pretty freely, though of course it is best
>> not to outright lose things.
>
> Absolutely: as I said, the individual files are just moved aside not to
> interfere with the likes of test_summary, but still left in place since
> dg-extract-results.* isn't always perfect in merging them.
>
> I'll go ahead and prepare a patch then.
Here's the mechanical part of such a patch, tested on
amd64-pc-solaris2.11 and making sure that both gdb make check and make
check-parallel-racy produce identical gdb.sum and racy.sum files (modulo
racy tests, of course).
However, looking at the resulting output, there are some issues to be
decided before the result can be fully useful. I'm attaching an example
output file for reference, from a Solaris 11/amd64 build with
--disable-binutils --disable-gas --disable-ld --disable-sim.
* Right now, the output talks of
Compiler version: 8.2.50.20180803-git ...
even when this is for gdb (or binutils) test results.
* The resulting script would mail testsuite results to
gcc-testresults@gcc.gnu.org; clearly not useful for gdb or binutils.
While we could use gdb-testers@sourceware.org for gdb (where Andreas
Krebbel already posts very similarly formatted results for
Linux/s390x) and there's even an (apparently unused)
gdb-testresults@sourceware.org list as I just happened to find at
https://sourceware.org/lists.html#gdb, there seems to be no equivalent
for binutils test results.
I think the contrib/test_summary script should provide decent defaults
depending on which tree it is run in.
* There are 3 scenarios, I believe:
gcc tree, already handled
binutils-gdb tree, binutils results
binutils-gdb tree, gdb results
I believe the result in the binutils-gdb tree should be the same as
what would happen in a binutils or gdb release tarball, i.e. the same
results summarized in each case. This could be done using the same
lists of subdirs used by src-release.sh to create the respective
tarballs.
In theory, there's still the combined tree case, although I have no
idea if it is still common these days.
My current thinking is that the toplevel mail-report.log target would
create all summaries suitable for the current tree, either by passing
corresponding -t [gcc|binutils|gdb] options to test_summary or letting
the script figure that out on its own, creating
* mail-report.log for gcc, keeping the name for backward compatibility,
* binutils-report.log and/or gdb-report.log if either or both
binutils/gas/ld and gdb/sim builds are present in the tree.
Thoughts?
Rainer
# HG changeset patch
# Parent 09a136c459e46cbf4007b68ce3ee2d2062b3448d
Support mail-report.log
new file mode 100755
@@ -0,0 +1,157 @@
+#! /bin/sh
+
+# (C) 1998, 1999, 2000, 2002, 2003, 2004, 2007, 2009, 2010
+# Free Software Foundation
+# Originally by Alexandre Oliva <oliva@dcc.unicamp.br>
+
+# This script is Free Software, and it can be copied, distributed and
+# modified as defined in the GNU General Public License. A copy of
+# its license can be downloaded from http://www.gnu.org/copyleft/gpl.html
+
+# This script processes *.{sum,log} files, producing a shell-script
+# that sends e-mail to the appropriate lists and renames files to
+# *.sent. It currently handles only gcc, but it should be quite easy
+# to modify it to handle other packages and its mailing lists.
+
+# The scripts assumes it is run in the root directory of the build
+# tree, and it will include all .sum files it finds in the mail
+# report.
+
+# configure flags are extracted from ./config.status
+
+# if the BOOT_CFLAGS environment variable is set, it will be included
+# in the mail report too.
+
+# The usage pattern of this script is as follows:
+
+# test_summary | more # so as to observe what should be done
+
+# test_summary | sh # so as to actually send e-mail and move log files
+
+# It accepts a few command line arguments. For example:
+if test x"$1" = "x-h"; then
+ cat <<_EOF
+ -o: re-reads logs that have been mailed already (.sum.sent)
+ -t: prevents logs from being renamed
+ -p: prepend specified file (or list of files: -p "a b") to the report
+ -i: append specified file (or list of files: -i "a b") to the report
+ -m: specify the e-mail address to send notes to. An appropriate default
+ should be selected from the log files.
+ -f: force reports to be mailed; if omitted, only reports that differ
+ from the sent.* version are sent.
+_EOF
+ exit 0
+fi
+
+# Find a good awk.
+if test -z "$AWK" ; then
+ for AWK in gawk nawk awk ; do
+ if type $AWK 2>&1 | grep 'not found' > /dev/null 2>&1 ; then
+ :
+ else
+ break
+ fi
+ done
+fi
+
+: ${filesuffix=}; export filesuffix
+: ${move=true}; export move
+: ${forcemail=false}; export forcemail
+while true; do
+ case "$1" in
+ -o) filesuffix=.sent; move=false; : ${mailto=nobody}; shift;;
+ -t) move=false; shift;;
+ -p) prepend_logs=${prepend_logs+"$prepend_logs "}"$2"; shift 2;;
+ -i) append_logs=${append_logs+"$append_logs "}"$2"; shift 2;;
+ -m) mailto=$2; forcemail=true; shift 2;;
+ -f) unset mailto; forcemail=true; shift;;
+ *) break;;
+ esac
+done
+: ${mailto="\" address \""}; export mailto
+files=`find . -name \*.sum$filesuffix -print | sort`
+anyfile=false anychange=$forcemail &&
+for file in $files; do
+ [ -f $file ] &&
+ anyfile=true &&
+ { $anychange ||
+ anychange=`diff $file.sent $file 2>/dev/null |
+ if test ! -f $file.sent ||
+ egrep '^[<>] (XPASS|FAIL)' >/dev/null; then
+ echo true
+ else
+ echo false
+ fi
+ `
+ }
+ true
+done &&
+$anyfile &&
+if $forcemail || $anychange; then :; else mailto=nobody; fi &&
+# We use cat instead of listing the files as arguments to AWK because
+# GNU awk 3.0.0 would break if any of the filenames contained `=' and
+# was preceded by an invalid ``variable'' name.
+( echo @TOPLEVEL_CONFIGURE_ARGUMENTS@ | ./config.status --file=-; cat $files ) |
+$AWK '
+BEGIN {
+ lang=""; configflags = "";
+ address="gcc-testresults@gcc.gnu.org";
+ version="gcc";
+ print "cat <<'"'"'EOF'"'"' |";
+'${prepend_logs+" system(\"cat $prepend_logs\"); "}'
+}
+NR == 1 {
+ configflags = $0 " ";
+ srcdir = configflags;
+ sub(/\/configure\047? .*/, "", srcdir);
+ sub(/^\047/, "", srcdir);
+ if ( system("test -f " srcdir "/LAST_UPDATED") == 0 ) {
+ printf "LAST_UPDATED: ";
+ system("tail -1 " srcdir "/LAST_UPDATED");
+ print "";
+ }
+
+ sub(/^[^ ]*\/configure\047? */, " ", configflags);
+ sub(/,;t t $/, " ", configflags);
+ sub(/ --with-gcc-version-trigger=[^ ]* /, " ", configflags);
+ sub(/ --norecursion /, " ", configflags);
+ sub(/ $/, "", configflags);
+ sub(/^ *$/, " none", configflags);
+ configflags = "configure flags:" configflags;
+}
+/^Running target / { print ""; print; }
+/^Target / { if (host != "") next; else host = $3; }
+/^Host / && host ~ /^unix\{.*\}$/ { host = $3 " " substr(host, 5); }
+/^Native / { if (host != "") next; else host = $4; }
+/^[ ]*=== [^ ]+ tests ===/ {
+ if (lang == "") lang = " "$2" "; else lang = " ";
+}
+$2 == "version" { save = $0; $1 = ""; $2 = ""; version = $0; gsub(/^ */, "", version); gsub(/\r$/, "", version); $0 = save; }
+/\===.*Summary/ { print ""; print; blanks=1; }
+/tests ===/ || /^(Target|Host|Native)/ || $2 == "version" { print; blanks=1; }
+/^(XPASS|FAIL|UNRESOLVED|WARNING|ERROR|# of )/ { sub ("\r", ""); print; }
+/^using:/ { print ""; print; print ""; }
+# dumpall != 0 && /^X?(PASS|FAIL|UNTESTED)|^testcase/ { dumpall=0; }
+# dumpall != 0 { print; }
+# /^FAIL/ { dumpall=1; }
+/^$/ && blanks>0 { print; --blanks; }
+END { if (lang != "") {
+ print "";
+ print "Compiler version: " prefix version lang;
+ print "Platform: " host;
+ print configflags;
+ '${BOOT_CFLAGS+'print "BOOT_CFLAGS='"${BOOT_CFLAGS}"'";'}'
+ if (boot_cflags != 0) print boot_cflags;
+'${append_logs+" system(\"cat $append_logs\"); "}'
+ print "EOF";
+ print "Mail -s \"Results for " prefix version lang "testsuite on " host "\" '"${mailto}"' &&";
+}}
+{ next; }
+' | sed "s/\([\`\$\\\\]\)/\\\\\\1/g" &&
+if $move; then
+ for file in $files `ls -1 $files | sed s/sum$/log/`; do
+ [ -f $file ] && echo "mv `${PWDCMD-pwd}`/$file `${PWDCMD-pwd}`/$file.sent &&"
+ done
+fi &&
+echo true
+exit 0
@@ -217,10 +217,12 @@ check-parallel:
-rm -rf cache outputs temp
$(MAKE) -k do-check-parallel; \
result=$$?; \
+ find outputs -name gdb.sum -exec mv \{\} \{\}.sep \; ; \
+ find outputs -name gdb.log -exec mv \{\} \{\}.sep \; ; \
$(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh \
- `find outputs -name gdb.sum -print` > gdb.sum; \
+ `find outputs -name gdb.sum.sep -print` > gdb.sum; \
$(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh -L \
- `find outputs -name gdb.log -print` > gdb.log; \
+ `find outputs -name gdb.log.sep -print` > gdb.log; \
sed -n '/=== gdb Summary ===/,$$ p' gdb.sum; \
exit $$result
@@ -237,16 +239,18 @@ check-parallel-racy:
for n in `seq $$racyiter` ; do \
$(MAKE) -k do-check-parallel-racy \
RACY_OUTPUT_N=$$n; \
+ find racy_outputs/$$n -name gdb.sum -exec mv \{\} \{\}.sep \; ; \
+ find racy_outputs/$$n -name gdb.log -exec mv \{\} \{\}.sep \; ; \
$(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh \
- `find racy_outputs/$$n -name gdb.sum -print` > \
- racy_outputs/$$n/gdb.sum; \
+ `find racy_outputs/$$n -name gdb.sum.sep -print` > \
+ racy_outputs/$$n/gdb.sum.sep; \
$(SHELL) $(srcdir)/../../dg-extract-results.sh -L \
- `find racy_outputs/$$n -name gdb.log -print` > \
- racy_outputs/$$n/gdb.log; \
- sed -n '/=== gdb Summary ===/,$$ p' racy_outputs/$$n/gdb.sum; \
+ `find racy_outputs/$$n -name gdb.log.sep -print` > \
+ racy_outputs/$$n/gdb.log.sep; \
+ sed -n '/=== gdb Summary ===/,$$ p' racy_outputs/$$n/gdb.sum.sep; \
done; \
$(srcdir)/analyze-racy-logs.py \
- `ls racy_outputs/*/gdb.sum` > racy.sum; \
+ `ls racy_outputs/*/gdb.sum.sep` > racy.sum; \
sed -n '/=== gdb Summary ===/,$$ p' racy.sum
# Turn a list of .exp files into "check/" targets. Only examine .exp