From: Eric Sandeen Date: Fri, 13 Jan 2017 14:56:15 +0000 (-0600) Subject: tools: compare-failures script X-Git-Tag: v2022.05.01~2219 X-Git-Url: http://git.apps.os.sepia.ceph.com/?p=xfstests-dev.git;a=commitdiff_plain;h=2ad51674753978db5c1a036eaf08b00994a218eb tools: compare-failures script This is a simple script to compare failures across runs. Given files containing stdout from several runs, each of which contains a Failures: line, it will print a table of all failures for each run. Test subdir names are abbreviated for compactness, i.e. generic->g. For 7 results files named test 1 through test 7: Failures: g/075 g/082 g/209 g/233 g/270 g/388 x/004 x/073 x/076 ----------------------------------------------------- g/082 g/233 x/004 x/073 test1 g/082 g/233 x/004 x/073 x/076 test2 g/082 x/004 x/073 x/076 test3 g/082 g/388 x/004 x/073 test4 g/082 g/270 x/004 x/073 test5 g/082 x/004 x/073 test6 g/075 g/082 g/209 g/233 x/004 x/073 test7 This lets us easily spot unique failures and outliers. This could be enhanced to output CSV etc, but for now I think it's helpful to visualize changes in failures across multiple runs. Signed-off-by: Eric Sandeen Reviewed-by: Eryu Guan Signed-off-by: Eryu Guan --- diff --git a/README b/README index f788e718..bb42985f 100644 --- a/README +++ b/README @@ -272,6 +272,9 @@ Pass/failure: The elapsed time for the most recent pass for each test is kept in "check.time". + The compare-failures script in tools/ may be used to compare failures + across multiple runs, given files containing stdout from those runs. + __________________ SUBMITTING PATCHES __________________ diff --git a/tools/compare-failures b/tools/compare-failures new file mode 100755 index 00000000..9604a638 --- /dev/null +++ b/tools/compare-failures @@ -0,0 +1,67 @@ +#!/bin/bash + +# Compare test failures across runs +# +# Takes multiple "results" files as arguments, comprised of the +# stdout from a ./check run, each containing a Failures: line. +# +# Outputs a table of failures for comparison across runs +# +#----------------------------------------------------------------------- +# Copyright (c) 2017 Red Hat, Inc. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# + +filter_names() { + sed -e s/btrfs/b/ \ + -e s/cifs/c/g \ + -e s/f2fs/f/g \ + -e s/generic/g/g \ + -e s/overlay/o/g \ + -e s/shared/s/g \ + -e s/udf/u/g \ + -e s/xfs/x/g +} + +# ALLFAILURES: A B C D E F G +# THESEFAILURES: A C D G + +# We want to print the header (ALLFAILURES) and then +# if a run didn't fail a particular test, print spaces instead + +# All tests that failed in any run, all on one line, unique +ALLFAILURES=`grep -h ^Failures: $* \ + | tr " " "\n" \ + | sort | uniq \ + | filter_names \ + | tr "\n" " " \ + | sed -e "s/^Failures: //g"` + +# Header +echo "Failures:" +echo $ALLFAILURES +echo $ALLFAILURES | sed -e "s/./-/g" + +# Per-file failures +for FILE in $*; do + THESEFAILURES=`grep ^Failures: $FILE | filter_names` + for FAILURE in $ALLFAILURES; do + CELL=`echo $THESEFAILURES \ + | grep -wo "$FAILURE" || echo $FAILURE | sed -e "s/./ /g"` + echo -n "$CELL " + done + echo $FILE +done