perffmwk

Class PerfComparer

java.lang.Object

perffmwk.Formatter

perffmwk.PerfComparer

All Implemented Interfaces:

ComparisonConstants

public class PerfComparer
extends Formatter
implements ComparisonConstants

Generates performance comparisons for the specified test container directories.

When only one test container directory is provided, the tests in that directory are compared to each other in lexicographic order of the result directory names.

When there is more than one test container directory, they are compared in the order given in the command-line arguments, with duplicates removed. Only tests with the same ID are compared with each other, where the test ID consists of the hydra test configuration file name and properties. The first available run of a particular test is considered the baseline. Each test container directory must contain at most one test run with a given test ID.

Comparisons are made using values read from the statistic archives for each test, as described in PerfReporter. By default, the statistics specification file used for a given test ID is the last run of the test in the last container that includes it. The statSpecFile system property can be used to override with a different specification file.

Tests can be compared in "raw" or "ratio" mode. Raw mode simply reports statistics values. Ratio mode indicates the percent change in value, with a sign indicating whether the change was "good" (positive) or "bad" (negative). Use ratioThreshold to skip reporting values under the threshold.

Set omitFailedTests true to omit reporting values for tests that hung or failed.

Set markFailedTests true to mark tests that hung or failed with "xxx". This overrides the setting for omitFailedTests to mark the failures but does not report their statistics values.

Set compareByKey true to compare tests with the same value of PerfSorter.COMPARISON_KEY_PROP.

Set addTestKey true to include a key of test descriptions in the report.

Set addConfigDiffs true to report differences in hydra test configuration files.

The comparison report is written by default to "perfcomparison.txt" in the working directory. The compReportFile system property can be used to override with a different file. Set compReportFile to "none" to write the report to stdout.

Set generateCSVFile true to generate a CSV file suitable for Excel. When there is a single test container directory, a row is created for each test. A column is generated for each property found in a test property file, and for each statistic value. When there are multiple test container directories, the rows are identical to the non-CSV report, and the report banner and various keys are excluded. The CSV file is written by default to "perfcomparison.csv" in the working directory. The csvFile system property can be used to override with a different file.

Log messages for the tool are appended to "perfcomparer.log" in the working directory. Use logLevel to change the verbosity of the log.

Usage:

   java -Dgemfire.home=<path_to_gemfire_product_tree>
        -DJTESTS=<path_to_test_classes>
        [-DcompReportFile=<report_filename(default:$pwd/perfcomparison.txt)>]
        [-DlogLevel=<level(default:info)>]
        [-Dmode=<comparison_mode(default:ratio)>]
        [-Dratio.threshold=<ratio_threshold(default:0.05)>]
        [-DomitFailedTests=<whether_to_omit_failed_tests(default:false)>]
        [-DmarkFailedTests=<whether_to_mark_failed_tests(default:false)>]
        [-DcompareByKey=<whether_to_compare_tests_by_key(default:false)>]
        [-DaddTestKey=<whether_to_include_test_key(default:false)>]
        [-DaddConfigDiffs=<whether_to_include_config_diffs(default:false)>]
        [-DstatSpecFile=<stat_spec_filename>]
        [-DgenerateCSVFile=<whether_to_generate_csv_file(default:false)>]
        [-DcsvFile=<csv_filename(default:$pwd/perfcomparison.csv)>]
        perffmwk.PerfComparer
        <two_or_more_comparable_test_directories>
 

Example:

    java -classpath $JTESTS:$GEMFIRE/lib/gemfire.jar
         -Dgemfire.home=$GEMFIRE -DJTESTS=$JTESTS
         perffmwk.PerfComparer test-081303-* test-081403-*
 

Example:

    java -classpath $JTESTS:$GEMFIRE/lib/gemfire.jar
         -Dgemfire.home=$GEMFIRE -DJTESTS=$JTESTS
         perffmwk.PerfComparer testsOnBuildA testsOnBuildB
 

Field Summary

Fields

Modifier and Type	Field and Description
protected static double	RatioThreshold

Fields inherited from class perffmwk.Formatter

DIVIDER, SUBDIVIDER, WIDTH

Fields inherited from interface perffmwk.ComparisonConstants

AVG_OP, BOGUS, ERR_OP, ERR_VAL, ERR_VAL_DOC, FAIL, FAIL_VAL, FAIL_VAL_DOC, HANG, MAX_OP, MIN_OP, MINUS_INFINITY, MINUS_INFINITY_DOC, MISSING_VAL, MISSING_VAL_DOC, MMM_OP, NIL_VAL, NIL_VAL_DOC, PASS, PLUS_INFINITY, PLUS_INFINITY_DOC, STD_OP

Constructor Summary

Constructors

Constructor and Description
PerfComparer()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)

Methods inherited from class perffmwk.Formatter

center, formatDecimal, maxLength, padLeft, padLeft, padLeft, padRight, padRight, padRight

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

RatioThreshold

protected static double RatioThreshold

Constructor Detail

PerfComparer

public PerfComparer()

Method Detail

main

public static void main(String[] args)