Skip navigation links
GemFire XD 1.5.6-rc1

Package hydra

Hydra is a test framework that makes it easy to create and run multiuser tests.

See: Description

Package hydra Description

Hydra is a test framework that makes it easy to create and run multiuser tests.

OVERVIEW

Hydra is a master-client test framework that uses RMI for interprocess communication. It is written mostly in Java and invokes a few scripts and utilities.

Hydra is distributed, concurrent, and asynchronous. Tests can specify multiple resources and multi-threaded client processes and distribute them across multiple hosts and platforms. Tasks (units of work) are scheduled asynchronously. Tasks run concurrently on client threads by default, but a test can optionally be configured to run tasks sequentially.

Testing with hydra involves:

The below diagram shows the architecture of a Hydra test. A configuration file controls the number of client VMs, the number of Hydra threads in each client VM, and the hosts on which the VMs run.

At runtime, the hydra master controller parses the configuration file, configures and starts the required processes, schedules tasks on clients, monitors test progress, shuts down processes, and reports the outcome. More specifically, the master:

During a test run, client processes report task results to the master. By default, the master will stop a test that has an error. If the error is a perceived hang, it will leave processes running so a debugger can be attached. Also, all test processes produce log output that can be monitored and analyzed. Test code can write to the logs as needed. The primary master log in a test run is Master_.log, which gives the final result of test execution.

In many cases, writing an effective hydra test is a matter of paying attention to just a small handful of items:

For more information, see the rest of this document and the javadocs for the various classes.

GETTING STARTED

Sample hydra environment for Solaris 

      setenv GEMFIRE   path_to_gemfire_checkout/build-artifacts/product
      setenv JAVA_HOME $GEMFIRE/jre;
      setenv JTESTS    $GEMFIRE/../tests/classes
      setenv JPROBE    /home/tools/jprobe40
      setenv CLASSPATH $JTESTS
      setenv LD_LIBRARY_PATH $GEMFIRE/lib:$GEMFIRE/../hidden/lib:$LD_LIBRARY_PATH
      set path = (. $GEMFIRE/bin $JAVA_HOME/bin vsd/vsdtree/bin $path)

Running hydra directly

To run an existing hydra test, first find a .conf file in the test tree. Then run hydra as described by MasterController.

Running hydra via batterytest

To run an existing batterytest, first find a .bt file in the test tree. Then run batterytest as described by BatteryTest.

Running hydra smoketests

To run the existing hydra smoke tests, run build.sh run-smoke-tests.

TEST CONFIGURATION

The configuration file is the key element in defining and controlling a hydra run. A hydra config file can specify most of the details of a GemFire configuration as well as several different kinds of test properties and activities.

Include files

...discuss include files, tests/hydraconfig...

Configuring hydra tasks

See TestTask.

Configuring hydra thread groups

See HydraThreadGroup.

Configuring hydra parameters

Hydra allows a test configuration file to specify values for parameters defined in any subclass of BasePrms on the classpath. The hydra package itself includes these parameter classes:

Test packages can easily define additional parameter classes by subclassing BasePrms. See the javadocs for BasePrms for how to do this. Also see the javadoc'd and non-javadoc'd subclasses of BasePrms in the test tree for examples.

Specifying parameters

Specify parameters in the hydra configuration file as key = value;, where key is fully_qualified_class_name-parameter_name. See the javadocs for the parameter to determine the type of value required, how it defaults, etc. For example: 

    hydra.Prms-totalTaskTimeSec   = 60; // run the test for 60 seconds

    hydra.ClientPrms-names        = clientA clientB; // specifies two client types
    hydra.ClientPrms-vmNames      = vm;   // logical vm name shared by both client types
                                          // the vm must be configured elsewhere in the file
    hydra.ClientPrms-vmQuantities = 1  2; // clientA has 1 VM and clientB has 2
    hydra.ClientPrms-vmThreads    = 1 10; // clientA has 1 thread per VM and clientB has 10

Individual tokens in a parameter value can sometimes be replaced with one of the special types: Range if the token is numerical, and OneOf or RobinG for just about anything. These types can also be nested. For example, 

    // produces random value between 1 and 10
    myPackage.MyTestPrms-prm1 = RANGE 1 10 EGNAR;

    // produces random value: true or false
    myPackage.MyTestPrms-prm3 = ONEOF true false FOENO;

    // produces global round robin values in this order: 1 2 3 4 5
    myPackage.MyTestPrms-prm4 = ROBING 1 2 3 4 5 GNIBOR;

    // produces random value: 0 or a random value between 100 and 200
    myPackage.MyTestPrms-prm2 = ONEOF 0 RANGE 100 200 EGNAR FOENO;

Parameters can also expect a list of lists, where a comma separates each list. The following line, for example, sets numLists to be a three-element list of lists each of which has two elements: 

  mypackage.MyPrms-numLists = 1 2, 3 4, 5 6;

For more detail on syntax, see hydra_grammar.txt. For examples, see the hydra include and configuration files in the hydraconfig and hydra.samples packages and elsewhere in the test tree.

Overriding parameters

...discuss local.conf...

Accessing parameters at runtime

The hydra master controller postprocesses the result of parsing the test configuration file into a TestConfig object for use by the master and its clients. Hydra clients have access to configuration parameters at runtime via this TestConfig object. The parameters are stored in a ConfigHashtable, which provides convenience methods for accessing various types of parameter values. For example, 

    ConfigHashtable prms = TestConfig.getInstance().getParameters();
    long totalTaskTimeSec = prms.longAt(Prms.totalTaskTimeSec);

If the parameter is a range or oneof type, and is read via ConfigHashtable methods, hydra will return a random value. For example, if the configuration file contains: 

    myPackage.MyTestPrms-size = RANGE 1 10;
Then at runtime, 
    int size = prms.intAt(MyTestPrms.size); // returns random int between 1 and 10

The master also postprocesses certain parameters into description objects for the various test processes and stores them in the object. The relationship (the "arrow" represents a "has a" relationship) between these objects is: 

                         HostDescription
                                ^
                     ___________|__________
                    |                      |
            GemFireDescription       VmDescription
                   ^                      ^ ^
           ________|              ________| |________
          |                      |                   |
                                      MasterDescription
    ClientDescription

Descriptions are used primarily by hydra internals and to log the postprocessed test configuration. Hydra makes commonly used information in the descriptions more directly available to clients via system properties and helper classes. For example, for a client thread to get its test directory, it can navigate descriptions like so: 

    String clientName = System.getProperty(ClientPrms.CLIENT_NAME_PROPERTY);
    File sysDir = TestConfig.getInstance()
                            .getClientDescription(clientName)
                            .getVmDescription()
                            .getHostDescription()
                            .getTestDir();
But it can get this particular information more easily using: 
    String testDir = System.getProperty("JTESTS");

As another example, suppose a client thread wanted to determine its actual host name. It could navigate its way through the TestConfig to the ClientDescription to the VmDescription to the HostDescription, but can more easily use InetAddress.getLocalHost() or, even better, the convenience method HostHelper.getLocalHost().

LOGGING

Hydra automatically logs task assignments, results, etc.

Tests can do additional logging as needed. Hydra makes a singleton instance of LogWriter available to clients at runtime via the static method Log.getLogWriter(). Messages written to this logger appear in the normal client logs. Messages written to System.out and System.err appear in bgexec files.

Logging is configured via LogPrms. GemFire system logging is configured via GemFirePrms.

Each hydra process produces one or more log files. All log files for all test processes on all hosts go the test directory where master controller is running. Startup information for master and client VMs goes to its corresponding bgexec log. Additional information goes to the Master, taskmaster, and client vm logs. The process id is included in the log file name and can be used to match the bgexec and vm_ files for a process. Watch Master_.log, taskmaster_.log, and bgexecmaster_.log for output from the master controller. Watch bgexec_.log and vm____.log for output from clients.

ANALYZING TEST RESULTS

Hydra reports test errors in the file "errors.txt" in the test directory.

USING JPROBE WITH HYDRA

Attaching to a process

JProbe allows attaching to and monitoring a running process. To do this, turn on monitoring for the desired JProbe configuration in the test configuration file, for example: 

    hydra.JProbePrms-monitor = true;
    hydra.JProbePrms-recordFromStart = false;
    hydra.JProbePrms-finalSnapshot = false;
This causes hydra to start the VMs that use this JProbe configuration using a randomly generated port. JProbe causes the VM to hang on startup until the jpprofiler or jpmemorydebugger is used to attach. Watch the log file for the process for the connection message. For clients, this is the bgexec file. Look for the port on which the process is waiting. The port is also given in the taskmaster log file. Use the desired JProbe tool to attach to the process. Check the log to see that it has received the connection and proceeded.
Skip navigation links
GemFire XD 1.5.6-rc1

Copyright © 2010-2015 Pivotal Software, Inc. All rights reserved.