OVERVIEW

Hydra is a master-client test framework that uses RMI for interprocess communication. It is written mostly in Java and invokes a few scripts and utilities.

Hydra is distributed, concurrent, and asynchronous. Tests can specify multiple resources and multi-threaded client processes and distribute them across multiple hosts and platforms. Tasks (units of work) are scheduled asynchronously. Tasks run concurrently on client threads by default, but a test can optionally be configured to run tasks sequentially.

Testing with hydra involves:

• Coding hydra tasks - Tasks are units of work written in Java as public static zero-argument methods. Hydra supports several different task types, each with their own scheduling algorithm: starttask, inittask, task, closetask, and endtask. Hydra also supports the task type unittest for use by the distributed unit test framework, {@link dunit}. Examples of hydra tasks can be found in the {@link hydratest.TaskClient} tryItOut methods, the client classes in {@link hydra.samples}, and throughout the test tree. Tasks are usually designed so that they can be used by many tests.

• Writing a hydra configuration file - A test configuration file tells the master what to do by defining the various client processes, tasks, scheduling instructions, work environments, resources, and parameter settings required by the test. It is a text file written according to hydra_grammar.txt. Example hydra configuration files can be found in hydraconfig, the .conf files in {@link hydra.samples}, and throughout the test tree.

• Running a hydra test - A hydra test is run by executing the test driver, {@link hydra.MasterController} on a test configuration file. Suites of tests can be run via {@link batterytest.BatteryTest}, which provides a layer on top of hydra. Hydra can also used to run tests in conjunction with the JUnit framework: {@link smoketest} for hydra smoke tests and {@link dunit} for distributed unit tests.

The below diagram shows the architecture of a Hydra test. A configuration file controls the {@link hydra.ClientPrms#vmQuantities number of client VMs}, the {@link hydra.ClientPrms#vmThreads number of Hydra threads in each client VM}, and the {@link hydra.HostPrms#hostNames hosts on which the VMs run}.

At runtime, the hydra master controller parses the configuration file, configures and starts the required processes, schedules tasks on clients, monitors test progress, shuts down processes, and reports the outcome. More specifically, the master:

• Parses the given configuration file and postprocesses the result into a test configuration object that can be accessed by the master and its clients via {@link hydra.TestConfig#getInstance}.

• Starts the RMI registry used for master-client communication.

• Starts hostagents needed to create and manage processes on remote hosts, if any.

• Creates, configures, and starts the resources, such as cache servers, if any.

• Starts single-threaded VMs for the starttasks, if any. Starttasks are typically used to do test initialization that produces side-effects visible to VMs started later for other task types. Starttasks use the {@link hydra.ClientNameTaskScheduler}. The master terminates these VMs when the starttasks complete.

• Starts multi-threaded VMs for the inittasks. Inittasks use the {@link hydra.SimpleTaskScheduler} and are typically used to initialize, on a per-VM and per-client-thread basis, any state needed for the main body of work to be carried out in the same VM by the tasks.

• Schedules tasks in the same VMs using the {@link hydra.ComplexTaskScheduler}. Tasks typically carry out the main body of work in a test. Their scheduler can be configured in great detail. For example, a task can be configured to run on particular client threads, at a particular interval, and with a particular priority. The scheduler continues scheduling the tasks over and over until a configurable termination condition is reached. Tasks can be configured to run for a certain amount of time, for a specified number of times, until an error condition is detected, and so forth.

• Schedules closetasks. Like starttasks, closetasks use the {@link hydra.SimpleTaskScheduler}. They are typically used to validate results and summarize statistics, on a per-client-thread and per-VM basis, produced by the main body of work. The master shuts down these VMs when all closetasks complete.

• Starts VMs for the endtasks. Endtasks are typically used to do statistic reporting, result analysis, and so forth, on a cross-VM basis, using information left behind by previous task VMs. Like starttasks, endtasks use the {@link hydra.ClientNameTaskScheduler}. The master terminates these VMs when the endtasks complete.

• Shuts down any remaining processes it started, including hostagents, the RMI registry, and resources such as cache servers.

• Reports the outcome of the test.

During a test run, client processes report task results to the master. By default, the master will stop a test that has an error. If the error is a perceived hang, it will leave processes running so a debugger can be attached. Also, all test processes produce log output that can be monitored and analyzed. Test code can write to the logs as needed. The primary master log in a test run is Master_.log, which gives the final result of test execution.

In many cases, writing an effective hydra test is a matter of paying attention to just a small handful of items:

• Encapsulating small chunks of work (tasks) in static methods.

• Using hydra configuration parameters to transmit values that control test activities. This lets you change your tests in important ways without rewriting your java code.

• Taking advantage of bounded randomness so that tests continue to do new things even without configuration file changes. This is achieved with {@link hydra.GsRandom}, {@link hydra.OneOf}, {@link hydra.Range}, {@link hydra.Fcn}, and system property variables ${property}.

• Using utilities to encapsulate common operations. This reduces the amount of code to write and brings those operations under the control of the test framework in a consistent and convenient way. See util and some of the classes in this package for examples.

GETTING STARTED

Sample hydra environment for Solaris

      setenv GEMFIRE   path_to_gemfire_checkout/build-artifacts/product
      setenv JAVA_HOME $GEMFIRE/jre;
      setenv JTESTS    $GEMFIRE/../tests/classes
      setenv JPROBE    /home/tools/jprobe40
      setenv CLASSPATH $JTESTS
      setenv LD_LIBRARY_PATH $GEMFIRE/lib:$GEMFIRE/../hidden/lib:$LD_LIBRARY_PATH
      set path = (. $GEMFIRE/bin $JAVA_HOME/bin vsd/vsdtree/bin $path)

Running hydra directly

To run an existing hydra test, first find a .conf file in the test tree. Then run hydra as described by {@link hydra.MasterController}.

Running hydra via batterytest

To run an existing batterytest, first find a .bt file in the test tree. Then run batterytest as described by {@link batterytest.BatteryTest}.

Running hydra smoketests

To run the existing hydra smoke tests, run build.sh run-smoke-tests.

TEST CONFIGURATION

The configuration file is the key element in defining and controlling a hydra run. A hydra config file can specify most of the details of a GemFire configuration as well as several different kinds of test properties and activities.

Include files

...discuss include files, tests/hydraconfig...

Configuring hydra tasks

See {@link hydra.TestTask}.

Configuring hydra thread groups

See {@link hydra.HydraThreadGroup}.

Configuring hydra parameters

Hydra allows a test configuration file to specify values for parameters defined in any subclass of {@link hydra.BasePrms} on the classpath. The hydra package itself includes these parameter classes:

• {@link hydra.HostPrms} to configure hosts. The operating system platform for each host is autogenerated by hydra.

• {@link hydra.VmPrms} to configure VM environments, including the logical host name for each.

• {@link hydra.GemFirePrms} to configure GemFire systems, including the logical host name for each.

• {@link hydra.Prms} to configure the master controller. The VM environment for the master is autogenerated by hydra.

• {@link hydra.ClientPrms} to configure clients, including the logical VM name and optional logical GemFire name for each.

• {@link hydra.log.LogPrms} to configure logging for hydra processes. Logging for GemFire processes is configured through GemFirePrms.

• {@link hydra.JProbePrms} to configure the JProbe performance and memory analysis tools.

Test packages can easily define additional parameter classes by subclassing {@link hydra.BasePrms}. See the javadocs for BasePrms for how to do this. Also see the javadoc'd and non-javadoc'd subclasses of BasePrms in the test tree for examples.

Specifying parameters

Specify parameters in the hydra configuration file as key = value;, where key is fully_qualified_class_name-parameter_name. See the javadocs for the parameter to determine the type of value required, how it defaults, etc. For example:

    hydra.Prms-totalTaskTimeSec   = 60; // run the test for 60 seconds

    hydra.ClientPrms-names        = clientA clientB; // specifies two client types
    hydra.ClientPrms-vmNames      = vm;   // logical vm name shared by both client types
                                          // the vm must be configured elsewhere in the file
    hydra.ClientPrms-vmQuantities = 1  2; // clientA has 1 VM and clientB has 2
    hydra.ClientPrms-vmThreads    = 1 10; // clientA has 1 thread per VM and clientB has 10

Individual tokens in a parameter value can sometimes be replaced with one of the special types: {@link hydra.Range} if the token is numerical, and {@link hydra.OneOf} or {@link hydra.RobinG} for just about anything. These types can also be nested. For example,

    // produces random value between 1 and 10
    myPackage.MyTestPrms-prm1 = RANGE 1 10 EGNAR;

    // produces random value: true or false
    myPackage.MyTestPrms-prm3 = ONEOF true false FOENO;

    // produces global round robin values in this order: 1 2 3 4 5
    myPackage.MyTestPrms-prm4 = ROBING 1 2 3 4 5 GNIBOR;

    // produces random value: 0 or a random value between 100 and 200
    myPackage.MyTestPrms-prm2 = ONEOF 0 RANGE 100 200 EGNAR FOENO;

Parameters can also expect a list of lists, where a comma separates each list. The following line, for example, sets numLists to be a three-element list of lists each of which has two elements:

  mypackage.MyPrms-numLists = 1 2, 3 4, 5 6;

For more detail on syntax, see hydra_grammar.txt. For examples, see the hydra include and configuration files in the hydraconfig and {@link hydra.samples} packages and elsewhere in the test tree.

Overriding parameters

...discuss local.conf...

Accessing parameters at runtime

The hydra master controller postprocesses the result of parsing the test configuration file into a {@link hydra.TestConfig} object for use by the master and its clients. Hydra clients have access to configuration parameters at runtime via this {@link hydra.TestConfig} object. The parameters are stored in a {@link hydra.ConfigHashtable}, which provides convenience methods for accessing various types of parameter values. For example,

    ConfigHashtable prms = TestConfig.getInstance().getParameters();
    long totalTaskTimeSec = prms.longAt(Prms.totalTaskTimeSec);

If the parameter is a range or oneof type, and is read via ConfigHashtable methods, hydra will return a random value. For example, if the configuration file contains:

    myPackage.MyTestPrms-size = RANGE 1 10;

    int size = prms.intAt(MyTestPrms.size); // returns random int between 1 and 10

The master also postprocesses certain parameters into description objects for the various test processes and stores them in the {@link hydra.TestConfig object}. The relationship (the "arrow" represents a "has a" relationship) between these objects is:

                         {@link hydra.HostDescription}
                                ^
                     ___________|__________
                    |                      |
            {@link hydra.GemFireDescription}       {@link hydra.VmDescription}
                   ^                      ^ ^
           ________|              ________| |________
          |                      |                   |
                                      {@link hydra.MasterDescription}
    {@link hydra.ClientDescription}

Descriptions are used primarily by hydra internals and to log the postprocessed test configuration. Hydra makes commonly used information in the descriptions more directly available to clients via system properties and helper classes. For example, for a client thread to get its test directory, it can navigate descriptions like so:

    String clientName = System.getProperty(ClientPrms.CLIENT_NAME_PROPERTY);
    File sysDir = TestConfig.getInstance()
                            .getClientDescription(clientName)
                            .getVmDescription()
                            .getHostDescription()
                            .getTestDir();

    String testDir = System.getProperty("JTESTS");

As another example, suppose a client thread wanted to determine its actual host name. It could navigate its way through the TestConfig to the ClientDescription to the VmDescription to the HostDescription, but can more easily use {@link java.net.InetAddress#getLocalHost} or, even better, the convenience method {@link hydra.HostHelper#getLocalHost}.

LOGGING

Hydra automatically logs task assignments, results, etc.

Tests can do additional logging as needed. Hydra makes a singleton instance of {@link com.gemstone.gemfire.LogWriter} available to clients at runtime via the static method {@link hydra.Log#getLogWriter()}. Messages written to this logger appear in the normal client logs. Messages written to System.out and System.err appear in bgexec files.

Logging is configured via {@link hydra.log.LogPrms}. GemFire system logging is configured via {@link hydra.GemFirePrms}.

Each hydra process produces one or more log files. All log files for all test processes on all hosts go the test directory where master controller is running. Startup information for master and client VMs goes to its corresponding bgexec log. Additional information goes to the Master, taskmaster, and client vm logs. The process id is included in the log file name and can be used to match the bgexec and vm_ files for a process. Watch Master_.log, taskmaster_.log, and bgexecmaster_.log for output from the master controller. Watch bgexec_.log and vm____.log for output from clients.

ANALYZING TEST RESULTS

Hydra reports test errors in the file "errors.txt" in the test directory.

USING JPROBE WITH HYDRA

Attaching to a process

JProbe allows attaching to and monitoring a running process. To do this, turn on monitoring for the desired JProbe configuration in the test configuration file, for example:

    hydra.JProbePrms-monitor = true;
    hydra.JProbePrms-recordFromStart = false;
    hydra.JProbePrms-finalSnapshot = false;