hdfs

Class HDFSUtil

java.lang.Object

hdfs.HDFSUtil

public class HDFSUtil
extends Object

Generic utility methods for testing with Hoplogs/HDFS

Field Summary

Fields

Modifier and Type	Field and Description
static String	HDFS_RESULT_REGION
static boolean	serverAlive Utility methods to monitor and wait for HDFS AEQ to drain

Constructor Summary

Constructors

Constructor and Description
HDFSUtil()

Method Summary

Modifier and Type	Method and Description
static void	configureAndStartHadoop()
static void	configureHadoopTask()
static void	configureHDFSTask()
static void	dumpHDFSResultRegion() Dump the HDFSResultRegion, created by the GetAllHDFSEventsForKey() mapreduce job.
static void	execMapReduceJob() Launch the MapReduce program specified by HDFSPrms.mapReduceClass by fgexec'ing a comand of the form: yarn --config jar .class libjars
static void	execMapReduceJob(String extraArgs)
static void	execMapReduceJob(String mapReduceClassName, String extraArgs)
static void	formatNameNodesTask()
static void	getAllHDFSEventsForKey(String keys) Exec a mapReduce job to find all updates for a particular entry and write those entries back out to a GemFire region (HDFS_RESULTS_REGION) as ("originalKey_EventNumber_Operation", value).
static String	getHDFSStoreName(com.gemstone.gemfire.cache.Region aRegion)
static void	HydraTask_startQueueMonitor() Starts a thread to monitor HDFS AEQ queues (may be active across multiple vms).
static void	HydraTask_waitForQueuesToDrain() Waits Prms.maxResultWaitSec for HDFS AEQs to drain.
static void	loadDataFromHDFS(com.gemstone.gemfire.cache.Region aRegion, String regionName) Utility method invoked by HydraTask_loadDataFromHDFS() for various streaming tests (e.g.
static void	loadSequenceFile(com.gemstone.gemfire.cache.Region aRegion, org.apache.hadoop.fs.FileSystem inputFS, org.apache.hadoop.fs.Path sequenceFileName)
static void	recycleDataNode()
static void	recycleNameNode()
static void	restartCluster()
static void	startCluster()
static void	startDataNodesTask()
static void	startHDFSCluster()
static void	startMapReduceCluster()
static void	startNameNodesTask()
static void	stopCluster()
static void	stopDataNodesTask()
static void	stopHDFSCluster()
static void	stopMapReduceCluster()
static void	stopNameNodesTask()
static void	stopStartDataNodes()
static void	stopStartNameNodes()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HDFS_RESULT_REGION

public static final String HDFS_RESULT_REGION

See Also:

Constant Field Values

serverAlive

public static boolean serverAlive

Utility methods to monitor and wait for HDFS AEQ to drain

Constructor Detail

HDFSUtil

public HDFSUtil()

Method Detail

startMapReduceCluster

public static void startMapReduceCluster()

stopMapReduceCluster

public static void stopMapReduceCluster()

startCluster

public static void startCluster()

stopCluster

public static void stopCluster()

startHDFSCluster

public static void startHDFSCluster()

stopHDFSCluster

public static void stopHDFSCluster()

configureHDFSTask

public static void configureHDFSTask()

configureHadoopTask

public static void configureHadoopTask()

formatNameNodesTask

public static void formatNameNodesTask()

startNameNodesTask

public static void startNameNodesTask()

startDataNodesTask

public static void startDataNodesTask()

restartCluster

public static void restartCluster()

stopNameNodesTask

public static void stopNameNodesTask()

stopDataNodesTask

public static void stopDataNodesTask()

configureAndStartHadoop

public static void configureAndStartHadoop()

stopStartDataNodes

public static void stopStartDataNodes()

stopStartNameNodes

public static void stopStartNameNodes()

recycleDataNode

public static void recycleDataNode()

recycleNameNode

public static void recycleNameNode()

loadDataFromHDFS

public static void loadDataFromHDFS(com.gemstone.gemfire.cache.Region aRegion,
                                    String regionName)

Utility method invoked by HydraTask_loadDataFromHDFS() for various streaming tests (e.g. and ParRegTest). Given the the original region name (so we can access in HDFS), load a region with the operations stored in HDFS. This is done in order (per bucket) ... so we should end up with the same region contents as the main workload vms.

loadSequenceFile

public static void loadSequenceFile(com.gemstone.gemfire.cache.Region aRegion,
                                    org.apache.hadoop.fs.FileSystem inputFS,
                                    org.apache.hadoop.fs.Path sequenceFileName)
                             throws Exception

Throws:

Exception

execMapReduceJob

public static void execMapReduceJob()

Launch the MapReduce program specified by HDFSPrms.mapReduceClass by fgexec'ing a comand of the form: yarn --config jar .class libjars

execMapReduceJob

public static void execMapReduceJob(String extraArgs)

execMapReduceJob

public static void execMapReduceJob(String mapReduceClassName,
                                    String extraArgs)

getAllHDFSEventsForKey

public static void getAllHDFSEventsForKey(String keys)

Exec a mapReduce job to find all updates for a particular entry and write those entries back out to a GemFire region (HDFS_RESULTS_REGION) as ("originalKey_EventNumber_Operation", value). Note that destroyed keys have a value of "DESTORYED". Creates the HDFSResultRegion if needed.

dumpHDFSResultRegion

public static void dumpHDFSResultRegion()

Dump the HDFSResultRegion, created by the GetAllHDFSEventsForKey() mapreduce job. Useful for debugging lost events, etc in HDFS

getHDFSStoreName

public static String getHDFSStoreName(com.gemstone.gemfire.cache.Region aRegion)

HydraTask_startQueueMonitor

public static void HydraTask_startQueueMonitor()

Starts a thread to monitor HDFS AEQ queues (may be active across multiple vms).

HydraTask_waitForQueuesToDrain

public static void HydraTask_waitForQueuesToDrain()

Waits Prms.maxResultWaitSec for HDFS AEQs to drain. Suitable for use as a helper method or hydra task.