hydra

Class HadoopHelper

java.lang.Object

hydra.HadoopHelper

public class HadoopHelper
extends Object

Helps hydra clients use HadoopDescription to manage a hadoop cluster. Methods are thread-safe but not distributed-thread-safe, so must be called by a single hydra client JVM to avoid problems at the hadoop cluster level.

Field Summary

Fields

Modifier and Type	Field and Description
static String	GFXD_SECURE_KEYTAB_FILE
static String	HDFS_SECURE_KEYTAB_FILE
static String	MAPRED_SECURE_KEYTAB_FILE
static String	YARN_SECURE_KEYTAB_FILE

Constructor Summary

Constructors

Constructor and Description
HadoopHelper()

Method Summary

Modifier and Type	Method and Description
static void	configureHadoop(String hadoopConfig) Builds the Hadoop configuration files for the given cluster configuration.
static void	configureHDFS(String hadoopConfig) Builds the Hadoop configuration files for the given cluster configuration.
static void	executeHadoopCommand(String host, HadoopDescription hdd, String command, String outfn) Executes a Hadoop command on a given host using the given HadoopDescription from HadoopPrms.names.
static void	executeHadoopCommand(String host, String hadoopConfig, String command, String outfn) Executes a Hadoop command on given host using the given cluster configuration from HadoopPrms.names.
static void	executeKdestroyCommand(String host, int maxWaitSec) Executes "kdestroy" on the specified host.
static void	executeKinitCommand(String host, String command, int maxWaitSec) Executes "kinit" on the given command on the specified host.
static void	formatNameNode(String hadoopConfig, String host) Formats the Hadoop NameNode on the given host using the given node configuration.
static void	formatNameNodes(String hadoopConfig) Formats Hadoop NameNodes using the given cluster configuration.
static HadoopDescription	getHadoopDescription(String hadoopConfig) Returns the HadoopDescription for the given cluster configuration.
static void	runHadoopCommand(HadoopDescription hdd, String command, String outfn) Runs a Hadoop command using the given HadoopDescription from HadoopPrms.names.
static void	runHadoopCommand(String hadoopConfig, String command, String outfn) Runs a Hadoop command using the given cluster configuration from HadoopPrms.names.
static void	runYarnCommand(String hadoopConfig, String command, String outfn) Runs a YARN command using the given cluster configuration from HadoopPrms.names.
static void	startCluster(String hadoopConfig) Starts the Hadoop cluster using the given cluster configuration.
static void	startDataNode(HadoopDescription hdd, HadoopDescription.NodeDescription nd) Starts the Hadoop DataNode using the given node configuration.
static void	startDataNode(String hadoopConfig, String host) Starts the Hadoop DataNode on the given host using the given cluster configuration.
static void	startDataNodes(List<HadoopInfo> hadoops) Starts all data nodes in the given list of configurations.
static void	startDataNodes(String hadoopConfig) Starts Hadoop DataNodes using the given cluster configuration.
static void	startHadoopProcess(HadoopDescription hdd, HadoopDescription.NodeDescription nd) Starts the Hadoop process using the given node configuration.
static void	startHadoopProcesses(List<HadoopInfo> hadoops) Starts all Hadoop process using the given list of configurations.
static void	startHDFSCluster(String hadoopConfig) Starts the HDFS cluster using the given cluster configuration.
static void	startNameNode(HadoopDescription hdd, HadoopDescription.NodeDescription nd) Starts the Hadoop NameNode using the given node configuration.
static void	startNameNode(String hadoopConfig, String host) Starts the Hadoop NameNode on the given host using the given cluster configuration.
static void	startNameNodes(List<HadoopInfo> hadoops) Starts all name nodes in the given list of configurations.
static void	startNameNodes(String hadoopConfig) Starts Hadoop NameNodes using the given cluster configuration.
static void	startNodeManager(HadoopDescription hdd, HadoopDescription.NodeDescription nd) Starts the YARN NodeManager using the given node configuration.
static void	startNodeManager(String hadoopConfig, String host) Starts the YARN NodeManager on the given host using the given cluster configuration.
static void	startNodeManagers(List<HadoopInfo> hadoops) Starts all node managers in the given list of configurations.
static void	startNodeManagers(String hadoopConfig) Starts YARN NodeManagers using the given cluster configuration.
static void	startResourceManager(HadoopDescription hdd, HadoopDescription.NodeDescription nd) Starts the YARN ResourceManager using the given node configuration.
static void	startResourceManager(String hadoopConfig) Starts the YARN ResourceManager using the given cluster configuration.
static void	startResourceManager(String hadoopConfig, String host) Starts the YARN ResourceManager on the given host using the given cluster configuration.
static void	startResourceManagers(List<HadoopInfo> hadoops) Starts all resource managers in the given list of configurations.
static void	stopCluster(String hadoopConfig) Stops the Hadoop cluster using the given cluster configuration, as found in HadoopPrms.names.
static void	stopDataNode(String hadoopConfig, String host) Stops the Hadoop DataNode on the given host using the given cluster configuration, as found in HadoopPrms.names.
static void	stopDataNodes(String hadoopConfig) Stops Hadoop DataNodes using the given cluster configuration, as found in HadoopPrms.names.
static void	stopHDFSCluster(String hadoopConfig) Stops the HDFS cluster using the given cluster configuration, as found in HadoopPrms.names.
static void	stopNameNode(String hadoopConfig, String host) Stops the Hadoop NameNode on the given host using the given cluster configuration, as found in HadoopPrms.names.
static void	stopNameNodes(String hadoopConfig) Stops the Hadoop NameNodes using the given cluster configuration, as found in HadoopPrms.names.
static void	stopNodeManager(String hadoopConfig, String host) Starts the YARN NodeManager on the given host using the given cluster configuration, as found in HadoopPrms.names.
static void	stopNodeManagers(String hadoopConfig) Stops YARN NodeManagers using the given cluster configuration, as found in HadoopPrms.names.
static void	stopResourceManager(String hadoopConfig) Stops the YARN ResourceManager using the given cluster configuration, as found in HadoopPrms.names.
static void	stopResourceManager(String hadoopConfig, String host) Stops the YARN ResourceManager on the given host using the given cluster configuration, as found in HadoopPrms.names.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

GFXD_SECURE_KEYTAB_FILE

public static final String GFXD_SECURE_KEYTAB_FILE

See Also:

Constant Field Values

HDFS_SECURE_KEYTAB_FILE

public static final String HDFS_SECURE_KEYTAB_FILE

See Also:

Constant Field Values

MAPRED_SECURE_KEYTAB_FILE

public static final String MAPRED_SECURE_KEYTAB_FILE

See Also:

Constant Field Values

YARN_SECURE_KEYTAB_FILE

public static final String YARN_SECURE_KEYTAB_FILE

See Also:

Constant Field Values

Constructor Detail

HadoopHelper

public HadoopHelper()

Method Detail

configureHadoop

public static void configureHadoop(String hadoopConfig)

Builds the Hadoop configuration files for the given cluster configuration. The configuration uses the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

The files are placed in the test directory in a subdirectory named for the cluster, node type, and host. For example: mycluster_DataNode_w2-2013-lin-08_conf

The subdirectory includes generated configuration files such as hadoop-env.sh and slaves, and other files that are copied from $JTESTS/hydra/hadoop/conf.

configureHDFS

public static void configureHDFS(String hadoopConfig)

Builds the Hadoop configuration files for the given cluster configuration. The configuration uses the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

The files are placed in the test directory in a subdirectory named for the cluster, node type, and host. For example: mycluster_DataNode_w2-2013-lin-08_conf

The subdirectory includes generated configuration files such as hadoop-env.sh and slaves, and other files that are copied from $JTESTS/hydra/hadoop/conf.

startCluster

public static void startCluster(String hadoopConfig)

Starts the Hadoop cluster using the given cluster configuration.

The cluster is configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

stopCluster

public static void stopCluster(String hadoopConfig)

Stops the Hadoop cluster using the given cluster configuration, as found in HadoopPrms.names.

startHDFSCluster

public static void startHDFSCluster(String hadoopConfig)

Starts the HDFS cluster using the given cluster configuration.

The cluster is configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

stopHDFSCluster

public static void stopHDFSCluster(String hadoopConfig)

Stops the HDFS cluster using the given cluster configuration, as found in HadoopPrms.names.

formatNameNodes

public static void formatNameNodes(String hadoopConfig)

Formats Hadoop NameNodes using the given cluster configuration.

The nodes are configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

formatNameNode

public static void formatNameNode(String hadoopConfig,
                                  String host)

Formats the Hadoop NameNode on the given host using the given node configuration.

The node is configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

startNameNodes

public static void startNameNodes(String hadoopConfig)

Starts Hadoop NameNodes using the given cluster configuration.

The nodes are configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

startNameNode

public static void startNameNode(String hadoopConfig,
                                 String host)

Starts the Hadoop NameNode on the given host using the given cluster configuration.

The node is configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

startNameNode

public static void startNameNode(HadoopDescription hdd,
                                 HadoopDescription.NodeDescription nd)

Starts the Hadoop NameNode using the given node configuration.

stopNameNodes

public static void stopNameNodes(String hadoopConfig)

Stops the Hadoop NameNodes using the given cluster configuration, as found in HadoopPrms.names.

stopNameNode

public static void stopNameNode(String hadoopConfig,
                                String host)

Stops the Hadoop NameNode on the given host using the given cluster configuration, as found in HadoopPrms.names.

startDataNodes

public static void startDataNodes(String hadoopConfig)

Starts Hadoop DataNodes using the given cluster configuration.

The nodes are configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

startDataNode

public static void startDataNode(String hadoopConfig,
                                 String host)

Starts the Hadoop DataNode on the given host using the given cluster configuration.

The node is configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

startDataNode

public static void startDataNode(HadoopDescription hdd,
                                 HadoopDescription.NodeDescription nd)

Starts the Hadoop DataNode using the given node configuration.

stopDataNodes

public static void stopDataNodes(String hadoopConfig)

Stops Hadoop DataNodes using the given cluster configuration, as found in HadoopPrms.names.

stopDataNode

public static void stopDataNode(String hadoopConfig,
                                String host)

Stops the Hadoop DataNode on the given host using the given cluster configuration, as found in HadoopPrms.names.

startResourceManager

public static void startResourceManager(String hadoopConfig)

Starts the YARN ResourceManager using the given cluster configuration.

The manager is configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

startResourceManager

public static void startResourceManager(String hadoopConfig,
                                        String host)

Starts the YARN ResourceManager on the given host using the given cluster configuration.

The manager is configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

startResourceManager

public static void startResourceManager(HadoopDescription hdd,
                                        HadoopDescription.NodeDescription nd)

Starts the YARN ResourceManager using the given node configuration.

stopResourceManager

public static void stopResourceManager(String hadoopConfig)

Stops the YARN ResourceManager using the given cluster configuration, as found in HadoopPrms.names.

stopResourceManager

public static void stopResourceManager(String hadoopConfig,
                                       String host)

Stops the YARN ResourceManager on the given host using the given cluster configuration, as found in HadoopPrms.names.

startNodeManagers

public static void startNodeManagers(String hadoopConfig)

Starts YARN NodeManagers using the given cluster configuration.

The managers are configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

startNodeManager

public static void startNodeManager(String hadoopConfig,
                                    String host)

Starts the YARN NodeManager on the given host using the given cluster configuration.

The node is configured using the HadoopDescription corresponding to the cluster configuration from HadoopPrms.names.

startNodeManager

public static void startNodeManager(HadoopDescription hdd,
                                    HadoopDescription.NodeDescription nd)

Starts the YARN NodeManager using the given node configuration.

stopNodeManagers

public static void stopNodeManagers(String hadoopConfig)

Stops YARN NodeManagers using the given cluster configuration, as found in HadoopPrms.names.

stopNodeManager

public static void stopNodeManager(String hadoopConfig,
                                   String host)

Starts the YARN NodeManager on the given host using the given cluster configuration, as found in HadoopPrms.names.

startHadoopProcesses

public static void startHadoopProcesses(List<HadoopInfo> hadoops)

Starts all Hadoop process using the given list of configurations. This is a convenience method useful with the RebootMgr API. It is up to the test to use this method in a thread-safe and timely fashion.

startNameNodes

public static void startNameNodes(List<HadoopInfo> hadoops)

Starts all name nodes in the given list of configurations. This is a convenience method useful with the RebootMgr API. It is up to the test to use this method in a thread-safe and timely fashion.

startDataNodes

public static void startDataNodes(List<HadoopInfo> hadoops)

Starts all data nodes in the given list of configurations. This is a convenience method useful with the RebootMgr API. It is up to the test to use this method in a thread-safe and timely fashion.

startResourceManagers

public static void startResourceManagers(List<HadoopInfo> hadoops)

Starts all resource managers in the given list of configurations. This is a convenience method useful with the RebootMgr API. It is up to the test to use this method in a thread-safe and timely fashion.

startNodeManagers

public static void startNodeManagers(List<HadoopInfo> hadoops)

Starts all node managers in the given list of configurations. This is a convenience method useful with the RebootMgr API. It is up to the test to use this method in a thread-safe and timely fashion.

startHadoopProcess

public static void startHadoopProcess(HadoopDescription hdd,
                                      HadoopDescription.NodeDescription nd)

Starts the Hadoop process using the given node configuration. This is a convenience method useful with the RebootMgr API. It is up to the test to use this method in a thread-safe and timely fashion.

runHadoopCommand

public static void runHadoopCommand(String hadoopConfig,
                                    String command,
                                    String outfn)

Runs a Hadoop command using the given cluster configuration from HadoopPrms.names. Writes the output to the specified file name.

runHadoopCommand

public static void runHadoopCommand(HadoopDescription hdd,
                                    String command,
                                    String outfn)

Runs a Hadoop command using the given HadoopDescription from HadoopPrms.names. Writes the output to the specified file name.

executeHadoopCommand

public static void executeHadoopCommand(String host,
                                        String hadoopConfig,
                                        String command,
                                        String outfn)

Executes a Hadoop command on given host using the given cluster configuration from HadoopPrms.names. Writes the output to the specified file name.

executeHadoopCommand

public static void executeHadoopCommand(String host,
                                        HadoopDescription hdd,
                                        String command,
                                        String outfn)

Executes a Hadoop command on a given host using the given HadoopDescription from HadoopPrms.names. Writes the output to the specified file name.

runYarnCommand

public static void runYarnCommand(String hadoopConfig,
                                  String command,
                                  String outfn)

Runs a YARN command using the given cluster configuration from HadoopPrms.names. Writes the output to the specified file name.

getHadoopDescription

public static HadoopDescription getHadoopDescription(String hadoopConfig)

Returns the HadoopDescription for the given cluster configuration.

executeKinitCommand

public static void executeKinitCommand(String host,
                                       String command,
                                       int maxWaitSec)

Executes "kinit" on the given command on the specified host.

executeKdestroyCommand

public static void executeKdestroyCommand(String host,
                                          int maxWaitSec)

Executes "kdestroy" on the specified host.