cascading.flow.hadoop3

Class Hadoop3MRPlanner

java.lang.Object

cascading.flow.planner.FlowPlanner<HadoopFlow,org.apache.hadoop.mapred.JobConf>

cascading.flow.hadoop.planner.HadoopPlanner

cascading.flow.hadoop3.Hadoop3MRPlanner

public class Hadoop3MRPlanner
extends HadoopPlanner

Class Hadoop3MRPlanner is the core Hadoop MapReduce planner used by default through the Hadoop3MRFlowConnector.

Notes:

Custom JobConf properties
A custom JobConf instance can be passed to this planner by calling HadoopPlanner.copyJobConf(java.util.Map, org.apache.hadoop.mapred.JobConf) on a map properties object before constructing a new Hadoop3MRFlowConnector.

A better practice would be to set Hadoop properties directly on the map properties object handed to the FlowConnector. All values in the map will be passed to a new default JobConf instance to be used as defaults for all resulting Flow instances.

For example, properties.set("mapred.child.java.opts","-Xmx512m"); would convince Hadoop to spawn all child jvms with a heap of 512MB.

Nested Class Summary

Nested classes/interfaces inherited from class cascading.flow.planner.FlowPlanner

cascading.flow.planner.FlowPlanner.IdentityElementFactgory, cascading.flow.planner.FlowPlanner.TempTapElementFactory

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	PLATFORM_NAME

Fields inherited from class cascading.flow.planner.FlowPlanner

checkpointTapRootPath, defaultAssertionLevel, defaultDebugLevel, defaultProperties, TRACE_PLAN_PATH, TRACE_PLAN_TRANSFORM_PATH, TRACE_STATS_PATH

Constructor Summary

Constructors

Constructor and Description
Hadoop3MRPlanner()

Method Summary

Modifier and Type	Method and Description
protected void	checkPlatform(org.apache.hadoop.conf.Configuration conf)
static void	copyConfiguration(java.util.Map<java.lang.Object,java.lang.Object> properties, org.apache.hadoop.conf.Configuration configuration) Method copyJobConf adds the given JobConf values to the given properties object.
static void	copyProperties(org.apache.hadoop.conf.Configuration configuration, java.util.Map<java.lang.Object,java.lang.Object> properties) Method copyProperties adds the given Map values to the given JobConf object.
cascading.flow.planner.PlannerInfo	getPlannerInfo(java.lang.String registryName)

Methods inherited from class cascading.flow.hadoop.planner.HadoopPlanner

configRuleRegistryDefaults, copyJobConf, copyProperties, createFlow, createJobConf, getDefaultConfig, getDefaultURIScheme, getFlowStepFactory, getPlatformInfo, getURIScheme, initialize, makeTempTap

Methods inherited from class cascading.flow.planner.FlowPlanner

buildFlow, createFlowElementGraph, getAssertionLevel, getDebugLevel, getDefaultProperties, getFlowNodeFactory, handleExceptionDuringPlanning, makeTempTap, resolveAssemblyPlanners, resolveTails, verifyAllTaps, verifyAssembly, verifyCheckpoints, verifyPipeAssemblyEndPoints, verifyResult, verifyResultInternal, verifySourceNotSinks, verifyTaps, verifyTraps

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PLATFORM_NAME

public static final java.lang.String PLATFORM_NAME

See Also:

Constant Field Values

Constructor Detail

Hadoop3MRPlanner

public Hadoop3MRPlanner()

Method Detail

copyConfiguration

public static void copyConfiguration(java.util.Map<java.lang.Object,java.lang.Object> properties,
                                     org.apache.hadoop.conf.Configuration configuration)

Method copyJobConf adds the given JobConf values to the given properties object. Use this method to pass custom default Hadoop JobConf properties to Hadoop.

Parameters:

properties - of type Map

configuration - of type JobConf

copyProperties

public static void copyProperties(org.apache.hadoop.conf.Configuration configuration,
                                  java.util.Map<java.lang.Object,java.lang.Object> properties)

Method copyProperties adds the given Map values to the given JobConf object.

Parameters:

configuration - of type JobConf

properties - of type Map

getPlannerInfo

public cascading.flow.planner.PlannerInfo getPlannerInfo(java.lang.String registryName)

Overrides:

getPlannerInfo in class HadoopPlanner

checkPlatform

protected void checkPlatform(org.apache.hadoop.conf.Configuration conf)

Overrides:

checkPlatform in class HadoopPlanner