cascading.flow.hadoop.planner

Class HadoopPlanner

java.lang.Object

cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>

cascading.flow.hadoop.planner.HadoopPlanner

Direct Known Subclasses:

Hadoop2MR1Planner

public class HadoopPlanner
extends cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>

Class HadoopPlanner is the core Hadoop MapReduce planner used by default through a FlowConnector sub-class.

Notes:

Custom JobConf properties
A custom JobConf instance can be passed to this planner by calling copyJobConf(java.util.Map, org.apache.hadoop.mapred.JobConf) on a map properties object before constructing a new FlowConnector sub-class.

A better practice would be to set Hadoop properties directly on the map properties object handed to the FlowConnector. All values in the map will be passed to a new default JobConf instance to be used as defaults for all resulting Flow instances.

For example, properties.set("mapred.child.java.opts","-Xmx512m"); would convince Hadoop to spawn all child jvms with a heap of 512MB.

Nested Class Summary

Nested classes/interfaces inherited from class cascading.flow.planner.FlowPlanner

cascading.flow.planner.FlowPlanner.IdentityElementFactgory, cascading.flow.planner.FlowPlanner.TempTapElementFactory

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	PLATFORM_NAME

Fields inherited from class cascading.flow.planner.FlowPlanner

checkpointTapRootPath, defaultAssertionLevel, defaultDebugLevel, defaultProperties, TRACE_PLAN_PATH, TRACE_PLAN_TRANSFORM_PATH, TRACE_STATS_PATH

Constructor Summary

Constructors

Constructor and Description
HadoopPlanner()

Method Summary

Modifier and Type	Method and Description
protected void	checkPlatform(Configuration conf)
void	configRuleRegistryDefaults(cascading.flow.planner.rule.RuleRegistry ruleRegistry)
static void	copyJobConf(java.util.Map<java.lang.Object,java.lang.Object> properties, JobConf jobConf) Method copyJobConf adds the given JobConf values to the given properties object.
static void	copyProperties(JobConf jobConf, java.util.Map<java.lang.Object,java.lang.Object> properties) Method copyProperties adds the given Map values to the given JobConf object.
protected HadoopFlow	createFlow(cascading.flow.FlowDef flowDef)
static JobConf	createJobConf(java.util.Map<java.lang.Object,java.lang.Object> properties) Method createJobConf returns a new JobConf instance using the values in the given properties argument.
JobConf	getDefaultConfig()
java.net.URI	getDefaultURIScheme(cascading.tap.Tap tap)
cascading.flow.planner.process.FlowStepFactory<JobConf>	getFlowStepFactory()
cascading.flow.planner.PlannerInfo	getPlannerInfo(java.lang.String registryName)
cascading.flow.planner.PlatformInfo	getPlatformInfo()
java.net.URI	getURIScheme(cascading.tap.Tap tap)
void	initialize(cascading.flow.FlowConnector flowConnector, java.util.Map<java.lang.Object,java.lang.Object> properties)
protected cascading.tap.Tap	makeTempTap(java.lang.String prefix, java.lang.String name)

Methods inherited from class cascading.flow.planner.FlowPlanner

buildFlow, createFlowElementGraph, getAssertionLevel, getDebugLevel, getDefaultProperties, getFlowNodeFactory, handleExceptionDuringPlanning, makeTempTap, resolveAssemblyPlanners, resolveTails, verifyAllTaps, verifyAssembly, verifyCheckpoints, verifyPipeAssemblyEndPoints, verifyResult, verifyResultInternal, verifySourceNotSinks, verifyTaps, verifyTraps

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PLATFORM_NAME

public static final java.lang.String PLATFORM_NAME

See Also:

Constant Field Values

Constructor Detail

HadoopPlanner

public HadoopPlanner()

Method Detail

copyJobConf

public static void copyJobConf(java.util.Map<java.lang.Object,java.lang.Object> properties,
                               JobConf jobConf)

Method copyJobConf adds the given JobConf values to the given properties object. Use this method to pass custom default Hadoop JobConf properties to Hadoop.

Parameters:

properties - of type Map

jobConf - of type JobConf

createJobConf

public static JobConf createJobConf(java.util.Map<java.lang.Object,java.lang.Object> properties)

Method createJobConf returns a new JobConf instance using the values in the given properties argument.

Parameters:

properties - of type Map

Returns:

a JobConf instance

copyProperties

public static void copyProperties(JobConf jobConf,
                                  java.util.Map<java.lang.Object,java.lang.Object> properties)

Method copyProperties adds the given Map values to the given JobConf object.

Parameters:

jobConf - of type JobConf

properties - of type Map

getPlannerInfo

public cascading.flow.planner.PlannerInfo getPlannerInfo(java.lang.String registryName)

Specified by:

getPlannerInfo in class cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>

getDefaultConfig

public JobConf getDefaultConfig()

Specified by:

getDefaultConfig in class cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>

getPlatformInfo

public cascading.flow.planner.PlatformInfo getPlatformInfo()

Specified by:

getPlatformInfo in class cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>

initialize

public void initialize(cascading.flow.FlowConnector flowConnector,
                       java.util.Map<java.lang.Object,java.lang.Object> properties)

Overrides:

initialize in class cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>

configRuleRegistryDefaults

public void configRuleRegistryDefaults(cascading.flow.planner.rule.RuleRegistry ruleRegistry)

Overrides:

configRuleRegistryDefaults in class cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>

checkPlatform

protected void checkPlatform(Configuration conf)

createFlow

protected HadoopFlow createFlow(cascading.flow.FlowDef flowDef)

Specified by:

createFlow in class cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>

getFlowStepFactory

public cascading.flow.planner.process.FlowStepFactory<JobConf> getFlowStepFactory()

Specified by:

getFlowStepFactory in class cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>

getDefaultURIScheme

public java.net.URI getDefaultURIScheme(cascading.tap.Tap tap)

getURIScheme

public java.net.URI getURIScheme(cascading.tap.Tap tap)

makeTempTap

protected cascading.tap.Tap makeTempTap(java.lang.String prefix,
                                        java.lang.String name)

Specified by:

makeTempTap in class cascading.flow.planner.FlowPlanner<HadoopFlow,JobConf>