cascading.flow.hadoop

Class HadoopFlow

java.lang.Object

cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>

cascading.flow.hadoop.HadoopFlow

All Implemented Interfaces:

cascading.flow.Flow<org.apache.hadoop.mapred.JobConf>, cascading.management.UnitOfWork<cascading.stats.FlowStats>, cascading.util.ProcessLogger

Direct Known Subclasses:

BaseMapReduceFlow

public class HadoopFlow
extends cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>

Class HadoopFlow is the Apache Hadoop specific implementation of a Flow.

HadoopFlow must be created through a FlowConnector sub-class instance.

If classpath paths are provided on the FlowDef, the Hadoop distributed cache mechanism will be used to augment the remote classpath.

Any path elements that are relative will be uploaded to HDFS, and the HDFS URI will be used on the JobConf. Note all paths are added as "files" to the JobConf, not archives, so they aren't needlessly uncompressed cluster side.

See Also:

FlowConnector

Nested Class Summary

Nested classes/interfaces inherited from class cascading.flow.BaseFlow

cascading.flow.BaseFlow.FlowHolder

Field Summary

Fields inherited from class cascading.flow.BaseFlow

completed, flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread, throwable

Fields inherited from interface cascading.flow.Flow

CASCADING_FLOW_ID

Fields inherited from interface cascading.util.ProcessLogger

NULL

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HadoopFlow()
	HadoopFlow(cascading.flow.planner.PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, org.apache.hadoop.mapred.JobConf jobConf, cascading.flow.FlowDef flowDef)
protected	HadoopFlow(cascading.flow.planner.PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, org.apache.hadoop.mapred.JobConf jobConf, java.lang.String name, java.util.Map<java.lang.String,java.lang.String> flowDescriptor)

Method Summary

Modifier and Type	Method and Description
protected void	copyToDistributedCache()
org.apache.hadoop.mapred.JobConf	getConfig()
java.util.Map<java.lang.Object,java.lang.Object>	getConfigAsProperties()
org.apache.hadoop.mapred.JobConf	getConfigCopy()
cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf>	getFlowProcess()
protected int	getMaxNumParallelSteps()
java.lang.String	getProperty(java.lang.String key) Method getProperty returns the value associated with the given key from the underlying properties system.
protected long	getTotalSliceCPUMilliSeconds()
protected void	initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties, org.apache.hadoop.mapred.JobConf parentConfig)
protected void	initFromProperties(java.util.Map<java.lang.Object,java.lang.Object> properties)
protected void	internalClean(boolean stop)
protected void	internalShutdown()
protected void	internalStart()
boolean	isPreserveTemporaryFiles() Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.
protected org.apache.hadoop.mapred.JobConf	newConfig(org.apache.hadoop.mapred.JobConf defaultConfig)
protected void	registerHadoopShutdownHook()
protected void	setConfigProperty(org.apache.hadoop.mapred.JobConf config, java.lang.Object key, java.lang.Object value)
boolean	stepsAreLocal()

Methods inherited from class cascading.flow.BaseFlow

addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getEligibleJobsSize, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowElementGraph, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowStepGraph, getFlowSteps, getFlowStepStrategy, getHolder, getID, getJobMapCallables, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, ifStoppingBlockUntilComplete, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isJobsMapInitialized, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeListeners, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowElementGraph, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, spawnSteps, start, stop, toString, updateJobsMap, updateSchemes, writeDOT, writeStepsDOT

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface cascading.flow.Flow

getSinkEntryStream, getSinkEntryStream, getSinkEntryStream, getSinkEntryStream, getSinkEntryStreamCopy, getSinkEntryStreamCopy, getSinkEntryStreamCopy, getSinkEntryStreamCopy, getSinkTupleStream, getSinkTupleStream, getSinkTupleStream, getSinkTupleStream, getSinkTupleStreamCopy, getSinkTupleStreamCopy, getSinkTupleStreamCopy, getSinkTupleStreamCopy, getSourceEntryStream, getSourceEntryStream, getSourceEntryStreamCopy, getSourceEntryStreamCopy, getSourceTupleStream, getSourceTupleStream, getSourceTupleStreamCopy, getSourceTupleStreamCopy

Constructor Detail

HadoopFlow

protected HadoopFlow()

HadoopFlow

protected HadoopFlow(cascading.flow.planner.PlatformInfo platformInfo,
                     java.util.Map<java.lang.Object,java.lang.Object> properties,
                     org.apache.hadoop.mapred.JobConf jobConf,
                     java.lang.String name,
                     java.util.Map<java.lang.String,java.lang.String> flowDescriptor)

HadoopFlow

public HadoopFlow(cascading.flow.planner.PlatformInfo platformInfo,
                  java.util.Map<java.lang.Object,java.lang.Object> properties,
                  org.apache.hadoop.mapred.JobConf jobConf,
                  cascading.flow.FlowDef flowDef)

Method Detail

initFromProperties

protected void initFromProperties(java.util.Map<java.lang.Object,java.lang.Object> properties)

Overrides:

initFromProperties in class cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>

initConfig

protected void initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties,
                          org.apache.hadoop.mapred.JobConf parentConfig)

Specified by:

initConfig in class cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>

setConfigProperty

protected void setConfigProperty(org.apache.hadoop.mapred.JobConf config,
                                 java.lang.Object key,
                                 java.lang.Object value)

Specified by:

setConfigProperty in class cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>

newConfig

protected org.apache.hadoop.mapred.JobConf newConfig(org.apache.hadoop.mapred.JobConf defaultConfig)

Specified by:

newConfig in class cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>

getConfig

@ProcessConfiguration
public org.apache.hadoop.mapred.JobConf getConfig()

getConfigCopy

public org.apache.hadoop.mapred.JobConf getConfigCopy()

getConfigAsProperties

public java.util.Map<java.lang.Object,java.lang.Object> getConfigAsProperties()

getProperty

public java.lang.String getProperty(java.lang.String key)

Method getProperty returns the value associated with the given key from the underlying properties system.

Parameters:

key - of type String

Returns:

String

getFlowProcess

public cascading.flow.FlowProcess<org.apache.hadoop.mapred.JobConf> getFlowProcess()

isPreserveTemporaryFiles

public boolean isPreserveTemporaryFiles()

Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.

Returns:

the preserveTemporaryFiles (type boolean) of this Flow object.

internalStart

protected void internalStart()

Specified by:

internalStart in class cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>

registerHadoopShutdownHook

protected void registerHadoopShutdownHook()

copyToDistributedCache

protected void copyToDistributedCache()

stepsAreLocal

public boolean stepsAreLocal()

internalClean

protected void internalClean(boolean stop)

Specified by:

internalClean in class cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>

internalShutdown

protected void internalShutdown()

Specified by:

internalShutdown in class cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>

getMaxNumParallelSteps

protected int getMaxNumParallelSteps()

Specified by:

getMaxNumParallelSteps in class cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>

getTotalSliceCPUMilliSeconds

protected long getTotalSliceCPUMilliSeconds()

Overrides:

getTotalSliceCPUMilliSeconds in class cascading.flow.BaseFlow<org.apache.hadoop.mapred.JobConf>