cascading.flow.hadoop

Class HadoopFlowProcess

java.lang.Object

cascading.flow.FlowProcess<JobConf>

cascading.flow.hadoop.HadoopFlowProcess

All Implemented Interfaces:

cascading.flow.hadoop.MapRed

public class HadoopFlowProcess
extends cascading.flow.FlowProcess<JobConf>
implements cascading.flow.hadoop.MapRed

Class HadoopFlowProcess is an implementation of FlowProcess for Hadoop. Use this interface to get direct access to the Hadoop JobConf and Reporter interfaces.

Be warned that coupling to this implementation will cause custom Operations to fail if they are executed on a system other than Hadoop.

See Also:

FlowSession, JobConf, Reporter

Nested Class Summary

Nested classes/interfaces inherited from class cascading.flow.FlowProcess

cascading.flow.FlowProcess.FlowProcessContext, cascading.flow.FlowProcess.NullFlowProcess

Field Summary

Fields inherited from class cascading.flow.FlowProcess

NULL

Constructor Summary

Constructors

Constructor and Description
HadoopFlowProcess()
HadoopFlowProcess(Configuration jobConf)
HadoopFlowProcess(cascading.flow.FlowSession flowSession, JobConf jobConf)
HadoopFlowProcess(cascading.flow.FlowSession flowSession, JobConf jobConf, boolean isMapper) Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance.
HadoopFlowProcess(HadoopFlowProcess flowProcess, JobConf jobConf)
HadoopFlowProcess(JobConf jobConf)

Method Summary

Modifier and Type	Method and Description
<C> C	copyConfig(C config)
cascading.flow.FlowProcess	copyWith(JobConf jobConf)
<C> java.util.Map<java.lang.String,java.lang.String>	diffConfigIntoMap(C defaultConfig, C updatedConfig)
JobConf	getConfig()
JobConf	getConfigCopy()
long	getCounterValue(java.lang.Enum counter)
long	getCounterValue(java.lang.String group, java.lang.String counter)
int	getCurrentNumMappers()
int	getCurrentNumReducers()
int	getCurrentSliceNum() Method getCurrentTaskNum returns the task number of this task.
JobConf	getJobConf() Method getJobConf returns the jobConf of this HadoopFlowProcess object.
int	getNumProcessSlices()
OutputCollector	getOutputCollector()
java.lang.Object	getProperty(java.lang.String key)
java.util.Collection<java.lang.String>	getPropertyKeys()
Reporter	getReporter()
void	increment(java.lang.Enum counter, long amount)
void	increment(java.lang.String group, java.lang.String counter, long amount)
boolean	isCounterStatusInitialized()
boolean	isMapper() Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper.
void	keepAlive()
JobConf	mergeMapIntoConfig(JobConf defaultConfig, java.util.Map<java.lang.String,java.lang.String> map)
java.lang.Object	newInstance(java.lang.String className)
cascading.tuple.TupleEntryCollector	openSystemIntermediateForWrite()
cascading.tuple.TupleEntryIterator	openTapForRead(cascading.tap.Tap tap)
cascading.tuple.TupleEntryCollector	openTapForWrite(cascading.tap.Tap tap)
cascading.tuple.TupleEntryCollector	openTrapForWrite(cascading.tap.Tap trap)
void	setOutputCollector(OutputCollector outputCollector)
void	setReporter(Reporter reporter) Method setReporter sets the reporter of this HadoopFlowProcess object.
void	setStatus(java.lang.String status)

Methods inherited from class cascading.flow.FlowProcess

closeTrapCollectors, getBooleanProperty, getBooleanProperty, getCurrentSession, getFlowProcessContext, getID, getIntegerProperty, getIntegerProperty, getStringProperty, getStringProperty, getTrapCollectorFor, getTrapCollectors, nullFlowProcess, setCurrentSession

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFlowProcess

public HadoopFlowProcess()

HadoopFlowProcess

public HadoopFlowProcess(Configuration jobConf)

HadoopFlowProcess

public HadoopFlowProcess(JobConf jobConf)

HadoopFlowProcess

public HadoopFlowProcess(cascading.flow.FlowSession flowSession,
                         JobConf jobConf)

HadoopFlowProcess

public HadoopFlowProcess(cascading.flow.FlowSession flowSession,
                         JobConf jobConf,
                         boolean isMapper)

Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance.

Parameters:

flowSession - of type FlowSession

jobConf - of type JobConf

HadoopFlowProcess

public HadoopFlowProcess(HadoopFlowProcess flowProcess,
                         JobConf jobConf)

Method Detail

copyWith

public cascading.flow.FlowProcess copyWith(JobConf jobConf)

Specified by:

copyWith in class cascading.flow.FlowProcess<JobConf>

getJobConf

public JobConf getJobConf()

Method getJobConf returns the jobConf of this HadoopFlowProcess object.

Returns:

the jobConf (type JobConf) of this HadoopFlowProcess object.

getConfig

public JobConf getConfig()

Specified by:

getConfig in class cascading.flow.FlowProcess<JobConf>

getConfigCopy

public JobConf getConfigCopy()

Specified by:

getConfigCopy in class cascading.flow.FlowProcess<JobConf>

isMapper

public boolean isMapper()

Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper. If false, it is a reducer.

Returns:

boolean

getCurrentNumMappers

public int getCurrentNumMappers()

getCurrentNumReducers

public int getCurrentNumReducers()

getCurrentSliceNum

public int getCurrentSliceNum()

Method getCurrentTaskNum returns the task number of this task. Task 0 is the first task.

Specified by:

getCurrentSliceNum in class cascading.flow.FlowProcess<JobConf>

Returns:

int

getNumProcessSlices

public int getNumProcessSlices()

Specified by:

getNumProcessSlices in class cascading.flow.FlowProcess<JobConf>

setReporter

public void setReporter(Reporter reporter)

Method setReporter sets the reporter of this HadoopFlowProcess object.

Parameters:

reporter - the reporter of this HadoopFlowProcess object.

getReporter

public Reporter getReporter()

Specified by:

getReporter in interface cascading.flow.hadoop.MapRed

setOutputCollector

public void setOutputCollector(OutputCollector outputCollector)

getOutputCollector

public OutputCollector getOutputCollector()

getProperty

public java.lang.Object getProperty(java.lang.String key)

Specified by:

getProperty in class cascading.flow.FlowProcess<JobConf>

getPropertyKeys

public java.util.Collection<java.lang.String> getPropertyKeys()

Specified by:

getPropertyKeys in class cascading.flow.FlowProcess<JobConf>

newInstance

public java.lang.Object newInstance(java.lang.String className)

Specified by:

newInstance in class cascading.flow.FlowProcess<JobConf>

keepAlive

public void keepAlive()

Specified by:

keepAlive in class cascading.flow.FlowProcess<JobConf>

increment

public void increment(java.lang.Enum counter,
                      long amount)

Specified by:

increment in class cascading.flow.FlowProcess<JobConf>

increment

public void increment(java.lang.String group,
                      java.lang.String counter,
                      long amount)

Specified by:

increment in class cascading.flow.FlowProcess<JobConf>

getCounterValue

public long getCounterValue(java.lang.Enum counter)

Specified by:

getCounterValue in class cascading.flow.FlowProcess<JobConf>

getCounterValue

public long getCounterValue(java.lang.String group,
                            java.lang.String counter)

Specified by:

getCounterValue in class cascading.flow.FlowProcess<JobConf>

setStatus

public void setStatus(java.lang.String status)

Specified by:

setStatus in class cascading.flow.FlowProcess<JobConf>

isCounterStatusInitialized

public boolean isCounterStatusInitialized()

Specified by:

isCounterStatusInitialized in class cascading.flow.FlowProcess<JobConf>

openTapForRead

public cascading.tuple.TupleEntryIterator openTapForRead(cascading.tap.Tap tap)
                                                  throws java.io.IOException

Specified by:

openTapForRead in class cascading.flow.FlowProcess<JobConf>

Throws:

java.io.IOException

openTapForWrite

public cascading.tuple.TupleEntryCollector openTapForWrite(cascading.tap.Tap tap)
                                                    throws java.io.IOException

Specified by:

openTapForWrite in class cascading.flow.FlowProcess<JobConf>

Throws:

java.io.IOException

openTrapForWrite

public cascading.tuple.TupleEntryCollector openTrapForWrite(cascading.tap.Tap trap)
                                                     throws java.io.IOException

Specified by:

openTrapForWrite in class cascading.flow.FlowProcess<JobConf>

Throws:

java.io.IOException

openSystemIntermediateForWrite

public cascading.tuple.TupleEntryCollector openSystemIntermediateForWrite()
                                                                   throws java.io.IOException

Specified by:

openSystemIntermediateForWrite in class cascading.flow.FlowProcess<JobConf>

Throws:

java.io.IOException

copyConfig

public <C> C copyConfig(C config)

Specified by:

copyConfig in class cascading.flow.FlowProcess<JobConf>

diffConfigIntoMap

public <C> java.util.Map<java.lang.String,java.lang.String> diffConfigIntoMap(C defaultConfig,
                                                                              C updatedConfig)

Specified by:

diffConfigIntoMap in class cascading.flow.FlowProcess<JobConf>

mergeMapIntoConfig

public JobConf mergeMapIntoConfig(JobConf defaultConfig,
                                  java.util.Map<java.lang.String,java.lang.String> map)

Specified by:

mergeMapIntoConfig in class cascading.flow.FlowProcess<JobConf>