cascading.flow.tez

Class Hadoop2TezFlowProcess

java.lang.Object

cascading.flow.FlowProcess<TezConfiguration>

cascading.flow.tez.Hadoop2TezFlowProcess

All Implemented Interfaces:

cascading.flow.hadoop.MapRed

public class Hadoop2TezFlowProcess
extends cascading.flow.FlowProcess<TezConfiguration>
implements cascading.flow.hadoop.MapRed

Class HadoopFlowProcess is an implementation of FlowProcess for Hadoop. Use this interface to get direct access to the Hadoop JobConf and Reporter interfaces.

Be warned that coupling to this implementation will cause custom Operations to fail if they are executed on a system other than Hadoop.

See Also:

FlowSession

Nested Class Summary

Nested classes/interfaces inherited from class cascading.flow.FlowProcess

cascading.flow.FlowProcess.FlowProcessContext, cascading.flow.FlowProcess.NullFlowProcess

Field Summary

Fields inherited from class cascading.flow.FlowProcess

NULL

Constructor Summary

Constructors

Constructor and Description
Hadoop2TezFlowProcess()
Hadoop2TezFlowProcess(cascading.flow.FlowSession flowSession, ProcessorContext context, TezConfiguration configuration)
Hadoop2TezFlowProcess(Hadoop2TezFlowProcess flowProcess, TezConfiguration configuration)
Hadoop2TezFlowProcess(TezConfiguration configuration)

Method Summary

Modifier and Type	Method and Description
<C> C	copyConfig(C config)
cascading.flow.FlowProcess	copyWith(TezConfiguration configuration)
<C> java.util.Map<java.lang.String,java.lang.String>	diffConfigIntoMap(C defaultConfig, C updatedConfig)
TezConfiguration	getConfig()
TezConfiguration	getConfigCopy()
TezConfiguration	getConfiguration() Method getJobConf returns the jobConf of this HadoopFlowProcess object.
ProcessorContext	getContext()
long	getCounterValue(java.lang.Enum counter)
long	getCounterValue(java.lang.String group, java.lang.String counter)
int	getCurrentSliceNum() Method getCurrentTaskNum returns the task number of this task.
int	getNumProcessSlices()
java.lang.Object	getProperty(java.lang.String key)
java.util.Collection<java.lang.String>	getPropertyKeys()
Reporter	getReporter() Method getReporter returns the reporter of this HadoopFlowProcess object.
void	increment(java.lang.Enum counter, long amount)
void	increment(java.lang.String group, java.lang.String counter, long amount)
boolean	isCounterStatusInitialized()
void	keepAlive()
TezConfiguration	mergeMapIntoConfig(TezConfiguration defaultConfig, java.util.Map<java.lang.String,java.lang.String> map)
java.lang.Object	newInstance(java.lang.String className)
cascading.tuple.TupleEntryCollector	openSystemIntermediateForWrite()
cascading.tuple.TupleEntryIterator	openTapForRead(cascading.tap.Tap tap)
cascading.tuple.TupleEntryCollector	openTapForWrite(cascading.tap.Tap tap)
cascading.tuple.TupleEntryCollector	openTrapForWrite(cascading.tap.Tap trap)
void	setStatus(java.lang.String status)
void	setWriter(Writer writer)

Methods inherited from class cascading.flow.FlowProcess

closeTrapCollectors, getBooleanProperty, getBooleanProperty, getCurrentSession, getFlowProcessContext, getID, getIntegerProperty, getIntegerProperty, getStringProperty, getStringProperty, getTrapCollectorFor, getTrapCollectors, nullFlowProcess, setCurrentSession

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Hadoop2TezFlowProcess

public Hadoop2TezFlowProcess()

Hadoop2TezFlowProcess

public Hadoop2TezFlowProcess(TezConfiguration configuration)

Hadoop2TezFlowProcess

public Hadoop2TezFlowProcess(cascading.flow.FlowSession flowSession,
                             ProcessorContext context,
                             TezConfiguration configuration)

Hadoop2TezFlowProcess

public Hadoop2TezFlowProcess(Hadoop2TezFlowProcess flowProcess,
                             TezConfiguration configuration)

Method Detail

getContext

public ProcessorContext getContext()

setWriter

public void setWriter(Writer writer)

copyWith

public cascading.flow.FlowProcess copyWith(TezConfiguration configuration)

Specified by:

copyWith in class cascading.flow.FlowProcess<TezConfiguration>

getConfiguration

public TezConfiguration getConfiguration()

Method getJobConf returns the jobConf of this HadoopFlowProcess object.

Returns:

the jobConf (type JobConf) of this HadoopFlowProcess object.

getConfig

public TezConfiguration getConfig()

Specified by:

getConfig in class cascading.flow.FlowProcess<TezConfiguration>

getConfigCopy

public TezConfiguration getConfigCopy()

Specified by:

getConfigCopy in class cascading.flow.FlowProcess<TezConfiguration>

getCurrentSliceNum

public int getCurrentSliceNum()

Method getCurrentTaskNum returns the task number of this task. Task 0 is the first task.

Specified by:

getCurrentSliceNum in class cascading.flow.FlowProcess<TezConfiguration>

Returns:

int

getNumProcessSlices

public int getNumProcessSlices()

Specified by:

getNumProcessSlices in class cascading.flow.FlowProcess<TezConfiguration>

getReporter

public Reporter getReporter()

Method getReporter returns the reporter of this HadoopFlowProcess object.

Specified by:

getReporter in interface cascading.flow.hadoop.MapRed

Returns:

the reporter (type Reporter) of this HadoopFlowProcess object.

getProperty

public java.lang.Object getProperty(java.lang.String key)

Specified by:

getProperty in class cascading.flow.FlowProcess<TezConfiguration>

getPropertyKeys

public java.util.Collection<java.lang.String> getPropertyKeys()

Specified by:

getPropertyKeys in class cascading.flow.FlowProcess<TezConfiguration>

newInstance

public java.lang.Object newInstance(java.lang.String className)

Specified by:

newInstance in class cascading.flow.FlowProcess<TezConfiguration>

keepAlive

public void keepAlive()

Specified by:

keepAlive in class cascading.flow.FlowProcess<TezConfiguration>

increment

public void increment(java.lang.Enum counter,
                      long amount)

Specified by:

increment in class cascading.flow.FlowProcess<TezConfiguration>

increment

public void increment(java.lang.String group,
                      java.lang.String counter,
                      long amount)

Specified by:

increment in class cascading.flow.FlowProcess<TezConfiguration>

getCounterValue

public long getCounterValue(java.lang.Enum counter)

Specified by:

getCounterValue in class cascading.flow.FlowProcess<TezConfiguration>

getCounterValue

public long getCounterValue(java.lang.String group,
                            java.lang.String counter)

Specified by:

getCounterValue in class cascading.flow.FlowProcess<TezConfiguration>

setStatus

public void setStatus(java.lang.String status)

Specified by:

setStatus in class cascading.flow.FlowProcess<TezConfiguration>

isCounterStatusInitialized

public boolean isCounterStatusInitialized()

Specified by:

isCounterStatusInitialized in class cascading.flow.FlowProcess<TezConfiguration>

openTapForRead

public cascading.tuple.TupleEntryIterator openTapForRead(cascading.tap.Tap tap)
                                                  throws java.io.IOException

Specified by:

openTapForRead in class cascading.flow.FlowProcess<TezConfiguration>

Throws:

java.io.IOException

openTapForWrite

public cascading.tuple.TupleEntryCollector openTapForWrite(cascading.tap.Tap tap)
                                                    throws java.io.IOException

Specified by:

openTapForWrite in class cascading.flow.FlowProcess<TezConfiguration>

Throws:

java.io.IOException

openTrapForWrite

public cascading.tuple.TupleEntryCollector openTrapForWrite(cascading.tap.Tap trap)
                                                     throws java.io.IOException

Specified by:

openTrapForWrite in class cascading.flow.FlowProcess<TezConfiguration>

Throws:

java.io.IOException

openSystemIntermediateForWrite

public cascading.tuple.TupleEntryCollector openSystemIntermediateForWrite()
                                                                   throws java.io.IOException

Specified by:

openSystemIntermediateForWrite in class cascading.flow.FlowProcess<TezConfiguration>

Throws:

java.io.IOException

copyConfig

public <C> C copyConfig(C config)

Specified by:

copyConfig in class cascading.flow.FlowProcess<TezConfiguration>

diffConfigIntoMap

public <C> java.util.Map<java.lang.String,java.lang.String> diffConfigIntoMap(C defaultConfig,
                                                                              C updatedConfig)

Specified by:

diffConfigIntoMap in class cascading.flow.FlowProcess<TezConfiguration>

mergeMapIntoConfig

public TezConfiguration mergeMapIntoConfig(TezConfiguration defaultConfig,
                                           java.util.Map<java.lang.String,java.lang.String> map)

Specified by:

mergeMapIntoConfig in class cascading.flow.FlowProcess<TezConfiguration>