io.dstream.tez

Class TaskDescriptor

java.lang.Object

io.dstream.tez.TaskDescriptor

public class TaskDescriptor
extends java.lang.Object

Constructor Summary

Constructors

Constructor and Description
TaskDescriptor(int id, java.lang.String name, java.lang.String operationName, java.util.Properties executionConfig, TaskDescriptor previousTaskDescriptor) Will create description of a Task from which Tez Vertex is created.

Method Summary

Modifier and Type	Method and Description
void	addDependentTasksChain(java.util.List<TaskDescriptor> dependentTasksChain)
void	andThen(io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> aFunction)
void	compose(io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> cFunction)
io.dstream.support.Classifier	getClassifier()
java.util.List<java.util.List<TaskDescriptor>>	getDependentTasksChains()
io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>>	getFunction()
int	getId()
java.lang.Class<?>	getInputFormatClass()
java.lang.String	getName()
java.lang.String	getOperationName()
int	getParallelism()
TaskDescriptor	getPreviousTaskDescriptor()
java.lang.Class<?>	getSourceElementType()
io.dstream.SerializableStreamAssets.SerSupplier<?>	getSourceSupplier()
void	setInputFormatClass(java.lang.Class<?> inputFormatClass)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TaskDescriptor

public TaskDescriptor(int id,
                      java.lang.String name,
                      java.lang.String operationName,
                      java.util.Properties executionConfig,
                      TaskDescriptor previousTaskDescriptor)

Will create description of a Task from which Tez Vertex is created. Parallelism and Classifier of the task (Vertex) is determined from DStreamConstants.PARALLELISM configuration which allows to configure both parallelism and Classifier. However, due to Tez way of doing things, the actual function itself should be applied to the previous task (Vertex) where the actual partitioning logic is invoked, while integer value representing parallelism should *also* be set on the current Vertex. To accommodate that the TaskDescriptor is created with reference to the previous TaskDescriptor. Upon determining partitioner configuration and parallelism for the current task, the actual Classifier is created and set on the previous TaskDescriptor while it's parallelism is set on this task.

Parameters:

id -

name -

operationName -

executionConfig -

previousTaskDescriptor -

Method Detail

getPreviousTaskDescriptor

public TaskDescriptor getPreviousTaskDescriptor()

Returns:

getParallelism

public int getParallelism()

Returns:

getDependentTasksChains

public java.util.List<java.util.List<TaskDescriptor>> getDependentTasksChains()

Returns:

getOperationName

public java.lang.String getOperationName()

Returns:

addDependentTasksChain

public void addDependentTasksChain(java.util.List<TaskDescriptor> dependentTasksChain)

Parameters:

dependentTasksChain -

getInputFormatClass

public java.lang.Class<?> getInputFormatClass()

Returns:

setInputFormatClass

public void setInputFormatClass(java.lang.Class<?> inputFormatClass)

Parameters:

inputFormatClass -

getSourceSupplier

public io.dstream.SerializableStreamAssets.SerSupplier<?> getSourceSupplier()

Returns:

getId

public int getId()

Returns:

getClassifier

public io.dstream.support.Classifier getClassifier()

Returns:

getFunction

public io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> getFunction()

Returns:

compose

public void compose(io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> cFunction)

Parameters:

cFunction -

andThen

public void andThen(io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> aFunction)

Parameters:

aFunction -

getName

public java.lang.String getName()

Returns:

getSourceElementType

public java.lang.Class<?> getSourceElementType()

Returns: