public class TaskDescriptor
extends java.lang.Object
| Constructor and Description |
|---|
TaskDescriptor(int id,
java.lang.String name,
java.lang.String operationName,
java.util.Properties executionConfig,
TaskDescriptor previousTaskDescriptor)
Will create description of a
Task from which Tez Vertex is created. |
| Modifier and Type | Method and Description |
|---|---|
void |
addDependentTasksChain(java.util.List<TaskDescriptor> dependentTasksChain) |
void |
andThen(io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> aFunction) |
void |
compose(io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> cFunction) |
io.dstream.support.Classifier |
getClassifier() |
java.util.List<java.util.List<TaskDescriptor>> |
getDependentTasksChains() |
io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> |
getFunction() |
int |
getId() |
java.lang.Class<?> |
getInputFormatClass() |
java.lang.String |
getName() |
java.lang.String |
getOperationName() |
int |
getParallelism() |
TaskDescriptor |
getPreviousTaskDescriptor() |
java.lang.Class<?> |
getSourceElementType() |
io.dstream.SerializableStreamAssets.SerSupplier<?> |
getSourceSupplier() |
void |
setInputFormatClass(java.lang.Class<?> inputFormatClass) |
public TaskDescriptor(int id,
java.lang.String name,
java.lang.String operationName,
java.util.Properties executionConfig,
TaskDescriptor previousTaskDescriptor)
Task from which Tez Vertex is created.
Parallelism and Classifier of the task (Vertex) is determined
from DStreamConstants.PARALLELISM configuration
which allows to configure both parallelism and Classifier. However, due to Tez way of
doing things, the actual function itself should be applied to the previous task (Vertex)
where the actual partitioning logic is invoked, while integer value representing parallelism should *also*
be set on the current Vertex.
To accommodate that the TaskDescriptor is created with reference to the previous
TaskDescriptor. Upon determining partitioner configuration and parallelism for the current task,
the actual Classifier is created and set on the previous TaskDescriptor while
it's parallelism is set on this task.id - name - operationName - executionConfig - previousTaskDescriptor - public TaskDescriptor getPreviousTaskDescriptor()
public int getParallelism()
public java.util.List<java.util.List<TaskDescriptor>> getDependentTasksChains()
public java.lang.String getOperationName()
public void addDependentTasksChain(java.util.List<TaskDescriptor> dependentTasksChain)
dependentTasksChain - public java.lang.Class<?> getInputFormatClass()
public void setInputFormatClass(java.lang.Class<?> inputFormatClass)
inputFormatClass - public io.dstream.SerializableStreamAssets.SerSupplier<?> getSourceSupplier()
public int getId()
public io.dstream.support.Classifier getClassifier()
public io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> getFunction()
public void compose(io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> cFunction)
cFunction - public void andThen(io.dstream.SerializableStreamAssets.SerFunction<java.util.stream.Stream<?>,java.util.stream.Stream<?>> aFunction)
aFunction - public java.lang.String getName()
public java.lang.Class<?> getSourceElementType()