org.apache.hadoop.hive.ql.parse.spark

Class SparkCompiler

java.lang.Object

org.apache.hadoop.hive.ql.parse.TaskCompiler

org.apache.hadoop.hive.ql.parse.spark.SparkCompiler

public class SparkCompiler
extends TaskCompiler

SparkCompiler translates the operator plan into SparkTasks. Cloned from TezCompiler.

Field Summary

Fields inherited from class org.apache.hadoop.hive.ql.parse.TaskCompiler

conf, console, db, LOG

Constructor Summary

Constructors

Constructor and Description
SparkCompiler()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	decideExecMode(List<Task<? extends Serializable>> rootTasks, Context ctx, GlobalLimitCtx globalLimitCtx)
protected void	generateTaskTree(List<Task<? extends Serializable>> rootTasks, ParseContext pCtx, List<Task<MoveWork>> mvTask, Set<ReadEntity> inputs, Set<WriteEntity> outputs) TODO: need to turn on rules that's commented out and add more if necessary.
protected void	optimizeOperatorPlan(ParseContext pCtx, Set<ReadEntity> inputs, Set<WriteEntity> outputs)
protected void	optimizeTaskPlan(List<Task<? extends Serializable>> rootTasks, ParseContext pCtx, Context ctx)
protected void	setInputFormat(Task<? extends Serializable> task)

Methods inherited from class org.apache.hadoop.hive.ql.parse.TaskCompiler

compile, genColumnStatsTask, getLeafTasks, getParseContext, init

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkCompiler

public SparkCompiler()

Method Detail

optimizeOperatorPlan

protected void optimizeOperatorPlan(ParseContext pCtx,
                        Set<ReadEntity> inputs,
                        Set<WriteEntity> outputs)
                             throws SemanticException

Overrides:

optimizeOperatorPlan in class TaskCompiler

Throws:

SemanticException

generateTaskTree

protected void generateTaskTree(List<Task<? extends Serializable>> rootTasks,
                    ParseContext pCtx,
                    List<Task<MoveWork>> mvTask,
                    Set<ReadEntity> inputs,
                    Set<WriteEntity> outputs)
                         throws SemanticException

TODO: need to turn on rules that's commented out and add more if necessary.

Specified by:

generateTaskTree in class TaskCompiler

Throws:

SemanticException

setInputFormat

protected void setInputFormat(Task<? extends Serializable> task)

Specified by:

setInputFormat in class TaskCompiler

decideExecMode

protected void decideExecMode(List<Task<? extends Serializable>> rootTasks,
                  Context ctx,
                  GlobalLimitCtx globalLimitCtx)
                       throws SemanticException

Specified by:

decideExecMode in class TaskCompiler

Throws:

SemanticException

optimizeTaskPlan

protected void optimizeTaskPlan(List<Task<? extends Serializable>> rootTasks,
                    ParseContext pCtx,
                    Context ctx)
                         throws SemanticException

Specified by:

optimizeTaskPlan in class TaskCompiler

Throws:

SemanticException