org.apache.hadoop.hive.ql.exec

Class AbstractFileMergeOperator<T extends FileMergeDesc>

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.AbstractFileMergeOperator<T>

All Implemented Interfaces:

Serializable, Cloneable, Node

Direct Known Subclasses:

OrcFileMergeOperator, RCFileMergeOperator

public abstract class AbstractFileMergeOperator<T extends FileMergeDesc>
extends Operator<T>
implements Serializable

Fast file merge operator for ORC and RCfile. This is an abstract class which does not process any rows. Refer OrcFileMergeOperator or RCFileMergeOperator for more details.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	autoDelete
static String	BACKUP_PREFIX
protected DynamicPartitionCtx	dpCtx
protected org.apache.hadoop.fs.Path	dpPath
protected boolean	exception
protected org.apache.hadoop.fs.Path	finalPath
protected org.apache.hadoop.fs.FileSystem	fs
protected boolean	hasDynamicPartitions
protected Set<org.apache.hadoop.fs.Path>	incompatFileSet
protected boolean	isListBucketingAlterTableConcatenate
protected org.apache.hadoop.mapred.JobConf	jc
protected int	listBucketingDepth
static org.apache.commons.logging.Log	LOG
protected org.apache.hadoop.fs.Path	outPath
protected org.apache.hadoop.fs.Path	taskTmpPath
protected org.apache.hadoop.fs.Path	tmpPath
protected boolean	tmpPathFixed
protected boolean	tmpPathFixedConcatenate

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

alias, childOperators, childOperatorsArray, childOperatorsTag, colExprMap, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVECOUNTERCREATEDFILES, HIVECOUNTERFATAL, id, inputObjInspectors, isLogDebugEnabled, isLogInfoEnabled, isLogTraceEnabled, operatorId, out, outputObjInspector, parentOperators, PLOG, reporter, state, statsMap

Constructor Summary

Constructors

Constructor and Description
AbstractFileMergeOperator()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	checkPartitionsMatch(org.apache.hadoop.fs.Path inputPath) Validates that each input path belongs to the same partition since each mapper merges the input to a single output directory
void	closeOp(boolean abort) Operator specific close routine.
protected void	fixTmpPath(org.apache.hadoop.fs.Path path)
protected void	fixTmpPath(org.apache.hadoop.fs.Path inputPath, int depthDiff) Fixes tmpPath to point to the correct partition.
Collection<Future<?>>	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
void	jobCloseOp(org.apache.hadoop.conf.Configuration hconf, boolean success)

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, cleanUpInputFileChangedOp, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, completeInitializationOp, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, endGroup, flush, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getConf, getConfiguration, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getName, getNextCntr, getNumChild, getNumParent, getOperatorId, getOperatorName, getOpTraits, getOutputObjInspector, getParentOperators, getSchema, getStatistics, getStats, getType, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, logStats, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, process, processGroup, removeChild, removeChildAndAdoptItsChildren, removeChildren, removeParent, replaceChild, replaceParent, reset, resetId, resetStats, setAlias, setChildOperators, setColumnExprMap, setConf, setDone, setExecContext, setGroupKeyObject, setId, setInputContext, setInputObjInspectors, setOperatorId, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, startGroup, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

BACKUP_PREFIX

public static final String BACKUP_PREFIX

See Also:

Constant Field Values

LOG

public static final org.apache.commons.logging.Log LOG

jc

protected org.apache.hadoop.mapred.JobConf jc

fs

protected org.apache.hadoop.fs.FileSystem fs

autoDelete

protected boolean autoDelete

exception

protected boolean exception

outPath

protected org.apache.hadoop.fs.Path outPath

finalPath

protected org.apache.hadoop.fs.Path finalPath

dpPath

protected org.apache.hadoop.fs.Path dpPath

tmpPath

protected org.apache.hadoop.fs.Path tmpPath

taskTmpPath

protected org.apache.hadoop.fs.Path taskTmpPath

listBucketingDepth

protected int listBucketingDepth

hasDynamicPartitions

protected boolean hasDynamicPartitions

isListBucketingAlterTableConcatenate

protected boolean isListBucketingAlterTableConcatenate

tmpPathFixedConcatenate

protected boolean tmpPathFixedConcatenate

tmpPathFixed

protected boolean tmpPathFixed

incompatFileSet

protected Set<org.apache.hadoop.fs.Path> incompatFileSet

dpCtx

protected transient DynamicPartitionCtx dpCtx

Constructor Detail

AbstractFileMergeOperator

public AbstractFileMergeOperator()

Method Detail

initializeOp

public Collection<Future<?>> initializeOp(org.apache.hadoop.conf.Configuration hconf)
                                   throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class Operator<T extends FileMergeDesc>

Throws:

HiveException

fixTmpPath

protected void fixTmpPath(org.apache.hadoop.fs.Path inputPath,
              int depthDiff)
                   throws IOException

Fixes tmpPath to point to the correct partition. Initialize operator will set tmpPath and taskTmpPath based on root table directory. So initially, tmpPath will be /_tmp.-ext-10000 and taskTmpPath will be /_task_tmp.-ext-10000. The depth of these two paths will be 0. Now, in case of dynamic partitioning or list bucketing the inputPath will have additional sub-directories under root table directory. This function updates the tmpPath and taskTmpPath to reflect these additional subdirectories. It updates tmpPath and taskTmpPath in the following way 1. finds out the difference in path based on depthDiff provided and saves the path difference in newPath 2. newPath is used to update the existing tmpPath and taskTmpPath similar to the way initializeOp() does. Note: The path difference between inputPath and tmpDepth can be DP or DP+LB. This method will automatically handle it. Continuing the example above, if inputPath is /-ext-10000/hr=a1/, newPath will be hr=a1/. Then, tmpPath and taskTmpPath will be updated to /-ext-10000/hr=a1/_tmp.ext-10000 and /-ext-10000/hr=a1/_task_tmp.ext-10000 respectively. We have list_bucket_dml_6.q cover this case: DP + LP + multiple skewed values + merge.

Parameters:

inputPath - - input path

Throws:

IOException

checkPartitionsMatch

protected void checkPartitionsMatch(org.apache.hadoop.fs.Path inputPath)
                             throws IOException

Validates that each input path belongs to the same partition since each mapper merges the input to a single output directory

Parameters:

inputPath - - input path

Throws:

IOException

fixTmpPath

protected void fixTmpPath(org.apache.hadoop.fs.Path path)
                   throws IOException

Throws:

IOException

closeOp

public void closeOp(boolean abort)
             throws HiveException

Description copied from class: Operator

Operator specific close routine. Operators which inherents this class should overwrite this funtion for their specific cleanup routine.

Overrides:

closeOp in class Operator<T extends FileMergeDesc>

Throws:

HiveException

jobCloseOp

public void jobCloseOp(org.apache.hadoop.conf.Configuration hconf,
              boolean success)
                throws HiveException

Overrides:

jobCloseOp in class Operator<T extends FileMergeDesc>

Throws:

HiveException