org.apache.hadoop.hive.ql.exec.vector.mapjoin

Class VectorMapJoinCommonOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator<MapJoinDesc>

org.apache.hadoop.hive.ql.exec.MapJoinOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

All Implemented Interfaces:

Serializable, Cloneable, VectorizationContextRegion, Node

Direct Known Subclasses:

VectorMapJoinGenerateResultOperator

public abstract class VectorMapJoinCommonOperator
extends MapJoinOperator
implements VectorizationContextRegion

This class is common operator class for native vectorized map join. It contain common initialization logic. It is used by both inner and outer joins.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	bigTableByteColumnVectorColumns
protected VectorExpression[]	bigTableFilterExpressions
protected int[]	bigTableKeyColumnMap
protected VectorExpression[]	bigTableKeyExpressions
protected ArrayList<String>	bigTableKeyTypeNames
protected VectorColumnOutputMapping	bigTableOuterKeyMapping
protected int[]	bigTableOuterKeyOutputVectorColumns
protected VectorColumnOutputMapping	bigTableRetainedMapping
protected VectorCopyRow	bigTableRetainedVectorCopy
protected int[]	bigTableValueColumnMap
protected VectorExpression[]	bigTableValueExpressions
protected ArrayList<String>	bigTableValueTypeNames
protected VectorCopyRow	bigTableVectorCopyOuterKeys
protected boolean	isOuterJoin
protected boolean	needCommonSetup
protected boolean	needHashTableSetup
protected int[]	outputProjection
protected String[]	outputTypeNames
protected VectorizedRowBatch	overflowBatch
protected int	overflowRepeatedThreshold
protected byte	posSingleVectorMapJoinSmallTable
protected int[]	smallTableByteColumnVectorColumns
protected VectorColumnSourceMapping	smallTableMapping
protected int[]	smallTableOutputVectorColumns
protected VectorDeserializeRow	smallTableVectorDeserializeRow
protected VectorizedRowBatch	spillReplayBatch
protected String	taskName
protected boolean	useOverflowRepeatedThreshold
protected VectorizationContext	vContext
protected VectorMapJoinHashTable	vectorMapJoinHashTable
protected VectorizationContext	vOutContext

Fields inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

firstSmallTable, hashMapRowGetters, loader, mapJoinTables, spilledMapJoinTables

Fields inherited from class org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator

emptyList, joinKeys, joinKeysObjectInspectors, posBigTable

Fields inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

alias, aliasFilterTags, condn, countAfterReport, dummyObj, dummyObjVectors, filterMaps, filterTags, forwardCache, heartbeatInterval, intermediate, joinFilterObjectInspectors, joinFilters, joinValues, joinValuesObjectInspectors, joinValuesStandardObjectInspectors, noOuterJoin, NOTSKIPBIGTABLE, nullsafes, numAliases, offsets, order, rowContainerStandardObjectInspectors, skipVectors, spillTableDesc, totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

childOperators, childOperatorsArray, childOperatorsTag, colExprMap, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVECOUNTERCREATEDFILES, HIVECOUNTERFATAL, id, inputObjInspectors, isLogDebugEnabled, isLogInfoEnabled, isLogTraceEnabled, operatorId, out, outputObjInspector, parentOperators, PLOG, reporter, state, statsMap

Constructor Summary

Constructors

Constructor and Description
VectorMapJoinCommonOperator()
VectorMapJoinCommonOperator(VectorizationContext vContext, OperatorDesc conf)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	commonSetup(VectorizedRowBatch batch)
protected void	determineCommonInfo(boolean isOuter)
protected void	displayBatchColumns(VectorizedRowBatch batch, String batchName)
protected HashTableLoader	getHashTableLoader(org.apache.hadoop.conf.Configuration hconf) This override lets us substitute our own fast vectorized hash table loader.
VectorizationContext	getOuputVectorizationContext()
OperatorType	getType() Return the type of the specific operator among the types in OperatorType.
protected Collection<Future<?>>	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
protected org.apache.commons.lang3.tuple.Pair<MapJoinTableContainer[],MapJoinTableContainerSerDe[]>	loadHashTable(ExecMapperContext mapContext, MapredContext mrContext)
protected VectorizedRowBatch	setupOverflowBatch()
protected void	setupVOutContext(List<String> outputColumnNames) Setup the vectorized row batch description of the output of the native vectorized map join operator.

Methods inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

cleanUpInputFileChangedOp, closeOp, completeInitializationOp, endGroup, generateMapMetaData, getName, getOperatorName, getRefKey, getValueObjectInspectors, process, reloadHashTable, reProcessBigTable, setMapJoinKey, spillBigTableRow, startGroup

Methods inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

checkAndGenObject, getFilteredValue, getFilterTag, getNextSize, getPosToAliasMap, hasFilter, internalForward, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, reportProgress, setPosToAliasMap

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getConf, getConfiguration, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeChildren, removeParent, replaceChild, replaceParent, reset, resetId, resetStats, setAlias, setChildOperators, setColumnExprMap, setConf, setDone, setExecContext, setGroupKeyObject, setId, setInputContext, setInputObjInspectors, setOperatorId, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

isOuterJoin

protected boolean isOuterJoin

posSingleVectorMapJoinSmallTable

protected byte posSingleVectorMapJoinSmallTable

vContext

protected VectorizationContext vContext

vOutContext

protected VectorizationContext vOutContext

outputProjection

protected int[] outputProjection

outputTypeNames

protected String[] outputTypeNames

bigTableFilterExpressions

protected VectorExpression[] bigTableFilterExpressions

bigTableKeyExpressions

protected VectorExpression[] bigTableKeyExpressions

bigTableValueExpressions

protected VectorExpression[] bigTableValueExpressions

bigTableKeyColumnMap

protected int[] bigTableKeyColumnMap

bigTableKeyTypeNames

protected ArrayList<String> bigTableKeyTypeNames

bigTableValueColumnMap

protected int[] bigTableValueColumnMap

bigTableValueTypeNames

protected ArrayList<String> bigTableValueTypeNames

bigTableRetainedMapping

protected VectorColumnOutputMapping bigTableRetainedMapping

bigTableOuterKeyMapping

protected VectorColumnOutputMapping bigTableOuterKeyMapping

smallTableMapping

protected VectorColumnSourceMapping smallTableMapping

smallTableOutputVectorColumns

protected int[] smallTableOutputVectorColumns

bigTableOuterKeyOutputVectorColumns

protected int[] bigTableOuterKeyOutputVectorColumns

bigTableByteColumnVectorColumns

protected int[] bigTableByteColumnVectorColumns

smallTableByteColumnVectorColumns

protected int[] smallTableByteColumnVectorColumns

taskName

protected transient String taskName

useOverflowRepeatedThreshold

protected transient boolean useOverflowRepeatedThreshold

overflowRepeatedThreshold

protected transient int overflowRepeatedThreshold

bigTableRetainedVectorCopy

protected transient VectorCopyRow bigTableRetainedVectorCopy

bigTableVectorCopyOuterKeys

protected transient VectorCopyRow bigTableVectorCopyOuterKeys

smallTableVectorDeserializeRow

protected transient VectorDeserializeRow smallTableVectorDeserializeRow

overflowBatch

protected transient VectorizedRowBatch overflowBatch

spillReplayBatch

protected transient VectorizedRowBatch spillReplayBatch

needCommonSetup

protected transient boolean needCommonSetup

needHashTableSetup

protected transient boolean needHashTableSetup

vectorMapJoinHashTable

protected transient VectorMapJoinHashTable vectorMapJoinHashTable

Constructor Detail

VectorMapJoinCommonOperator

public VectorMapJoinCommonOperator()

VectorMapJoinCommonOperator

public VectorMapJoinCommonOperator(VectorizationContext vContext,
                           OperatorDesc conf)
                            throws HiveException

Throws:

HiveException

Method Detail

determineCommonInfo

protected void determineCommonInfo(boolean isOuter)

setupVOutContext

protected void setupVOutContext(List<String> outputColumnNames)

Setup the vectorized row batch description of the output of the native vectorized map join operator. Use the output projection we previously built from a mixture of input big table columns and new scratch columns.

getHashTableLoader

protected HashTableLoader getHashTableLoader(org.apache.hadoop.conf.Configuration hconf)

This override lets us substitute our own fast vectorized hash table loader.

Overrides:

getHashTableLoader in class MapJoinOperator

initializeOp

protected Collection<Future<?>> initializeOp(org.apache.hadoop.conf.Configuration hconf)
                                      throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class MapJoinOperator

Throws:

HiveException

loadHashTable

protected org.apache.commons.lang3.tuple.Pair<MapJoinTableContainer[],MapJoinTableContainerSerDe[]> loadHashTable(ExecMapperContext mapContext,
                                                                                                      MapredContext mrContext)
                                                                                                           throws HiveException

Overrides:

loadHashTable in class MapJoinOperator

Throws:

HiveException

setupOverflowBatch

protected VectorizedRowBatch setupOverflowBatch()
                                         throws HiveException

Throws:

HiveException

commonSetup

protected void commonSetup(VectorizedRowBatch batch)
                    throws HiveException

Throws:

HiveException

displayBatchColumns

protected void displayBatchColumns(VectorizedRowBatch batch,
                       String batchName)

getType

public OperatorType getType()

Description copied from class: Operator

Return the type of the specific operator among the types in OperatorType.

Overrides:

getType in class MapJoinOperator

Returns:

OperatorType.*

getOuputVectorizationContext

public VectorizationContext getOuputVectorizationContext()

Specified by:

getOuputVectorizationContext in interface VectorizationContextRegion