org.apache.hadoop.hive.ql.exec.vector.mapjoin

Class VectorMapJoinLeftSemiGenerateResultOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator<MapJoinDesc>

org.apache.hadoop.hive.ql.exec.MapJoinOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinLeftSemiGenerateResultOperator

All Implemented Interfaces:

Serializable, Cloneable, VectorizationContextRegion, Node

Direct Known Subclasses:

VectorMapJoinLeftSemiLongOperator, VectorMapJoinLeftSemiMultiKeyOperator, VectorMapJoinLeftSemiStringOperator

public abstract class VectorMapJoinLeftSemiGenerateResultOperator
extends VectorMapJoinGenerateResultOperator

This class has methods for generating vectorized join results for left semi joins. The big difference between inner joins and left semi joins is existence testing. Inner joins use a hash map to lookup the 1 or more small table values. Left semi joins are a specialized join for outputting big table rows whose key exists in the small table. No small table values are needed for left semi join since they would be empty. So, we use a hash set as the hash table. Hash sets just report whether a key exists. This is a big performance optimization.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	allMatchs
protected VectorMapJoinHashSetResult[]	hashSetResults
protected int[]	spillHashMapResultIndices
protected int[]	spills

Fields inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

batchCounter

Fields inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

bigTableByteColumnVectorColumns, bigTableFilterExpressions, bigTableKeyColumnMap, bigTableKeyExpressions, bigTableKeyTypeNames, bigTableOuterKeyMapping, bigTableOuterKeyOutputVectorColumns, bigTableRetainedMapping, bigTableRetainedVectorCopy, bigTableValueColumnMap, bigTableValueExpressions, bigTableValueTypeNames, bigTableVectorCopyOuterKeys, isOuterJoin, needCommonSetup, needHashTableSetup, outputProjection, outputTypeNames, overflowBatch, overflowRepeatedThreshold, posSingleVectorMapJoinSmallTable, smallTableByteColumnVectorColumns, smallTableMapping, smallTableOutputVectorColumns, smallTableVectorDeserializeRow, spillReplayBatch, taskName, useOverflowRepeatedThreshold, vContext, vectorMapJoinHashTable, vOutContext

Fields inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

firstSmallTable, hashMapRowGetters, loader, mapJoinTables, spilledMapJoinTables

Fields inherited from class org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator

emptyList, joinKeys, joinKeysObjectInspectors, posBigTable

Fields inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

alias, aliasFilterTags, condn, countAfterReport, dummyObj, dummyObjVectors, filterMaps, filterTags, forwardCache, heartbeatInterval, intermediate, joinFilterObjectInspectors, joinFilters, joinValues, joinValuesObjectInspectors, joinValuesStandardObjectInspectors, noOuterJoin, NOTSKIPBIGTABLE, nullsafes, numAliases, offsets, order, rowContainerStandardObjectInspectors, skipVectors, spillTableDesc, totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

childOperators, childOperatorsArray, childOperatorsTag, colExprMap, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVECOUNTERCREATEDFILES, HIVECOUNTERFATAL, id, inputObjInspectors, isLogDebugEnabled, isLogInfoEnabled, isLogTraceEnabled, operatorId, out, outputObjInspector, parentOperators, PLOG, reporter, state, statsMap

Constructor Summary

Constructors

Constructor and Description
VectorMapJoinLeftSemiGenerateResultOperator()
VectorMapJoinLeftSemiGenerateResultOperator(VectorizationContext vContext, OperatorDesc conf)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	commonSetup(VectorizedRowBatch batch)
protected void	finishLeftSemi(VectorizedRowBatch batch, int allMatchCount, int spillCount, VectorMapJoinHashTableResult[] hashTableResults) Generate the left semi join output results for one vectorized row batch.
protected void	finishLeftSemiRepeated(VectorizedRowBatch batch, JoinUtil.JoinResult joinResult, VectorMapJoinHashTableResult hashSetResult)
protected int	generateHashSetResultRepeatedAll(VectorizedRowBatch batch) Generate the left semi join output results for one vectorized row batch with a repeated key.

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

closeOp, forwardBigTableBatch, forwardOverflow, generateHashMapResultMultiValue, generateHashMapResultRepeatedAll, generateHashMapResultSingleValue, intArrayToRangesString, longArrayToRangesString, performValueExpressions, reloadHashTable, reProcessBigTable, spillBatchRepeated, spillHashMapBatch, verifyMonotonicallyIncreasing

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

determineCommonInfo, displayBatchColumns, getHashTableLoader, getOuputVectorizationContext, getType, initializeOp, loadHashTable, setupOverflowBatch, setupVOutContext

Methods inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

cleanUpInputFileChangedOp, completeInitializationOp, endGroup, generateMapMetaData, getName, getOperatorName, getRefKey, getValueObjectInspectors, process, setMapJoinKey, spillBigTableRow, startGroup

Methods inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

checkAndGenObject, getFilteredValue, getFilterTag, getNextSize, getPosToAliasMap, hasFilter, internalForward, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, reportProgress, setPosToAliasMap

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getConf, getConfiguration, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeChildren, removeParent, replaceChild, replaceParent, reset, resetId, resetStats, setAlias, setChildOperators, setColumnExprMap, setConf, setDone, setExecContext, setGroupKeyObject, setId, setInputContext, setInputObjInspectors, setOperatorId, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

hashSetResults

protected transient VectorMapJoinHashSetResult[] hashSetResults

allMatchs

protected transient int[] allMatchs

spills

protected transient int[] spills

spillHashMapResultIndices

protected transient int[] spillHashMapResultIndices

Constructor Detail

VectorMapJoinLeftSemiGenerateResultOperator

public VectorMapJoinLeftSemiGenerateResultOperator()

VectorMapJoinLeftSemiGenerateResultOperator

public VectorMapJoinLeftSemiGenerateResultOperator(VectorizationContext vContext,
                                           OperatorDesc conf)
                                            throws HiveException

Throws:

HiveException

Method Detail

commonSetup

protected void commonSetup(VectorizedRowBatch batch)
                    throws HiveException

Overrides:

commonSetup in class VectorMapJoinGenerateResultOperator

Throws:

HiveException

finishLeftSemi

protected void finishLeftSemi(VectorizedRowBatch batch,
                  int allMatchCount,
                  int spillCount,
                  VectorMapJoinHashTableResult[] hashTableResults)
                       throws HiveException,
                              IOException

Generate the left semi join output results for one vectorized row batch.

Parameters:

batch - The big table batch with any matching and any non matching rows both as selected in use.

allMatchCount - Number of matches in allMatchs.

spillCount - Number of spills in spills.

hashTableResults - The array of all hash table results for the batch. We need the VectorMapJoinHashTableResult for the spill information.

Throws:

HiveException

IOException

generateHashSetResultRepeatedAll

protected int generateHashSetResultRepeatedAll(VectorizedRowBatch batch)
                                        throws HiveException

Generate the left semi join output results for one vectorized row batch with a repeated key.

Parameters:

batch - The big table batch whose repeated key matches.

Throws:

HiveException

finishLeftSemiRepeated

protected void finishLeftSemiRepeated(VectorizedRowBatch batch,
                          JoinUtil.JoinResult joinResult,
                          VectorMapJoinHashTableResult hashSetResult)
                               throws HiveException,
                                      IOException

Throws:

HiveException

IOException