org.apache.hadoop.hive.ql.exec.vector.mapjoin

Class VectorMapJoinInnerBigOnlyGenerateResultOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator<MapJoinDesc>

org.apache.hadoop.hive.ql.exec.MapJoinOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerBigOnlyGenerateResultOperator

All Implemented Interfaces:

Serializable, Cloneable, VectorizationContextRegion, Node

Direct Known Subclasses:

VectorMapJoinInnerBigOnlyLongOperator, VectorMapJoinInnerBigOnlyMultiKeyOperator, VectorMapJoinInnerBigOnlyStringOperator

public abstract class VectorMapJoinInnerBigOnlyGenerateResultOperator
extends VectorMapJoinGenerateResultOperator

This class has methods for generating vectorized join results for the big table only variation of inner joins. When an inner join does not have any small table columns in the join result, we use this variation we call inner big only. This variation uses a hash multi-set instead of hash map since there are no values (just a count). Note that if a inner key appears in the small table results area, we use the inner join projection optimization and are able to use this variation.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	allMatchs
protected int[]	equalKeySeriesAllMatchIndices
protected int[]	equalKeySeriesDuplicateCounts
protected long[]	equalKeySeriesValueCounts
protected VectorMapJoinHashMultiSetResult[]	hashMultiSetResults
protected int[]	spillHashMapResultIndices
protected int[]	spills

Fields inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

batchCounter

Fields inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

bigTableByteColumnVectorColumns, bigTableFilterExpressions, bigTableKeyColumnMap, bigTableKeyExpressions, bigTableKeyTypeNames, bigTableOuterKeyMapping, bigTableOuterKeyOutputVectorColumns, bigTableRetainedMapping, bigTableRetainedVectorCopy, bigTableValueColumnMap, bigTableValueExpressions, bigTableValueTypeNames, bigTableVectorCopyOuterKeys, isOuterJoin, needCommonSetup, needHashTableSetup, outputProjection, outputTypeNames, overflowBatch, overflowRepeatedThreshold, posSingleVectorMapJoinSmallTable, smallTableByteColumnVectorColumns, smallTableMapping, smallTableOutputVectorColumns, smallTableVectorDeserializeRow, spillReplayBatch, taskName, useOverflowRepeatedThreshold, vContext, vectorMapJoinHashTable, vOutContext

Fields inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

firstSmallTable, hashMapRowGetters, loader, mapJoinTables, spilledMapJoinTables

Fields inherited from class org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator

emptyList, joinKeys, joinKeysObjectInspectors, posBigTable

Fields inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

alias, aliasFilterTags, condn, countAfterReport, dummyObj, dummyObjVectors, filterMaps, filterTags, forwardCache, heartbeatInterval, intermediate, joinFilterObjectInspectors, joinFilters, joinValues, joinValuesObjectInspectors, joinValuesStandardObjectInspectors, noOuterJoin, NOTSKIPBIGTABLE, nullsafes, numAliases, offsets, order, rowContainerStandardObjectInspectors, skipVectors, spillTableDesc, totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

childOperators, childOperatorsArray, childOperatorsTag, colExprMap, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVECOUNTERCREATEDFILES, HIVECOUNTERFATAL, id, inputObjInspectors, isLogDebugEnabled, isLogInfoEnabled, isLogTraceEnabled, operatorId, out, outputObjInspector, parentOperators, PLOG, reporter, state, statsMap

Constructor Summary

Constructors

Constructor and Description
VectorMapJoinInnerBigOnlyGenerateResultOperator()
VectorMapJoinInnerBigOnlyGenerateResultOperator(VectorizationContext vContext, OperatorDesc conf)

Method Summary

Methods

Modifier and Type	Method and Description
protected void	commonSetup(VectorizedRowBatch batch)
protected void	finishInnerBigOnly(VectorizedRowBatch batch, int allMatchCount, int equalKeySeriesCount, int spillCount, VectorMapJoinHashTableResult[] hashTableResults, int hashMapResultCount) Generate the inner big table only join output results for one vectorized row batch.
protected void	finishInnerBigOnlyRepeated(VectorizedRowBatch batch, JoinUtil.JoinResult joinResult, VectorMapJoinHashMultiSetResult hashMultiSetResult)
protected int	generateHashMultiSetResultRepeatedAll(VectorizedRowBatch batch, VectorMapJoinHashMultiSetResult hashMultiSetResult) Generate the inner big table only join output results for one vectorized row batch with a repeated key.

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

closeOp, forwardBigTableBatch, forwardOverflow, generateHashMapResultMultiValue, generateHashMapResultRepeatedAll, generateHashMapResultSingleValue, intArrayToRangesString, longArrayToRangesString, performValueExpressions, reloadHashTable, reProcessBigTable, spillBatchRepeated, spillHashMapBatch, verifyMonotonicallyIncreasing

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

determineCommonInfo, displayBatchColumns, getHashTableLoader, getOuputVectorizationContext, getType, initializeOp, loadHashTable, setupOverflowBatch, setupVOutContext

Methods inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

cleanUpInputFileChangedOp, completeInitializationOp, endGroup, generateMapMetaData, getName, getOperatorName, getRefKey, getValueObjectInspectors, process, setMapJoinKey, spillBigTableRow, startGroup

Methods inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

checkAndGenObject, getFilteredValue, getFilterTag, getNextSize, getPosToAliasMap, hasFilter, internalForward, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, reportProgress, setPosToAliasMap

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getConf, getConfiguration, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeChildren, removeParent, replaceChild, replaceParent, reset, resetId, resetStats, setAlias, setChildOperators, setColumnExprMap, setConf, setDone, setExecContext, setGroupKeyObject, setId, setInputContext, setInputObjInspectors, setOperatorId, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

hashMultiSetResults

protected transient VectorMapJoinHashMultiSetResult[] hashMultiSetResults

allMatchs

protected transient int[] allMatchs

equalKeySeriesValueCounts

protected transient long[] equalKeySeriesValueCounts

equalKeySeriesAllMatchIndices

protected transient int[] equalKeySeriesAllMatchIndices

equalKeySeriesDuplicateCounts

protected transient int[] equalKeySeriesDuplicateCounts

spills

protected transient int[] spills

spillHashMapResultIndices

protected transient int[] spillHashMapResultIndices

Constructor Detail

VectorMapJoinInnerBigOnlyGenerateResultOperator

public VectorMapJoinInnerBigOnlyGenerateResultOperator()

VectorMapJoinInnerBigOnlyGenerateResultOperator

public VectorMapJoinInnerBigOnlyGenerateResultOperator(VectorizationContext vContext,
                                               OperatorDesc conf)
                                                throws HiveException

Throws:

HiveException

Method Detail

commonSetup

protected void commonSetup(VectorizedRowBatch batch)
                    throws HiveException

Overrides:

commonSetup in class VectorMapJoinGenerateResultOperator

Throws:

HiveException

finishInnerBigOnly

protected void finishInnerBigOnly(VectorizedRowBatch batch,
                      int allMatchCount,
                      int equalKeySeriesCount,
                      int spillCount,
                      VectorMapJoinHashTableResult[] hashTableResults,
                      int hashMapResultCount)
                           throws HiveException,
                                  IOException

Generate the inner big table only join output results for one vectorized row batch.

Parameters:

batch - The big table batch with any matching and any non matching rows both as selected in use.

allMatchCount - Number of matches in allMatchs.

equalKeySeriesCount - Number of single value matches.

spillCount - Number of spills in spills.

hashTableResults - The array of all hash table results for the batch. We need the VectorMapJoinHashTableResult for the spill information.

hashMapResultCount - Number of entries in hashMapResults.

Throws:

HiveException

IOException

generateHashMultiSetResultRepeatedAll

protected int generateHashMultiSetResultRepeatedAll(VectorizedRowBatch batch,
                                        VectorMapJoinHashMultiSetResult hashMultiSetResult)
                                             throws HiveException

Generate the inner big table only join output results for one vectorized row batch with a repeated key.

Parameters:

batch - The big table batch with any matching and any non matching rows both as selected in use.

hashMultiSetResult - The hash multi-set results for the batch.

Throws:

HiveException

finishInnerBigOnlyRepeated

protected void finishInnerBigOnlyRepeated(VectorizedRowBatch batch,
                              JoinUtil.JoinResult joinResult,
                              VectorMapJoinHashMultiSetResult hashMultiSetResult)
                                   throws HiveException,
                                          IOException

Throws:

HiveException

IOException