org.apache.hadoop.hive.ql.optimizer.stats.annotation

Class StatsRulesProcFactory.FilterStatsRule

java.lang.Object

org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory.DefaultStatsRule

org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory.FilterStatsRule

All Implemented Interfaces:

NodeProcessor

Enclosing class:

StatsRulesProcFactory

public static class StatsRulesProcFactory.FilterStatsRule
extends StatsRulesProcFactory.DefaultStatsRule
implements NodeProcessor

FILTER operator does not change the average row size but it does change the number of rows emitted. The reduction in the number of rows emitted is dependent on the filter expression.

Notations:
• T(S) - Number of tuples in relations S
• V(S,A) - Number of distinct values of attribute A in relation S

Rules:
• Column equals a constant
T(S) = T(R) / V(R,A)
• Inequality conditions
T(S) = T(R) / 3
• Not equals comparison
- Simple formula T(S) = T(R)

- Alternate formula T(S) = T(R) (V(R,A) - 1) / V(R,A)

• NOT condition
T(S) = 1 - T(S'), where T(S') is the satisfying condition
• Multiple AND conditions
Cascadingly apply the rules 1 to 3 (order doesn't matter)
• Multiple OR conditions
- Simple formula is to evaluate conditions independently and sum the results T(S) = m1 + m2

- Alternate formula T(S) = T(R) * ( 1 - ( 1 - m1/T(R) ) * ( 1 - m2/T(R) ))

where, m1 is the number of tuples that satisfy condition1 and m2 is the number of tuples that satisfy condition2

Worst case: If no column statistics are available, then evaluation of predicate expression will assume worst case (i.e; half the input rows) for each of predicate expression.

For more information, refer 'Estimating The Cost Of Operations' chapter in "Database Systems: The Complete Book" by Garcia-Molina et. al.

Constructor Summary

Constructors

Constructor and Description
StatsRulesProcFactory.FilterStatsRule()

Method Summary

Methods

Modifier and Type	Method and Description
Object	process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, Object... nodeOutputs) Generic process for all ops that don't have specific implementations.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StatsRulesProcFactory.FilterStatsRule

public StatsRulesProcFactory.FilterStatsRule()

Method Detail

process

public Object process(Node nd,
             Stack<Node> stack,
             NodeProcessorCtx procCtx,
             Object... nodeOutputs)
               throws SemanticException

Description copied from interface: NodeProcessor

Generic process for all ops that don't have specific implementations.

Specified by:

process in interface NodeProcessor

Overrides:

process in class StatsRulesProcFactory.DefaultStatsRule

Parameters:

nd - operator to process

procCtx - operator processor context

nodeOutputs - A variable argument list of outputs from other nodes in the walk

Returns:

Object to be returned by the process call

Throws:

SemanticException