org.verdictdb.core.scrambling

Class FastConvergeScramblingMethod

java.lang.Object

org.verdictdb.core.scrambling.ScramblingMethodBase

org.verdictdb.core.scrambling.FastConvergeScramblingMethod

All Implemented Interfaces:

ScramblingMethod

public class FastConvergeScramblingMethod
extends ScramblingMethodBase

Policy: 1. Tier 0: tuples containing outlier values. 2. Tier 1: tuples containing rare groups 3. Tier 2: other tuples

Outlier values: Bottom 0.1% and 99.9% percentile values for every numeric column. Suppose a table has 20 numeric columns; then, in the worst case, 20 * 0.2% = 4% of the tuples are the tuples containing outlier values.

Blocks for Tier 0: Tier 0 takes up to 50% of each block.

Blocks for Tier 1: Tier 0 + Tier 1 take up to 80% of each block.

Blocks for Tier 2: Tier 2 takes the rest of the space in each block.

Author:

Yongjoo Park

Field Summary

Fields

Modifier and Type	Field and Description
static String	MAIN_TABLE_SOURCE_ALIAS_NAME
static String	RIGHT_TABLE_SOURCE_ALIAS_NAME

Fields inherited from class org.verdictdb.core.scrambling.ScramblingMethodBase

blockSize

Constructor Summary

Constructors

Constructor and Description
FastConvergeScramblingMethod(long blockSize, String scratchpadSchemaName)
FastConvergeScramblingMethod(long blockSize, String scratchpadSchemaName, String primaryColumnName)

Method Summary

Modifier and Type	Method and Description
int	getBlockCount()
List<Double>	getCumulativeProbabilityDistributionForTier(Map<String,Object> metaData, int tier)
String	getMainTableAlias()
AbstractRelation	getScramblingSource(String originalSchema, String originalTable, Map<String,Object> metaData) Returns the table that should be used in the final scrambling stage.
List<ExecutableNodeBase>	getStatisticsNode(String oldSchemaName, String oldTableName, String columnMetaTokenKey, String partitionMetaTokenKey) Computes three nodes.
int	getTierCount()
List<UnnamedColumn>	getTierExpressions(Map<String,Object> metaData)

Methods inherited from class org.verdictdb.core.scrambling.ScramblingMethodBase

getStoredCumulativeProbabilityDistributionForTier, storeCumulativeProbabilityDistribution

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAIN_TABLE_SOURCE_ALIAS_NAME

public static final String MAIN_TABLE_SOURCE_ALIAS_NAME

See Also:

Constant Field Values

RIGHT_TABLE_SOURCE_ALIAS_NAME

public static final String RIGHT_TABLE_SOURCE_ALIAS_NAME

See Also:

Constant Field Values

Constructor Detail

FastConvergeScramblingMethod

public FastConvergeScramblingMethod(long blockSize,
                                    String scratchpadSchemaName)

FastConvergeScramblingMethod

public FastConvergeScramblingMethod(long blockSize,
                                    String scratchpadSchemaName,
                                    String primaryColumnName)

Method Detail

getStatisticsNode

public List<ExecutableNodeBase> getStatisticsNode(String oldSchemaName,
                                                  String oldTableName,
                                                  String columnMetaTokenKey,
                                                  String partitionMetaTokenKey)

Computes three nodes. They compute: (1) 0.1% and 99.9% percentiles of numeric columns and the total count, (for this, we compute standard deviations and estimate those percentiles based on the standard deviations and normal distribution assumptions. \pm 3.09 * stddev is the 99.9 and 0.1 percentiles of the standard normal distribution.) (2) the list of "large" groups, and (3) the sizes of "large" groups

Recall that channels 100 and 101 are reserved for column meta and partition meta, respectively.

This method generates up to three nodes, and the token keys set up by those nodes are: 1. queryResult: this contains avg, std, and count 2. schemaName, tableName: this is the name of the temporary tables that contains a list of large groups. 3. queryResult: this contains the sum of the sizes of large groups.

getTierExpressions

public List<UnnamedColumn> getTierExpressions(Map<String,Object> metaData)

Returns:

A list of sql expressions (boolean predicates) that must be evaluated true at the step indicating a particular tier

getCumulativeProbabilityDistributionForTier

public List<Double> getCumulativeProbabilityDistributionForTier(Map<String,Object> metaData,
                                                                int tier)

tier - 0, 1, ..., getTierCount()-1

Returns:

A list of doubles. The values should be increasing; the last value must be 1.0; and the size of the list must be equal to "length".

getScramblingSource

public AbstractRelation getScramblingSource(String originalSchema,
                                            String originalTable,
                                            Map<String,Object> metaData)

Description copied from interface: ScramblingMethod

Returns the table that should be used in the final scrambling stage. This can be a join of the main table and some auxiliary tables.

Returns:

getMainTableAlias

public String getMainTableAlias()

getBlockCount

public int getBlockCount()

getTierCount

public int getTierCount()