io.warp10.continuum.gts

Class GTSOutliersHelper

java.lang.Object

io.warp10.continuum.gts.GTSOutliersHelper

public class GTSOutliersHelper
extends java.lang.Object

Class that implements tests to detect outliers in GTS

Constructor Summary

Constructors

Constructor and Description
GTSOutliersHelper()

Method Summary

Modifier and Type	Method and Description
protected static void	doubleCheck(GeoTimeSerie gts)
static java.util.List<java.lang.Long>	entropyHybridTest(GeoTimeSerie gts, int buckets_per_period, int periods_per_piece, int k, double alpha) Applying Seasonal Entropy Hybrid ESD test This test is based on piecewise decomposition where trend components are approximated by median and seasonal components are factored by the entropy of the cycle sub-series.
static java.util.List<java.lang.Long>	ESDTest(GeoTimeSerie gts, int k, boolean useMedian)
static java.util.List<java.lang.Long>	ESDTest(GeoTimeSerie gts, int k, boolean useMedian, double alpha) Applying generalized extreme Studentized deviate test using mean/std or median/mad
static java.util.List<java.lang.Long>	grubbsTest(GeoTimeSerie gts, boolean useMedian)
static java.util.List<java.lang.Long>	grubbsTest(GeoTimeSerie gts, boolean useMedian, double alpha) Applying Grubbs' test using mean/std or median/mad
static java.util.List<java.lang.Long>	hybridTest(GeoTimeSerie gts, int buckets_per_period, int periods_per_piece, int k, double alpha, java.util.Map<java.lang.String,java.lang.Object> params) Applying Seasonal Hybrid ESD.
protected static double[]	madsigma(GeoTimeSerie gts, boolean useMedian) Compute mu and sigma or median and mad, given useMedian is true or false Mad is returned modified as an estimate of sigma
protected static double	max(GeoTimeSerie gts)
protected static double	mean(GeoTimeSerie gts)
protected static double	median(GeoTimeSerie gts)
protected static double	medianAbsoluteDeviation(GeoTimeSerie gts, double median)
protected static double	min(GeoTimeSerie gts)
static java.util.List<java.lang.Long>	STLESDTest(GeoTimeSerie gts, int buckets_per_period, int k, double alpha, java.util.Map<java.lang.String,java.lang.Object> params) Applying STL-ESD test.
static java.util.List<java.lang.Long>	thresholdTest(GeoTimeSerie gts, double threshold)
static java.util.List<java.lang.Long>	thresholdTest(GeoTimeSerie gts, double threshold, boolean abs) Applying a simple threshold test.
static GeoTimeSerie	zScore(GeoTimeSerie gts, boolean useMedian, boolean inplace) Compute Z-score or modified Z-score
static java.util.List<java.lang.Long>	zScoreTest(GeoTimeSerie gts, boolean useMedian, double d) Applying Z-score test If useMedian is true, then use modified z-score.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GTSOutliersHelper

public GTSOutliersHelper()

Method Detail

doubleCheck

protected static void doubleCheck(GeoTimeSerie gts)
                           throws WarpScriptException

Throws:

WarpScriptException

mean

protected static double mean(GeoTimeSerie gts)
                      throws WarpScriptException

Throws:

WarpScriptException

median

protected static double median(GeoTimeSerie gts)

medianAbsoluteDeviation

protected static double medianAbsoluteDeviation(GeoTimeSerie gts,
                                                double median)

max

protected static double max(GeoTimeSerie gts)
                     throws WarpScriptException

Throws:

WarpScriptException

min

protected static double min(GeoTimeSerie gts)
                     throws WarpScriptException

Throws:

WarpScriptException

madsigma

protected static double[] madsigma(GeoTimeSerie gts,
                                   boolean useMedian)

Compute mu and sigma or median and mad, given useMedian is true or false Mad is returned modified as an estimate of sigma

zScore

public static GeoTimeSerie zScore(GeoTimeSerie gts,
                                  boolean useMedian,
                                  boolean inplace)
                           throws WarpScriptException

Compute Z-score or modified Z-score

Parameters:

gts -

useMedian - Should we compute Z-score using median/mad rather than mean/std

inplace - Should the output gts be the input instance or a new one

Returns:

gts_

Throws:

WarpScriptException

See Also:

http://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm

thresholdTest

public static java.util.List<java.lang.Long> thresholdTest(GeoTimeSerie gts,
                                                           double threshold,
                                                           boolean abs)
                                                    throws WarpScriptException

Applying a simple threshold test.

Parameters:

gts -

threshold -

abs - Should we compare threshold with absolute value

Returns:

anomalous_ticks

Throws:

WarpScriptException

thresholdTest

public static java.util.List<java.lang.Long> thresholdTest(GeoTimeSerie gts,
                                                           double threshold)
                                                    throws WarpScriptException

Throws:

WarpScriptException

zScoreTest

public static java.util.List<java.lang.Long> zScoreTest(GeoTimeSerie gts,
                                                        boolean useMedian,
                                                        double d)
                                                 throws WarpScriptException

Applying Z-score test If useMedian is true, then use modified z-score.

Parameters:

gts -

useMedian - Should we compute Z-score using median/mad rather than mean/std

d - Threshold. Default set at 3.5

Returns:

anomalous_ticks

Throws:

WarpScriptException

See Also:

http://www.itl.nist.gov/div898/handbook/eda/section3/eda35h.htm

grubbsTest

public static java.util.List<java.lang.Long> grubbsTest(GeoTimeSerie gts,
                                                        boolean useMedian,
                                                        double alpha)
                                                 throws WarpScriptException

Applying Grubbs' test using mean/std or median/mad

Parameters:

gts -

useMedian - Should the test use median/mad instead of mean/std

alpha - Significance level with which to accept or reject anomalies. Default is 0.05

Returns:

anomalous_ticks

Throws:

WarpScriptException

See Also:

http://www.itl.nist.gov/div898/handbook/eda/section3/eda35h1.htm

grubbsTest

public static java.util.List<java.lang.Long> grubbsTest(GeoTimeSerie gts,
                                                        boolean useMedian)
                                                 throws WarpScriptException

Throws:

WarpScriptException

ESDTest

public static java.util.List<java.lang.Long> ESDTest(GeoTimeSerie gts,
                                                     int k,
                                                     boolean useMedian,
                                                     double alpha)
                                              throws WarpScriptException

Applying generalized extreme Studentized deviate test using mean/std or median/mad

Parameters:

gts -

k - Upper bound of suspected number of outliers

useMedian - Should the test use median/mad instead of mean/std

alpha - Significance level with which to accept or reject anomalies. Default is 0.05

Returns:

anomalous_ticks

Throws:

WarpScriptException

See Also:

http://www.itl.nist.gov/div898/handbook/eda/section3/eda35h3.htm

ESDTest

public static java.util.List<java.lang.Long> ESDTest(GeoTimeSerie gts,
                                                     int k,
                                                     boolean useMedian)
                                              throws WarpScriptException

Throws:

WarpScriptException

STLESDTest

public static java.util.List<java.lang.Long> STLESDTest(GeoTimeSerie gts,
                                                        int buckets_per_period,
                                                        int k,
                                                        double alpha,
                                                        java.util.Map<java.lang.String,java.lang.Object> params)
                                                 throws WarpScriptException

Applying STL-ESD test. ESD test is passed on residual of STL decomposition.

Parameters:

gts -

k - Upper bound of suspected number of outliers

alpha - Significance level with which to accept or reject anomalies. Default is 0.05

params - Optional map of parameters used for stl calls

Returns:

anomalous_ticks

Throws:

WarpScriptException

hybridTest

public static java.util.List<java.lang.Long> hybridTest(GeoTimeSerie gts,
                                                        int buckets_per_period,
                                                        int periods_per_piece,
                                                        int k,
                                                        double alpha,
                                                        java.util.Map<java.lang.String,java.lang.Object> params)
                                                 throws WarpScriptException

Applying Seasonal Hybrid ESD. This test is based on piecewise median and STL seasonal extraction, completed by an ESD test. It was developed at Twitter.

Parameters:

gts -

k - Upper bound of suspected number of outliers

alpha - Significance level with which to accept or reject anomalies. Default is 0.05

params - Optional map of parameters used for stl calls

Returns:

anomalous_ticks

Throws:

WarpScriptException

See Also:

https://www.usenix.org/system/files/conference/hotcloud14/hotcloud14-vallis.pdf, https://github.com/twitter/AnomalyDetection

entropyHybridTest

public static java.util.List<java.lang.Long> entropyHybridTest(GeoTimeSerie gts,
                                                               int buckets_per_period,
                                                               int periods_per_piece,
                                                               int k,
                                                               double alpha)
                                                        throws WarpScriptException

Applying Seasonal Entropy Hybrid ESD test This test is based on piecewise decomposition where trend components are approximated by median and seasonal components are factored by the entropy of the cycle sub-series. An ESD test is passed upon the residuals. It differs from hybridTest by approximating seasonal component instead of using STL. But in many cases this approximation is more useful than estimation of STL.

Parameters:

gts -

k - Upper bound of suspected number of outliers

alpha - Significance level with which to accept or reject anomalies. Default is 0.05

Returns:

anomalous_ticks

Throws:

WarpScriptException