io.redlink.lucene.search.suggest

Class FreeTextSuggester

java.lang.Object

org.apache.lucene.search.suggest.Lookup

io.redlink.lucene.search.suggest.FreeTextSuggester

All Implemented Interfaces:

org.apache.lucene.util.Accountable

public class FreeTextSuggester
extends org.apache.lucene.search.suggest.Lookup
implements org.apache.lucene.util.Accountable

Builds an ngram model from the text sent to build(org.apache.lucene.search.suggest.InputIterator) and predicts based on the last grams-1 tokens in the request sent to lookup(java.lang.CharSequence, boolean, int). This tries to handle the "long tail" of suggestions for when the incoming query is a never before seen query string.

Likely this suggester would only be used as a fallback, when the primary suggester fails to find any suggestions.

Note that the weight for each suggestion is unused, and the suggestions are the analyzed forms (so your analysis process should normally be very "light").

This uses the stupid backoff language model to smooth scores across ngram models; see "Large language models in machine translation", http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.76.1126 for details.

From lookup(java.lang.CharSequence, boolean, int), the key of each result is the ngram token; the value is Long.MAX_VALUE * score (fixed point, cast to long). Divide by Long.MAX_VALUE to get the score back, which ranges from 0.0 to 1.0. onlyMorePopular is unused.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.search.suggest.Lookup

org.apache.lucene.search.suggest.Lookup.LookupPriorityQueue, org.apache.lucene.search.suggest.Lookup.LookupResult

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CODEC_NAME Codec name used in the header for the saved model.
static double	DEFAULT_ALPHA The constant used for backoff smoothing; during lookup, this means that if a given trigram did not occur, and we backoff to the bigram, the overall score will be 0.4 times what the bigram model would have assigned.
static int	DEFAULT_GRAMS By default we use a bigram model.
static byte	DEFAULT_SEPARATOR The default character used to join multiple tokens into a single ngram token.
static int	VERSION_CURRENT Current version of the the saved model file format.
static int	VERSION_START Initial version of the the saved model file format.

Fields inherited from class org.apache.lucene.search.suggest.Lookup

CHARSEQUENCE_COMPARATOR

Constructor Summary

Constructors

Constructor and Description
FreeTextSuggester(org.apache.lucene.analysis.Analyzer analyzer) Instantiate, using the provided analyzer for both indexing and lookup, using bigram model by default.
FreeTextSuggester(org.apache.lucene.analysis.Analyzer indexAnalyzer, org.apache.lucene.analysis.Analyzer queryAnalyzer) Instantiate, using the provided indexing and lookup analyzers, using bigram model by default.
FreeTextSuggester(org.apache.lucene.analysis.Analyzer indexAnalyzer, org.apache.lucene.analysis.Analyzer queryAnalyzer, int grams) Instantiate, using the provided indexing and lookup analyzers, with the specified model (2 = bigram, 3 = trigram, etc.).
FreeTextSuggester(org.apache.lucene.analysis.Analyzer indexAnalyzer, org.apache.lucene.analysis.Analyzer queryAnalyzer, int grams, byte separator) Instantiate, using the provided indexing and lookup analyzers, and specified model (2 = bigram, 3 = trigram ,etc.).
FreeTextSuggester(org.apache.lucene.analysis.Analyzer indexAnalyzer, org.apache.lucene.analysis.Analyzer queryAnalyzer, int grams, byte separator, double alpha)

Method Summary

Modifier and Type	Method and Description
void	build(org.apache.lucene.search.suggest.InputIterator iterator)
void	build(org.apache.lucene.search.suggest.InputIterator iterator, double ramBufferSizeMB) Build the suggest index, using up to the specified amount of temporary RAM while building.
java.lang.Object	get(java.lang.CharSequence key) Returns the weight associated with an input string, or null if it does not exist.
java.util.Collection<org.apache.lucene.util.Accountable>	getChildResources()
long	getCount()
boolean	load(org.apache.lucene.store.DataInput input)
java.util.List<org.apache.lucene.search.suggest.Lookup.LookupResult>	lookup(java.lang.CharSequence key, boolean onlyMorePopular, int num)
java.util.List<org.apache.lucene.search.suggest.Lookup.LookupResult>	lookup(java.lang.CharSequence key, int num) Lookup, without any context.
java.util.List<org.apache.lucene.search.suggest.Lookup.LookupResult>	lookup(java.lang.CharSequence key, java.util.Set<org.apache.lucene.util.BytesRef> contexts, boolean onlyMorePopular, int num)
java.util.List<org.apache.lucene.search.suggest.Lookup.LookupResult>	lookup(java.lang.CharSequence key, java.util.Set<org.apache.lucene.util.BytesRef> contexts, int num) Retrieve suggestions.
long	ramBytesUsed() Returns byte size of the underlying FST.
boolean	store(org.apache.lucene.store.DataOutput output)

Methods inherited from class org.apache.lucene.search.suggest.Lookup

build, load, lookup, store

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CODEC_NAME

public static final java.lang.String CODEC_NAME

Codec name used in the header for the saved model.

See Also:

Constant Field Values

VERSION_START

public static final int VERSION_START

Initial version of the the saved model file format.

See Also:

Constant Field Values

VERSION_CURRENT

public static final int VERSION_CURRENT

Current version of the the saved model file format.

See Also:

Constant Field Values

DEFAULT_GRAMS

public static final int DEFAULT_GRAMS

By default we use a bigram model.

See Also:

Constant Field Values

DEFAULT_ALPHA

public static final double DEFAULT_ALPHA

The constant used for backoff smoothing; during lookup, this means that if a given trigram did not occur, and we backoff to the bigram, the overall score will be 0.4 times what the bigram model would have assigned.

See Also:

Constant Field Values

DEFAULT_SEPARATOR

public static final byte DEFAULT_SEPARATOR

The default character used to join multiple tokens into a single ngram token. The input tokens produced by the analyzer must not contain this character.

See Also:

Constant Field Values

Constructor Detail

FreeTextSuggester

public FreeTextSuggester(org.apache.lucene.analysis.Analyzer analyzer)

Instantiate, using the provided analyzer for both indexing and lookup, using bigram model by default.

FreeTextSuggester

public FreeTextSuggester(org.apache.lucene.analysis.Analyzer indexAnalyzer,
                         org.apache.lucene.analysis.Analyzer queryAnalyzer)

Instantiate, using the provided indexing and lookup analyzers, using bigram model by default.

FreeTextSuggester

public FreeTextSuggester(org.apache.lucene.analysis.Analyzer indexAnalyzer,
                         org.apache.lucene.analysis.Analyzer queryAnalyzer,
                         int grams)

Instantiate, using the provided indexing and lookup analyzers, with the specified model (2 = bigram, 3 = trigram, etc.).

FreeTextSuggester

public FreeTextSuggester(org.apache.lucene.analysis.Analyzer indexAnalyzer,
                         org.apache.lucene.analysis.Analyzer queryAnalyzer,
                         int grams,
                         byte separator)

Instantiate, using the provided indexing and lookup analyzers, and specified model (2 = bigram, 3 = trigram ,etc.). The separator is passed to ShingleFilter.setTokenSeparator(java.lang.String) to join multiple tokens into a single ngram token; it must be an ascii (7-bit-clean) byte. No input tokens should have this byte, otherwise IllegalArgumentException is thrown.

FreeTextSuggester

public FreeTextSuggester(org.apache.lucene.analysis.Analyzer indexAnalyzer,
                         org.apache.lucene.analysis.Analyzer queryAnalyzer,
                         int grams,
                         byte separator,
                         double alpha)

Method Detail

ramBytesUsed

public long ramBytesUsed()

Returns byte size of the underlying FST.

Specified by:

ramBytesUsed in interface org.apache.lucene.util.Accountable

getChildResources

public java.util.Collection<org.apache.lucene.util.Accountable> getChildResources()

Specified by:

getChildResources in interface org.apache.lucene.util.Accountable

build

public void build(org.apache.lucene.search.suggest.InputIterator iterator)
           throws java.io.IOException

Specified by:

build in class org.apache.lucene.search.suggest.Lookup

Throws:

java.io.IOException

build

public void build(org.apache.lucene.search.suggest.InputIterator iterator,
                  double ramBufferSizeMB)
           throws java.io.IOException

Build the suggest index, using up to the specified amount of temporary RAM while building. Note that the weights for the suggestions are ignored.

Parameters:

iterator - the input interator

ramBufferSizeMB - the buffer size in MByte

Throws:

java.io.IOException - on any IO releated error

store

public boolean store(org.apache.lucene.store.DataOutput output)
              throws java.io.IOException

Specified by:

store in class org.apache.lucene.search.suggest.Lookup

Throws:

java.io.IOException

load

public boolean load(org.apache.lucene.store.DataInput input)
             throws java.io.IOException

Specified by:

load in class org.apache.lucene.search.suggest.Lookup

Throws:

java.io.IOException

lookup

public java.util.List<org.apache.lucene.search.suggest.Lookup.LookupResult> lookup(java.lang.CharSequence key,
                                                                                   boolean onlyMorePopular,
                                                                                   int num)

Overrides:

lookup in class org.apache.lucene.search.suggest.Lookup

lookup

public java.util.List<org.apache.lucene.search.suggest.Lookup.LookupResult> lookup(java.lang.CharSequence key,
                                                                                   int num)

Lookup, without any context.

lookup

public java.util.List<org.apache.lucene.search.suggest.Lookup.LookupResult> lookup(java.lang.CharSequence key,
                                                                                   java.util.Set<org.apache.lucene.util.BytesRef> contexts,
                                                                                   boolean onlyMorePopular,
                                                                                   int num)

Specified by:

lookup in class org.apache.lucene.search.suggest.Lookup

getCount

public long getCount()

Specified by:

getCount in class org.apache.lucene.search.suggest.Lookup

lookup

public java.util.List<org.apache.lucene.search.suggest.Lookup.LookupResult> lookup(java.lang.CharSequence key,
                                                                                   java.util.Set<org.apache.lucene.util.BytesRef> contexts,
                                                                                   int num)
                                                                            throws java.io.IOException

Retrieve suggestions.

Throws:

java.io.IOException

get

public java.lang.Object get(java.lang.CharSequence key)

Returns the weight associated with an input string, or null if it does not exist.