io.earcam.utilitarian.site.search.offline

Class SimpleTokenizer

java.lang.Object

io.earcam.utilitarian.site.search.offline.SimpleTokenizer

All Implemented Interfaces:

Component, Processor, Function<Document,Document>

public class SimpleTokenizer
extends Object
implements Processor

Constructor Summary

Constructors

Constructor and Description
SimpleTokenizer()

Method Summary

Modifier and Type	Method and Description
protected org.apache.lucene.analysis.Analyzer	createAnalyzer() Override this method to return a custom Analyzer.
void	process(Document document)
List<String>	tokenize(String input)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.earcam.utilitarian.site.search.offline.Processor

apply

Methods inherited from interface io.earcam.utilitarian.site.search.offline.Component

configure, getOrDefault, getOrDefault, id, mandatory

Methods inherited from interface java.util.function.Function

andThen, compose, identity

Constructor Detail

SimpleTokenizer

public SimpleTokenizer()

Method Detail

tokenize

public List<String> tokenize(String input)

createAnalyzer

protected org.apache.lucene.analysis.Analyzer createAnalyzer()

Override this method to return a custom Analyzer.

Note; Use of Lucene for stemming, stopword filtering, etc must match whatever is configured for lunrjs.

Returns:

an Analyzer for tokenizing

process

public void process(Document document)

Specified by:

process in interface Processor