Package org.opensearch.analysis.common

Class KeywordMarkerTokenFilterFactory

java.lang.Object
org.opensearch.index.AbstractIndexComponent
org.opensearch.index.analysis.AbstractTokenFilterFactory
org.opensearch.analysis.common.KeywordMarkerTokenFilterFactory
All Implemented Interfaces:
TokenFilterFactory, IndexComponent
public class KeywordMarkerTokenFilterFactory
extends AbstractTokenFilterFactory
A factory for creating keyword marker token filters that prevent tokens from being modified by stemmers. Two types of keyword marker filters are available: the SetKeywordMarkerFilter and the PatternKeywordMarkerFilter. The SetKeywordMarkerFilter uses a set of keywords to denote which tokens should be excluded from stemming. This filter is created if the settings include keywords, which contains the list of keywords, or `keywords_path`, which contains a path to a file in the config directory with the keywords. The PatternKeywordMarkerFilter uses a regular expression pattern to match against tokens that should be excluded from stemming. This filter is created if the settings include keywords_pattern, which contains the regular expression to match against.

  • Method Details

    • create

      public org.apache.lucene.analysis.TokenStream create​(org.apache.lucene.analysis.TokenStream tokenStream)