RegexTokenizer (org.apache.directory.api_api-all_sources-and-javadoc 2.1.1 API)

java.lang.Object
- org.apache.commons.text.similarity.RegexTokenizer

All Implemented Interfaces:

Tokenizer<CharSequence>
```
class RegexTokenizer
extends Object
implements Tokenizer<CharSequence>
```
A simple word tokenizer that utilizes regex to find words. It applies a regex (\w)+ over the input text to extract words from a given character sequence.

Since:

1.0

- Constructor Summary
  
  Constructors
  Constructor Description
  
  RegexTokenizer()
- Method Summary
  
  All Methods Instance Methods Concrete Methods
  Modifier and Type Method Description
  
  CharSequence[] tokenize(CharSequence text)
  Returns an array of tokens.
  - Methods inherited from class java.lang.Object
    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - RegexTokenizer
```
RegexTokenizer()
```
- Method Detail
  - tokenize
```
public CharSequence[] tokenize(CharSequence text)
```
    Returns an array of tokens.
    
    Specified by:
    
    tokenize in interface Tokenizer<CharSequence>
    
    Parameters:
    
    text - input text
    
    Returns:
    
    array of tokens
    
    Throws:
    
    IllegalArgumentException - if the input text is blank