Tokenizer (org.apache.directory.api_api-all_sources-and-javadoc 2.1.1 API)

Type Parameters:

T - given type

All Known Implementing Classes:

RegexTokenizer
```
interface Tokenizer<T>
```
A tokenizer. Can produce arrays of tokens from a given type.

Since:

1.0

- Method Summary
  
  All Methods Instance Methods Abstract Methods
  Modifier and Type Method Description
  
  T[] tokenize(CharSequence text)
  Returns an array of tokens.

- Method Detail
  - tokenize
```
T[] tokenize(CharSequence text)
```
    Returns an array of tokens.
    
    Parameters:
    
    text - input text
    
    Returns:
    
    array of tokens