public class JapaneseTokenizer extends Tokenizer
bosNode, dictionary, eosNode, unknownCToken, unknownPartOfSpeechDescription| Constructor and Description |
|---|
JapaneseTokenizer(Dictionary dictionary,
String unknownPartOfSpeechDescription)
Creates a JapaneseTokenizer with the given Dictionary
|
| Modifier and Type | Method and Description |
|---|---|
Node |
lookup(SentenceIterator iterator,
char[] surface)
Searches for possible morphemes from the given SentenceIterator.
|
getBOSNode, getDictionary, getEOSNode, getUnknownNodepublic JapaneseTokenizer(Dictionary dictionary, String unknownPartOfSpeechDescription)
dictionary - The Dictionary in which to search for possible morphemesunknownPartOfSpeechDescription - The part-of-speech code to use for unknown tokenspublic Node lookup(SentenceIterator iterator, char[] surface)
TokenizerNode that is returned links through
Node.rnext to a list of matches which may be of varying
lengths