Class GenericTokenizer
- java.lang.Object
-
- org.pipservices3.expressions.tokenizers.AbstractTokenizer
-
- org.pipservices3.expressions.tokenizers.generic.GenericTokenizer
-
- All Implemented Interfaces:
ITokenizer
public class GenericTokenizer extends AbstractTokenizer
Implements a default tokenizer class.
-
-
Field Summary
-
Fields inherited from class org.pipservices3.expressions.tokenizers.AbstractTokenizer
_lastTokenType, _nextToken, _scanner
-
-
Constructor Summary
Constructors Constructor Description GenericTokenizer()
-
Method Summary
-
Methods inherited from class org.pipservices3.expressions.tokenizers.AbstractTokenizer
clearCharacterStates, getCharacterState, getCommentState, getDecodeStrings, getMergeWhitespaces, getNumberState, getQuoteState, getScanner, getSkipComments, getSkipEof, getSkipUnknown, getSkipWhitespaces, getSymbolState, getUnifyNumbers, getWhitespaceState, getWordState, hasNextToken, nextToken, readNextToken, setCharacterState, setCommentState, setDecodeStrings, setMergeWhitespaces, setNumberState, setQuoteState, setScanner, setSkipComments, setSkipEof, setSkipUnknown, setSkipWhitespaces, setSymbolState, setUnifyNumbers, setWhitespaceState, setWordState, tokenizeBuffer, tokenizeBufferToStrings, tokenizeStream, tokenizeStreamToStrings
-
-