Package org.pipservices3.expressions.tokenizers

Interface Summary

Interface	Description
ICommentState	Defines an interface for tokenizer state that processes comments.
INumberState	Defines interface for tokenizer state that processes numbers - Integers, Floats, HexDec..
IQuoteState	Defines an interface for tokenizer state that processes quoted strings.
ISymbolState	Defines an interface for tokenizer state that processes delimiters.
ITokenizer
ITokenizerState	A tokenizerState returns a token, given a scanner, an initial character read from the scanner, and a tokenizer that is conducting an overall tokenization of the scanner.
IWhitespaceState	Defines an interface for tokenizer state that processes whitespaces (' ', '\t')
IWordState	Defines an interface for tokenizer state that processes words, identificators or keywords

Class Summary

Class	Description
AbstractTokenizer	Implements an abstract tokenizer class.
Token	A token represents a logical chunk of a string.

Enum Summary

Enum	Description
TokenType	Types (categories) of tokens such as "number", "symbol" or "word".