- All Known Subinterfaces:
ICommentState, INumberState, IQuoteState, ISymbolState, IWhitespaceState, IWordState
- All Known Implementing Classes:
CCommentState, CppCommentState, CsvQuoteState, CsvSymbolState, CsvWordState, ExpressionNumberState, ExpressionQuoteState, ExpressionSymbolState, ExpressionWordState, GenericCommentState, GenericNumberState, GenericQuoteState, GenericSymbolState, GenericWhitespaceState, GenericWordState, MustacheSpecialState
public interface ITokenizerState
A tokenizerState returns a token, given a scanner, an initial character read from the scanner,
and a tokenizer that is conducting an overall tokenization of the scanner. The tokenizer will
typically have a character state table that decides which state to use, depending on an initial
character. If a single character is insufficient, a state such as SlashState
will read a second character, and may delegate to another state, such as SlashStarState.
This prospect of delegation is the reason that the nextToken()
method has a tokenizer argument.