Class AbstractTokenizer
- java.lang.Object
-
- org.pipservices3.expressions.tokenizers.AbstractTokenizer
-
- All Implemented Interfaces:
ITokenizer
- Direct Known Subclasses:
CsvTokenizer,ExpressionTokenizer,GenericTokenizer,MustacheTokenizer
public abstract class AbstractTokenizer extends Object implements ITokenizer
Implements an abstract tokenizer class.
-
-
Field Summary
Fields Modifier and Type Field Description protected TokenType_lastTokenTypeprotected Token_nextTokenprotected IScanner_scanner
-
Constructor Summary
Constructors Constructor Description AbstractTokenizer()
-
Method Summary
Modifier and Type Method Description voidclearCharacterStates()ITokenizerStategetCharacterState(int symbol)ICommentStategetCommentState()A token state to process comments.BooleangetDecodeStrings()Decodes quoted strings.BooleangetMergeWhitespaces()Merges whitespaces.INumberStategetNumberState()A token state to process numbers.IQuoteStategetQuoteState()A token state to process quoted strings.IScannergetScanner()The stream scanner to tokenize.BooleangetSkipComments()Skips comments.BooleangetSkipEof()Skips End-Of-File token at the end of stream.BooleangetSkipUnknown()Skip unknown charactersBooleangetSkipWhitespaces()Skips whitespaces.ISymbolStategetSymbolState()A token state to process symbols (single like "=" or muti-character like "<>")BooleangetUnifyNumbers()Unifies numbers: "Integers" and "Floats" makes just "Numbers"IWhitespaceStategetWhitespaceState()A token state to process white space delimiters.IWordStategetWordState()A token state to process words or indentificators.BooleanhasNextToken()Checks if there is the next token exist.TokennextToken()Gets the next token from the scanner.protected TokenreadNextToken()voidsetCharacterState(int fromSymbol, int toSymbol, ITokenizerState state)voidsetCommentState(ICommentState value)A token state to process comments.voidsetDecodeStrings(boolean value)Decodes quoted strings.voidsetMergeWhitespaces(boolean value)Merges whitespaces.voidsetNumberState(INumberState value)A token state to process numbers.voidsetQuoteState(IQuoteState value)A token state to process quoted strings.voidsetScanner(IScanner value)The stream scanner to tokenize.voidsetSkipComments(boolean value)Skips comments.voidsetSkipEof(boolean value)Skips End-Of-File token at the end of stream.voidsetSkipUnknown(boolean value)Skip unknown charactersvoidsetSkipWhitespaces(boolean value)Skips whitespaces.voidsetSymbolState(ISymbolState value)A token state to process symbols (single like "=" or muti-character like "<>")voidsetUnifyNumbers(boolean value)Unifies numbers: "Integers" and "Floats" makes just "Numbers"voidsetWhitespaceState(IWhitespaceState value)A token state to process white space delimiters.voidsetWordState(IWordState value)A token state to process words or indentificators.List<Token>tokenizeBuffer(String buffer)Tokenizes a string buffer into a list of tokens structures.List<String>tokenizeBufferToStrings(String buffer)Tokenizes a string buffer into a list of strings.List<Token>tokenizeStream(IScanner scanner)Tokenizes a textual stream into a list of token structures.List<String>tokenizeStreamToStrings(IScanner scanner)Tokenizes a textual stream into a list of strings.
-
-
-
Method Detail
-
getSkipUnknown
public Boolean getSkipUnknown()
Description copied from interface:ITokenizerSkip unknown characters- Specified by:
getSkipUnknownin interfaceITokenizer
-
setSkipUnknown
public void setSkipUnknown(boolean value)
Description copied from interface:ITokenizerSkip unknown characters- Specified by:
setSkipUnknownin interfaceITokenizer
-
getSkipWhitespaces
public Boolean getSkipWhitespaces()
Description copied from interface:ITokenizerSkips whitespaces.- Specified by:
getSkipWhitespacesin interfaceITokenizer
-
setSkipWhitespaces
public void setSkipWhitespaces(boolean value)
Description copied from interface:ITokenizerSkips whitespaces.- Specified by:
setSkipWhitespacesin interfaceITokenizer
-
getSkipComments
public Boolean getSkipComments()
Description copied from interface:ITokenizerSkips comments.- Specified by:
getSkipCommentsin interfaceITokenizer
-
setSkipComments
public void setSkipComments(boolean value)
Description copied from interface:ITokenizerSkips comments.- Specified by:
setSkipCommentsin interfaceITokenizer
-
getSkipEof
public Boolean getSkipEof()
Description copied from interface:ITokenizerSkips End-Of-File token at the end of stream.- Specified by:
getSkipEofin interfaceITokenizer
-
setSkipEof
public void setSkipEof(boolean value)
Description copied from interface:ITokenizerSkips End-Of-File token at the end of stream.- Specified by:
setSkipEofin interfaceITokenizer
-
getMergeWhitespaces
public Boolean getMergeWhitespaces()
Description copied from interface:ITokenizerMerges whitespaces.- Specified by:
getMergeWhitespacesin interfaceITokenizer
-
setMergeWhitespaces
public void setMergeWhitespaces(boolean value)
Description copied from interface:ITokenizerMerges whitespaces.- Specified by:
setMergeWhitespacesin interfaceITokenizer
-
getUnifyNumbers
public Boolean getUnifyNumbers()
Description copied from interface:ITokenizerUnifies numbers: "Integers" and "Floats" makes just "Numbers"- Specified by:
getUnifyNumbersin interfaceITokenizer
-
setUnifyNumbers
public void setUnifyNumbers(boolean value)
Description copied from interface:ITokenizerUnifies numbers: "Integers" and "Floats" makes just "Numbers"- Specified by:
setUnifyNumbersin interfaceITokenizer
-
getDecodeStrings
public Boolean getDecodeStrings()
Description copied from interface:ITokenizerDecodes quoted strings.- Specified by:
getDecodeStringsin interfaceITokenizer
-
setDecodeStrings
public void setDecodeStrings(boolean value)
Description copied from interface:ITokenizerDecodes quoted strings.- Specified by:
setDecodeStringsin interfaceITokenizer
-
getCommentState
public ICommentState getCommentState()
Description copied from interface:ITokenizerA token state to process comments.- Specified by:
getCommentStatein interfaceITokenizer
-
setCommentState
public void setCommentState(ICommentState value)
Description copied from interface:ITokenizerA token state to process comments.- Specified by:
setCommentStatein interfaceITokenizer
-
getNumberState
public INumberState getNumberState()
Description copied from interface:ITokenizerA token state to process numbers.- Specified by:
getNumberStatein interfaceITokenizer
-
setNumberState
public void setNumberState(INumberState value)
Description copied from interface:ITokenizerA token state to process numbers.- Specified by:
setNumberStatein interfaceITokenizer
-
getQuoteState
public IQuoteState getQuoteState()
Description copied from interface:ITokenizerA token state to process quoted strings.- Specified by:
getQuoteStatein interfaceITokenizer
-
setQuoteState
public void setQuoteState(IQuoteState value)
Description copied from interface:ITokenizerA token state to process quoted strings.- Specified by:
setQuoteStatein interfaceITokenizer
-
getSymbolState
public ISymbolState getSymbolState()
Description copied from interface:ITokenizerA token state to process symbols (single like "=" or muti-character like "<>")- Specified by:
getSymbolStatein interfaceITokenizer
-
setSymbolState
public void setSymbolState(ISymbolState value)
Description copied from interface:ITokenizerA token state to process symbols (single like "=" or muti-character like "<>")- Specified by:
setSymbolStatein interfaceITokenizer
-
getWhitespaceState
public IWhitespaceState getWhitespaceState()
Description copied from interface:ITokenizerA token state to process white space delimiters.- Specified by:
getWhitespaceStatein interfaceITokenizer
-
setWhitespaceState
public void setWhitespaceState(IWhitespaceState value)
Description copied from interface:ITokenizerA token state to process white space delimiters.- Specified by:
setWhitespaceStatein interfaceITokenizer
-
getWordState
public IWordState getWordState()
Description copied from interface:ITokenizerA token state to process words or indentificators.- Specified by:
getWordStatein interfaceITokenizer
-
setWordState
public void setWordState(IWordState value)
Description copied from interface:ITokenizerA token state to process words or indentificators.- Specified by:
setWordStatein interfaceITokenizer
-
getScanner
public IScanner getScanner()
Description copied from interface:ITokenizerThe stream scanner to tokenize.- Specified by:
getScannerin interfaceITokenizer
-
setScanner
public void setScanner(IScanner value)
Description copied from interface:ITokenizerThe stream scanner to tokenize.- Specified by:
setScannerin interfaceITokenizer
-
getCharacterState
public ITokenizerState getCharacterState(int symbol)
-
setCharacterState
public void setCharacterState(int fromSymbol, int toSymbol, ITokenizerState state) throws Exception- Throws:
Exception
-
clearCharacterStates
public void clearCharacterStates()
-
hasNextToken
public Boolean hasNextToken() throws Exception
Description copied from interface:ITokenizerChecks if there is the next token exist.- Specified by:
hasNextTokenin interfaceITokenizer- Returns:
trueif scanner has the next token.- Throws:
Exception
-
nextToken
public Token nextToken() throws Exception
Description copied from interface:ITokenizerGets the next token from the scanner.- Specified by:
nextTokenin interfaceITokenizer- Returns:
- Next token of
nullif there are no more tokens left. - Throws:
Exception
-
tokenizeStream
public List<Token> tokenizeStream(IScanner scanner) throws Exception
Description copied from interface:ITokenizerTokenizes a textual stream into a list of token structures.- Specified by:
tokenizeStreamin interfaceITokenizer- Parameters:
scanner- A textual stream to be tokenized.- Returns:
- A list of token structures.
- Throws:
Exception
-
tokenizeBuffer
public List<Token> tokenizeBuffer(String buffer) throws Exception
Description copied from interface:ITokenizerTokenizes a string buffer into a list of tokens structures.- Specified by:
tokenizeBufferin interfaceITokenizer- Parameters:
buffer- A string buffer to be tokenized.- Returns:
- A list of token structures.
- Throws:
Exception
-
tokenizeStreamToStrings
public List<String> tokenizeStreamToStrings(IScanner scanner) throws Exception
Description copied from interface:ITokenizerTokenizes a textual stream into a list of strings.- Specified by:
tokenizeStreamToStringsin interfaceITokenizer- Parameters:
scanner- A textual stream to be tokenized.- Returns:
- A list of token strings.
- Throws:
Exception
-
tokenizeBufferToStrings
public List<String> tokenizeBufferToStrings(String buffer) throws Exception
Description copied from interface:ITokenizerTokenizes a string buffer into a list of strings.- Specified by:
tokenizeBufferToStringsin interfaceITokenizer- Parameters:
buffer- A string buffer to be tokenized.- Returns:
- A list of token strings.
- Throws:
Exception
-
-