Package org.pipservices3.expressions.tokenizers

Interface ITokenizer

    • Method Detail

      • getSkipUnknown

        Boolean getSkipUnknown()
        Skip unknown characters

      • setSkipUnknown

        void setSkipUnknown​(boolean value)
        Skip unknown characters

      • getSkipWhitespaces

        Boolean getSkipWhitespaces()
        Skips whitespaces.

      • setSkipWhitespaces

        void setSkipWhitespaces​(boolean value)
        Skips whitespaces.

      • getSkipComments

        Boolean getSkipComments()
        Skips comments.

      • setSkipComments

        void setSkipComments​(boolean value)
        Skips comments.

      • getSkipEof

        Boolean getSkipEof()
        Skips End-Of-File token at the end of stream.

      • setSkipEof

        void setSkipEof​(boolean value)
        Skips End-Of-File token at the end of stream.

      • getMergeWhitespaces

        Boolean getMergeWhitespaces()
        Merges whitespaces.

      • setMergeWhitespaces

        void setMergeWhitespaces​(boolean value)
        Merges whitespaces.

      • getUnifyNumbers

        Boolean getUnifyNumbers()
        Unifies numbers: "Integers" and "Floats" makes just "Numbers"

      • setUnifyNumbers

        void setUnifyNumbers​(boolean value)
        Unifies numbers: "Integers" and "Floats" makes just "Numbers"

      • getDecodeStrings

        Boolean getDecodeStrings()
        Decodes quoted strings.

      • setDecodeStrings

        void setDecodeStrings​(boolean value)
        Decodes quoted strings.

      • getCommentState

        ICommentState getCommentState()
        A token state to process comments.

      • setCommentState

        void setCommentState​(ICommentState value)
        A token state to process comments.

      • getNumberState

        INumberState getNumberState()
        A token state to process numbers.

      • setNumberState

        void setNumberState​(INumberState value)
        A token state to process numbers.

      • getQuoteState

        IQuoteState getQuoteState()
        A token state to process quoted strings.

      • setQuoteState

        void setQuoteState​(IQuoteState value)
        A token state to process quoted strings.

      • getSymbolState

        ISymbolState getSymbolState()
        A token state to process symbols (single like "=" or muti-character like "<>")

      • setSymbolState

        void setSymbolState​(ISymbolState value)
        A token state to process symbols (single like "=" or muti-character like "<>")

      • getWhitespaceState

        IWhitespaceState getWhitespaceState()
        A token state to process white space delimiters.

      • setWhitespaceState

        void setWhitespaceState​(IWhitespaceState value)
        A token state to process white space delimiters.

      • getWordState

        IWordState getWordState()
        A token state to process words or indentificators.

      • setWordState

        void setWordState​(IWordState value)
        A token state to process words or indentificators.

      • getScanner

        IScanner getScanner()
        The stream scanner to tokenize.

      • setScanner

        void setScanner​(IScanner scanner)
        The stream scanner to tokenize.

      • hasNextToken

        Boolean hasNextToken()
              throws Exception
        Checks if there is the next token exist.
        Returns:
        true if scanner has the next token.
        Throws:
        Exception

      • nextToken

        Token nextToken()
         throws Exception
        Gets the next token from the scanner.
        Returns:
        Next token of null if there are no more tokens left.
        Throws:
        Exception

      • tokenizeStream

        List<Token> tokenizeStream​(IScanner scanner)
                    throws Exception
        Tokenizes a textual stream into a list of token structures.
        Parameters:
        scanner - A textual stream to be tokenized.
        Returns:
        A list of token structures.
        Throws:
        Exception

      • tokenizeBuffer

        List<Token> tokenizeBuffer​(String buffer)
                    throws Exception
        Tokenizes a string buffer into a list of tokens structures.
        Parameters:
        buffer - A string buffer to be tokenized.
        Returns:
        A list of token structures.
        Throws:
        Exception

      • tokenizeStreamToStrings

        List<String> tokenizeStreamToStrings​(IScanner scanner)
                              throws Exception
        Tokenizes a textual stream into a list of strings.
        Parameters:
        scanner - A textual stream to be tokenized.
        Returns:
        A list of token strings.
        Throws:
        Exception

      • tokenizeBufferToStrings

        List<String> tokenizeBufferToStrings​(String buffer)
                              throws Exception
        Tokenizes a string buffer into a list of strings.
        Parameters:
        buffer - A string buffer to be tokenized.
        Returns:
        A list of token strings.
        Throws:
        Exception