io.debezium.relational.ddl

Class DdlTokenizer

java.lang.Object

io.debezium.relational.ddl.DdlTokenizer

All Implemented Interfaces:

TokenStream.Tokenizer

public class DdlTokenizer
extends Object
implements TokenStream.Tokenizer

A TokenStream.Tokenizer that is specialized for tokenizing DDL streams.

Author:

Randall Hauch, Horia Chiorean, Barry LaFond, Jure Kajzer

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	DdlTokenizer.TokenTypeFunction

Field Summary

Fields

Modifier and Type	Field and Description
static int	COMMENT The token type for tokens that consist of all the characters between "/*" and "*/", between "//" and the next line terminator (e.g., '\n', '\r' or "\r\n"), between "#" and the next line terminator (e.g., '\n', '\r' or "\r\n"), or between "--" and the next line terminator (e.g., '\n', '\r' or "\r\n").
static int	DECIMAL The token type for tokens that consist of an individual '.' character.
static int	DOUBLE_QUOTED_STRING The token type for tokens that consist of all the characters within double-quotes.
static int	KEYWORD The token type for tokens that represent key words or reserved words for a given DDL dialect.
private boolean	removeQuotes
private DdlTokenizer.TokenTypeFunction	retypingFunction
static int	SINGLE_QUOTED_STRING The token type for tokens that consist of all the characters within single-quotes.
static int	STATEMENT_KEY The token type for tokens that represent the start of a DDL statement.
static int	STATEMENT_TERMINATOR The token type for tokens that represent the end of a DDL statement.
static int	SYMBOL The token type for tokens that consist of an individual "symbol" character.
private boolean	useComments
static int	WORD The token type for tokens that represent an unquoted string containing a character sequence made up of non-whitespace and non-symbol characters.

Constructor Summary

Constructors

Constructor and Description
DdlTokenizer(boolean useComments)
DdlTokenizer(boolean useComments, DdlTokenizer.TokenTypeFunction retypingFunction)

Method Summary

Modifier and Type	Method and Description
protected TokenStream.Tokens	adapt(TokenStream.CharacterStream input, TokenStream.Tokens output)
boolean	includeComments()
void	tokenize(TokenStream.CharacterStream input, TokenStream.Tokens tokens) Process the supplied characters and construct the appropriate TokenStream.Token objects.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WORD

public static final int WORD

The token type for tokens that represent an unquoted string containing a character sequence made up of non-whitespace and non-symbol characters.

See Also:

Constant Field Values

SYMBOL

public static final int SYMBOL

The token type for tokens that consist of an individual "symbol" character. The set of characters includes: -(){}*,;+%?$[]!<>|=:

See Also:

Constant Field Values

DECIMAL

public static final int DECIMAL

The token type for tokens that consist of an individual '.' character.

See Also:

Constant Field Values

SINGLE_QUOTED_STRING

public static final int SINGLE_QUOTED_STRING

The token type for tokens that consist of all the characters within single-quotes. Single quote characters are included if they are preceded (escaped) by a '\' character.

See Also:

Constant Field Values

DOUBLE_QUOTED_STRING

public static final int DOUBLE_QUOTED_STRING

The token type for tokens that consist of all the characters within double-quotes. Double quote characters are included if they are preceded (escaped) by a '\' character.

See Also:

Constant Field Values

COMMENT

public static final int COMMENT

The token type for tokens that consist of all the characters between "/*" and "*/", between "//" and the next line terminator (e.g., '\n', '\r' or "\r\n"), between "#" and the next line terminator (e.g., '\n', '\r' or "\r\n"), or between "--" and the next line terminator (e.g., '\n', '\r' or "\r\n").

See Also:

Constant Field Values

KEYWORD

public static final int KEYWORD

The token type for tokens that represent key words or reserved words for a given DDL dialect.

Examples would be: "CREATE", "TABLE", "ALTER", "SCHEMA", "DROP", etc...

See Also:

Constant Field Values

STATEMENT_KEY

public static final int STATEMENT_KEY

The token type for tokens that represent the start of a DDL statement.

Examples would be: {"CREATE", "TABLE"} {"CREATE", "OR", "REPLACE", "VIEW"}

See Also:

Constant Field Values

STATEMENT_TERMINATOR

public static final int STATEMENT_TERMINATOR

The token type for tokens that represent the end of a DDL statement.

See Also:

Constant Field Values

removeQuotes

private final boolean removeQuotes

See Also:

Constant Field Values

useComments

private final boolean useComments

retypingFunction

private final DdlTokenizer.TokenTypeFunction retypingFunction

Constructor Detail

DdlTokenizer

public DdlTokenizer(boolean useComments)

DdlTokenizer

public DdlTokenizer(boolean useComments,
                    DdlTokenizer.TokenTypeFunction retypingFunction)

Method Detail

includeComments

public boolean includeComments()

Returns:

useComments

adapt

protected TokenStream.Tokens adapt(TokenStream.CharacterStream input,
                                   TokenStream.Tokens output)

tokenize

public void tokenize(TokenStream.CharacterStream input,
                     TokenStream.Tokens tokens)
              throws ParsingException

Description copied from interface: TokenStream.Tokenizer

Process the supplied characters and construct the appropriate TokenStream.Token objects.

Specified by:

tokenize in interface TokenStream.Tokenizer

Parameters:

input - the character input stream; never null

tokens - the factory for TokenStream.Token objects, which records the order in which the tokens are created

Throws:

ParsingException - if there is an error while processing the character stream (e.g., a quote is not closed, etc.)