io.debezium.relational.ddl

Class DdlParser

java.lang.Object

io.debezium.relational.ddl.DdlParser

Direct Known Subclasses:

DdlParserSql2003

@NotThreadSafe
public class DdlParser
extends Object

A parser for DDL statements.

Author:

Randall Hauch, Horia Chiorean, Barry LaFond

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static interface	DdlParser.TokenSet

Field Summary

Fields

Modifier and Type	Field and Description
private String	currentSchema
protected Tables	databaseTables
protected DataTypeParser	dataTypeParser
private Set<String>	keywords
private List<DdlParserListener>	listeners
protected org.slf4j.Logger	logger
protected boolean	skipViews
private Set<String>	statementStarts
private String	terminator
protected TokenStream	tokens

Constructor Summary

Constructors

Constructor and Description
DdlParser(String terminator) Create a new parser that uses the supplied DataTypeParser, but that does not include view definitions.
DdlParser(String terminator, boolean includeViews) Create a new parser that uses the supplied DataTypeParser.

Method Summary

Modifier and Type	Method and Description
void	addListener(DdlParserListener listener) Add a listener.
protected boolean	canConsumeJoin(TokenStream.Marker start)
protected boolean	canConsumeJoinCondition(TokenStream.Marker start)
protected String	consumeDoubleQuotedString() Consume the next token that is a double-quoted string.
protected String	consumeQuotedString() Consume the next token that is either a single-quoted string or a double-quoted string.
protected void	consumeRemainingStatement(TokenStream.Marker start) Consume all tokens from the current position until and including either the end-of-statement terminator token or one of the tokens that is registered as the start of a statement.
protected String	consumeSingleQuotedString() Consume the next token that is a single-quoted string.
protected void	consumeStatement() Consume all tokens from the current position that is a starting-statement token until either the end-of-statement terminator token or before the next starting-statement token.
protected Column	createColumnFromConstant(String columnName, String constantValue)
String	currentSchema() Get the name of the current schema.
protected void	debugParsed(TokenStream.Marker statementStart)
protected void	debugSkipped(TokenStream.Marker statementStart)
protected int	determineTokenType(int type, String token)
protected String	determineTypeNameForConstant(BigDecimal value)
protected String	determineTypeNameForConstant(double value)
protected String	determineTypeNameForConstant(float value)
protected String	determineTypeNameForConstant(long value)
protected void	initializeDataTypes(DataTypeParser dataTypeParser)
protected void	initializeKeywords(DdlParser.TokenSet keywords)
protected void	initializeStatementStarts(DdlParser.TokenSet statementStartTokens)
void	parse(String ddlContent, Tables databaseTables) Examine the supplied string containing DDL statements, and apply those statements to the specified database table definitions.
void	parse(TokenStream ddlContent, Tables databaseTables) Examine the stream starting at its current position for DDL statements, and apply those statements to the specified database table definitions.
private void	parseAliasedTableInFrom(TokenStream.Marker start, Map<String,Table> tablesByAlias) Parse a potentially qualified table name along with an optional alias.
protected void	parseAlter(TokenStream.Marker marker) Parse the a DDL "ALTER" statement.
protected String	parseCharacterLiteral(TokenStream.Marker start)
protected String	parseCharacterSetName(TokenStream.Marker start)
protected void	parseColumnName(TokenStream.Marker start, Map<String,String> tableAliasByColumnAliases, Map<String,String> columnNameByAliases) Parse the potentially qualified and aliased column information, and add the information to the supplied maps.
protected Map<String,Column>	parseColumnsInSelectClause(TokenStream.Marker start) Parse the column information in the SELECT clause.
protected void	parseComment(TokenStream.Marker marker) Parse the a DDL line comment.
protected void	parseCreate(TokenStream.Marker marker) Parse the a DDL "CREATE" statement.
protected String	parseDateLiteral(TokenStream.Marker start)
protected void	parseDrop(TokenStream.Marker marker) Parse the a DDL "DROP" statement.
protected Object	parseLiteral(TokenStream.Marker start)
protected void	parseNextStatement(TokenStream.Marker marker) Parse the next DDL statement.
protected Object	parseNumericLiteral(TokenStream.Marker start, boolean signed)
protected TableId	parseQualifiedTableName(TokenStream.Marker start) Parse the next tokens for a possibly qualified table name.
protected List<TableId>	parseQualifiedTableNames(TokenStream.Marker start) Parse the next tokens for one or more comma-separated qualified table names.
protected String	parseSchemaQualifiedName(TokenStream.Marker start) Parse the next tokens for a possibly qualified name.
protected Map<String,Table>	parseSelectFromClause(TokenStream.Marker start) Returns the tables keyed by their aliases that appear in a SELECT clause's "FROM" list.
protected String	parseTimeLiteral(TokenStream.Marker start)
protected String	parseTimestampLiteral(TokenStream.Marker start)
protected void	parseUnknownStatement(TokenStream.Marker marker) Parse a DDL statement that is not known by the parseNextStatement(io.debezium.text.TokenStream.Marker).
protected void	parsingFailed(Position position, Collection<ParsingException> errors, String msg) Generate a ParsingException or MultipleParsingExceptions with the supplied error or errors and the supplied message, which is appended by this method with additional information about the position's line and column.
protected void	parsingFailed(Position position, String msg) Generate a ParsingException with the supplied message, which is appended by this method with additional information about the position's line and column.
protected void	parsingFailed(Position position, String msg, Throwable t) Generate a ParsingException with the supplied message, which is appended by this method with additional information about the position's line and column.
private String	removeLineFeeds(String input)
boolean	removeListener(DdlParserListener listener) Remove an existing listener.
void	removeListeners() Remove all existing listeners.
protected TableId	resolveTableId(String schemaName, String tableName) Create a TableId from the supplied schema and table names.
void	setCurrentSchema(String name) Set the name of the current schema used when resolving TableIds.
protected void	setTypeInfoForConstant(String constantValue, ColumnEditor column)
protected void	signalAlterDatabase(String databaseName, String previousDatabaseName, TokenStream.Marker statementStart) Signal an alter database event to all listeners.
protected void	signalAlterTable(TableId id, TableId previousId, String statement) Signal an alter table event to all listeners.
protected void	signalAlterTable(TableId id, TableId previousId, TokenStream.Marker statementStart) Signal an alter table event to all listeners.
protected void	signalAlterView(TableId id, TableId previousId, TokenStream.Marker statementStart) Signal an alter view event to all listeners.
protected void	signalCreateDatabase(String databaseName, TokenStream.Marker statementStart) Signal a create database event to all listeners.
protected void	signalCreateIndex(String indexName, TableId id, TokenStream.Marker statementStart) Signal a create index event to all listeners.
protected void	signalCreateTable(TableId id, TokenStream.Marker statementStart) Signal a create table event to all listeners.
protected void	signalCreateView(TableId id, TokenStream.Marker statementStart) Signal a create view event to all listeners.
protected void	signalDropDatabase(String databaseName, TokenStream.Marker statementStart) Signal a drop database event to all listeners.
protected void	signalDropIndex(String indexName, TableId id, TokenStream.Marker statementStart) Signal a drop index event to all listeners.
protected void	signalDropTable(TableId id, String statement) Signal a drop table event to all listeners.
protected void	signalDropTable(TableId id, TokenStream.Marker statementStart) Signal a drop table event to all listeners.
protected void	signalDropView(TableId id, String statement) Signal a drop view event to all listeners.
protected void	signalDropView(TableId id, TokenStream.Marker statementStart) Signal a drop view event to all listeners.
protected void	signalEvent(DdlParserListener.Event event) Signal an event to all listeners.
protected boolean	skipComments() Determine whether parsing should exclude comments from the token stream.
protected String	statement(TokenStream.Marker statementStart)
String	terminator() The token used to terminate a DDL statement.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

keywords

private final Set<String> keywords

statementStarts

private final Set<String> statementStarts

terminator

private final String terminator

currentSchema

private String currentSchema

skipViews

protected final boolean skipViews

logger

protected final org.slf4j.Logger logger

dataTypeParser

protected final DataTypeParser dataTypeParser

databaseTables

protected Tables databaseTables

tokens

protected TokenStream tokens

listeners

private final List<DdlParserListener> listeners

Constructor Detail

DdlParser

public DdlParser(String terminator)

Create a new parser that uses the supplied DataTypeParser, but that does not include view definitions.

Parameters:

terminator - the terminator character sequence; may be null if the default terminator (;) should be used

DdlParser

public DdlParser(String terminator,
                 boolean includeViews)

Create a new parser that uses the supplied DataTypeParser.

Parameters:

terminator - the terminator character sequence; may be null if the default terminator (;) should be used

includeViews - true if view definitions should be included, or false if they should be skipped

Method Detail

addListener

public void addListener(DdlParserListener listener)

Add a listener. This method should not be called more than once with the same listener object, since the result will be that object will be called multiple times for each event.

Parameters:

listener - the listener; if null nothing is done

removeListener

public boolean removeListener(DdlParserListener listener)

Remove an existing listener.

Parameters:

listener - the listener; if null nothing is done

Returns:

true if the listener was removed, or false otherwise

removeListeners

public void removeListeners()

Remove all existing listeners.

initializeDataTypes

protected void initializeDataTypes(DataTypeParser dataTypeParser)

initializeKeywords

protected void initializeKeywords(DdlParser.TokenSet keywords)

initializeStatementStarts

protected void initializeStatementStarts(DdlParser.TokenSet statementStartTokens)

terminator

public final String terminator()

The token used to terminate a DDL statement.

Returns:

the terminating token; never null

determineTokenType

protected int determineTokenType(int type,
                                 String token)

setCurrentSchema

public void setCurrentSchema(String name)

Set the name of the current schema used when resolving TableIds.

Parameters:

name - the name of the current schema; may be null

currentSchema

public String currentSchema()

Get the name of the current schema.

Returns:

the current schema name, or null if the current schema name has not been set

parseSchemaQualifiedName

protected String parseSchemaQualifiedName(TokenStream.Marker start)

Parse the next tokens for a possibly qualified name. This method builds up a string containing the schema name (or if none is found the the current schema), a '.' delimiter, and the object name. If no schema name is found, just the object name is returned.

Parameters:

start - the start of the statement

Returns:

the qualified schema name.

parseQualifiedTableName

protected TableId parseQualifiedTableName(TokenStream.Marker start)

Parse the next tokens for a possibly qualified table name. This method uses the schema name that appears in the token stream, or if none is found the currentSchema(), and then calls resolveTableId(String, String) with the values.

Parameters:

start - the start of the statement

Returns:

the resolved TableId

parseQualifiedTableNames

protected List<TableId> parseQualifiedTableNames(TokenStream.Marker start)

Parse the next tokens for one or more comma-separated qualified table names. This method uses the schema name that appears in the token stream, or if none is found the currentSchema(), and then calls resolveTableId(String, String) with the values.

Parameters:

start - the start of the statement

Returns:

the resolved TableId

resolveTableId

protected TableId resolveTableId(String schemaName,
                                 String tableName)

Create a TableId from the supplied schema and table names. By default, this method uses the supplied schema name as the TableId's catalog, which often matches the catalog name in JDBC database metadata.

Parameters:

schemaName - the name of the schema; may be null if not specified

tableName - the name of the table; should not be null

Returns:

the table identifier; never null

skipComments

protected boolean skipComments()

Determine whether parsing should exclude comments from the token stream. By default, this method returns true.

Returns:

true if comments should be skipped/excluded, or false if they should not be skipped

parse

public final void parse(String ddlContent,
                        Tables databaseTables)

Examine the supplied string containing DDL statements, and apply those statements to the specified database table definitions.

Parameters:

ddlContent - the stream of tokens containing the DDL statements; may not be null

databaseTables - the database's table definitions, which should be used by this method to create, change, or remove tables as defined in the DDL content; may not be null

Throws:

ParsingException - if there is a problem parsing the supplied content

parse

public final void parse(TokenStream ddlContent,
                        Tables databaseTables)
                 throws ParsingException,
                        IllegalStateException

Examine the stream starting at its current position for DDL statements, and apply those statements to the specified database table definitions.

Parameters:

ddlContent - the stream of tokens containing the DDL statements; may not be null

databaseTables - the database's table definitions, which should be used by this method to create, change, or remove tables as defined in the DDL content; may not be null

Throws:

ParsingException - if there is a problem parsing the supplied content

IllegalStateException - if the supplied token stream is in an invalid state

parseNextStatement

protected void parseNextStatement(TokenStream.Marker marker)

Parse the next DDL statement. This is the primary entry point for subclasses.

Parameters:

marker - the start of the statement; never null

Throws:

ParsingException - if there is an error parsing the statement

parseComment

protected void parseComment(TokenStream.Marker marker)

Parse the a DDL line comment. This is generally called by parseNextStatement(io.debezium.text.TokenStream.Marker) for line comments that appear between other DDL statements, and is not typically called for comments that appear within DDL statements.

Parameters:

marker - the start of the statement; never null

Throws:

ParsingException - if there is an error parsing the statement

parseCreate

protected void parseCreate(TokenStream.Marker marker)

Parse the a DDL "CREATE" statement. This method is intended to be overridden by subclasses.

By default this method simply consumes the complete statement.

Parameters:

marker - the start of the statement; never null

Throws:

ParsingException - if there is an error parsing the statement

parseAlter

protected void parseAlter(TokenStream.Marker marker)

Parse the a DDL "ALTER" statement. This method is intended to be overridden by subclasses.

By default this method simply consumes the complete statement.

Parameters:

marker - the start of the statement; never null

Throws:

ParsingException - if there is an error parsing the statement

parseDrop

protected void parseDrop(TokenStream.Marker marker)

Parse the a DDL "DROP" statement. This method is intended to be overridden by subclasses.

By default this method simply consumes the complete statement.

Parameters:

marker - the start of the statement; never null

Throws:

ParsingException - if there is an error parsing the statement

parseUnknownStatement

protected void parseUnknownStatement(TokenStream.Marker marker)

Parse a DDL statement that is not known by the parseNextStatement(io.debezium.text.TokenStream.Marker). This method can be overridden by subclasses, although it will be more common for subclasses to override parseNextStatement(io.debezium.text.TokenStream.Marker).

By default this method simply consumes the complete statement.

Parameters:

marker - the start of the statement; never null

Throws:

ParsingException - if there is an error parsing the statement

signalEvent

protected void signalEvent(DdlParserListener.Event event)

Signal an event to all listeners.

Parameters:

event - the event; may not be null

signalCreateDatabase

protected void signalCreateDatabase(String databaseName,
                                    TokenStream.Marker statementStart)

Signal a create database event to all listeners.

Parameters:

databaseName - the database name; may not be null

statementStart - the start of the statement; may not be null

signalAlterDatabase

protected void signalAlterDatabase(String databaseName,
                                   String previousDatabaseName,
                                   TokenStream.Marker statementStart)

Signal an alter database event to all listeners.

Parameters:

databaseName - the database name; may not be null

previousDatabaseName - the previous name of the database if it was renamed, or null if it was not renamed

statementStart - the start of the statement; may not be null

signalDropDatabase

protected void signalDropDatabase(String databaseName,
                                  TokenStream.Marker statementStart)

Signal a drop database event to all listeners.

Parameters:

databaseName - the database name; may not be null

statementStart - the start of the statement; may not be null

signalCreateTable

protected void signalCreateTable(TableId id,
                                 TokenStream.Marker statementStart)

Signal a create table event to all listeners.

Parameters:

id - the table identifier; may not be null

statementStart - the start of the statement; may not be null

signalAlterTable

protected void signalAlterTable(TableId id,
                                TableId previousId,
                                TokenStream.Marker statementStart)

Signal an alter table event to all listeners.

Parameters:

id - the table identifier; may not be null

previousId - the previous name of the view if it was renamed, or null if it was not renamed

statementStart - the start of the statement; may not be null

signalAlterTable

protected void signalAlterTable(TableId id,
                                TableId previousId,
                                String statement)

Signal an alter table event to all listeners.

Parameters:

id - the table identifier; may not be null

previousId - the previous name of the view if it was renamed, or null if it was not renamed

statement - the DDL statement; may not be null

signalDropTable

protected void signalDropTable(TableId id,
                               TokenStream.Marker statementStart)

Signal a drop table event to all listeners.

Parameters:

id - the table identifier; may not be null

statementStart - the start of the statement; may not be null

signalDropTable

protected void signalDropTable(TableId id,
                               String statement)

Signal a drop table event to all listeners.

Parameters:

id - the table identifier; may not be null

statement - the statement; may not be null

signalCreateView

protected void signalCreateView(TableId id,
                                TokenStream.Marker statementStart)

Signal a create view event to all listeners.

Parameters:

id - the table identifier; may not be null

statementStart - the start of the statement; may not be null

signalAlterView

protected void signalAlterView(TableId id,
                               TableId previousId,
                               TokenStream.Marker statementStart)

Signal an alter view event to all listeners.

Parameters:

id - the table identifier; may not be null

previousId - the previous name of the view if it was renamed, or null if it was not renamed

statementStart - the start of the statement; may not be null

signalDropView

protected void signalDropView(TableId id,
                              TokenStream.Marker statementStart)

Signal a drop view event to all listeners.

Parameters:

id - the table identifier; may not be null

statementStart - the start of the statement; may not be null

signalDropView

protected void signalDropView(TableId id,
                              String statement)

Signal a drop view event to all listeners.

Parameters:

id - the table identifier; may not be null

statement - the statement; may not be null

signalCreateIndex

protected void signalCreateIndex(String indexName,
                                 TableId id,
                                 TokenStream.Marker statementStart)

Signal a create index event to all listeners.

Parameters:

indexName - the name of the index; may not be null

id - the table identifier; may be null if the index does not apply to a single table

statementStart - the start of the statement; may not be null

signalDropIndex

protected void signalDropIndex(String indexName,
                               TableId id,
                               TokenStream.Marker statementStart)

Signal a drop index event to all listeners.

Parameters:

indexName - the name of the index; may not be null

id - the table identifier; may not be null

statementStart - the start of the statement; may not be null

debugParsed

protected void debugParsed(TokenStream.Marker statementStart)

debugSkipped

protected void debugSkipped(TokenStream.Marker statementStart)

statement

protected String statement(TokenStream.Marker statementStart)

removeLineFeeds

private String removeLineFeeds(String input)

consumeStatement

protected void consumeStatement()
                         throws ParsingException

Consume all tokens from the current position that is a starting-statement token until either the end-of-statement terminator token or before the next starting-statement token.

Throws:

ParsingException - if the next token is not a starting-statement token

consumeRemainingStatement

protected void consumeRemainingStatement(TokenStream.Marker start)

Consume all tokens from the current position until and including either the end-of-statement terminator token or one of the tokens that is registered as the start of a statement.

Parameters:

start - the marker at which the statement was begun

consumeSingleQuotedString

protected String consumeSingleQuotedString()

Consume the next token that is a single-quoted string.

Returns:

the quoted string; never null

Throws:

ParsingException - if there is no single-quoted string at the current position

consumeDoubleQuotedString

protected String consumeDoubleQuotedString()

Consume the next token that is a double-quoted string.

Returns:

the quoted string; never null

Throws:

ParsingException - if there is no double-quoted string at the current position

consumeQuotedString

protected String consumeQuotedString()

Consume the next token that is either a single-quoted string or a double-quoted string.

Returns:

the quoted string; never null

Throws:

ParsingException - if there is no single- or double-quoted string at the current position

parsingFailed

protected void parsingFailed(Position position,
                             String msg)

Generate a ParsingException with the supplied message, which is appended by this method with additional information about the position's line and column.

Parameters:

position - the position at which the error occurred; may not be null

msg - the leading portion of the message; may not be null

parsingFailed

protected void parsingFailed(Position position,
                             String msg,
                             Throwable t)

Generate a ParsingException with the supplied message, which is appended by this method with additional information about the position's line and column.

Parameters:

position - the position at which the error occurred; may not be null

msg - the leading portion of the message; may not be null

t - the exception that occurred; may be null

parsingFailed

protected void parsingFailed(Position position,
                             Collection<ParsingException> errors,
                             String msg)

Generate a ParsingException or MultipleParsingExceptions with the supplied error or errors and the supplied message, which is appended by this method with additional information about the position's line and column.

Parameters:

position - the position at which the error occurred; may not be null

errors - the multiple parsing exception errors; may not be null

msg - the leading portion of the message; may not be null

parseLiteral

protected Object parseLiteral(TokenStream.Marker start)

parseNumericLiteral

protected Object parseNumericLiteral(TokenStream.Marker start,
                                     boolean signed)

parseCharacterLiteral

protected String parseCharacterLiteral(TokenStream.Marker start)

parseCharacterSetName

protected String parseCharacterSetName(TokenStream.Marker start)

parseDateLiteral

protected String parseDateLiteral(TokenStream.Marker start)

parseTimeLiteral

protected String parseTimeLiteral(TokenStream.Marker start)

parseTimestampLiteral

protected String parseTimestampLiteral(TokenStream.Marker start)

parseColumnsInSelectClause

protected Map<String,Column> parseColumnsInSelectClause(TokenStream.Marker start)

Parse the column information in the SELECT clause. This statement stops before consuming the FROM clause.

Parameters:

start - the start of the statement

Returns:

the map of resolved Columns keyed by the column alias (or name) used in the SELECT statement; never null but possibly empty if we couldn't parse the SELECT clause correctly

createColumnFromConstant

protected Column createColumnFromConstant(String columnName,
                                          String constantValue)

setTypeInfoForConstant

protected void setTypeInfoForConstant(String constantValue,
                                      ColumnEditor column)

determineTypeNameForConstant

protected String determineTypeNameForConstant(long value)

determineTypeNameForConstant

protected String determineTypeNameForConstant(float value)

determineTypeNameForConstant

protected String determineTypeNameForConstant(double value)

determineTypeNameForConstant

protected String determineTypeNameForConstant(BigDecimal value)

parseColumnName

protected void parseColumnName(TokenStream.Marker start,
                               Map<String,String> tableAliasByColumnAliases,
                               Map<String,String> columnNameByAliases)

Parse the potentially qualified and aliased column information, and add the information to the supplied maps.

Parameters:

start - the start of the statement

tableAliasByColumnAliases - the map to which is added the column's alias (or name) keyed by the alias of the table in which the column should appear; may not be null

columnNameByAliases - the map to which is added the column's name keyed by the its alias (or itself if there is no alias); may not be null

parseSelectFromClause

protected Map<String,Table> parseSelectFromClause(TokenStream.Marker start)

Returns the tables keyed by their aliases that appear in a SELECT clause's "FROM" list. This method handles the various standard joins.

Parameters:

start - the start of the statement

Returns:

the map of resolved tables keyed by the alias (or table name) used in the SELECT statement; never null but possibly empty if we couldn't parse the from clause correctly

canConsumeJoin

protected boolean canConsumeJoin(TokenStream.Marker start)

canConsumeJoinCondition

protected boolean canConsumeJoinCondition(TokenStream.Marker start)

parseAliasedTableInFrom

private void parseAliasedTableInFrom(TokenStream.Marker start,
                                     Map<String,Table> tablesByAlias)

Parse a potentially qualified table name along with an optional alias.

Parameters:

start - the start of the statement

tablesByAlias - the map to which this method should add the table keyed by its alias (or name if there is no alias); may not be null