io.debezium.connector.mysql

Class MySqlSchema

java.lang.Object

io.debezium.connector.mysql.MySqlSchema

@NotThreadSafe
public class MySqlSchema
extends Object

Component that records the schema history for databases hosted by a MySQL database server. The schema information includes the table definitions and the Kafka Connect Schemas for each table, where the Schema excludes any columns that have been specified in the configuration.

The history is changed by applying DDL statements, and every change is persisted as defined in the supplied MySQL connector configuration. This component can be reconstructed (e.g., on connector restart) and the history loaded from persisted storage.

Note that when applying DDL statements, the caller is able to supply a consumer function that will be called with the DDL statements and the database to which they apply, grouped by database names. However, these will only be called based when the databases are included by the database filters defined in the MySQL connector configuration.

Author:

Randall Hauch

Field Summary

Fields

Modifier and Type	Field and Description
private DatabaseHistory	dbHistory
private DdlChanges	ddlChanges
private MySqlDdlParser	ddlParser
private Filters	filters
private HistoryRecordComparator	historyComparator
private Set<String>	ignoredQueryStatements
private org.slf4j.Logger	logger
private TableSchemaBuilder	schemaBuilder
private AvroValidator	schemaNameValidator
private String	schemaPrefix
private String	serverName
private Tables	tables
private Map<TableId,TableSchema>	tableSchemaByTableId

Constructor Summary

Constructors

Constructor and Description
MySqlSchema(Configuration config, String serverName, Predicate<String> gtidFilter) Create a schema component given the supplied MySQL connector configuration.

Method Summary

Modifier and Type	Method and Description
protected void	appendCreateTableStatement(StringBuilder sb, Table table)
protected void	appendDropTableStatement(StringBuilder sb, TableId tableId)
boolean	applyDdl(SourceInfo source, String databaseName, String ddlStatements, DdlChanges.DatabaseStatementStringConsumer statementConsumer) Apply the supplied DDL statements to this database schema and record the history.
protected void	changeTablesAndRecordInHistory(SourceInfo source, Callable<Void> changeFunction) Apply the given function to change or alter the current set of table definitions, and record the new state of the table definitions in the database history by dropping tables that were removed and dropping and re-creating tables that were changed.
Filters	filters() Get the database and table filters defined by the configuration.
protected HistoryRecordComparator	historyComparator()
String	historyLocation() Get the information about where the DDL statement history is recorded.
void	loadFromDatabase(JdbcConnection jdbc, SourceInfo source) Load the schema for the databases using JDBC database metadata.
void	loadHistory(SourceInfo startingPoint) Load the database schema information using the previously-recorded history, and stop reading the history when the the history reaches the supplied starting point.
protected void	refreshSchemas() Discard any currently-cached schemas and rebuild them using the filters.
TableSchema	schemaFor(TableId id) Get the Schema information for the table with the given identifier, if that table exists and is included by the filter.
void	setSystemVariables(Map<String,String> variables) Set the system variables on the DDL parser.
void	shutdown() Stop recording history and release any resources acquired since start().
void	start() Start by acquiring resources needed to persist the database history
MySqlSystemVariables	systemVariables() Get the system variables as known by the DDL parser.
Table	tableFor(TableId id) Get the Schema information for the table with the given identifier, if that table exists and is included by the filter.
Tables	tables() Get all of the table definitions for all database tables as defined by applied DDL statements, including those that have been excluded by the filters.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

private final org.slf4j.Logger logger

schemaNameValidator

private final AvroValidator schemaNameValidator

ignoredQueryStatements

private final Set<String> ignoredQueryStatements

ddlParser

private final MySqlDdlParser ddlParser

tableSchemaByTableId

private final Map<TableId,TableSchema> tableSchemaByTableId

filters

private final Filters filters

dbHistory

private final DatabaseHistory dbHistory

schemaBuilder

private final TableSchemaBuilder schemaBuilder

ddlChanges

private final DdlChanges ddlChanges

serverName

private final String serverName

schemaPrefix

private final String schemaPrefix

historyComparator

private final HistoryRecordComparator historyComparator

tables

private Tables tables

Constructor Detail

MySqlSchema

public MySqlSchema(Configuration config,
                   String serverName,
                   Predicate<String> gtidFilter)

Create a schema component given the supplied MySQL connector configuration.

Parameters:

config - the connector configuration, which is presumed to be valid

serverName - the name of the server

gtidFilter - the predicate function that should be applied to GTID sets in database history, and which returns true if a GTID source is to be included, or false if a GTID source is to be excluded; may be null if not needed

Method Detail

historyComparator

protected HistoryRecordComparator historyComparator()

start

public void start()

Start by acquiring resources needed to persist the database history

shutdown

public void shutdown()

Stop recording history and release any resources acquired since start().

filters

public Filters filters()

Get the database and table filters defined by the configuration.

Returns:

the filters; never null

tables

public Tables tables()

Get all of the table definitions for all database tables as defined by applied DDL statements, including those that have been excluded by the filters.

Returns:

the table definitions; never null

tableFor

public Table tableFor(TableId id)

Get the Schema information for the table with the given identifier, if that table exists and is included by the filter.

Parameters:

id - the fully-qualified table identifier; may be null

Returns:

the current table definition, or null if there is no table with the given identifier, if the identifier is null, or if the table has been excluded by the filters

schemaFor

public TableSchema schemaFor(TableId id)

Get the Schema information for the table with the given identifier, if that table exists and is included by the filter.

Note that the Schema will not contain any columns that have been filtered out.

Parameters:

id - the fully-qualified table identifier; may be null

Returns:

the schema information, or null if there is no table with the given identifier, if the identifier is null, or if the table has been excluded by the filters

historyLocation

public String historyLocation()

Get the information about where the DDL statement history is recorded.

Returns:

the history description; never null

setSystemVariables

public void setSystemVariables(Map<String,String> variables)

Set the system variables on the DDL parser.

Parameters:

variables - the system variables; may not be null but may be empty

systemVariables

public MySqlSystemVariables systemVariables()

Get the system variables as known by the DDL parser.

Returns:

the system variables; never null

loadFromDatabase

public void loadFromDatabase(JdbcConnection jdbc,
                             SourceInfo source)
                      throws SQLException

Load the schema for the databases using JDBC database metadata. If there are changes relative to any table definitions that existed when this method is called, those changes are recorded in the database history and the schemas for the affected tables are updated.

Parameters:

jdbc - the JDBC connection; may not be null

source - the source information that should be recorded with the history; may not be null

Throws:

SQLException - if there is a failure reading the JDBC database metadata

changeTablesAndRecordInHistory

protected void changeTablesAndRecordInHistory(SourceInfo source,
                                              Callable<Void> changeFunction)
                                       throws SQLException

Apply the given function to change or alter the current set of table definitions, and record the new state of the table definitions in the database history by dropping tables that were removed and dropping and re-creating tables that were changed.

This method is written this way so that the complex logic can be easily tested without actually requiring a database.

Parameters:

source - the source information that should be recorded with the history; may not be null

changeFunction - the function that changes the table definitions and returns true if at least one table definition was modified in some way; may not be null

Throws:

SQLException - if there is a failure reading the JDBC database metadata

appendDropTableStatement

protected void appendDropTableStatement(StringBuilder sb,
                                        TableId tableId)

appendCreateTableStatement

protected void appendCreateTableStatement(StringBuilder sb,
                                          Table table)

loadHistory

public void loadHistory(SourceInfo startingPoint)

Load the database schema information using the previously-recorded history, and stop reading the history when the the history reaches the supplied starting point.

Parameters:

startingPoint - the source information with the current SourceInfo.partition() and offset at which the database schemas are to reflect; may not be null

refreshSchemas

protected void refreshSchemas()

Discard any currently-cached schemas and rebuild them using the filters.

applyDdl

public boolean applyDdl(SourceInfo source,
                        String databaseName,
                        String ddlStatements,
                        DdlChanges.DatabaseStatementStringConsumer statementConsumer)

Apply the supplied DDL statements to this database schema and record the history. If a statementConsumer is supplied, then call it for each sub-sequence of the DDL statements that all apply to the same database.

Typically DDL statements are applied using a connection to a single database, and unless the statements use fully-qualified names, the DDL statements apply to this database.

Parameters:

source - the current SourceInfo.partition() and offset at which these changes are found; may not be null

databaseName - the name of the default database under which these statements are applied; may not be null

ddlStatements - the ;-separated DDL statements; may be null or empty

statementConsumer - the consumer that should be called with each sub-sequence of DDL statements that apply to a single database; may be null if no action is to be performed with the changes

Returns:

true if changes were made to the database schema, or false if the DDL statements had no effect on the database schema