io.debezium.connector.mysql.legacy

Class MySqlSchema

java.lang.Object

io.debezium.relational.RelationalDatabaseSchema

io.debezium.connector.mysql.legacy.MySqlSchema

All Implemented Interfaces:

DatabaseSchema<TableId>

@NotThreadSafe
public class MySqlSchema
extends RelationalDatabaseSchema

Component that records the schema history for databases hosted by a MySQL database server. The schema information includes the table definitions and the Kafka Connect Schemas for each table, where the Schema excludes any columns that have been specified in the configuration.

The history is changed by applying DDL statements, and every change is persisted as defined in the supplied MySQL connector configuration. This component can be reconstructed (e.g., on connector restart) and the history loaded from persisted storage.

Note that when applying DDL statements, the caller is able to supply a consumer function that will be called with the DDL statements and the database to which they apply, grouped by database names. However, these will only be called based when the databases are included by the database filters defined in the MySQL connector configuration.

Author:

Randall Hauch

Field Summary

Fields

Modifier and Type	Field and Description
private DatabaseHistory	dbHistory
private DdlChanges	ddlChanges
private DdlParser	ddlParser
private Filters	filters
private HistoryRecordComparator	historyComparator
private Set<String>	ignoredQueryStatements
private static org.slf4j.Logger	logger
private boolean	recoveredTables
private boolean	skipUnparseableDDL
private boolean	storeOnlyMonitoredTablesDdl

Fields inherited from interface io.debezium.schema.DatabaseSchema

NO_CAPTURED_DATA_COLLECTIONS_WARNING

Constructor Summary

Constructors

Constructor and Description
MySqlSchema(MySqlConnectorConfig configuration, Predicate<String> gtidFilter, boolean tableIdCaseInsensitive, TopicSelector<TableId> topicSelector, Filters tableFilters) Create a schema component given the supplied MySQL connector configuration.

Method Summary

Modifier and Type	Method and Description
protected void	appendCreateTableStatement(StringBuilder sb, Table table)
protected void	appendDropTableStatement(StringBuilder sb, TableId tableId)
boolean	applyDdl(SourceInfo source, String databaseName, String ddlStatements, DdlChanges.DatabaseStatementStringConsumer statementConsumer) Apply the supplied DDL statements to this database schema and record the history.
Filters	filters() Get the database and table filters defined by the configuration.
private static MySqlValueConverters	getValueConverters(MySqlConnectorConfig configuration)
HistoryRecordComparator	historyComparator()
boolean	historyExists() Return true if the database history entity exists
String	historyLocation() Get the information about where the DDL statement history is recorded.
void	intializeHistoryStorage() Initialize permanent storage for database history
boolean	isGlobalSetVariableStatement(String ddl, String databaseName)
boolean	isStoreOnlyMonitoredTablesDdl()
boolean	isTableMonitored(TableId id) Decide whether events should be captured for a given table
void	loadHistory(SourceInfo startingPoint) Load the database schema information using the previously-recorded history, and stop reading the history when the the history reaches the supplied starting point.
String[]	monitoredTablesAsStringArray() Get all table names for all databases that are monitored whose events are captured by Debezium
void	refreshSchemas() Discard any currently-cached schemas and rebuild them using the filters.
void	setSystemVariables(Map<String,String> variables) Set the system variables on the DDL parser.
void	shutdown() Stop recording history and release any resources acquired since start().
void	start() Start by acquiring resources needed to persist the database history
SystemVariables	systemVariables() Get the system variables as known by the DDL parser.
boolean	tableInformationComplete()

Methods inherited from class io.debezium.relational.RelationalDatabaseSchema

assureNonEmptySchema, buildAndRegisterSchema, clearSchemas, close, getTableFilter, refresh, refreshSchema, removeSchema, schemaFor, tableFor, tableIds, tables

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

private static final org.slf4j.Logger logger

ignoredQueryStatements

private final Set<String> ignoredQueryStatements

ddlParser

private final DdlParser ddlParser

filters

private final Filters filters

dbHistory

private final DatabaseHistory dbHistory

ddlChanges

private final DdlChanges ddlChanges

historyComparator

private final HistoryRecordComparator historyComparator

skipUnparseableDDL

private final boolean skipUnparseableDDL

storeOnlyMonitoredTablesDdl

private final boolean storeOnlyMonitoredTablesDdl

recoveredTables

private boolean recoveredTables

Constructor Detail

MySqlSchema

public MySqlSchema(MySqlConnectorConfig configuration,
                   Predicate<String> gtidFilter,
                   boolean tableIdCaseInsensitive,
                   TopicSelector<TableId> topicSelector,
                   Filters tableFilters)

Create a schema component given the supplied MySQL connector configuration.

Parameters:

configuration - the connector configuration, which is presumed to be valid

gtidFilter - the predicate function that should be applied to GTID sets in database history, and which returns true if a GTID source is to be included, or false if a GTID source is to be excluded; may be null if not needed

tableIdCaseInsensitive - true if table lookup ignores letter case

Method Detail

getValueConverters

private static MySqlValueConverters getValueConverters(MySqlConnectorConfig configuration)

historyComparator

public HistoryRecordComparator historyComparator()

start

public void start()

Start by acquiring resources needed to persist the database history

shutdown

public void shutdown()

Stop recording history and release any resources acquired since start().

filters

public Filters filters()

Get the database and table filters defined by the configuration.

Returns:

the filters; never null

monitoredTablesAsStringArray

public String[] monitoredTablesAsStringArray()

Get all table names for all databases that are monitored whose events are captured by Debezium

Returns:

the array with the table names

isTableMonitored

public boolean isTableMonitored(TableId id)

Decide whether events should be captured for a given table

Parameters:

id - the fully-qualified table identifier; may be null

Returns:

true if events from the table are captured

historyLocation

public String historyLocation()

Get the information about where the DDL statement history is recorded.

Returns:

the history description; never null

setSystemVariables

public void setSystemVariables(Map<String,String> variables)

Set the system variables on the DDL parser.

Parameters:

variables - the system variables; may not be null but may be empty

systemVariables

public SystemVariables systemVariables()

Get the system variables as known by the DDL parser.

Returns:

the system variables; never null

appendDropTableStatement

protected void appendDropTableStatement(StringBuilder sb,
                                        TableId tableId)

appendCreateTableStatement

protected void appendCreateTableStatement(StringBuilder sb,
                                          Table table)

loadHistory

public void loadHistory(SourceInfo startingPoint)

Load the database schema information using the previously-recorded history, and stop reading the history when the the history reaches the supplied starting point.

Parameters:

startingPoint - the source information with the current SourceInfo.partition() and offset at which the database schemas are to reflect; may not be null

historyExists

public boolean historyExists()

Return true if the database history entity exists

intializeHistoryStorage

public void intializeHistoryStorage()

Initialize permanent storage for database history

refreshSchemas

public void refreshSchemas()

Discard any currently-cached schemas and rebuild them using the filters.

applyDdl

public boolean applyDdl(SourceInfo source,
                        String databaseName,
                        String ddlStatements,
                        DdlChanges.DatabaseStatementStringConsumer statementConsumer)

Apply the supplied DDL statements to this database schema and record the history. If a statementConsumer is supplied, then call it for each sub-sequence of the DDL statements that all apply to the same database.

Typically DDL statements are applied using a connection to a single database, and unless the statements use fully-qualified names, the DDL statements apply to this database.

Parameters:

source - the current SourceInfo.partition() and offset at which these changes are found; may not be null

databaseName - the name of the default database under which these statements are applied; may not be null

ddlStatements - the ;-separated DDL statements; may be null or empty

statementConsumer - the consumer that should be called with each sub-sequence of DDL statements that apply to a single database; may be null if no action is to be performed with the changes

Returns:

true if changes were made to the database schema, or false if the DDL statements had no effect on the database schema

isGlobalSetVariableStatement

public boolean isGlobalSetVariableStatement(String ddl,
                                            String databaseName)

isStoreOnlyMonitoredTablesDdl

public boolean isStoreOnlyMonitoredTablesDdl()

Returns:

true if only monitored tables should be stored in database history, false if all tables should be stored

tableInformationComplete

public boolean tableInformationComplete()

Specified by:

tableInformationComplete in interface DatabaseSchema<TableId>

Overrides:

tableInformationComplete in class RelationalDatabaseSchema