io.debezium.connector.mysql

Class MySqlSnapshotChangeEventSource

java.lang.Object

io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

io.debezium.relational.RelationalSnapshotChangeEventSource

io.debezium.connector.mysql.MySqlSnapshotChangeEventSource

All Implemented Interfaces:

ChangeEventSource, SnapshotChangeEventSource

public class MySqlSnapshotChangeEventSource
extends RelationalSnapshotChangeEventSource

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	MySqlSnapshotChangeEventSource.MySqlSnapshotContext Mutable context which is populated in the course of snapshotting.

Nested classes/interfaces inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

RelationalSnapshotChangeEventSource.RelationalSnapshotContext

Nested classes/interfaces inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

AbstractSnapshotChangeEventSource.SnapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field and Description
private MySqlConnection	connection
private MySqlConnectorConfig	connectorConfig
private MySqlDatabaseSchema	databaseSchema
private Set<TableId>	delayedSchemaSnapshotTables
private RelationalTableFilters	filters
private long	globalLockAcquiredAt
private BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>>	lastEventProcessor
private static org.slf4j.Logger	LOGGER
private MySqlSnapshotChangeEventSourceMetrics	metrics
private MySqlOffsetContext	previousOffset
private List<SchemaChangeEvent>	schemaEvents
private long	tableLockAcquiredAt

Fields inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

clock, dispatcher

Constructor Summary

Constructors

Constructor and Description
MySqlSnapshotChangeEventSource(MySqlConnectorConfig connectorConfig, MySqlOffsetContext previousOffset, MySqlConnection connection, MySqlDatabaseSchema schema, EventDispatcher<TableId> dispatcher, Clock clock, MySqlSnapshotChangeEventSourceMetrics metrics, BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor)

Method Summary

Modifier and Type	Method and Description
private void	addSchemaEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, String database, String ddl)
protected void	complete(AbstractSnapshotChangeEventSource.SnapshotContext snapshotContext)
protected void	connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
protected void	createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
(package private) void	createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, Collection<TableId> tablesToRead, boolean firstPhase)
private Statement	createStatementWithLargeResultSet() Create a JDBC statement that can be used for large result sets.
protected void	determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext ctx)
protected Set<TableId>	getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext ctx)
protected Object	getColumnValue(ResultSet rs, int columnIndex, Column column, Table table)
protected SchemaChangeEvent	getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, Table table)
protected Optional<String>	getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, TableId tableId) Generate a valid sqlserver query string for the specified table
protected AbstractSnapshotChangeEventSource.SnapshottingTask	getSnapshottingTask(OffsetContext previousOffset)
private void	globalLock()
private void	globalUnlock()
private boolean	isGloballyLocked()
private boolean	isTablesLocked()
protected void	lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
protected void	lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
protected void	postSnapshot()
protected AbstractSnapshotChangeEventSource.SnapshotContext	prepare(ChangeEventSource.ChangeEventSourceContext context)
private String	quote(String dbOrTableName)
private String	quote(TableId id)
private Object	readDateField(ResultSet rs, int fieldNo, Column column, Table table) In non-string mode the date field can contain zero in any of the date part which we need to handle as all-zero
protected Statement	readTableStatement(OptionalLong rowCount)
protected void	readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
private Object	readTimeField(ResultSet rs, int fieldNo) As MySQL connector/J implementation is broken for MySQL type "TIME" we have to use a binary-ish workaround
private Object	readTimestampField(ResultSet rs, int fieldNo, Column column, Table table) In non-string mode the time field can contain zero in any of the date part which we need to handle as all-zero
protected void	releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
protected void	releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
protected OptionalLong	rowCountForTable(TableId tableId)
private void	tableLock(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
private void	tableUnlock()
private boolean	twoPhaseSchemaSnapshot()

Methods inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

createSnapshotConnection, doExecute, enhanceOverriddenSelect, getChangeRecordEmitter, getClock, getColumnValue, tryStartingSnapshot

Methods inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

delaySnapshotIfNeeded, determineDataCollectionsToBeSnapshotted, execute

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

connectorConfig

private final MySqlConnectorConfig connectorConfig

connection

private final MySqlConnection connection

globalLockAcquiredAt

private long globalLockAcquiredAt

tableLockAcquiredAt

private long tableLockAcquiredAt

filters

private final RelationalTableFilters filters

metrics

private final MySqlSnapshotChangeEventSourceMetrics metrics

previousOffset

private final MySqlOffsetContext previousOffset

databaseSchema

private final MySqlDatabaseSchema databaseSchema

schemaEvents

private final List<SchemaChangeEvent> schemaEvents

delayedSchemaSnapshotTables

private Set<TableId> delayedSchemaSnapshotTables

lastEventProcessor

private final BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor

Constructor Detail

MySqlSnapshotChangeEventSource

public MySqlSnapshotChangeEventSource(MySqlConnectorConfig connectorConfig,
                                      MySqlOffsetContext previousOffset,
                                      MySqlConnection connection,
                                      MySqlDatabaseSchema schema,
                                      EventDispatcher<TableId> dispatcher,
                                      Clock clock,
                                      MySqlSnapshotChangeEventSourceMetrics metrics,
                                      BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor)

Method Detail

getSnapshottingTask

protected AbstractSnapshotChangeEventSource.SnapshottingTask getSnapshottingTask(OffsetContext previousOffset)

Specified by:

getSnapshottingTask in class AbstractSnapshotChangeEventSource

prepare

protected AbstractSnapshotChangeEventSource.SnapshotContext prepare(ChangeEventSource.ChangeEventSourceContext context)
                                                             throws Exception

Specified by:

prepare in class AbstractSnapshotChangeEventSource

Throws:

Exception

connectionCreated

protected void connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                          throws Exception

Overrides:

connectionCreated in class RelationalSnapshotChangeEventSource

Throws:

Exception

getAllTableIds

protected Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext ctx)
                               throws Exception

Specified by:

getAllTableIds in class RelationalSnapshotChangeEventSource

Throws:

Exception

lockTablesForSchemaSnapshot

protected void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                           RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                                    throws SQLException,
                                           InterruptedException

Specified by:

lockTablesForSchemaSnapshot in class RelationalSnapshotChangeEventSource

Throws:

SQLException

InterruptedException

releaseSchemaSnapshotLocks

protected void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                                   throws SQLException

Specified by:

releaseSchemaSnapshotLocks in class RelationalSnapshotChangeEventSource

Throws:

SQLException

releaseDataSnapshotLocks

protected void releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                                 throws Exception

Overrides:

releaseDataSnapshotLocks in class RelationalSnapshotChangeEventSource

Throws:

Exception

determineSnapshotOffset

protected void determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext ctx)
                                throws Exception

Specified by:

determineSnapshotOffset in class RelationalSnapshotChangeEventSource

Throws:

Exception

addSchemaEvent

private void addSchemaEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                            String database,
                            String ddl)

readTableStructure

protected void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                  RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                           throws Exception

Specified by:

readTableStructure in class RelationalSnapshotChangeEventSource

Throws:

Exception

createSchemaEventsForTables

void createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                                 Collection<TableId> tablesToRead,
                                 boolean firstPhase)
                          throws SQLException

Throws:

SQLException

twoPhaseSchemaSnapshot

private boolean twoPhaseSchemaSnapshot()

getCreateTableEvent

protected SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                                                Table table)
                                         throws SQLException

Specified by:

getCreateTableEvent in class RelationalSnapshotChangeEventSource

Throws:

SQLException

complete

protected void complete(AbstractSnapshotChangeEventSource.SnapshotContext snapshotContext)

Overrides:

complete in class AbstractSnapshotChangeEventSource

getSnapshotSelect

protected Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                                             TableId tableId)

Generate a valid sqlserver query string for the specified table

Specified by:

getSnapshotSelect in class RelationalSnapshotChangeEventSource

Parameters:

tableId - the table to generate a query for

Returns:

a valid query string

getColumnValue

protected Object getColumnValue(ResultSet rs,
                                int columnIndex,
                                Column column,
                                Table table)
                         throws SQLException

Overrides:

getColumnValue in class RelationalSnapshotChangeEventSource

Throws:

SQLException

readTimeField

private Object readTimeField(ResultSet rs,
                             int fieldNo)
                      throws SQLException

As MySQL connector/J implementation is broken for MySQL type "TIME" we have to use a binary-ish workaround

Throws:

SQLException

See Also:

https://issues.jboss.org/browse/DBZ-342

readDateField

private Object readDateField(ResultSet rs,
                             int fieldNo,
                             Column column,
                             Table table)
                      throws SQLException

In non-string mode the date field can contain zero in any of the date part which we need to handle as all-zero

Throws:

SQLException

readTimestampField

private Object readTimestampField(ResultSet rs,
                                  int fieldNo,
                                  Column column,
                                  Table table)
                           throws SQLException

In non-string mode the time field can contain zero in any of the date part which we need to handle as all-zero

Throws:

SQLException

isGloballyLocked

private boolean isGloballyLocked()

isTablesLocked

private boolean isTablesLocked()

globalLock

private void globalLock()
                 throws SQLException

Throws:

SQLException

globalUnlock

private void globalUnlock()
                   throws SQLException

Throws:

SQLException

tableLock

private void tableLock(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                throws SQLException

Throws:

SQLException

tableUnlock

private void tableUnlock()
                  throws SQLException

Throws:

SQLException

quote

private String quote(String dbOrTableName)

quote

private String quote(TableId id)

rowCountForTable

protected OptionalLong rowCountForTable(TableId tableId)

Overrides:

rowCountForTable in class RelationalSnapshotChangeEventSource

readTableStatement

protected Statement readTableStatement(OptionalLong rowCount)
                                throws SQLException

Overrides:

readTableStatement in class RelationalSnapshotChangeEventSource

Throws:

SQLException

createStatementWithLargeResultSet

private Statement createStatementWithLargeResultSet()
                                             throws SQLException

Create a JDBC statement that can be used for large result sets.

By default, the MySQL Connector/J driver retrieves all rows for ResultSets and stores them in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. However, when ResultSets that have a large number of rows or large values, the driver may not be able to allocate heap space in the JVM and may result in an OutOfMemoryError. See DBZ-94 for details.

This method handles such cases using the recommended technique for MySQL by creating the JDBC Statement with forward-only cursor and read-only concurrency flags, and with a minimum value fetch size hint.

Returns:

the statement; never null

Throws:

SQLException - if there is a problem creating the statement

createSchemaChangeEventsForTables

protected void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                                 RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                                                 AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
                                          throws Exception

Overrides:

createSchemaChangeEventsForTables in class RelationalSnapshotChangeEventSource

Throws:

Exception

lastSnapshotRecord

protected void lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)

Overrides:

lastSnapshotRecord in class RelationalSnapshotChangeEventSource

postSnapshot

protected void postSnapshot()
                     throws InterruptedException

Overrides:

postSnapshot in class RelationalSnapshotChangeEventSource

Throws:

InterruptedException