io.debezium.connector.mysql

Class SnapshotReader

java.lang.Object

io.debezium.connector.mysql.AbstractReader

io.debezium.connector.mysql.SnapshotReader

All Implemented Interfaces:

Reader

public class SnapshotReader
extends AbstractReader

A component that performs a snapshot of a MySQL server, and records the schema changes in MySqlSchema.

Author:

Randall Hauch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static interface	SnapshotReader.RecordRecorder

Nested classes/interfaces inherited from class io.debezium.connector.mysql.AbstractReader

AbstractReader.AcceptAllPredicate

Nested classes/interfaces inherited from interface io.debezium.connector.mysql.Reader

Reader.State

Field Summary

Fields

Modifier and Type	Field and Description
private ExecutorService	executorService
private boolean	includeData
private SnapshotReaderMetrics	metrics
private SnapshotReader.RecordRecorder	recorder
private MySqlConnectorConfig.SnapshotLockingMode	snapshotLockingMode
private boolean	useGlobalLock

Fields inherited from class io.debezium.connector.mysql.AbstractReader

changeEventQueueMetrics, connectionContext, context, logger

Constructor Summary

Constructors

Constructor and Description
SnapshotReader(String name, MySqlTaskContext context) Create a snapshot reader.
SnapshotReader(String name, MySqlTaskContext context, boolean useGlobalLock) Create a snapshot reader that can use global locking onyl optionally.

Method Summary

Modifier and Type	Method and Description
private Statement	createStatement(Connection connection)
private Statement	createStatementWithLargeResultSet(Connection connection) Create a JDBC statement that can be used for large result sets.
protected void	doCleanup() The reader has completed all processing and all enqueued records have been consumed, so this reader should clean up any resources that might remain.
void	doDestroy() The reader has been requested to de-initialize resources after stopping.
protected void	doInitialize() The reader has been requested to initialize resources prior to starting.
protected void	doStart() Start the snapshot and return immediately.
protected void	doStop() The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started.
protected void	enqueueSchemaChanges(String dbName, Set<TableId> tables, String ddlStatement)
protected void	execute() Perform the snapshot using the same logic as the "mysqldump" utility.
SnapshotReader	generateInsertEvents() Set this reader's execution to produce a Envelope.Operation.CREATE event for each row.
SnapshotReader	generateReadEvents() Set this reader's execution to produce a Envelope.Operation.READ event for each row.
private Filters	getCreateTableFilters(Filters filters) Get the filters for table creation.
private void	logRolesForCurrentUser(JdbcConnection mysql)
private void	logServerInformation(JdbcConnection mysql)
protected String	quote(String dbOrTableName)
protected String	quote(TableId id)
protected void	readBinlogPosition(int step, SourceInfo source, JdbcConnection mysql, AtomicReference<String> sql)
private Object	readDateField(ResultSet rs, int fieldNo, Column column, Table table) In non-string mode the date field can contain zero in any of the date part which we need to handle as all-zero
protected Object	readField(ResultSet rs, int fieldNo, Column actualColumn, Table actualTable)
private void	readTableSchema(AtomicReference<String> sql, JdbcConnection mysql, MySqlSchema schema, SourceInfo source, String dbName, TableId tableId)
private Object	readTimeField(ResultSet rs, int fieldNo) As MySQL connector/J implementation is broken for MySQL type "TIME" we have to use a binary-ish workaround
private Object	readTimestampField(ResultSet rs, int fieldNo, Column column, Table table) In non-string mode the time field can contain zero in any of the date part which we need to handle as all-zero
protected void	recordRowAsInsert(RecordMakers.RecordsForTable recordMaker, Object[] row, Instant ts)
protected void	recordRowAsRead(RecordMakers.RecordsForTable recordMaker, Object[] row, Instant ts)
protected org.apache.kafka.connect.source.SourceRecord	replaceOffsetAndSource(org.apache.kafka.connect.source.SourceRecord record) Utility method to replace the offset and the source in the given record with the latest.
private boolean	shouldRecordTableSchema(MySqlSchema schema, Filters filters, TableId id)

Methods inherited from class io.debezium.connector.mysql.AbstractReader

cleanupResources, completeSuccessfully, destroy, enqueueRecord, failed, failed, initialize, isRunning, name, poll, pollComplete, start, state, stop, toString, uponCompletion, wrap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

includeData

private final boolean includeData

recorder

private SnapshotReader.RecordRecorder recorder

metrics

private final SnapshotReaderMetrics metrics

executorService

private ExecutorService executorService

useGlobalLock

private final boolean useGlobalLock

snapshotLockingMode

private final MySqlConnectorConfig.SnapshotLockingMode snapshotLockingMode

Constructor Detail

SnapshotReader

public SnapshotReader(String name,
                      MySqlTaskContext context)

Create a snapshot reader.

Parameters:

name - the name of this reader; may not be null

context - the task context in which this reader is running; may not be null

SnapshotReader

SnapshotReader(String name,
               MySqlTaskContext context,
               boolean useGlobalLock)

Create a snapshot reader that can use global locking onyl optionally. Used mostly for testing.

Parameters:

name - the name of this reader; may not be null

context - the task context in which this reader is running; may not be null

useGlobalLock - false to simulate cloud (Amazon RDS) restrictions

Method Detail

generateReadEvents

public SnapshotReader generateReadEvents()

Set this reader's execution to produce a Envelope.Operation.READ event for each row.

Returns:

this object for method chaining; never null

generateInsertEvents

public SnapshotReader generateInsertEvents()

Set this reader's execution to produce a Envelope.Operation.CREATE event for each row.

Returns:

this object for method chaining; never null

doInitialize

protected void doInitialize()

Description copied from class: AbstractReader

The reader has been requested to initialize resources prior to starting. This should only be called once before AbstractReader.doStart().

Overrides:

doInitialize in class AbstractReader

doDestroy

public void doDestroy()

Description copied from class: AbstractReader

The reader has been requested to de-initialize resources after stopping. This should only be called once after AbstractReader.doStop().

Overrides:

doDestroy in class AbstractReader

doStart

protected void doStart()

Start the snapshot and return immediately. Once started, the records read from the database can be retrieved using AbstractReader.poll() until that method returns null.

Specified by:

doStart in class AbstractReader

doStop

protected void doStop()

Description copied from class: AbstractReader

The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started.

This method is always called when AbstractReader.stop() is called, and the first time AbstractReader.isRunning() will return true the first time and false for any subsequent calls.

Specified by:

doStop in class AbstractReader

doCleanup

protected void doCleanup()

Description copied from class: AbstractReader

The reader has completed all processing and all enqueued records have been consumed, so this reader should clean up any resources that might remain.

Specified by:

doCleanup in class AbstractReader

readField

protected Object readField(ResultSet rs,
                           int fieldNo,
                           Column actualColumn,
                           Table actualTable)
                    throws SQLException

Throws:

SQLException

readTimeField

private Object readTimeField(ResultSet rs,
                             int fieldNo)
                      throws SQLException

As MySQL connector/J implementation is broken for MySQL type "TIME" we have to use a binary-ish workaround

Throws:

SQLException

See Also:

https://issues.jboss.org/browse/DBZ-342

readDateField

private Object readDateField(ResultSet rs,
                             int fieldNo,
                             Column column,
                             Table table)
                      throws SQLException

In non-string mode the date field can contain zero in any of the date part which we need to handle as all-zero

Throws:

SQLException

readTimestampField

private Object readTimestampField(ResultSet rs,
                                  int fieldNo,
                                  Column column,
                                  Table table)
                           throws SQLException

In non-string mode the time field can contain zero in any of the date part which we need to handle as all-zero

Throws:

SQLException

execute

protected void execute()

Perform the snapshot using the same logic as the "mysqldump" utility.

readTableSchema

private void readTableSchema(AtomicReference<String> sql,
                             JdbcConnection mysql,
                             MySqlSchema schema,
                             SourceInfo source,
                             String dbName,
                             TableId tableId)
                      throws SQLException

Throws:

SQLException

shouldRecordTableSchema

private boolean shouldRecordTableSchema(MySqlSchema schema,
                                        Filters filters,
                                        TableId id)

readBinlogPosition

protected void readBinlogPosition(int step,
                                  SourceInfo source,
                                  JdbcConnection mysql,
                                  AtomicReference<String> sql)
                           throws SQLException

Throws:

SQLException

getCreateTableFilters

private Filters getCreateTableFilters(Filters filters)

Get the filters for table creation. Depending on the configuration, this may not be the default filter set.

Parameters:

filters - the default filters of this SnapshotReader

Returns:

Filters that represent all the tables that this snapshot reader should CREATE

quote

protected String quote(String dbOrTableName)

quote

protected String quote(TableId id)

createStatementWithLargeResultSet

private Statement createStatementWithLargeResultSet(Connection connection)
                                             throws SQLException

Create a JDBC statement that can be used for large result sets.

By default, the MySQL Connector/J driver retrieves all rows for ResultSets and stores them in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. However, when ResultSets that have a large number of rows or large values, the driver may not be able to allocate heap space in the JVM and may result in an OutOfMemoryError. See DBZ-94 for details.

This method handles such cases using the recommended technique for MySQL by creating the JDBC Statement with forward-only cursor and read-only concurrency flags, and with a minimum value fetch size hint.

Parameters:

connection - the JDBC connection; may not be null

Returns:

the statement; never null

Throws:

SQLException - if there is a problem creating the statement

createStatement

private Statement createStatement(Connection connection)
                           throws SQLException

Throws:

SQLException

logServerInformation

private void logServerInformation(JdbcConnection mysql)

logRolesForCurrentUser

private void logRolesForCurrentUser(JdbcConnection mysql)

replaceOffsetAndSource

protected org.apache.kafka.connect.source.SourceRecord replaceOffsetAndSource(org.apache.kafka.connect.source.SourceRecord record)

Utility method to replace the offset and the source in the given record with the latest. This is used on the last record produced during the snapshot.

Parameters:

record - the record

Returns:

the updated record

enqueueSchemaChanges

protected void enqueueSchemaChanges(String dbName,
                                    Set<TableId> tables,
                                    String ddlStatement)

recordRowAsRead

protected void recordRowAsRead(RecordMakers.RecordsForTable recordMaker,
                               Object[] row,
                               Instant ts)
                        throws InterruptedException

Throws:

InterruptedException

recordRowAsInsert

protected void recordRowAsInsert(RecordMakers.RecordsForTable recordMaker,
                                 Object[] row,
                                 Instant ts)
                          throws InterruptedException

Throws:

InterruptedException