io.debezium.connector.mysql

Class SnapshotReader

java.lang.Object

io.debezium.connector.mysql.AbstractReader

io.debezium.connector.mysql.SnapshotReader

public class SnapshotReader
extends AbstractReader

A component that performs a snapshot of a MySQL server, and records the schema changes in MySqlSchema.

Author:

Randall Hauch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static interface	SnapshotReader.RecordRecorder

Field Summary

Fields

Modifier and Type	Field and Description
private boolean	minimalBlocking
private Runnable	onSuccessfulCompletion
private SnapshotReader.RecordRecorder	recorder
private Thread	thread

Fields inherited from class io.debezium.connector.mysql.AbstractReader

context, logger

Constructor Summary

Constructors

Constructor and Description
SnapshotReader(MySqlTaskContext context) Create a snapshot reader.

Method Summary

Modifier and Type	Method and Description
private Statement	createStatement(Connection connection)
private Statement	createStatementWithLargeResultSet(Connection connection) Create a JDBC statement that can be used for large result sets.
protected void	doCleanup() The reader has completed sending all enqueued records, so clean up any resources that remain.
protected void	doStart() Start the snapshot and return immediately.
protected void	doStop() Stop the snapshot from running.
protected void	enqueueSchemaChanges(String dbName, String ddlStatement)
protected void	execute() Perform the snapshot using the same logic as the "mysqldump" utility.
SnapshotReader	generateInsertEvents() Set this reader's execution to produce a Envelope.Operation.CREATE event for each row.
SnapshotReader	generateReadEvents() Set this reader's execution to produce a Envelope.Operation.READ event for each row.
private void	logRolesForCurrentUser(JdbcConnection mysql)
private void	logServerInformation(JdbcConnection mysql)
SnapshotReader	onSuccessfulCompletion(Runnable onSuccessfulCompletion) Set the non-blocking function that should be called upon successful completion of the snapshot, which is after the snapshot generates its final record and all such records have been polled.
protected void	recordRowAsInsert(RecordMakers.RecordsForTable recordMaker, Object[] row, long ts)
protected void	recordRowAsRead(RecordMakers.RecordsForTable recordMaker, Object[] row, long ts)
protected org.apache.kafka.connect.source.SourceRecord	replaceOffset(org.apache.kafka.connect.source.SourceRecord record) Utility method to replace the offset in the given record with the latest.
SnapshotReader	useMinimalBlocking(boolean minimalBlocking) Set whether this reader's execution should block other transactions as minimally as possible by releasing the read lock as early as possible.

Methods inherited from class io.debezium.connector.mysql.AbstractReader

completeSuccessfully, enqueueRecord, failed, failed, isRunning, poll, start, stop, wrap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

minimalBlocking

private boolean minimalBlocking

recorder

private SnapshotReader.RecordRecorder recorder

thread

private volatile Thread thread

onSuccessfulCompletion

private volatile Runnable onSuccessfulCompletion

Constructor Detail

SnapshotReader

public SnapshotReader(MySqlTaskContext context)

Create a snapshot reader.

Parameters:

context - the task context in which this reader is running; may not be null

Method Detail

onSuccessfulCompletion

public SnapshotReader onSuccessfulCompletion(Runnable onSuccessfulCompletion)

Set the non-blocking function that should be called upon successful completion of the snapshot, which is after the snapshot generates its final record and all such records have been polled.

Parameters:

onSuccessfulCompletion - the function; may be null

Returns:

this object for method chaining; never null

useMinimalBlocking

public SnapshotReader useMinimalBlocking(boolean minimalBlocking)

Set whether this reader's execution should block other transactions as minimally as possible by releasing the read lock as early as possible. Although the snapshot process should obtain a consistent snapshot even when releasing the lock as early as possible, it may be desirable to explicitly hold onto the read lock until execution completes. In such cases, holding onto the lock will prevent all updates to the database during the snapshot process.

Parameters:

minimalBlocking - true if the lock is to be released as early as possible, or false if the lock is to be held for the entire execution

Returns:

this object for method chaining; never null

generateReadEvents

public SnapshotReader generateReadEvents()

Set this reader's execution to produce a Envelope.Operation.READ event for each row.

Returns:

this object for method chaining; never null

generateInsertEvents

public SnapshotReader generateInsertEvents()

Set this reader's execution to produce a Envelope.Operation.CREATE event for each row.

Returns:

this object for method chaining; never null

doStart

protected void doStart()

Start the snapshot and return immediately. Once started, the records read from the database can be retrieved using AbstractReader.poll() until that method returns null.

Specified by:

doStart in class AbstractReader

doStop

protected void doStop()

Stop the snapshot from running.

Specified by:

doStop in class AbstractReader

doCleanup

protected void doCleanup()

Description copied from class: AbstractReader

The reader has completed sending all enqueued records, so clean up any resources that remain.

Specified by:

doCleanup in class AbstractReader

execute

protected void execute()

Perform the snapshot using the same logic as the "mysqldump" utility.

createStatementWithLargeResultSet

private Statement createStatementWithLargeResultSet(Connection connection)
                                             throws SQLException

Create a JDBC statement that can be used for large result sets.

By default, the MySQL Connector/J driver retrieves all rows for ResultSets and stores them in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. However, when ResultSets that have a large number of rows or large values, the driver may not be able to allocate heap space in the JVM and may result in an OutOfMemoryError. See DBZ-94 for details.

This method handles such cases using the recommended technique for MySQL by creating the JDBC Statement with forward-only cursor and read-only concurrency flags, and with a minimum value fetch size hint.

Parameters:

connection - the JDBC connection; may not be null

Returns:

the statement; never null

Throws:

SQLException - if there is a problem creating the statement

createStatement

private Statement createStatement(Connection connection)
                           throws SQLException

Throws:

SQLException

logServerInformation

private void logServerInformation(JdbcConnection mysql)

logRolesForCurrentUser

private void logRolesForCurrentUser(JdbcConnection mysql)

replaceOffset

protected org.apache.kafka.connect.source.SourceRecord replaceOffset(org.apache.kafka.connect.source.SourceRecord record)

Utility method to replace the offset in the given record with the latest. This is used on the last record produced during the snapshot.

Parameters:

record - the record

Returns:

the updated record

enqueueSchemaChanges

protected void enqueueSchemaChanges(String dbName,
                                    String ddlStatement)

recordRowAsRead

protected void recordRowAsRead(RecordMakers.RecordsForTable recordMaker,
                               Object[] row,
                               long ts)
                        throws InterruptedException

Throws:

InterruptedException

recordRowAsInsert

protected void recordRowAsInsert(RecordMakers.RecordsForTable recordMaker,
                                 Object[] row,
                                 long ts)
                          throws InterruptedException

Throws:

InterruptedException