io.debezium.connector.mysql

Class SnapshotReader

java.lang.Object

io.debezium.connector.mysql.AbstractReader

io.debezium.connector.mysql.SnapshotReader

All Implemented Interfaces:

Reader

public class SnapshotReader
extends AbstractReader

A component that performs a snapshot of a MySQL server, and records the schema changes in MySqlSchema.

Author:

Randall Hauch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static interface	SnapshotReader.RecordRecorder

Nested classes/interfaces inherited from interface io.debezium.connector.mysql.Reader

Reader.State

Field Summary

Fields

Modifier and Type	Field and Description
private boolean	includeData
private SnapshotReaderMetrics	metrics
private boolean	minimalBlocking
private SnapshotReader.RecordRecorder	recorder
private Thread	thread

Fields inherited from class io.debezium.connector.mysql.AbstractReader

context, logger

Constructor Summary

Constructors

Constructor and Description
SnapshotReader(String name, MySqlTaskContext context) Create a snapshot reader.

Method Summary

Modifier and Type	Method and Description
private Statement	createStatement(Connection connection)
private Statement	createStatementWithLargeResultSet(Connection connection) Create a JDBC statement that can be used for large result sets.
protected void	doCleanup() The reader has completed all processing and all enqueued records have been consumed, so this reader should clean up any resources that might remain.
protected void	doStart() Start the snapshot and return immediately.
protected void	doStop() The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started.
protected void	enqueueSchemaChanges(String dbName, String ddlStatement)
protected void	execute() Perform the snapshot using the same logic as the "mysqldump" utility.
SnapshotReader	generateInsertEvents() Set this reader's execution to produce a Envelope.Operation.CREATE event for each row.
SnapshotReader	generateReadEvents() Set this reader's execution to produce a Envelope.Operation.READ event for each row.
private void	logRolesForCurrentUser(JdbcConnection mysql)
private void	logServerInformation(JdbcConnection mysql)
protected String	quote(String dbOrTableName)
protected String	quote(TableId id)
protected void	readBinlogPosition(int step, SourceInfo source, JdbcConnection mysql, AtomicReference<String> sql)
protected void	recordRowAsInsert(RecordMakers.RecordsForTable recordMaker, Object[] row, long ts)
protected void	recordRowAsRead(RecordMakers.RecordsForTable recordMaker, Object[] row, long ts)
protected org.apache.kafka.connect.source.SourceRecord	replaceOffset(org.apache.kafka.connect.source.SourceRecord record) Utility method to replace the offset in the given record with the latest.
SnapshotReader	useMinimalBlocking(boolean minimalBlocking) Set whether this reader's execution should block other transactions as minimally as possible by releasing the read lock as early as possible.

Methods inherited from class io.debezium.connector.mysql.AbstractReader

cleanupResources, completeSuccessfully, enqueueRecord, failed, failed, isRunning, name, poll, pollComplete, start, state, stop, uponCompletion, wrap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

minimalBlocking

private boolean minimalBlocking

includeData

private final boolean includeData

recorder

private SnapshotReader.RecordRecorder recorder

thread

private volatile Thread thread

metrics

private final SnapshotReaderMetrics metrics

Constructor Detail

SnapshotReader

public SnapshotReader(String name,
                      MySqlTaskContext context)

Create a snapshot reader.

Parameters:

name - the name of this reader; may not be null

context - the task context in which this reader is running; may not be null

Method Detail

useMinimalBlocking

public SnapshotReader useMinimalBlocking(boolean minimalBlocking)

Set whether this reader's execution should block other transactions as minimally as possible by releasing the read lock as early as possible. Although the snapshot process should obtain a consistent snapshot even when releasing the lock as early as possible, it may be desirable to explicitly hold onto the read lock until execution completes. In such cases, holding onto the lock will prevent all updates to the database during the snapshot process.

Parameters:

minimalBlocking - true if the lock is to be released as early as possible, or false if the lock is to be held for the entire execution

Returns:

this object for method chaining; never null

generateReadEvents

public SnapshotReader generateReadEvents()

Set this reader's execution to produce a Envelope.Operation.READ event for each row.

Returns:

this object for method chaining; never null

generateInsertEvents

public SnapshotReader generateInsertEvents()

Set this reader's execution to produce a Envelope.Operation.CREATE event for each row.

Returns:

this object for method chaining; never null

doStart

protected void doStart()

Start the snapshot and return immediately. Once started, the records read from the database can be retrieved using AbstractReader.poll() until that method returns null.

Specified by:

doStart in class AbstractReader

doStop

protected void doStop()

Description copied from class: AbstractReader

The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started.

This method is always called when AbstractReader.stop() is called, and the first time AbstractReader.isRunning() will return true the first time and false for any subsequent calls.

Specified by:

doStop in class AbstractReader

doCleanup

protected void doCleanup()

Description copied from class: AbstractReader

The reader has completed all processing and all enqueued records have been consumed, so this reader should clean up any resources that might remain.

Specified by:

doCleanup in class AbstractReader

execute

protected void execute()

Perform the snapshot using the same logic as the "mysqldump" utility.

readBinlogPosition

protected void readBinlogPosition(int step,
                                  SourceInfo source,
                                  JdbcConnection mysql,
                                  AtomicReference<String> sql)
                           throws SQLException

Throws:

SQLException

quote

protected String quote(String dbOrTableName)

quote

protected String quote(TableId id)

createStatementWithLargeResultSet

private Statement createStatementWithLargeResultSet(Connection connection)
                                             throws SQLException

Create a JDBC statement that can be used for large result sets.

By default, the MySQL Connector/J driver retrieves all rows for ResultSets and stores them in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. However, when ResultSets that have a large number of rows or large values, the driver may not be able to allocate heap space in the JVM and may result in an OutOfMemoryError. See DBZ-94 for details.

This method handles such cases using the recommended technique for MySQL by creating the JDBC Statement with forward-only cursor and read-only concurrency flags, and with a minimum value fetch size hint.

Parameters:

connection - the JDBC connection; may not be null

Returns:

the statement; never null

Throws:

SQLException - if there is a problem creating the statement

createStatement

private Statement createStatement(Connection connection)
                           throws SQLException

Throws:

SQLException

logServerInformation

private void logServerInformation(JdbcConnection mysql)

logRolesForCurrentUser

private void logRolesForCurrentUser(JdbcConnection mysql)

replaceOffset

protected org.apache.kafka.connect.source.SourceRecord replaceOffset(org.apache.kafka.connect.source.SourceRecord record)

Utility method to replace the offset in the given record with the latest. This is used on the last record produced during the snapshot.

Parameters:

record - the record

Returns:

the updated record

enqueueSchemaChanges

protected void enqueueSchemaChanges(String dbName,
                                    String ddlStatement)

recordRowAsRead

protected void recordRowAsRead(RecordMakers.RecordsForTable recordMaker,
                               Object[] row,
                               long ts)
                        throws InterruptedException

Throws:

InterruptedException

recordRowAsInsert

protected void recordRowAsInsert(RecordMakers.RecordsForTable recordMaker,
                                 Object[] row,
                                 long ts)
                          throws InterruptedException

Throws:

InterruptedException