io.debezium.connector.mysql

Class BinlogReader

java.lang.Object

io.debezium.connector.mysql.AbstractReader

io.debezium.connector.mysql.BinlogReader

public class BinlogReader
extends AbstractReader

A component that reads the binlog of a MySQL server, and records any schema changes in MySqlSchema.

Author:

Randall Hauch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	BinlogReader.ReaderThreadLifecycleListener

Field Summary

Fields

Modifier and Type	Field and Description
private com.github.shyiko.mysql.binlog.BinaryLogClient	client
private Clock	clock
private EnumMap<com.github.shyiko.mysql.binlog.event.EventType,BlockingConsumer<com.github.shyiko.mysql.binlog.event.Event>>	eventHandlers
private com.github.shyiko.mysql.binlog.GtidSet	gtidSet
private static long	INITIAL_POLL_PERIOD_IN_MILLIS
private Map<String,?>	lastOffset
private static long	MAX_POLL_PERIOD_IN_MILLIS
private ElapsedTimeStrategy	pollOutputDelay
private long	previousOutputMillis
private long	recordCounter
private RecordMakers	recordMakers
private boolean	recordSchemaChangesInSourceRecords
private SourceInfo	source
private int	startingRowNumber
private AtomicLong	totalRecordCounter

Fields inherited from class io.debezium.connector.mysql.AbstractReader

context, logger

Constructor Summary

Constructors

Constructor and Description
BinlogReader(MySqlTaskContext context) Create a binlog reader.

Method Summary

Modifier and Type	Method and Description
protected void	doCleanup() The reader has completed sending all enqueued records, so clean up any resources that remain.
protected void	doStart() The reader has been requested to start, so initialize any resources required by the reader.
protected void	doStop() The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started..
protected void	handleDelete(com.github.shyiko.mysql.binlog.event.Event event) Generate source records for the supplied event with an DeleteRowsEventData.
protected void	handleEvent(com.github.shyiko.mysql.binlog.event.Event event)
protected void	handleGtidEvent(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with a GtidEventData that signals the beginning of a GTID transaction.
protected void	handleInsert(com.github.shyiko.mysql.binlog.event.Event event) Generate source records for the supplied event with an WriteRowsEventData.
protected void	handleQueryEvent(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the MySQL schemas.
protected void	handleRotateLogsEvent(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with a RotateEventData that signals the logs are being rotated.
protected void	handleServerHeartbeat(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event that is sent by a master to a slave to let the slave know that the master is still alive.
protected void	handleServerIncident(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event that signals that an out of the ordinary event that occurred on the master.
protected void	handleServerStop(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event that signals that mysqld has stopped.
protected void	handleUpdate(com.github.shyiko.mysql.binlog.event.Event event) Generate source records for the supplied event with an UpdateRowsEventData.
protected void	handleUpdateTableMetadata(com.github.shyiko.mysql.binlog.event.Event event) Handle a change in the table metadata.
protected void	ignoreEvent(com.github.shyiko.mysql.binlog.event.Event event)
protected void	logEvent(com.github.shyiko.mysql.binlog.event.Event event)
protected void	pollComplete(List<org.apache.kafka.connect.source.SourceRecord> batch) Method called when AbstractReader.poll() completes sending a non-zero-sized batch of records.
protected com.github.shyiko.mysql.binlog.network.SSLMode	sslModeFor(MySqlConnectorConfig.SecureConnectionMode mode)
protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T	unwrapData(com.github.shyiko.mysql.binlog.event.Event event)

Methods inherited from class io.debezium.connector.mysql.AbstractReader

completeSuccessfully, enqueueRecord, failed, failed, isRunning, poll, start, stop, wrap

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INITIAL_POLL_PERIOD_IN_MILLIS

private static final long INITIAL_POLL_PERIOD_IN_MILLIS

MAX_POLL_PERIOD_IN_MILLIS

private static final long MAX_POLL_PERIOD_IN_MILLIS

recordSchemaChangesInSourceRecords

private final boolean recordSchemaChangesInSourceRecords

recordMakers

private final RecordMakers recordMakers

source

private final SourceInfo source

eventHandlers

private final EnumMap<com.github.shyiko.mysql.binlog.event.EventType,BlockingConsumer<com.github.shyiko.mysql.binlog.event.Event>> eventHandlers

client

private com.github.shyiko.mysql.binlog.BinaryLogClient client

startingRowNumber

private int startingRowNumber

clock

private final Clock clock

pollOutputDelay

private final ElapsedTimeStrategy pollOutputDelay

recordCounter

private long recordCounter

previousOutputMillis

private long previousOutputMillis

totalRecordCounter

private final AtomicLong totalRecordCounter

lastOffset

private volatile Map<String,?> lastOffset

gtidSet

private com.github.shyiko.mysql.binlog.GtidSet gtidSet

Constructor Detail

BinlogReader

public BinlogReader(MySqlTaskContext context)

Create a binlog reader.

Parameters:

context - the task context in which this reader is running; may not be null

Method Detail

doStart

protected void doStart()

Description copied from class: AbstractReader

The reader has been requested to start, so initialize any resources required by the reader.

Specified by:

doStart in class AbstractReader

doStop

protected void doStop()

Description copied from class: AbstractReader

The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started..

Specified by:

doStop in class AbstractReader

doCleanup

protected void doCleanup()

Description copied from class: AbstractReader

The reader has completed sending all enqueued records, so clean up any resources that remain.

Specified by:

doCleanup in class AbstractReader

pollComplete

protected void pollComplete(List<org.apache.kafka.connect.source.SourceRecord> batch)

Description copied from class: AbstractReader

Method called when AbstractReader.poll() completes sending a non-zero-sized batch of records.

Overrides:

pollComplete in class AbstractReader

Parameters:

batch - the batch of records being recorded

logEvent

protected void logEvent(com.github.shyiko.mysql.binlog.event.Event event)

ignoreEvent

protected void ignoreEvent(com.github.shyiko.mysql.binlog.event.Event event)

handleEvent

protected void handleEvent(com.github.shyiko.mysql.binlog.event.Event event)

unwrapData

protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T unwrapData(com.github.shyiko.mysql.binlog.event.Event event)

handleServerStop

protected void handleServerStop(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event that signals that mysqld has stopped.

Parameters:

event - the server stopped event to be processed; may not be null

handleServerHeartbeat

protected void handleServerHeartbeat(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event that is sent by a master to a slave to let the slave know that the master is still alive. Not written to a binary log.

Parameters:

event - the server stopped event to be processed; may not be null

handleServerIncident

protected void handleServerIncident(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event that signals that an out of the ordinary event that occurred on the master. It notifies the slave that something happened on the master that might cause data to be in an inconsistent state.

Parameters:

event - the server stopped event to be processed; may not be null

handleRotateLogsEvent

protected void handleRotateLogsEvent(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event with a RotateEventData that signals the logs are being rotated. This means that either the server was restarted, or the binlog has transitioned to a new file. In either case, subsequent table numbers will be different than those seen to this point, so we need to discard the cache of record makers.

Parameters:

event - the database change data event to be processed; may not be null

handleGtidEvent

protected void handleGtidEvent(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event with a GtidEventData that signals the beginning of a GTID transaction. We don't yet know whether this transaction contains any events we're interested in, but we have to record it so that we know the position of this event and know we've processed the binlog to this point.

Note that this captures the current GTID and complete GTID set, regardless of whether the connector is filtering the GTID set upon connection. We do this because we actually want to capture all GTID set values found in the binlog, whether or not we process them. However, only when we connect do we actually want to pass to MySQL only those GTID ranges that are applicable per the configuration.

Parameters:

event - the GTID event to be processed; may not be null

handleQueryEvent

protected void handleQueryEvent(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the MySQL schemas.

Parameters:

event - the database change data event to be processed; may not be null

handleUpdateTableMetadata

protected void handleUpdateTableMetadata(com.github.shyiko.mysql.binlog.event.Event event)

Handle a change in the table metadata.

This method should be called whenever we consume a TABLE_MAP event, and every transaction in the log should include one of these for each table affected by the transaction. Each table map event includes a monotonically-increasing numeric identifier, and this identifier is used within subsequent events within the same transaction. This table identifier can change when:

1. the table structure is modified (e.g., via an ALTER TABLE ... command); or
2. MySQL rotates to a new binary log file, even if the table structure does not change.

Parameters:

event - the update event; never null

handleInsert

protected void handleInsert(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException

Generate source records for the supplied event with an WriteRowsEventData.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

handleUpdate

protected void handleUpdate(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException

Generate source records for the supplied event with an UpdateRowsEventData.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

handleDelete

protected void handleDelete(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException

Generate source records for the supplied event with an DeleteRowsEventData.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

sslModeFor

protected com.github.shyiko.mysql.binlog.network.SSLMode sslModeFor(MySqlConnectorConfig.SecureConnectionMode mode)