io.debezium.connector.mysql

Class MySqlStreamingChangeEventSource

java.lang.Object

io.debezium.connector.mysql.MySqlStreamingChangeEventSource

All Implemented Interfaces:

ChangeEventSource, StreamingChangeEventSource

public class MySqlStreamingChangeEventSource
extends Object
implements StreamingChangeEventSource

Author:

Jiri Pechanec

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static interface	MySqlStreamingChangeEventSource.BinlogChangeEmitter<T>
static class	MySqlStreamingChangeEventSource.BinlogPosition
protected class	MySqlStreamingChangeEventSource.ReaderThreadLifecycleListener
private static interface	MySqlStreamingChangeEventSource.RowsProvider<E extends com.github.shyiko.mysql.binlog.event.EventData,U>
private static interface	MySqlStreamingChangeEventSource.TableIdProvider<E extends com.github.shyiko.mysql.binlog.event.EventData>

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field and Description
private Map<String,Thread>	binaryLogClientThreads
private com.github.shyiko.mysql.binlog.BinaryLogClient	client
private Clock	clock
private MySqlConnection	connection
private MySqlConnectorConfig	connectorConfig
private ErrorHandler	errorHandler
private CommonConnectorConfig.EventProcessingFailureHandlingMode	eventDeserializationFailureHandlingMode
private EventDispatcher<TableId>	eventDispatcher
private EnumMap<com.github.shyiko.mysql.binlog.event.EventType,BlockingConsumer<com.github.shyiko.mysql.binlog.event.Event>>	eventHandlers
private Instant	eventTimestamp
private Predicate<String>	gtidDmlSourceFilter
private com.github.shyiko.mysql.binlog.GtidSet	gtidSet
private float	heartbeatIntervalFactor
private boolean	ignoreDmlEventByGtidSource
private CommonConnectorConfig.EventProcessingFailureHandlingMode	inconsistentSchemaHandlingMode
private long	initialEventsToSkip
private static String	KEEPALIVE_THREAD_NAME
private Map<String,?>	lastOffset
private static org.slf4j.Logger	LOGGER
private MySqlStreamingChangeEventSourceMetrics	metrics
private MySqlOffsetContext	offsetContext
private boolean	skipEvent
private int	startingRowNumber
private MySqlTaskContext	taskContext
private AtomicLong	totalRecordCounter

Constructor Summary

Constructors

Constructor and Description
MySqlStreamingChangeEventSource(MySqlConnectorConfig connectorConfig, MySqlOffsetContext offsetContext, MySqlConnection connection, EventDispatcher<TableId> dispatcher, ErrorHandler errorHandler, Clock clock, MySqlTaskContext taskContext, MySqlStreamingChangeEventSourceMetrics metrics)

Method Summary

Modifier and Type	Method and Description
void	execute(ChangeEventSource.ChangeEventSourceContext context)
GtidSet	filterGtidSet(GtidSet availableServerGtidSet, GtidSet purgedServerGtid) Apply the include/exclude GTID source filters to the current GTID set and merge them onto the currently available GTID set from a MySQL server.
private com.github.shyiko.mysql.binlog.network.SSLSocketFactory	getBinlogSslSocketFactory(MySqlConnectorConfig connectorConfig, MySqlConnection connection)
(package private) MySqlStreamingChangeEventSource.BinlogPosition	getCurrentBinlogPosition()
(package private) MySqlStreamingChangeEventSourceMetrics	getMetrics()
private <T extends com.github.shyiko.mysql.binlog.event.EventData,U> void	handleChange(com.github.shyiko.mysql.binlog.event.Event event, String changeType, Class<T> eventDataClass, MySqlStreamingChangeEventSource.TableIdProvider<T> tableIdProvider, MySqlStreamingChangeEventSource.RowsProvider<T,U> rowsProvider, MySqlStreamingChangeEventSource.BinlogChangeEmitter<U> changeEmitter)
protected void	handleDelete(com.github.shyiko.mysql.binlog.event.Event event) Generate source records for the supplied event with an DeleteRowsEventData.
protected void	handleEvent(com.github.shyiko.mysql.binlog.event.Event event)
protected void	handleGtidEvent(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with a GtidEventData that signals the beginning of a GTID transaction.
protected void	handleInsert(com.github.shyiko.mysql.binlog.event.Event event) Generate source records for the supplied event with an WriteRowsEventData.
protected void	handleQueryEvent(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the MySQL schemas.
protected void	handleRotateLogsEvent(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with a RotateEventData that signals the logs are being rotated.
protected void	handleRowsQuery(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event with an RowsQueryEventData by recording the original SQL query that generated the event.
protected void	handleServerHeartbeat(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event that is sent by a primary to a replica to let the replica know that the primary is still alive.
protected void	handleServerIncident(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event that signals that an out of the ordinary event that occurred on the master.
protected void	handleServerStop(com.github.shyiko.mysql.binlog.event.Event event) Handle the supplied event that signals that mysqld has stopped.
private void	handleTransactionCompletion(com.github.shyiko.mysql.binlog.event.Event event)
protected void	handleUpdate(com.github.shyiko.mysql.binlog.event.Event event) Generate source records for the supplied event with an UpdateRowsEventData.
protected void	handleUpdateTableMetadata(com.github.shyiko.mysql.binlog.event.Event event) Handle a change in the table metadata.
protected void	ignoreEvent(com.github.shyiko.mysql.binlog.event.Event event)
private void	informAboutUnknownTableIfRequired(com.github.shyiko.mysql.binlog.event.Event event, TableId tableId, String typeToLog) If we receive an event for a table that is monitored but whose metadata we don't know, either ignore that event or raise a warning or error as per the MySqlConnectorConfig.INCONSISTENT_SCHEMA_HANDLING_MODE configuration.
protected void	logEvent(com.github.shyiko.mysql.binlog.event.Event event)
private void	logStreamingSourceState()
private void	logStreamingSourceState(org.slf4j.event.Level severity)
protected void	onEvent(com.github.shyiko.mysql.binlog.event.Event event)
protected void	prepareTransaction(com.github.shyiko.mysql.binlog.event.Event event) Handle a EventType.XA_PREPARE event.
(package private) void	rewindBinaryLogClient(ChangeEventSource.ChangeEventSourceContext context, MySqlStreamingChangeEventSource.BinlogPosition position)
private com.github.shyiko.mysql.binlog.network.SSLMode	sslModeFor(MySqlConnectorConfig.SecureConnectionMode mode)
protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T	unwrapData(com.github.shyiko.mysql.binlog.event.Event event)
protected void	viewChange(com.github.shyiko.mysql.binlog.event.Event event) Handle a EventType.VIEW_CHANGE event.
protected io.debezium.DebeziumException	wrap(Throwable error) Wraps the specified exception in a DebeziumException, ensuring that all useful state is captured inside the new exception's message.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.debezium.pipeline.source.spi.StreamingChangeEventSource

commitOffset

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

KEEPALIVE_THREAD_NAME

private static final String KEEPALIVE_THREAD_NAME

See Also:

Constant Field Values

eventHandlers

private final EnumMap<com.github.shyiko.mysql.binlog.event.EventType,BlockingConsumer<com.github.shyiko.mysql.binlog.event.Event>> eventHandlers

client

private final com.github.shyiko.mysql.binlog.BinaryLogClient client

metrics

private final MySqlStreamingChangeEventSourceMetrics metrics

clock

private final Clock clock

eventDeserializationFailureHandlingMode

private final CommonConnectorConfig.EventProcessingFailureHandlingMode eventDeserializationFailureHandlingMode

inconsistentSchemaHandlingMode

private final CommonConnectorConfig.EventProcessingFailureHandlingMode inconsistentSchemaHandlingMode

startingRowNumber

private int startingRowNumber

initialEventsToSkip

private long initialEventsToSkip

skipEvent

private boolean skipEvent

ignoreDmlEventByGtidSource

private boolean ignoreDmlEventByGtidSource

gtidDmlSourceFilter

private final Predicate<String> gtidDmlSourceFilter

totalRecordCounter

private final AtomicLong totalRecordCounter

lastOffset

private volatile Map<String,?> lastOffset

gtidSet

private com.github.shyiko.mysql.binlog.GtidSet gtidSet

heartbeatIntervalFactor

private final float heartbeatIntervalFactor

See Also:

Constant Field Values

binaryLogClientThreads

private final Map<String,Thread> binaryLogClientThreads

taskContext

private final MySqlTaskContext taskContext

connectorConfig

private final MySqlConnectorConfig connectorConfig

connection

private final MySqlConnection connection

eventDispatcher

private final EventDispatcher<TableId> eventDispatcher

offsetContext

private final MySqlOffsetContext offsetContext

errorHandler

private final ErrorHandler errorHandler

eventTimestamp

private Instant eventTimestamp

Constructor Detail

MySqlStreamingChangeEventSource

public MySqlStreamingChangeEventSource(MySqlConnectorConfig connectorConfig,
                                       MySqlOffsetContext offsetContext,
                                       MySqlConnection connection,
                                       EventDispatcher<TableId> dispatcher,
                                       ErrorHandler errorHandler,
                                       Clock clock,
                                       MySqlTaskContext taskContext,
                                       MySqlStreamingChangeEventSourceMetrics metrics)

Method Detail

onEvent

protected void onEvent(com.github.shyiko.mysql.binlog.event.Event event)

ignoreEvent

protected void ignoreEvent(com.github.shyiko.mysql.binlog.event.Event event)

handleEvent

protected void handleEvent(com.github.shyiko.mysql.binlog.event.Event event)

unwrapData

protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T unwrapData(com.github.shyiko.mysql.binlog.event.Event event)

handleServerStop

protected void handleServerStop(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event that signals that mysqld has stopped.

Parameters:

event - the server stopped event to be processed; may not be null

handleServerHeartbeat

protected void handleServerHeartbeat(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event that is sent by a primary to a replica to let the replica know that the primary is still alive. Not written to a binary log.

Parameters:

event - the server stopped event to be processed; may not be null

handleServerIncident

protected void handleServerIncident(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event that signals that an out of the ordinary event that occurred on the master. It notifies the replica that something happened on the primary that might cause data to be in an inconsistent state.

Parameters:

event - the server stopped event to be processed; may not be null

handleRotateLogsEvent

protected void handleRotateLogsEvent(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event with a RotateEventData that signals the logs are being rotated. This means that either the server was restarted, or the binlog has transitioned to a new file. In either case, subsequent table numbers will be different than those seen to this point.

Parameters:

event - the database change data event to be processed; may not be null

handleGtidEvent

protected void handleGtidEvent(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event with a GtidEventData that signals the beginning of a GTID transaction. We don't yet know whether this transaction contains any events we're interested in, but we have to record it so that we know the position of this event and know we've processed the binlog to this point.

Note that this captures the current GTID and complete GTID set, regardless of whether the connector is filtering the GTID set upon connection. We do this because we actually want to capture all GTID set values found in the binlog, whether or not we process them. However, only when we connect do we actually want to pass to MySQL only those GTID ranges that are applicable per the configuration.

Parameters:

event - the GTID event to be processed; may not be null

handleRowsQuery

protected void handleRowsQuery(com.github.shyiko.mysql.binlog.event.Event event)

Handle the supplied event with an RowsQueryEventData by recording the original SQL query that generated the event.

Parameters:

event - the database change data event to be processed; may not be null

handleQueryEvent

protected void handleQueryEvent(com.github.shyiko.mysql.binlog.event.Event event)
                         throws InterruptedException

Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the MySQL schemas.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while recording the DDL statements

handleTransactionCompletion

private void handleTransactionCompletion(com.github.shyiko.mysql.binlog.event.Event event)
                                  throws InterruptedException

Throws:

InterruptedException

handleUpdateTableMetadata

protected void handleUpdateTableMetadata(com.github.shyiko.mysql.binlog.event.Event event)

Handle a change in the table metadata.

This method should be called whenever we consume a TABLE_MAP event, and every transaction in the log should include one of these for each table affected by the transaction. Each table map event includes a monotonically-increasing numeric identifier, and this identifier is used within subsequent events within the same transaction. This table identifier can change when:

1. the table structure is modified (e.g., via an ALTER TABLE ... command); or
2. MySQL rotates to a new binary log file, even if the table structure does not change.

Parameters:

event - the update event; never null

informAboutUnknownTableIfRequired

private void informAboutUnknownTableIfRequired(com.github.shyiko.mysql.binlog.event.Event event,
                                               TableId tableId,
                                               String typeToLog)

If we receive an event for a table that is monitored but whose metadata we don't know, either ignore that event or raise a warning or error as per the MySqlConnectorConfig.INCONSISTENT_SCHEMA_HANDLING_MODE configuration.

handleInsert

protected void handleInsert(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException

Generate source records for the supplied event with an WriteRowsEventData.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

handleUpdate

protected void handleUpdate(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException

Generate source records for the supplied event with an UpdateRowsEventData.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

handleDelete

protected void handleDelete(com.github.shyiko.mysql.binlog.event.Event event)
                     throws InterruptedException

Generate source records for the supplied event with an DeleteRowsEventData.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

handleChange

private <T extends com.github.shyiko.mysql.binlog.event.EventData,U> void handleChange(com.github.shyiko.mysql.binlog.event.Event event,
                                                                                       String changeType,
                                                                                       Class<T> eventDataClass,
                                                                                       MySqlStreamingChangeEventSource.TableIdProvider<T> tableIdProvider,
                                                                                       MySqlStreamingChangeEventSource.RowsProvider<T,U> rowsProvider,
                                                                                       MySqlStreamingChangeEventSource.BinlogChangeEmitter<U> changeEmitter)
                                                                                throws InterruptedException

Throws:

InterruptedException

viewChange

protected void viewChange(com.github.shyiko.mysql.binlog.event.Event event)
                   throws InterruptedException

Handle a EventType.VIEW_CHANGE event.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

prepareTransaction

protected void prepareTransaction(com.github.shyiko.mysql.binlog.event.Event event)
                           throws InterruptedException

Handle a EventType.XA_PREPARE event.

Parameters:

event - the database change data event to be processed; may not be null

Throws:

InterruptedException - if this thread is interrupted while blocking

sslModeFor

private com.github.shyiko.mysql.binlog.network.SSLMode sslModeFor(MySqlConnectorConfig.SecureConnectionMode mode)

execute

public void execute(ChangeEventSource.ChangeEventSourceContext context)
             throws InterruptedException

Specified by:

execute in interface StreamingChangeEventSource

Throws:

InterruptedException

getBinlogSslSocketFactory

private com.github.shyiko.mysql.binlog.network.SSLSocketFactory getBinlogSslSocketFactory(MySqlConnectorConfig connectorConfig,
                                                                                          MySqlConnection connection)

logStreamingSourceState

private void logStreamingSourceState()

logEvent

protected void logEvent(com.github.shyiko.mysql.binlog.event.Event event)

logStreamingSourceState

private void logStreamingSourceState(org.slf4j.event.Level severity)

filterGtidSet

public GtidSet filterGtidSet(GtidSet availableServerGtidSet,
                             GtidSet purgedServerGtid)

Apply the include/exclude GTID source filters to the current GTID set and merge them onto the currently available GTID set from a MySQL server. The merging behavior of this method might seem a bit strange at first. It's required in order for Debezium to consume a MySQL binlog that has multi-source replication enabled, if a failover has to occur. In such a case, the server that Debezium is failed over to might have a different set of sources, but still include the sources required for Debezium to continue to function. MySQL does not allow downstream replicas to connect if the GTID set does not contain GTIDs for all channels that the server is replicating from, even if the server does have the data needed by the client. To get around this, we can have Debezium merge its GTID set with whatever is on the server, so that MySQL will allow it to connect. See DBZ-143 for details. This method does not mutate any state in the context.

Parameters:

availableServerGtidSet - the GTID set currently available in the MySQL server

purgedServerGtid - the GTID set already purged by the MySQL server

Returns:

A GTID set meant for consuming from a MySQL binlog; may return null if the SourceInfo has no GTIDs and therefore none were filtered

getMetrics

MySqlStreamingChangeEventSourceMetrics getMetrics()

rewindBinaryLogClient

void rewindBinaryLogClient(ChangeEventSource.ChangeEventSourceContext context,
                           MySqlStreamingChangeEventSource.BinlogPosition position)

getCurrentBinlogPosition

MySqlStreamingChangeEventSource.BinlogPosition getCurrentBinlogPosition()

wrap

protected io.debezium.DebeziumException wrap(Throwable error)

Wraps the specified exception in a DebeziumException, ensuring that all useful state is captured inside the new exception's message.

Parameters:

error - the exception; may not be null

Returns:

the wrapped Kafka Connect exception