Package io.debezium.connector.mysql.legacy

Class BinlogReader

java.lang.Object
io.debezium.connector.mysql.legacy.AbstractReader
io.debezium.connector.mysql.legacy.BinlogReader
All Implemented Interfaces:
Reader
public class BinlogReader extends AbstractReader
A component that reads the binlog of a MySQL server, and records any schema changes in MySqlSchema.
Author:
Randall Hauch

  • Field Details

    • INITIAL_POLL_PERIOD_IN_MILLIS

      private static final long INITIAL_POLL_PERIOD_IN_MILLIS

    • MAX_POLL_PERIOD_IN_MILLIS

      private static final long MAX_POLL_PERIOD_IN_MILLIS

    • KEEPALIVE_THREAD_NAME

      private static final String KEEPALIVE_THREAD_NAME
      See Also:

    • recordSchemaChangesInSourceRecords

      private final boolean recordSchemaChangesInSourceRecords

    • recordMakers

      private final RecordMakers recordMakers

    • source

      private final SourceInfo source

    • eventHandlers

      private final EnumMap<com.github.shyiko.mysql.binlog.event.EventType,BlockingConsumer<com.github.shyiko.mysql.binlog.event.Event>> eventHandlers

    • client

      private final com.github.shyiko.mysql.binlog.BinaryLogClient client

    • metrics

      private final BinlogReaderMetrics metrics

    • clock

      private final Clock clock

    • pollOutputDelay

      private final ElapsedTimeStrategy pollOutputDelay

    • eventDeserializationFailureHandlingMode

      private final CommonConnectorConfig.EventProcessingFailureHandlingMode eventDeserializationFailureHandlingMode

    • inconsistentSchemaHandlingMode

      private final CommonConnectorConfig.EventProcessingFailureHandlingMode inconsistentSchemaHandlingMode

    • startingRowNumber

      private int startingRowNumber

    • recordCounter

      private long recordCounter

    • previousOutputMillis

      private long previousOutputMillis

    • initialEventsToSkip

      private long initialEventsToSkip

    • skipEvent

      private boolean skipEvent

    • ignoreDmlEventByGtidSource

      private boolean ignoreDmlEventByGtidSource

    • gtidDmlSourceFilter

      private final Predicate<String> gtidDmlSourceFilter

    • totalRecordCounter

      private final AtomicLong totalRecordCounter

    • lastOffset

      private volatile Map<String,?> lastOffset

    • gtidSet

      private com.github.shyiko.mysql.binlog.GtidSet gtidSet

    • heartbeat

      private Heartbeat heartbeat

    • connectionContext

      private MySqlJdbcContext connectionContext

    • heartbeatIntervalFactor

      private final float heartbeatIntervalFactor
      See Also:

    • binaryLogClientThreads

      private final Map<String,Thread> binaryLogClientThreads

  • Constructor Details

  • Method Details

    • doInitialize

      protected void doInitialize()
      Description copied from class: AbstractReader
      The reader has been requested to initialize resources prior to starting. This should only be called once before AbstractReader.doStart(MySqlPartition).
      Overrides:
      doInitialize in class AbstractReader

    • doDestroy

      public void doDestroy()
      Description copied from class: AbstractReader
      The reader has been requested to de-initialize resources after stopping. This should only be called once after AbstractReader.doStop(MySqlPartition).
      Overrides:
      doDestroy in class AbstractReader

    • doStart

      protected void doStart(MySqlPartition partition)
      Description copied from class: AbstractReader
      The reader has been requested to start, so initialize any un-initialized resources required by the reader.
      Specified by:
      doStart in class AbstractReader

    • rewindBinaryLogClient

      protected void rewindBinaryLogClient(BinlogReader.BinlogPosition position)

    • getLastOffset

      public Map<String,?> getLastOffset()
      Returns:
      a copy of the last offset of this reader, or null if this reader has not completed a poll.

    • doStop

      protected void doStop(MySqlPartition partition)
      Description copied from class: AbstractReader
      The reader has been requested to stop, so perform any work required to stop the reader's resources that were previously started.

      This method is always called when Reader.stop() is called, and the first time AbstractReader.isRunning() will return true the first time and false for any subsequent calls.

      Specified by:
      doStop in class AbstractReader

    • doCleanup

      protected void doCleanup()
      Description copied from class: AbstractReader
      The reader has completed all processing and all enqueued records have been consumed, so this reader should clean up any resources that might remain.
      Specified by:
      doCleanup in class AbstractReader

    • pollComplete

      protected void pollComplete(List<org.apache.kafka.connect.source.SourceRecord> batch)
      Description copied from class: AbstractReader
      Method called when AbstractReader.poll() completes sending a non-zero-sized batch of records.
      Overrides:
      pollComplete in class AbstractReader
      Parameters:
      batch - the batch of records being recorded

    • logEvent

      protected void logEvent(com.github.shyiko.mysql.binlog.event.Event event)

    • onEvent

      protected void onEvent(com.github.shyiko.mysql.binlog.event.Event event)

    • ignoreEvent

      protected void ignoreEvent(com.github.shyiko.mysql.binlog.event.Event event)

    • handleEvent

      protected void handleEvent(com.github.shyiko.mysql.binlog.event.Event event)

    • unwrapData

      protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T unwrapData(com.github.shyiko.mysql.binlog.event.Event event)

    • handleServerStop

      protected void handleServerStop(com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event that signals that mysqld has stopped.
      Parameters:
      event - the server stopped event to be processed; may not be null

    • handleServerHeartbeat

      protected void handleServerHeartbeat(com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event that is sent by a primary to a replica to let the replica know that the primary is still alive. Not written to a binary log.
      Parameters:
      event - the server stopped event to be processed; may not be null

    • handleServerIncident

      protected void handleServerIncident(MySqlPartition partition, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event that signals that an out of the ordinary event that occurred on the master. It notifies the replica that something happened on the primary that might cause data to be in an inconsistent state.
      Parameters:
      event - the server stopped event to be processed; may not be null

    • handleRotateLogsEvent

      protected void handleRotateLogsEvent(com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event with a RotateEventData that signals the logs are being rotated. This means that either the server was restarted, or the binlog has transitioned to a new file. In either case, subsequent table numbers will be different than those seen to this point, so we need to discard the cache of record makers.
      Parameters:
      event - the database change data event to be processed; may not be null

    • handleGtidEvent

      protected void handleGtidEvent(com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event with a GtidEventData that signals the beginning of a GTID transaction. We don't yet know whether this transaction contains any events we're interested in, but we have to record it so that we know the position of this event and know we've processed the binlog to this point.

      Note that this captures the current GTID and complete GTID set, regardless of whether the connector is filtering the GTID set upon connection. We do this because we actually want to capture all GTID set values found in the binlog, whether or not we process them. However, only when we connect do we actually want to pass to MySQL only those GTID ranges that are applicable per the configuration.

      Parameters:
      event - the GTID event to be processed; may not be null

    • handleRowsQuery

      protected void handleRowsQuery(com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event with an RowsQueryEventData by recording the original SQL query that generated the event.
      Parameters:
      event - the database change data event to be processed; may not be null

    • handleQueryEvent

      protected void handleQueryEvent(com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the MySQL schemas.
      Parameters:
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while recording the DDL statements

    • handleTransactionCompletion

      private void handleTransactionCompletion(com.github.shyiko.mysql.binlog.event.Event event)

    • handleUpdateTableMetadata

      protected void handleUpdateTableMetadata(MySqlPartition partition, com.github.shyiko.mysql.binlog.event.Event event)
      Handle a change in the table metadata.

      This method should be called whenever we consume a TABLE_MAP event, and every transaction in the log should include one of these for each table affected by the transaction. Each table map event includes a monotonically-increasing numeric identifier, and this identifier is used within subsequent events within the same transaction. This table identifier can change when:

      1. the table structure is modified (e.g., via an ALTER TABLE ... command); or
      2. MySQL rotates to a new binary log file, even if the table structure does not change.
      Parameters:
      event - the update event; never null

    • informAboutUnknownTableIfRequired

      private void informAboutUnknownTableIfRequired(MySqlPartition partition, com.github.shyiko.mysql.binlog.event.Event event, TableId tableId, String typeToLog, Envelope.Operation operation)
      If we receive an event for a table that is monitored but whose metadata we don't know, either ignore that event or raise a warning or error as per the MySqlConnectorConfig.INCONSISTENT_SCHEMA_HANDLING_MODE configuration.

    • informAboutUnknownTableIfRequired

      private void informAboutUnknownTableIfRequired(MySqlPartition partition, com.github.shyiko.mysql.binlog.event.Event event, TableId tableId, String typeToLog)

    • handleInsert

      protected void handleInsert(MySqlPartition partition, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Generate source records for the supplied event with an WriteRowsEventData.
      Parameters:
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while blocking

    • handleUpdate

      protected void handleUpdate(MySqlPartition partition, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Generate source records for the supplied event with an UpdateRowsEventData.
      Parameters:
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while blocking

    • handleDelete

      protected void handleDelete(MySqlPartition partition, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Generate source records for the supplied event with an DeleteRowsEventData.
      Parameters:
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while blocking

    • viewChange

      protected void viewChange(com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Handle a EventType.VIEW_CHANGE event.
      Parameters:
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while blocking

    • prepareTransaction

      protected void prepareTransaction(com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Handle a EventType.XA_PREPARE event.
      Parameters:
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while blocking

    • sslModeFor

      protected static com.github.shyiko.mysql.binlog.network.SSLMode sslModeFor(MySqlConnectorConfig.SecureConnectionMode mode)

    • logReaderState

      private void logReaderState()

    • logReaderState

      private void logReaderState(org.slf4j.event.Level severity)

    • getMetrics

      protected BinlogReaderMetrics getMetrics()

    • getBinlogClient

      protected com.github.shyiko.mysql.binlog.BinaryLogClient getBinlogClient()

    • getCurrentBinlogPosition

      public BinlogReader.BinlogPosition getCurrentBinlogPosition()

    • getBinlogSslSocketFactory

      private com.github.shyiko.mysql.binlog.network.SSLSocketFactory getBinlogSslSocketFactory(MySqlJdbcContext connectionContext)