Package io.debezium.connector.binlog

Class BinlogStreamingChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext>

java.lang.Object
io.debezium.connector.binlog.BinlogStreamingChangeEventSource<P,O>
All Implemented Interfaces:
ChangeEventSource, StreamingChangeEventSource<P,O>
public abstract class BinlogStreamingChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext> extends Object implements StreamingChangeEventSource<P,O>
An abstract common implementation of StreamingChangeEventSource for binlog-based connectors.
Author:
Jiri Pechanec, Chris Cranford

  • Field Details

  • Constructor Details

  • Method Details

    • execute

      public void execute(ChangeEventSource.ChangeEventSourceContext context, P partition, O offsetContext) throws InterruptedException
      Specified by:
      execute in interface StreamingChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext>
      Throws:
      InterruptedException

    • getOffsetContext

      public O getOffsetContext()
      Specified by:
      getOffsetContext in interface StreamingChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext>

    • setEffectiveOffsetContext

      protected void setEffectiveOffsetContext(O offsetContext)

    • setIgnoreDmlEventByGtidSource

      protected void setIgnoreDmlEventByGtidSource(boolean ignoreDmlEventByGtidSource)

    • setGtidChanged

      protected void setGtidChanged(String gtid)

    • isGtidModeEnabled

      protected boolean isGtidModeEnabled()

    • createBinaryLogClient

      protected com.github.shyiko.mysql.binlog.BinaryLogClient createBinaryLogClient(BinlogTaskContext<?> taskContext, BinlogConnectorConfig connectorConfig, Map<String,Thread> clientThreads, BinlogConnectorConnection connection)

    • configureReplicaCompatibility

      protected void configureReplicaCompatibility(com.github.shyiko.mysql.binlog.BinaryLogClient client)

    • createEventDeserializer

      protected com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer createEventDeserializer()

    • onEvent

      protected void onEvent(O offsetContext, com.github.shyiko.mysql.binlog.event.Event event)

    • setEventTimestamp

      protected abstract void setEventTimestamp(com.github.shyiko.mysql.binlog.event.Event event, long eventTs)

    • ignoreEvent

      protected void ignoreEvent(O offsetContext, com.github.shyiko.mysql.binlog.event.Event event)

    • handleEvent

      protected void handleEvent(P partition, O offsetContext, ChangeEventSource.ChangeEventSourceContext context, com.github.shyiko.mysql.binlog.event.Event event)

    • unwrapData

      protected <T extends com.github.shyiko.mysql.binlog.event.EventData> T unwrapData(com.github.shyiko.mysql.binlog.event.Event event)

    • handleServerStop

      protected void handleServerStop(O offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event that signals that the database has stopped.
      Parameters:
      event - the server stopped event to be processed; may not be null

    • handleServerHeartbeat

      protected void handleServerHeartbeat(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Handle the supplied event that is sent by a primary to a replica to let the replica know that the primary is still alive. Not written to a binary log.
      Parameters:
      event - the server stopped event to be processed; may not be null
      Throws:
      InterruptedException

    • handleServerIncident

      protected void handleServerIncident(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event that signals that an out of the ordinary event that occurred on the master. It notifies the replica that something happened on the primary that might cause data to be in an inconsistent state.
      Parameters:
      event - the server stopped event to be processed; may not be null

    • handleRotateLogsEvent

      protected void handleRotateLogsEvent(O offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event with a RotateEventData that signals the logs are being rotated. This means that either the server was restarted, or the binlog has transitioned to a new file. In either case, subsequent table numbers will be different from those seen to this point.
      Parameters:
      event - the database change data event to be processed; may not be null

    • handleQueryEvent

      protected void handleQueryEvent(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Handle the supplied event with an QueryEventData by possibly recording the DDL statements as changes in the database schemas.
      Parameters:
      partition - the partition in which the even occurred
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while recording the DDL statements

    • removeSetStatement

      private String removeSetStatement(String sql)

    • handleUpdateTableMetadata

      protected void handleUpdateTableMetadata(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Handle a change in the table metadata.

      This method should be called whenever we consume a TABLE_MAP event, and every transaction in the log should include one of these for each table affected by the transaction. Each table map event includes a monotonically-increasing numeric identifier, and this identifier is used within subsequent events within the same transaction. This table identifier can change when:
      1. the table structure is modified (e.g., via an ALTER TABLE ... command); or
      2. MySQL rotates to a new binary log file, even if the table structure does not change.
      Parameters:
      partition - the partition; never null
      offsetContext - the offset context; never null
      event - the update event; never null
      Throws:
      InterruptedException

    • handleTransactionPayload

      protected void handleTransactionPayload(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Handle an event of type TRANSACTION_PAYLOAD_EVENT

      This method should be called whenever a transaction payload event is encountered by the mysql binlog connector. A Transaction payload event is propagated from the binlog connector when compression is turned on over binlog. This method loops over the individual events in the compressed binlog and calls the respective atomic event handlers.
      Parameters:
      partition - the partition; never null
      offsetContext - the offset context; never null
      event - the transaction payload event; never null
      Throws:
      InterruptedException

    • handleInsert

      protected void handleInsert(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Generate source records for the supplied event with an WriteRowsEventData.
      Parameters:
      partition - the partition in which the even occurred
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while blocking

    • handleUpdate

      protected void handleUpdate(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Generate source records for the supplied event with an UpdateRowsEventData.
      Parameters:
      partition - the partition in which the even occurred
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while blocking

    • handleDelete

      protected void handleDelete(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Generate source records for the supplied event with an DeleteRowsEventData.
      Parameters:
      partition - the partition in which the even occurred
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while blocking

    • viewChange

      protected void viewChange(O offsetContext, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Handle a EventType.VIEW_CHANGE event.
      Parameters:
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while blocking

    • prepareTransaction

      protected void prepareTransaction(O offsetContext, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Handle a EventType.XA_PREPARE event.
      Parameters:
      event - the database change data event to be processed; may not be null
      Throws:
      InterruptedException - if this thread is interrupted while blocking

    • handleTransactionBegin

      protected void handleTransactionBegin(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event, Long threadId) throws InterruptedException
      Throws:
      InterruptedException

    • getConnectorClass

      protected abstract Class<? extends org.apache.kafka.connect.source.SourceConnector> getConnectorClass()
      Returns:
      the connector class type

    • handleGtidEvent

      protected abstract void handleGtidEvent(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event, Predicate<String> gtidSourceFilter) throws InterruptedException
      Handle the GTID event.
      Parameters:
      partition - the partition; never null
      offsetContext - the offset context; never null
      event - the event
      gtidSourceFilter - the GTID source filter
      Throws:
      InterruptedException - if the thread is interrupted

    • handleRecordingQuery

      protected abstract void handleRecordingQuery(O offsetContext, com.github.shyiko.mysql.binlog.event.Event event)
      Handle the supplied event with a RowsQueryEventData or AnnotateRowsEventData by recording the original SQL query that generated the event.
      Parameters:
      offsetContext - the offset context; never null
      event - the database change data event to be processed; may not be null

    • getIncludeQueryEventType

      protected abstract com.github.shyiko.mysql.binlog.event.EventType getIncludeQueryEventType()
      Returns:
      the event type for reading query events from

    • getGtidEventType

      protected abstract com.github.shyiko.mysql.binlog.event.EventType getGtidEventType()
      Returns:
      the event type for the global transaction identifier events

    • initializeGtidSet

      protected abstract void initializeGtidSet(String value)
      Initialize the connector's global transaction identifier set.
      Parameters:
      value - the value to initialize the set based upon

    • getGtidDmlSourceFilter

      private Predicate<String> getGtidDmlSourceFilter()

    • handleTransactionCompletion

      private void handleTransactionCompletion(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event) throws InterruptedException
      Throws:
      InterruptedException

    • informAboutUnknownTableIfRequired

      private void informAboutUnknownTableIfRequired(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event, TableId tableId, Envelope.Operation operation) throws InterruptedException
      If we receive an event for a table that is monitored but whose metadata we don't know, either ignore that event or raise a warning or error per the BinlogConnectorConfig.INCONSISTENT_SCHEMA_HANDLING_MODE configuration.
      Parameters:
      partition - the partition; never null
      offsetContext - the offset context; never null
      event - the binlog event
      tableId - the table identifier
      operation - the operation being performed
      Throws:
      InterruptedException

    • informAboutUnknownTableIfRequired

      private void informAboutUnknownTableIfRequired(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event, TableId tableId) throws InterruptedException
      If we receive an event for a table that is monitored but whose metadata we don't know, either ignore that event or raise a warning or error per the BinlogConnectorConfig.INCONSISTENT_SCHEMA_HANDLING_MODE configuration.
      Parameters:
      partition - the partition; never null
      offsetContext - the offset context; never null
      event - the binlog event
      tableId - the table identifier
      Throws:
      InterruptedException - if the thread is interrupted

    • validateChangeEventWithTable

      private void validateChangeEventWithTable(Table table, Object[] before, Object[] after)

    • handleChange

      private <T extends com.github.shyiko.mysql.binlog.event.EventData, U> void handleChange(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event, Envelope.Operation operation, Class<T> eventDataClass, BinlogStreamingChangeEventSource.TableIdProvider<T> tableIdProvider, BinlogStreamingChangeEventSource.RowsProvider<T,U> rowsProvider, BinlogStreamingChangeEventSource.BinlogChangeEmitter<U> changeEmitter, BinlogStreamingChangeEventSource.ChangeEventValidator<U> changeEventValidator) throws InterruptedException
      Throws:
      InterruptedException

    • waitWhenStreamingPaused

      private void waitWhenStreamingPaused(ChangeEventSource.ChangeEventSourceContext context) throws InterruptedException
      Throws:
      InterruptedException

    • logStreamingSourceState

      private void logStreamingSourceState()

    • logEvent

      protected void logEvent(O offsetContext, com.github.shyiko.mysql.binlog.event.Event event)

    • logStreamingSourceState

      private void logStreamingSourceState(org.slf4j.event.Level severity)

    • getMetrics

      BinlogStreamingChangeEventSourceMetrics<?,P> getMetrics()

    • rewindBinaryLogClient

      void rewindBinaryLogClient(ChangeEventSource.ChangeEventSourceContext context, BinlogStreamingChangeEventSource.BinlogPosition position)

    • getCurrentBinlogPosition

      BinlogStreamingChangeEventSource.BinlogPosition getCurrentBinlogPosition()

    • wrap

      protected DebeziumException wrap(Throwable error)
      Wraps the specified exception in a DebeziumException, ensuring that all useful state is captured inside the new exception's message.
      Parameters:
      error - the exception; may not be null
      Returns:
      the wrapped Kafka Connect exception

    • sslModeFor

      private com.github.shyiko.mysql.binlog.network.SSLMode sslModeFor(BinlogConnectorConfig.SecureConnectionMode mode)

    • getBinlogSslSocketFactory

      private com.github.shyiko.mysql.binlog.network.SSLSocketFactory getBinlogSslSocketFactory(BinlogConnectorConfig connectorConfig, BinlogConnectorConnection connection)