Package io.debezium.connector.common

Class BaseSourceTask<P extends Partition,O extends OffsetContext>

java.lang.Object
org.apache.kafka.connect.source.SourceTask
io.debezium.connector.common.BaseSourceTask<P,O>
All Implemented Interfaces:
org.apache.kafka.connect.connector.Task
public abstract class BaseSourceTask<P extends Partition,O extends OffsetContext> extends org.apache.kafka.connect.source.SourceTask
Base class for Debezium's CDC SourceTask implementations. Provides functionality common to all connectors, such as validation of the configuration.
Author:
Gunnar Morling

  • Field Details

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • INITIAL_POLL_PERIOD_IN_MILLIS

      private static final Duration INITIAL_POLL_PERIOD_IN_MILLIS

    • MAX_POLL_PERIOD_IN_MILLIS

      private static final Duration MAX_POLL_PERIOD_IN_MILLIS

    • state

      private final AtomicReference<BaseSourceTask.State> state

    • stateLock

      private final ReentrantLock stateLock
      Used to ensure that start(), stop() and commitRecord() calls are serialized.

    • restartDelay

      private volatile ElapsedTimeStrategy restartDelay

    • props

      private volatile Map<String,String> props
      Raw connector properties, kept here so they can be passed again in case of a restart.

    • coordinator

      private ChangeEventSourceCoordinator<P extends Partition,O extends OffsetContext> coordinator
      The change event source coordinator for those connectors adhering to the new framework structure, null for legacy-style connectors.

    • lastOffsets

      private final Map<Map<String,?>,Map<String,?>> lastOffsets
      The latest offsets that have been acknowledged by the Kafka producer. Will be acknowledged with the source database in commit() (which may be a no-op depending on the connector).

    • retriableRestartWait

      private Duration retriableRestartWait

    • pollOutputDelay

      private final ElapsedTimeStrategy pollOutputDelay

    • clock

      private final Clock clock

    • previousOutputInstant

      private Instant previousOutputInstant

    • previousOutputBatchSize

      private int previousOutputBatchSize

  • Constructor Details

    • BaseSourceTask

      protected BaseSourceTask()

  • Method Details

    • start

      public final void start(Map<String,String> props)
      Specified by:
      start in interface org.apache.kafka.connect.connector.Task
      Specified by:
      start in class org.apache.kafka.connect.source.SourceTask

    • start

      protected abstract ChangeEventSourceCoordinator<P,O> start(Configuration config)
      Called once when starting this source task.
      Parameters:
      config - the task configuration; implementations should wrap it in a dedicated implementation of CommonConnectorConfig and work with typed access to configuration properties that way

    • poll

      public final List<org.apache.kafka.connect.source.SourceRecord> poll() throws InterruptedException
      Specified by:
      poll in class org.apache.kafka.connect.source.SourceTask
      Throws:
      InterruptedException

    • logStatistics

      void logStatistics(List<org.apache.kafka.connect.source.SourceRecord> records)

    • updateLastOffset

      private void updateLastOffset(Map<String,?> partition, Map<String,?> lastOffset)

    • doPoll

      protected abstract List<org.apache.kafka.connect.source.SourceRecord> doPoll() throws InterruptedException
      Returns the next batch of source records, if any are available.
      Throws:
      InterruptedException

    • startIfNeededAndPossible

      private boolean startIfNeededAndPossible()
      Starts this connector in case it has been stopped after a retriable error, and the backoff period has passed.

    • stop

      public final void stop()
      Specified by:
      stop in interface org.apache.kafka.connect.connector.Task
      Specified by:
      stop in class org.apache.kafka.connect.source.SourceTask

    • stop

      private void stop(boolean restart)

    • doStop

      protected abstract void doStop()

    • commitRecord

      public void commitRecord(org.apache.kafka.connect.source.SourceRecord record) throws InterruptedException
      Overrides:
      commitRecord in class org.apache.kafka.connect.source.SourceTask
      Throws:
      InterruptedException

    • commit

      public void commit() throws InterruptedException
      Overrides:
      commit in class org.apache.kafka.connect.source.SourceTask
      Throws:
      InterruptedException

    • getAllConfigurationFields

      protected abstract Iterable<Field> getAllConfigurationFields()
      Returns all configuration Field supported by this source task.

    • getPreviousOffsets

      protected Offsets<P,O> getPreviousOffsets(Partition.Provider<P> provider, OffsetContext.Loader<O> loader)
      Loads the connector's persistent offsets (if present) via the given loader.