Package io.debezium.embedded

Class EmbeddedEngine

java.lang.Object
io.debezium.embedded.EmbeddedEngine
All Implemented Interfaces:
EmbeddedEngineConfig, io.debezium.engine.DebeziumEngine<org.apache.kafka.connect.source.SourceRecord>, Closeable, AutoCloseable, Runnable
@ThreadSafe public final class EmbeddedEngine extends Object implements io.debezium.engine.DebeziumEngine<org.apache.kafka.connect.source.SourceRecord>, EmbeddedEngineConfig
A mechanism for running a single Kafka Connect SourceConnector within an application's process. An embedded connector is entirely standalone and only talks with the source system; no Kafka, Kafka Connect, or Zookeeper processes are needed. Applications using an embedded connector simply set one up and supply a consumer function to which the connector will pass all SourceRecords containing database change events.

With an embedded connector, the application that runs the connector assumes all responsibility for fault tolerance, scalability, and durability. Additionally, applications must specify how the connector can store its relational database schema history and offsets. By default, this information will be stored in memory and will thus be lost upon application restart.

Embedded connectors are designed to be submitted to an Executor or ExecutorService for execution by a single thread, and a running connector can be stopped either by calling stop() from another thread or by interrupting the running thread (e.g., as is the case with ExecutorService.shutdownNow()).

Author:
Randall Hauch

  • Field Details

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • config

      private final Configuration config

    • clock

      private final Clock clock

    • classLoader

      private final ClassLoader classLoader

    • handler

      private final io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler

    • completionCallback

      private final io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback

    • connectorCallback

      private final io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback

    • runningThread

      private final AtomicReference<Thread> runningThread

    • latch

      private final VariableLatch latch

    • keyConverter

      private final org.apache.kafka.connect.storage.Converter keyConverter

    • valueConverter

      private final org.apache.kafka.connect.storage.Converter valueConverter

    • workerConfig

      private final org.apache.kafka.connect.runtime.WorkerConfig workerConfig

    • completionResult

      private final EmbeddedEngine.CompletionResult completionResult

    • recordsSinceLastCommit

      private long recordsSinceLastCommit

    • timeOfLastCommitMillis

      private long timeOfLastCommitMillis

    • offsetCommitPolicy

      private io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy

    • task

      private org.apache.kafka.connect.source.SourceTask task

    • transformations

      private final Transformations transformations

  • Constructor Details

    • EmbeddedEngine

      private EmbeddedEngine(Configuration config, ClassLoader classLoader, Clock clock, io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler, io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback, io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback, io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy)

  • Method Details

    • buildDefaultChangeConsumer

      private static io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> buildDefaultChangeConsumer(Consumer<org.apache.kafka.connect.source.SourceRecord> consumer)

    • isRunning

      public boolean isRunning()
      Determine if this embedded connector is currently running.
      Returns:
      true if running, or false otherwise

    • fail

      private void fail(String msg)

    • fail

      private void fail(String msg, Throwable error)

    • failAndThrow

      private void failAndThrow(String msg, Throwable error) throws EmbeddedEngine.EmbeddedEngineRuntimeException
      Throws:
      EmbeddedEngine.EmbeddedEngineRuntimeException

    • succeed

      private void succeed(String msg)

    • run

      public void run()
      Run this embedded connector and deliver database changes to the registered Consumer. This method blocks until the connector is stopped.

      First, the method checks to see if this instance is currently running, and if so immediately returns.

      If the configuration is valid, this method starts the connector and starts polling the connector for change events. All messages are delivered in batches to the Consumer registered with this embedded connector. The batch size, polling frequency, and other parameters are controlled via configuration settings. This continues until this connector is stopped.

      Note that there are two ways to stop a connector running on a thread: calling stop() from another thread, or interrupting the thread (e.g., via ExecutorService.shutdownNow()).

      This method can be called repeatedly as needed.

      Specified by:
      run in interface Runnable

    • instantiateConnector

      private org.apache.kafka.connect.source.SourceConnector instantiateConnector(String connectorClassName) throws EmbeddedEngine.EmbeddedEngineRuntimeException
      Throws:
      EmbeddedEngine.EmbeddedEngineRuntimeException

    • getConnectorConfig

      private Map<String,String> getConnectorConfig(org.apache.kafka.connect.source.SourceConnector connector, String connectorClassName) throws EmbeddedEngine.EmbeddedEngineRuntimeException
      Throws:
      EmbeddedEngine.EmbeddedEngineRuntimeException

    • initializeOffsetStore

      private org.apache.kafka.connect.storage.OffsetBackingStore initializeOffsetStore(Map<String,String> connectorConfig) throws EmbeddedEngine.EmbeddedEngineRuntimeException
      Determines, which offset backing store should be used, instantiate it and start the offset store.
      Throws:
      EmbeddedEngine.EmbeddedEngineRuntimeException

    • setOffsetCommitPolicy

      private void setOffsetCommitPolicy() throws EmbeddedEngine.EmbeddedEngineRuntimeException
      Throws:
      EmbeddedEngine.EmbeddedEngineRuntimeException

    • initializeConnector

      private void initializeConnector(org.apache.kafka.connect.source.SourceConnector connector, org.apache.kafka.connect.storage.OffsetStorageReader offsetReader)

    • createSourceTask

      private org.apache.kafka.connect.source.SourceTask createSourceTask(org.apache.kafka.connect.source.SourceConnector connector, List<Map<String,String>> taskConfigs, Class<? extends org.apache.kafka.connect.connector.Task> taskClass) throws EmbeddedEngine.EmbeddedEngineRuntimeException, NoSuchMethodException, InvocationTargetException
      Throws:
      EmbeddedEngine.EmbeddedEngineRuntimeException
      NoSuchMethodException
      InvocationTargetException

    • startSourceTask

      private void startSourceTask(List<Map<String,String>> taskConfigs, org.apache.kafka.connect.storage.OffsetStorageReader offsetReader)

    • stopSourceTask

      private void stopSourceTask()

    • handleRetries

      private Throwable handleRetries(org.apache.kafka.connect.errors.RetriableException e, List<Map<String,String>> taskConfigs)

    • pollRecords

      private void pollRecords(List<Map<String,String>> taskConfigs, io.debezium.engine.DebeziumEngine.RecordCommitter committer, EmbeddedEngine.HandlerErrors errors) throws Throwable
      Throws:
      Throwable

    • setCompletionResult

      private void setCompletionResult(String connectorClassName, EmbeddedEngine.HandlerErrors errors)

    • stopTaskAndCommitOffset

      private void stopTaskAndCommitOffset(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, Duration commitTimeout, Optional<io.debezium.engine.DebeziumEngine.ConnectorCallback> connectorCallback)

    • stopOffsetStoreAndConnector

      private void stopOffsetStoreAndConnector(org.apache.kafka.connect.source.SourceConnector connector, String connectorClassName, org.apache.kafka.connect.storage.OffsetBackingStore offsetStore, Optional<io.debezium.engine.DebeziumEngine.ConnectorCallback> connectorCallback)

    • getErrorsMaxRetries

      private int getErrorsMaxRetries()

    • buildRecordCommitter

      protected io.debezium.engine.DebeziumEngine.RecordCommitter buildRecordCommitter(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, org.apache.kafka.connect.source.SourceTask task, Duration commitTimeout)
      Creates a new RecordCommitter that is responsible for informing the engine about the updates to the given batch
      Parameters:
      offsetWriter - the offsetWriter current in use
      task - the sourcetask
      commitTimeout - the time in ms until a commit times out
      Returns:
      the new recordCommitter to be used for a given batch

    • maybeFlush

      protected void maybeFlush(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, io.debezium.engine.spi.OffsetCommitPolicy policy, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) throws InterruptedException
      Determine if we should flush offsets to storage, and if so then attempt to flush offsets.
      Parameters:
      offsetWriter - the offset storage writer; may not be null
      policy - the offset commit policy; may not be null
      commitTimeout - the timeout to wait for commit results
      task - the task which produced the records for which the offsets have been committed
      Throws:
      InterruptedException

    • commitOffsets

      protected void commitOffsets(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) throws InterruptedException
      Flush offsets to storage.
      Parameters:
      offsetWriter - the offset storage writer; may not be null
      commitTimeout - the timeout to wait for commit results
      task - the task which produced the records for which the offsets have been committed
      Throws:
      InterruptedException

    • completedFlush

      protected void completedFlush(Throwable error, Void result)

    • stop

      public boolean stop()
      Stop the execution of this embedded connector. This method does not block until the connector is stopped; use await(long, TimeUnit) for this purpose.
      Returns:
      true if the connector was running and will eventually stop, or false if it was not running when this method is called
      See Also:

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • await

      public boolean await(long timeout, TimeUnit unit) throws InterruptedException
      Wait for the connector to complete processing. If the processor is not running, this method returns immediately; however, if the processor is stopped and restarted before this method is called, this method will return only when it completes the second time.
      Parameters:
      timeout - the maximum amount of time to wait before returning
      unit - the unit of time; may not be null
      Returns:
      true if the connector completed within the timeout (or was not running), or false if it is still running when the timeout occurred
      Throws:
      InterruptedException - if this thread is interrupted while waiting for the completion of the connector

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • runWithTask

      public void runWithTask(Consumer<org.apache.kafka.connect.source.SourceTask> consumer)

    • delayStrategy

      private DelayStrategy delayStrategy(Configuration config)