Package io.debezium.embedded

Class EmbeddedEngine

java.lang.Object
io.debezium.embedded.EmbeddedEngine
All Implemented Interfaces:
io.debezium.engine.DebeziumEngine<org.apache.kafka.connect.source.SourceRecord>, Closeable, AutoCloseable, Runnable
@ThreadSafe public final class EmbeddedEngine extends Object implements io.debezium.engine.DebeziumEngine<org.apache.kafka.connect.source.SourceRecord>
A mechanism for running a single Kafka Connect SourceConnector within an application's process. An embedded connector is entirely standalone and only talks with the source system; no Kafka, Kafka Connect, or Zookeeper processes are needed. Applications using an embedded connector simply set one up and supply a consumer function to which the connector will pass all SourceRecords containing database change events.

With an embedded connector, the application that runs the connector assumes all responsibility for fault tolerance, scalability, and durability. Additionally, applications must specify how the connector can store its relational database schema history and offsets. By default, this information will be stored in memory and will thus be lost upon application restart.

Embedded connectors are designed to be submitted to an Executor or ExecutorService for execution by a single thread, and a running connector can be stopped either by calling stop() from another thread or by interrupting the running thread (e.g., as is the case with ExecutorService.shutdownNow()).

Author:
Randall Hauch

  • Field Details

    • ENGINE_NAME

      public static final Field ENGINE_NAME
      A required field for an embedded connector that specifies the unique name for the connector instance.

    • CONNECTOR_CLASS

      public static final Field CONNECTOR_CLASS
      A required field for an embedded connector that specifies the name of the normal Debezium connector's Java class.

    • OFFSET_STORAGE

      public static final Field OFFSET_STORAGE
      An optional field that specifies the name of the class that implements the OffsetBackingStore interface, and that will be used to store offsets recorded by the connector.

    • OFFSET_STORAGE_FILE_FILENAME

      public static final Field OFFSET_STORAGE_FILE_FILENAME
      An optional field that specifies the file location for the FileOffsetBackingStore.
      See Also:

    • OFFSET_STORAGE_KAFKA_TOPIC

      public static final Field OFFSET_STORAGE_KAFKA_TOPIC
      An optional field that specifies the topic name for the KafkaOffsetBackingStore.
      See Also:

    • OFFSET_STORAGE_KAFKA_PARTITIONS

      public static final Field OFFSET_STORAGE_KAFKA_PARTITIONS
      An optional field that specifies the number of partitions for the KafkaOffsetBackingStore.
      See Also:

    • OFFSET_STORAGE_KAFKA_REPLICATION_FACTOR

      public static final Field OFFSET_STORAGE_KAFKA_REPLICATION_FACTOR
      An optional field that specifies the replication factor for the KafkaOffsetBackingStore.
      See Also:

    • OFFSET_FLUSH_INTERVAL_MS

      public static final Field OFFSET_FLUSH_INTERVAL_MS
      An optional advanced field that specifies the maximum amount of time that the embedded connector should wait for an offset commit to complete.

    • OFFSET_COMMIT_TIMEOUT_MS

      public static final Field OFFSET_COMMIT_TIMEOUT_MS
      An optional advanced field that specifies the maximum amount of time that the embedded connector should wait for an offset commit to complete.

    • OFFSET_COMMIT_POLICY

      public static final Field OFFSET_COMMIT_POLICY

    • PREDICATES

      public static final Field PREDICATES
      A list of Predicates that can be assigned to transformations.

    • TRANSFORMS

      public static final Field TRANSFORMS
      A list of SMTs to be applied on the messages generated by the engine.

    • DEFAULT_ERROR_MAX_RETRIES

      private static final int DEFAULT_ERROR_MAX_RETRIES
      See Also:

    • ERRORS_MAX_RETRIES

      private static final Field ERRORS_MAX_RETRIES

    • ERRORS_RETRY_DELAY_INITIAL_MS

      private static final Field ERRORS_RETRY_DELAY_INITIAL_MS

    • ERRORS_RETRY_DELAY_MAX_MS

      private static final Field ERRORS_RETRY_DELAY_MAX_MS

    • CONNECTOR_FIELDS

      public static final Field.Set CONNECTOR_FIELDS
      The array of fields that are required by each connectors.

    • ALL_FIELDS

      protected static final Field.Set ALL_FIELDS
      The array of all exposed fields.

    • WAIT_FOR_COMPLETION_BEFORE_INTERRUPT_DEFAULT

      private static final Duration WAIT_FOR_COMPLETION_BEFORE_INTERRUPT_DEFAULT
      How long we wait before forcefully stopping the connector thread when shutting down. Must be longer than ChangeEventSourceCoordinator.SHUTDOWN_WAIT_TIMEOUT * 2.

    • WAIT_FOR_COMPLETION_BEFORE_INTERRUPT_PROP

      private static final String WAIT_FOR_COMPLETION_BEFORE_INTERRUPT_PROP
      See Also:

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • config

      private final Configuration config

    • clock

      private final Clock clock

    • classLoader

      private final ClassLoader classLoader

    • handler

      private final io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler

    • completionCallback

      private final io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback

    • connectorCallback

      private final io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback

    • runningThread

      private final AtomicReference<Thread> runningThread

    • latch

      private final VariableLatch latch

    • keyConverter

      private final org.apache.kafka.connect.storage.Converter keyConverter

    • valueConverter

      private final org.apache.kafka.connect.storage.Converter valueConverter

    • workerConfig

      private final org.apache.kafka.connect.runtime.WorkerConfig workerConfig

    • completionResult

      private final EmbeddedEngine.CompletionResult completionResult

    • recordsSinceLastCommit

      private long recordsSinceLastCommit

    • timeOfLastCommitMillis

      private long timeOfLastCommitMillis

    • offsetCommitPolicy

      private io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy

    • task

      private org.apache.kafka.connect.source.SourceTask task

    • transformations

      private final Transformations transformations

  • Constructor Details

    • EmbeddedEngine

      private EmbeddedEngine(Configuration config, ClassLoader classLoader, Clock clock, io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler, io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback, io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback, io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy)

  • Method Details

    • buildDefaultChangeConsumer

      private static EmbeddedEngine.ChangeConsumer buildDefaultChangeConsumer(Consumer<org.apache.kafka.connect.source.SourceRecord> consumer)

    • create

      @Deprecated public static EmbeddedEngine.Builder create()
      Deprecated.
      Obtain a new EmbeddedEngine.Builder instance that can be used to construct runnable EmbeddedEngine instances.
      Returns:
      the new builder; never null

    • isRunning

      public boolean isRunning()
      Determine if this embedded connector is currently running.
      Returns:
      true if running, or false otherwise

    • fail

      private void fail(String msg)

    • fail

      private void fail(String msg, Throwable error)

    • succeed

      private void succeed(String msg)

    • run

      public void run()
      Run this embedded connector and deliver database changes to the registered Consumer. This method blocks until the connector is stopped.

      First, the method checks to see if this instance is currently running, and if so immediately returns.

      If the configuration is valid, this method starts the connector and starts polling the connector for change events. All messages are delivered in batches to the Consumer registered with this embedded connector. The batch size, polling frequency, and other parameters are controlled via configuration settings. This continues until this connector is stopped.

      Note that there are two ways to stop a connector running on a thread: calling stop() from another thread, or interrupting the thread (e.g., via ExecutorService.shutdownNow()).

      This method can be called repeatedly as needed.

      Specified by:
      run in interface Runnable

    • getErrorsMaxRetries

      private int getErrorsMaxRetries()

    • buildRecordCommitter

      protected EmbeddedEngine.RecordCommitter buildRecordCommitter(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, org.apache.kafka.connect.source.SourceTask task, Duration commitTimeout)
      Creates a new RecordCommitter that is responsible for informing the engine about the updates to the given batch
      Parameters:
      offsetWriter - the offsetWriter current in use
      task - the sourcetask
      commitTimeout - the time in ms until a commit times out
      Returns:
      the new recordCommitter to be used for a given batch

    • maybeFlush

      protected void maybeFlush(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, io.debezium.engine.spi.OffsetCommitPolicy policy, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) throws InterruptedException
      Determine if we should flush offsets to storage, and if so then attempt to flush offsets.
      Parameters:
      offsetWriter - the offset storage writer; may not be null
      policy - the offset commit policy; may not be null
      commitTimeout - the timeout to wait for commit results
      task - the task which produced the records for which the offsets have been committed
      Throws:
      InterruptedException

    • commitOffsets

      protected void commitOffsets(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, Duration commitTimeout, org.apache.kafka.connect.source.SourceTask task) throws InterruptedException
      Flush offsets to storage.
      Parameters:
      offsetWriter - the offset storage writer; may not be null
      commitTimeout - the timeout to wait for commit results
      task - the task which produced the records for which the offsets have been committed
      Throws:
      InterruptedException

    • completedFlush

      protected void completedFlush(Throwable error, Void result)

    • stop

      public boolean stop()
      Stop the execution of this embedded connector. This method does not block until the connector is stopped; use await(long, TimeUnit) for this purpose.
      Returns:
      true if the connector was running and will eventually stop, or false if it was not running when this method is called
      See Also:

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • await

      public boolean await(long timeout, TimeUnit unit) throws InterruptedException
      Wait for the connector to complete processing. If the processor is not running, this method returns immediately; however, if the processor is stopped and restarted before this method is called, this method will return only when it completes the second time.
      Parameters:
      timeout - the maximum amount of time to wait before returning
      unit - the unit of time; may not be null
      Returns:
      true if the connector completed within the timeout (or was not running), or false if it is still running when the timeout occurred
      Throws:
      InterruptedException - if this thread is interrupted while waiting for the completion of the connector

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • runWithTask

      public void runWithTask(Consumer<org.apache.kafka.connect.source.SourceTask> consumer)

    • delayStrategy

      private DelayStrategy delayStrategy(Configuration config)