Package io.debezium.embedded

Class EmbeddedEngine

  • All Implemented Interfaces:
    io.debezium.engine.DebeziumEngine<org.apache.kafka.connect.source.SourceRecord>, Closeable, AutoCloseable, Runnable
    @ThreadSafe
public final class EmbeddedEngine
extends Object
implements io.debezium.engine.DebeziumEngine<org.apache.kafka.connect.source.SourceRecord>
    A mechanism for running a single Kafka Connect SourceConnector within an application's process. An embedded connector is entirely standalone and only talks with the source system; no Kafka, Kafka Connect, or Zookeeper processes are needed. Applications using an embedded connector simply set one up and supply a consumer function to which the connector will pass all SourceRecords containing database change events.

    With an embedded connector, the application that runs the connector assumes all responsibility for fault tolerance, scalability, and durability. Additionally, applications must specify how the connector can store its relational database schema history and offsets. By default, this information will be stored in memory and will thus be lost upon application restart.

    Embedded connectors are designed to be submitted to an Executor or ExecutorService for execution by a single thread, and a running connector can be stopped either by calling stop() from another thread or by interrupting the running thread (e.g., as is the case with ExecutorService.shutdownNow()).

    Author:
    Randall Hauch

    • Field Detail

      • ENGINE_NAME

        public static final Field ENGINE_NAME
        A required field for an embedded connector that specifies the unique name for the connector instance.

      • CONNECTOR_CLASS

        public static final Field CONNECTOR_CLASS
        A required field for an embedded connector that specifies the name of the normal Debezium connector's Java class.

      • OFFSET_STORAGE

        public static final Field OFFSET_STORAGE
        An optional field that specifies the name of the class that implements the OffsetBackingStore interface, and that will be used to store offsets recorded by the connector.

      • OFFSET_STORAGE_FILE_FILENAME

        public static final Field OFFSET_STORAGE_FILE_FILENAME
        An optional field that specifies the file location for the FileOffsetBackingStore.
        See Also:
        OFFSET_STORAGE

      • OFFSET_STORAGE_KAFKA_TOPIC

        public static final Field OFFSET_STORAGE_KAFKA_TOPIC
        An optional field that specifies the topic name for the KafkaOffsetBackingStore.
        See Also:
        OFFSET_STORAGE

      • OFFSET_STORAGE_KAFKA_PARTITIONS

        public static final Field OFFSET_STORAGE_KAFKA_PARTITIONS
        An optional field that specifies the number of partitions for the KafkaOffsetBackingStore.
        See Also:
        OFFSET_STORAGE

      • OFFSET_STORAGE_KAFKA_REPLICATION_FACTOR

        public static final Field OFFSET_STORAGE_KAFKA_REPLICATION_FACTOR
        An optional field that specifies the replication factor for the KafkaOffsetBackingStore.
        See Also:
        OFFSET_STORAGE

      • OFFSET_FLUSH_INTERVAL_MS

        public static final Field OFFSET_FLUSH_INTERVAL_MS
        An optional advanced field that specifies the maximum amount of time that the embedded connector should wait for an offset commit to complete.

      • OFFSET_COMMIT_TIMEOUT_MS

        public static final Field OFFSET_COMMIT_TIMEOUT_MS
        An optional advanced field that specifies the maximum amount of time that the embedded connector should wait for an offset commit to complete.

      • OFFSET_COMMIT_POLICY

        public static final Field OFFSET_COMMIT_POLICY

      • INTERNAL_KEY_CONVERTER_CLASS

        protected static final Field INTERNAL_KEY_CONVERTER_CLASS

      • INTERNAL_VALUE_CONVERTER_CLASS

        protected static final Field INTERNAL_VALUE_CONVERTER_CLASS

      • TRANSFORMS

        public static final Field TRANSFORMS
        A list of SMTs to be applied on the messages generated by the engine.

      • CONNECTOR_FIELDS

        public static final Field.Set CONNECTOR_FIELDS
        The array of fields that are required by each connectors.

      • ALL_FIELDS

        protected static final Field.Set ALL_FIELDS
        The array of all exposed fields.

      • WAIT_FOR_COMPLETION_BEFORE_INTERRUPT_PROP

        private static final String WAIT_FOR_COMPLETION_BEFORE_INTERRUPT_PROP
        See Also:
        Constant Field Values

      • LOGGER

        private static final org.slf4j.Logger LOGGER

      • clock

        private final Clock clock

      • handler

        private final io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler

      • completionCallback

        private final io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback

      • connectorCallback

        private final io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback

      • keyConverter

        private final org.apache.kafka.connect.storage.Converter keyConverter

      • valueConverter

        private final org.apache.kafka.connect.storage.Converter valueConverter

      • workerConfig

        private final org.apache.kafka.connect.runtime.WorkerConfig workerConfig

      • recordsSinceLastCommit

        private long recordsSinceLastCommit

      • timeOfLastCommitMillis

        private long timeOfLastCommitMillis

      • offsetCommitPolicy

        private io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy

      • task

        private org.apache.kafka.connect.source.SourceTask task

    • Constructor Detail

      • EmbeddedEngine

        private EmbeddedEngine​(Configuration config,
                       ClassLoader classLoader,
                       Clock clock,
                       io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> handler,
                       io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback,
                       io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback,
                       io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy)

    • Method Detail

      • isRunning

        public boolean isRunning()
        Determine if this embedded connector is currently running.
        Returns:
        true if running, or false otherwise

      • fail

        private void fail​(String msg)

      • succeed

        private void succeed​(String msg)

      • run

        public void run()
        Run this embedded connector and deliver database changes to the registered Consumer. This method blocks until the connector is stopped.

        First, the method checks to see if this instance is currently running, and if so immediately returns.

        If the configuration is valid, this method starts the connector and starts polling the connector for change events. All messages are delivered in batches to the Consumer registered with this embedded connector. The batch size, polling frequency, and other parameters are controlled via configuration settings. This continues until this connector is stopped.

        Note that there are two ways to stop a connector running on a thread: calling stop() from another thread, or interrupting the thread (e.g., via ExecutorService.shutdownNow()).

        This method can be called repeatedly as needed.

        Specified by:
        run in interface Runnable

      • buildRecordCommitter

        protected EmbeddedEngine.RecordCommitter buildRecordCommitter​(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter,
                                                              org.apache.kafka.connect.source.SourceTask task,
                                                              Duration commitTimeout)
        Creates a new RecordCommitter that is responsible for informing the engine about the updates to the given batch
        Parameters:
        offsetWriter - the offsetWriter current in use
        task - the sourcetask
        commitTimeout - the time in ms until a commit times out
        Returns:
        the new recordCommitter to be used for a given batch

      • maybeFlush

        protected void maybeFlush​(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter,
                          io.debezium.engine.spi.OffsetCommitPolicy policy,
                          Duration commitTimeout,
                          org.apache.kafka.connect.source.SourceTask task)
                   throws InterruptedException
        Determine if we should flush offsets to storage, and if so then attempt to flush offsets.
        Parameters:
        offsetWriter - the offset storage writer; may not be null
        policy - the offset commit policy; may not be null
        commitTimeout - the timeout to wait for commit results
        task - the task which produced the records for which the offsets have been committed
        Throws:
        InterruptedException

      • commitOffsets

        protected void commitOffsets​(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter,
                             Duration commitTimeout,
                             org.apache.kafka.connect.source.SourceTask task)
                      throws InterruptedException
        Flush offsets to storage.
        Parameters:
        offsetWriter - the offset storage writer; may not be null
        commitTimeout - the timeout to wait for commit results
        task - the task which produced the records for which the offsets have been committed
        Throws:
        InterruptedException

      • completedFlush

        protected void completedFlush​(Throwable error,
                              Void result)

      • stop

        public boolean stop()
        Stop the execution of this embedded connector. This method does not block until the connector is stopped; use await(long, TimeUnit) for this purpose.
        Returns:
        true if the connector was running and will eventually stop, or false if it was not running when this method is called
        See Also:
        await(long, TimeUnit)

      • await

        public boolean await​(long timeout,
                     TimeUnit unit)
              throws InterruptedException
        Wait for the connector to complete processing. If the processor is not running, this method returns immediately; however, if the processor is stopped and restarted before this method is called, this method will return only when it completes the second time.
        Parameters:
        timeout - the maximum amount of time to wait before returning
        unit - the unit of time; may not be null
        Returns:
        true if the connector completed within the timeout (or was not running), or false if it is still running when the timeout occurred
        Throws:
        InterruptedException - if this thread is interrupted while waiting for the completion of the connector

      • runWithTask

        public void runWithTask​(Consumer<org.apache.kafka.connect.source.SourceTask> consumer)