Package io.debezium.embedded.async

Class AsyncEmbeddedEngine<R>

java.lang.Object
io.debezium.embedded.async.AsyncEmbeddedEngine<R>
All Implemented Interfaces:
AsyncEngineConfig, EmbeddedEngineConfig, io.debezium.engine.DebeziumEngine<R>, Closeable, AutoCloseable, Runnable
public final class AsyncEmbeddedEngine<R> extends Object implements io.debezium.engine.DebeziumEngine<R>, AsyncEngineConfig
Implementation of DebeziumEngine which allows to run multiple tasks in parallel and also allows to process part or whole record processing pipeline in parallel. For more detail see DDD-7 (TODO link).
Author:
vjuranek

  • Field Details

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • config

      private final Configuration config

    • clock

      private final Clock clock

    • classLoader

      private final ClassLoader classLoader

    • consumer

      private final Consumer<R> consumer

    • handler

      private final io.debezium.engine.DebeziumEngine.ChangeConsumer<R> handler

    • completionCallback

      private final io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback

    • connectorCallback

      private final Optional<io.debezium.engine.DebeziumEngine.ConnectorCallback> connectorCallback

    • offsetKeyConverter

      private final org.apache.kafka.connect.storage.Converter offsetKeyConverter

    • offsetValueConverter

      private final org.apache.kafka.connect.storage.Converter offsetValueConverter

    • workerConfig

      private final org.apache.kafka.connect.runtime.WorkerConfig workerConfig

    • offsetCommitPolicy

      private final io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy

    • connector

      private final EngineSourceConnector connector

    • transformations

      private final Transformations transformations

    • headerConverter

      private final org.apache.kafka.connect.storage.HeaderConverter headerConverter

    • recordConverter

      private final Function<org.apache.kafka.connect.source.SourceRecord,R> recordConverter

    • sourceConverter

      private final Function<R,org.apache.kafka.connect.source.SourceRecord> sourceConverter

    • state

      private final AtomicReference<AsyncEmbeddedEngine.State> state

    • tasks

      private final List<EngineSourceTask> tasks

    • pollingFutures

      private final List<Future<Void>> pollingFutures

    • taskService

      private final ExecutorService taskService

    • recordService

      private final ExecutorService recordService

    • shutDownLatch

      private final CountDownLatch shutDownLatch

  • Constructor Details

    • AsyncEmbeddedEngine

      private AsyncEmbeddedEngine(Properties config, Consumer<R> consumer, io.debezium.engine.DebeziumEngine.ChangeConsumer<R> handler, ClassLoader classLoader, Clock clock, io.debezium.engine.DebeziumEngine.CompletionCallback completionCallback, io.debezium.engine.DebeziumEngine.ConnectorCallback connectorCallback, io.debezium.engine.spi.OffsetCommitPolicy offsetCommitPolicy, org.apache.kafka.connect.storage.HeaderConverter headerConverter, Function<org.apache.kafka.connect.source.SourceRecord,R> recordConverter)

  • Method Details

    • run

      public void run()
      Specified by:
      run in interface Runnable

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • runWithTask

      @VisibleForTesting public void runWithTask(Consumer<org.apache.kafka.connect.source.SourceTask> consumer)
      For backward compatibility with tests and for testing purposes ONLY. MUST NOT be used in user applications! Exposes tasks to a use defined consumer, which allows to run the tasks in tests.
      Parameters:
      consumer - Consumer for running tasks.

    • close

      private void close(AsyncEmbeddedEngine.State stateBeforeStop)
      Shuts down the engine. Currently, it's limited only to closing header converter and stopping the source connector.
      Parameters:
      stateBeforeStop - AsyncEmbeddedEngine.State of the engine when the shutdown was requested.

    • closeEngineWithException

      private void closeEngineWithException(Throwable exitError)
      Stops engine if needed upon the failure. The engine is stopped if it's not already stopped, or it's being stopped in right now.
      Parameters:
      exitError - Throwable which was thrown during the engine run and propagated to the main thread.

    • finishShutDown

      private void finishShutDown(Throwable exitError)
      Finish engine shut down - move it into STOPPED state and call the completion callback.
      Parameters:
      exitError - Throwable which was thrown during engine run, null is engine finished without any issue.

    • initializeConnector

      private Map<String,String> initializeConnector() throws Exception
      Initialize all the required pieces for initialization of the connector and returns configuration of the connector.
      Returns:
      Map<String,String> with connector configuration.
      Throws:
      Exception

    • createSourceTasks

      private void createSourceTasks(EngineSourceConnector connector, List<EngineSourceTask> tasks) throws NoSuchMethodException, InvocationTargetException, InstantiationException, IllegalAccessException
      Creates list of connector tasks to be started as the sources of records.
      Parameters:
      connector - EngineSourceConnector to which the source tasks belong to.
      tasks - List<EngineSourceTask> to be populated by the source tasks create in this method.
      Throws:
      NoSuchMethodException
      InvocationTargetException
      InstantiationException
      IllegalAccessException

    • startSourceTasks

      private void startSourceTasks(List<EngineSourceTask> tasks) throws Exception
      Starts the source tasks. The caller is responsible for handling possible error states. However, all the tasks are awaited to either start of fail.
      Parameters:
      tasks - List<EngineSourceTask> of tasks to be started
      Throws:
      Exception

    • runTasksPolling

      private void runTasksPolling(List<EngineSourceTask> tasks) throws ExecutionException
      Schedules polling of provided tasks and wait until all polling tasks eventually finish.
      Parameters:
      tasks - List<EngineSourceTask> of source tasks which should poll for the records.
      Throws:
      ExecutionException

    • selectRecordProcessor

      private String selectRecordProcessor()
      Select RecordProcessor class based on the user configuration.
      Returns:
      Name of the class which should be used for creating RecordProcessor instances.

    • createRecordProcessor

      private RecordProcessor createRecordProcessor(String processorClassName, EngineSourceTask task)
      Instantiate RecordProcessor based on the class name deremined in selectRecordProcessor() method.
      Returns:
      RecordProcessor instance which will be used for processing the records.

    • stopRecordService

      private void stopRecordService()
      Shuts down the ExecutorService which processes the change event records. Waits RECORD_PROCESSING_SHUTDOWN_TIMEOUT_MS milliseconds for already submitted records to finish. If the specified timeout is exceeded, the service is shut down immediately.

    • stopPollingIfNeeded

      private void stopPollingIfNeeded()
      Stops task polling if they haven't stopped yet. Some tasks may be stuck in the polling, we should interrupt such tasks.

    • stopSourceTasks

      private void stopSourceTasks(List<EngineSourceTask> tasks)
      Stops all the connector's tasks. There are no checks if the tasks were fully stated or already running, stop is always called. Also tries to stop all the other tasks which may be still running or awaiting execution in the task's thread pool.
      Parameters:
      tasks - List<EngineSourceTask> of source tasks which should be stopped.

    • stopOffsetStore

      private void stopOffsetStore(DebeziumSourceConnectorContext connectorContext)
      Stops OffsetBackingStore used by engine if there is any. If engine fails during initialization phase before connector context is created, the reference may be null and in such this method does nothing.
      Parameters:
      connectorContext - DebeziumSourceConnectorContext used by the connector or null if the context hasn't been initialized yet.

    • stopConnector

      private void stopConnector(List<EngineSourceTask> tasks, AsyncEmbeddedEngine.State engineState)
      Stops connector's tasks if they are already running and then stops connector itself.
      Parameters:
      tasks - List<EngineSourceTask> of source task should be stopped now.

    • callCompletionHandler

      private void callCompletionHandler(Throwable error)
      Calls provided implementation of DebeziumEngine.CompletionCallback.
      Parameters:
      error - Error with which the engine has failed, null if the engine has finished successfully.

    • getEngineState

      private AsyncEmbeddedEngine.State getEngineState()
      Gets the current state of the engine.
      Returns:
      current AsyncEmbeddedEngine.State of the AsyncEmbeddedEngine.

    • setEngineState

      private void setEngineState(AsyncEmbeddedEngine.State expectedState, AsyncEmbeddedEngine.State requestedState)
      Sets the new state of AsyncEmbeddedEngine. Initial state is always State.CREATING. State transition checks current engine state and if it doesn't correspond with expected state an exception is thrown as there is likely a bug in engine implementation or usage.
      Parameters:
      expectedState - expected current AsyncEmbeddedEngine.State of the AsyncEmbeddedEngine.
      requestedState - new AsyncEmbeddedEngine.State of the AsyncEmbeddedEngine to be set.
      Throws:
      IllegalStateException - The exception is thrown when expected engine state doesn't match the actual engine state.

    • validateAndGetConnectorConfig

      private Map<String,String> validateAndGetConnectorConfig(org.apache.kafka.connect.source.SourceConnector connector, String connectorClassName)
      Validates provided configuration of the Kafka Connect connector and returns its configuration if it's a valid config.
      Parameters:
      connector - Kafka Connect SourceConnector.
      connectorClassName - Class name of Kafka Connect SourceConnector.
      Returns:
      Map<String,String> with connector configuration.

    • createAndStartOffsetStore

      private org.apache.kafka.connect.storage.OffsetBackingStore createAndStartOffsetStore(Map<String,String> connectorConfig) throws Exception
      Determines which offset backing store should be used, instantiate it and starts the offset store.
      Parameters:
      connectorConfig - Map<String,String> with the connector configuration.
      Returns:
      OffsetBackingStore instance used by the engine.
      Throws:
      Exception

    • commitOffsets

      private static boolean commitOffsets(org.apache.kafka.connect.storage.OffsetStorageWriter offsetWriter, Clock clock, long commitTimeout, org.apache.kafka.connect.source.SourceTask task) throws InterruptedException, TimeoutException
      Commits the offset to OffsetBackingStore via OffsetStorageWriter.
      Parameters:
      offsetWriter - OffsetStorageWriter which performs the flushing the offset into OffsetBackingStore.
      commitTimeout - amount of time to wait for offset flush to finish before it's aborted.
      task - SourceTask which performs the offset commit.
      Returns:
      true if the offset was successfully committed, false otherwise.
      Throws:
      InterruptedException
      TimeoutException

    • buildDefaultChangeConsumer

      private static io.debezium.engine.DebeziumEngine.ChangeConsumer<org.apache.kafka.connect.source.SourceRecord> buildDefaultChangeConsumer(Consumer<org.apache.kafka.connect.source.SourceRecord> consumer)
      Build the default DebeziumEngine.ChangeConsumer from provided custom Consumer.
      Parameters:
      consumer - Consumer provided by the user.
      Returns:
      DebeziumEngine.ChangeConsumer which use user-provided Consumer for processing the Debezium records.

    • buildConvertingChangeConsumer

      private static io.debezium.engine.DebeziumEngine.ChangeConsumer buildConvertingChangeConsumer(Consumer consumer, Function<org.apache.kafka.connect.source.SourceRecord,?> recordConverter)
      Build the DebeziumEngine.ChangeConsumer from provided custom Consumer which convert records to requested format before passing them to the custom Consumer.
      Parameters:
      consumer - Consumer provided by the user.
      Returns:
      DebeziumEngine.ChangeConsumer which use user-provided Consumer for processing the Debezium records.

    • computeRecordThreads

      private int computeRecordThreads(String recordProcessingThreads)
      Determines the size of the thread pool which will be used for processing records. The value can be either number (provided as a String value) or a predefined placeholder from AsyncEmbeddedEngine.ProcessingCores enumeration. If the number of threads is provided as a number, it will be eventually limited to AsyncEngineConfig.RECORD_PROCESSING_THREADS_CAP to avoid possible overhead with too many context switches on a beefy machines with many cores, but running many other tasks.
      Parameters:
      recordProcessingThreads - Requested number of processing threads as a String. It can be a number or predefined placeholder.
      Returns:
      Either requested number of threads or minimum of AsyncEngineConfig.RECORD_PROCESSING_THREADS_CAP and AsyncEngineConfig.AVAILABLE_CORES.