Package io.debezium.relational.history

Class KafkaDatabaseHistory

java.lang.Object
io.debezium.relational.history.AbstractDatabaseHistory
io.debezium.relational.history.KafkaDatabaseHistory
All Implemented Interfaces:
DatabaseHistory
@NotThreadSafe public class KafkaDatabaseHistory extends AbstractDatabaseHistory
A DatabaseHistory implementation that records schema changes as normal SourceRecords on the specified topic, and that recovers the history by establishing a Kafka Consumer re-processing all messages on that topic.
Author:
Randall Hauch

  • Field Details

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • CLEANUP_POLICY_NAME

      private static final String CLEANUP_POLICY_NAME
      See Also:

    • CLEANUP_POLICY_VALUE

      private static final String CLEANUP_POLICY_VALUE
      See Also:

    • RETENTION_MS_NAME

      private static final String RETENTION_MS_NAME
      See Also:

    • RETENTION_MS_MAX

      private static final long RETENTION_MS_MAX
      See Also:

    • RETENTION_MS_MIN

      private static final long RETENTION_MS_MIN

    • RETENTION_BYTES_NAME

      private static final String RETENTION_BYTES_NAME
      See Also:

    • UNLIMITED_VALUE

      private static final int UNLIMITED_VALUE
      See Also:

    • PARTITION_COUNT

      private static final int PARTITION_COUNT
      See Also:

    • DEFAULT_TOPIC_REPLICATION_FACTOR_PROP_NAME

      private static final String DEFAULT_TOPIC_REPLICATION_FACTOR_PROP_NAME
      The name of broker property defining default replication factor for topics without the explicit setting.
      See Also:

    • DEFAULT_TOPIC_REPLICATION_FACTOR

      private static final short DEFAULT_TOPIC_REPLICATION_FACTOR
      The default replication factor for the history topic which is used in case the value couldn't be retrieved from the broker.
      See Also:

    • TOPIC

      public static final Field TOPIC

    • BOOTSTRAP_SERVERS

      public static final Field BOOTSTRAP_SERVERS

    • RECOVERY_POLL_INTERVAL_MS

      public static final Field RECOVERY_POLL_INTERVAL_MS

    • RECOVERY_POLL_ATTEMPTS

      public static final Field RECOVERY_POLL_ATTEMPTS

    • INTERNAL_CONNECTOR_CLASS

      public static final Field INTERNAL_CONNECTOR_CLASS

    • INTERNAL_CONNECTOR_ID

      public static final Field INTERNAL_CONNECTOR_ID

    • KAFKA_QUERY_TIMEOUT_MS

      public static final Field KAFKA_QUERY_TIMEOUT_MS

    • ALL_FIELDS

      public static Field.Set ALL_FIELDS

    • CONSUMER_PREFIX

      private static final String CONSUMER_PREFIX
      See Also:

    • PRODUCER_PREFIX

      private static final String PRODUCER_PREFIX
      See Also:

    • PARTITION

      private static final Integer PARTITION
      The one and only partition of the history topic.

    • reader

      private final DocumentReader reader

    • topicName

      private String topicName

    • consumerConfig

      private Configuration consumerConfig

    • producerConfig

      private Configuration producerConfig

    • producer

      private volatile org.apache.kafka.clients.producer.KafkaProducer<String,String> producer

    • maxRecoveryAttempts

      private int maxRecoveryAttempts

    • pollInterval

      private Duration pollInterval

    • checkTopicSettingsExecutor

      private ExecutorService checkTopicSettingsExecutor

    • kafkaQueryTimeout

      private Duration kafkaQueryTimeout

    • USE_KAFKA_24_NEW_TOPIC_CONSTRUCTOR

      private static final boolean USE_KAFKA_24_NEW_TOPIC_CONSTRUCTOR

  • Constructor Details

    • KafkaDatabaseHistory

      public KafkaDatabaseHistory()

  • Method Details

    • hasNewTopicConstructorWithOptionals

      private static boolean hasNewTopicConstructorWithOptionals()

    • configure

      public void configure(Configuration config, HistoryRecordComparator comparator, DatabaseHistoryListener listener, boolean useCatalogBeforeSchema)
      Description copied from interface: DatabaseHistory
      Configure this instance.
      Specified by:
      configure in interface DatabaseHistory
      Overrides:
      configure in class AbstractDatabaseHistory
      Parameters:
      config - the configuration for this history store
      comparator - the function that should be used to compare history records during recovery; may be null if the default comparator is to be used
      listener - TODO
      useCatalogBeforeSchema - true if the parsed string for a table contains only 2 items and the first should be used as the catalog and the second as the table name, or false if the first should be used as the schema and the second as the table name

    • start

      public void start()
      Description copied from interface: DatabaseHistory
      Start the history.
      Specified by:
      start in interface DatabaseHistory
      Overrides:
      start in class AbstractDatabaseHistory

    • storeRecord

      protected void storeRecord(HistoryRecord record) throws DatabaseHistoryException
      Specified by:
      storeRecord in class AbstractDatabaseHistory
      Throws:
      DatabaseHistoryException

    • recoverRecords

      protected void recoverRecords(Consumer<HistoryRecord> records)
      Specified by:
      recoverRecords in class AbstractDatabaseHistory

    • getEndOffsetOfDbHistoryTopic

      private Long getEndOffsetOfDbHistoryTopic(Long previousEndOffset, org.apache.kafka.clients.consumer.KafkaConsumer<String,String> historyConsumer)

    • storageExists

      public boolean storageExists()
      Description copied from interface: DatabaseHistory
      Determines if the underlying storage exists (e.g. a Kafka topic, file or similar). Note: storage may exist while history entities not yet written, see DatabaseHistory.exists()

    • exists

      public boolean exists()
      Description copied from interface: DatabaseHistory
      Determines if the database history entity exists; i.e. the storage must have been initialized and the history must have been populated.

    • checkTopicSettings

      private void checkTopicSettings(String topicName)

    • stop

      public void stop()
      Description copied from interface: DatabaseHistory
      Stop recording history and release any resources acquired since #configure(Configuration, HistoryRecordComparator, DatabaseHistoryListener).
      Specified by:
      stop in interface DatabaseHistory
      Overrides:
      stop in class AbstractDatabaseHistory

    • stopCheckTopicSettingsExecutor

      private void stopCheckTopicSettingsExecutor()

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • consumerConfigPropertyName

      protected static String consumerConfigPropertyName(String kafkaConsumerPropertyName)

    • initializeStorage

      public void initializeStorage()
      Description copied from interface: DatabaseHistory
      Called to initialize permanent storage of the history.
      Specified by:
      initializeStorage in interface DatabaseHistory
      Overrides:
      initializeStorage in class AbstractDatabaseHistory

    • getDefaultTopicReplicationFactor

      private short getDefaultTopicReplicationFactor(org.apache.kafka.clients.admin.AdminClient admin) throws Exception
      Throws:
      Exception

    • getKafkaBrokerConfig

      private org.apache.kafka.clients.admin.Config getKafkaBrokerConfig(org.apache.kafka.clients.admin.AdminClient admin) throws Exception
      Throws:
      Exception

    • forKafka

      private static Field.Validator forKafka(Field.Validator validator)