io.debezium.relational.history

Class KafkaDatabaseHistory

java.lang.Object

io.debezium.relational.history.AbstractDatabaseHistory

io.debezium.relational.history.KafkaDatabaseHistory

All Implemented Interfaces:

DatabaseHistory

@NotThreadSafe
public class KafkaDatabaseHistory
extends AbstractDatabaseHistory

A DatabaseHistory implementation that records schema changes as normal SourceRecords on the specified topic, and that recovers the history by establishing a Kafka Consumer re-processing all messages on that topic.

Author:

Randall Hauch

Field Summary

Fields

Modifier and Type	Field and Description
static Field.Set	ALL_FIELDS
static Field	BOOTSTRAP_SERVERS
private static String	CONSUMER_PREFIX
private Configuration	consumerConfig
private static String	DEFAULT_TOPIC_REPLICATION_FACTOR_PROP_NAME The name of broker property defining default replication factor for topics without the explicit setting.
private static Duration	KAFKA_QUERY_TIMEOUT
private int	maxRecoveryAttempts
private static Integer	PARTITION The one and only partition of the history topic.
private int	pollIntervalMs
private org.apache.kafka.clients.producer.KafkaProducer<String,String>	producer
private static String	PRODUCER_PREFIX
private Configuration	producerConfig
private DocumentReader	reader
static Field	RECOVERY_POLL_ATTEMPTS
static Field	RECOVERY_POLL_INTERVAL_MS
static Field	TOPIC
private String	topicName

Fields inherited from class io.debezium.relational.history.AbstractDatabaseHistory

config, logger

Fields inherited from interface io.debezium.relational.history.DatabaseHistory

CONFIGURATION_FIELD_PREFIX_STRING, DDL_FILTER, NAME, SKIP_UNPARSEABLE_DDL_STATEMENTS, STORE_ONLY_MONITORED_TABLES_DDL

Constructor Summary

Constructors

Constructor and Description
KafkaDatabaseHistory()

Method Summary

Modifier and Type	Method and Description
void	configure(Configuration config, HistoryRecordComparator comparator) Configure this instance.
protected static String	consumerConfigPropertyName(String kafkaConsumerPropertyName)
boolean	exists() Determines if the DatabaseHistory entity exists
private Long	getEndOffsetOfDbHistoryTopic(Long previousEndOffset, org.apache.kafka.clients.consumer.KafkaConsumer<String,String> historyConsumer)
private org.apache.kafka.clients.admin.Config	getKafkaBrokerConfig(org.apache.kafka.clients.admin.AdminClient admin)
void	initializeStorage() Called to initialize permanent storage of the history.
protected void	recoverRecords(Consumer<HistoryRecord> records)
void	start() Start the history.
void	stop() Stop recording history and release any resources acquired since DatabaseHistory.configure(Configuration, HistoryRecordComparator).
protected void	storeRecord(HistoryRecord record)
String	toString()

Methods inherited from class io.debezium.relational.history.AbstractDatabaseHistory

record, record, recover

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_TOPIC_REPLICATION_FACTOR_PROP_NAME

private static final String DEFAULT_TOPIC_REPLICATION_FACTOR_PROP_NAME

The name of broker property defining default replication factor for topics without the explicit setting.

See Also:

kafka.server.KafkaConfig.DefaultReplicationFactorProp, Constant Field Values

TOPIC

public static final Field TOPIC

BOOTSTRAP_SERVERS

public static final Field BOOTSTRAP_SERVERS

RECOVERY_POLL_INTERVAL_MS

public static final Field RECOVERY_POLL_INTERVAL_MS

RECOVERY_POLL_ATTEMPTS

public static final Field RECOVERY_POLL_ATTEMPTS

ALL_FIELDS

public static Field.Set ALL_FIELDS

CONSUMER_PREFIX

private static final String CONSUMER_PREFIX

See Also:

Constant Field Values

PRODUCER_PREFIX

private static final String PRODUCER_PREFIX

See Also:

Constant Field Values

KAFKA_QUERY_TIMEOUT

private static final Duration KAFKA_QUERY_TIMEOUT

PARTITION

private static final Integer PARTITION

The one and only partition of the history topic.

reader

private final DocumentReader reader

topicName

private String topicName

consumerConfig

private Configuration consumerConfig

producerConfig

private Configuration producerConfig

producer

private volatile org.apache.kafka.clients.producer.KafkaProducer<String,String> producer

maxRecoveryAttempts

private int maxRecoveryAttempts

pollIntervalMs

private int pollIntervalMs

Constructor Detail

KafkaDatabaseHistory

public KafkaDatabaseHistory()

Method Detail

configure

public void configure(Configuration config,
                      HistoryRecordComparator comparator)

Description copied from interface: DatabaseHistory

Configure this instance.

Specified by:

configure in interface DatabaseHistory

Overrides:

configure in class AbstractDatabaseHistory

Parameters:

config - the configuration for this history store

comparator - the function that should be used to compare history records during recovery; may be null if the default comparator is to be used

start

public void start()

Description copied from interface: DatabaseHistory

Start the history.

Specified by:

start in interface DatabaseHistory

Overrides:

start in class AbstractDatabaseHistory

storeRecord

protected void storeRecord(HistoryRecord record)
                    throws DatabaseHistoryException

Specified by:

storeRecord in class AbstractDatabaseHistory

Throws:

DatabaseHistoryException

recoverRecords

protected void recoverRecords(Consumer<HistoryRecord> records)

Specified by:

recoverRecords in class AbstractDatabaseHistory

getEndOffsetOfDbHistoryTopic

private Long getEndOffsetOfDbHistoryTopic(Long previousEndOffset,
                                          org.apache.kafka.clients.consumer.KafkaConsumer<String,String> historyConsumer)

exists

public boolean exists()

Description copied from interface: DatabaseHistory

Determines if the DatabaseHistory entity exists

Returns:

stop

public void stop()

Description copied from interface: DatabaseHistory

Stop recording history and release any resources acquired since DatabaseHistory.configure(Configuration, HistoryRecordComparator).

Specified by:

stop in interface DatabaseHistory

Overrides:

stop in class AbstractDatabaseHistory

toString

public String toString()

Overrides:

toString in class Object

consumerConfigPropertyName

protected static String consumerConfigPropertyName(String kafkaConsumerPropertyName)

initializeStorage

public void initializeStorage()

Description copied from interface: DatabaseHistory

Called to initialize permanent storage of the history.

Specified by:

initializeStorage in interface DatabaseHistory

Overrides:

initializeStorage in class AbstractDatabaseHistory

getKafkaBrokerConfig

private org.apache.kafka.clients.admin.Config getKafkaBrokerConfig(org.apache.kafka.clients.admin.AdminClient admin)
                                                            throws Exception

Throws:

Exception