Package io.debezium.pipeline

Class EventDispatcher<P extends Partition,T extends DataCollectionId>

java.lang.Object

io.debezium.pipeline.EventDispatcher<P,T>

All Implemented Interfaces:

AutoCloseable

public class EventDispatcher<P extends Partition,T extends DataCollectionId>
extends Object
implements AutoCloseable

Central dispatcher for data change and schema change events. The former will be routed to the change event queue, the latter will be routed to the DatabaseSchema. But based on the applying include/exclude list configuration, events may be not be dispatched at all.

This router is also in charge of emitting heartbeat messages, exposing of metrics via JMX etc.

Author:

Gunnar Morling

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private final class	EventDispatcher.BufferingSnapshotChangeRecordReceiver
static interface	EventDispatcher.InconsistentSchemaHandler<P extends Partition,T extends DataCollectionId>	Reaction to an incoming change event for which schema is not found
private final class	EventDispatcher.IncrementalSnapshotChangeRecordReceiver
private final class	EventDispatcher.SchemaChangeEventReceiver
static interface	EventDispatcher.SnapshotReceiver<P extends Partition>	Change record receiver used during snapshotting.
private final class	EventDispatcher.StreamingChangeRecordReceiver

Field Summary

Fields

Modifier and Type	Field	Description
private final ChangeEventCreator	changeEventCreator
private final CommonConnectorConfig	connectorConfig
private final boolean	emitTombstonesOnDelete
private DataChangeEventListener<P>	eventListener
private final DataCollectionFilters.DataCollectionFilter<T>	filter
private final Heartbeat	heartbeat
private final HistorizedDatabaseSchema<T>	historizedSchema
private final EventDispatcher.InconsistentSchemaHandler<P,T>	inconsistentSchemaHandler
private IncrementalSnapshotChangeEventSource<P,T>	incrementalSnapshotChangeEventSource
private static final org.slf4j.Logger	LOGGER
private final boolean	neverSkip
private final ChangeEventQueue<DataChangeEvent>	queue
private final DatabaseSchema<T>	schema
private final org.apache.kafka.connect.data.Schema	schemaChangeKeySchema
private final org.apache.kafka.connect.data.Schema	schemaChangeValueSchema
private final Signal<P>	signal
private final EnumSet<Envelope.Operation>	skippedOperations
private final EventDispatcher<P,T>.StreamingChangeRecordReceiver	streamingReceiver	Change event receiver for events dispatched from a streaming change event source.
private final ConnectTableChangeSerializer	tableChangesSerializer
private final TopicNamingStrategy<T>	topicNamingStrategy
private final TransactionMonitor	transactionMonitor

Constructor Summary

Constructors

Constructor	Description
EventDispatcher(CommonConnectorConfig connectorConfig, TopicNamingStrategy<T> topicNamingStrategy, DatabaseSchema<T> schema, ChangeEventQueue<DataChangeEvent> queue, DataCollectionFilters.DataCollectionFilter<T> filter, ChangeEventCreator changeEventCreator, EventDispatcher.InconsistentSchemaHandler<P,T> inconsistentSchemaHandler, Heartbeat heartbeat, SchemaNameAdjuster schemaNameAdjuster, TransactionMonitor transactionMonitor)
EventDispatcher(CommonConnectorConfig connectorConfig, TopicNamingStrategy<T> topicNamingStrategy, DatabaseSchema<T> schema, ChangeEventQueue<DataChangeEvent> queue, DataCollectionFilters.DataCollectionFilter<T> filter, ChangeEventCreator changeEventCreator, EventDispatcher.InconsistentSchemaHandler<P,T> inconsistentSchemaHandler, EventMetadataProvider metadataProvider, Heartbeat heartbeat, SchemaNameAdjuster schemaNameAdjuster)
EventDispatcher(CommonConnectorConfig connectorConfig, TopicNamingStrategy<T> topicNamingStrategy, DatabaseSchema<T> schema, ChangeEventQueue<DataChangeEvent> queue, DataCollectionFilters.DataCollectionFilter<T> filter, ChangeEventCreator changeEventCreator, EventMetadataProvider metadataProvider, Heartbeat heartbeat, SchemaNameAdjuster schemaNameAdjuster)
EventDispatcher(CommonConnectorConfig connectorConfig, TopicNamingStrategy<T> topicNamingStrategy, DatabaseSchema<T> schema, ChangeEventQueue<DataChangeEvent> queue, DataCollectionFilters.DataCollectionFilter<T> filter, ChangeEventCreator changeEventCreator, EventMetadataProvider metadataProvider, SchemaNameAdjuster schemaNameAdjuster)

Method Summary

Modifier and Type	Method	Description
void	alwaysDispatchHeartbeatEvent(P partition, OffsetContext offset)
void	close()
void	dispatchConnectorEvent(P partition, ConnectorEvent event)
boolean	dispatchDataChangeEvent(P partition, T dataCollectionId, ChangeRecordEmitter<P> changeRecordEmitter)	Dispatches one or more DataChangeEvents.
void	dispatchFilteredEvent(P partition, OffsetContext offset)
void	dispatchHeartbeatEvent(P partition, OffsetContext offset)
void	dispatchSchemaChangeEvent(Collection<T> dataCollectionIds, SchemaChangeEventEmitter schemaChangeEventEmitter)
void	dispatchSchemaChangeEvent(P partition, T dataCollectionId, SchemaChangeEventEmitter schemaChangeEventEmitter)
void	dispatchServerHeartbeatEvent(P partition, OffsetContext offset)
void	dispatchSnapshotEvent(P partition, T dataCollectionId, ChangeRecordEmitter<P> changeRecordEmitter, EventDispatcher.SnapshotReceiver<P> receiver)
void	dispatchTransactionCommittedEvent(P partition, OffsetContext offset, Instant timestamp)
void	dispatchTransactionStartedEvent(P partition, String transactionId, OffsetContext offset, Instant timestamp)
private void	enqueueHeartbeat(org.apache.kafka.connect.source.SourceRecord record)
private void	enqueueSchemaChangeMessage(org.apache.kafka.connect.source.SourceRecord record)
private void	enqueueTransactionMessage(org.apache.kafka.connect.source.SourceRecord record)
Optional<DataCollectionSchema>	errorOnMissingSchema(P partition, T dataCollectionId, ChangeRecordEmitter<P> changeRecordEmitter)
HistorizedDatabaseSchema<T>	getHistorizedSchema()
EventDispatcher.SnapshotReceiver<P>	getIncrementalSnapshotChangeEventReceiver(DataChangeEventListener<P> dataListener)
IncrementalSnapshotChangeEventSource<P,T>	getIncrementalSnapshotChangeEventSource()
DatabaseSchema<T>	getSchema()
EventDispatcher.SnapshotReceiver<P>	getSnapshotChangeEventReceiver()
boolean	heartbeatsEnabled()
Optional<DataCollectionSchema>	ignoreMissingSchema(T dataCollectionId, ChangeRecordEmitter<P> changeRecordEmitter)
void	setEventListener(DataChangeEventListener<P> eventListener)	Provide a listener that is invoked for every incoming event to be processed.
void	setIncrementalSnapshotChangeEventSource(Optional<IncrementalSnapshotChangeEventSource<P,? extends DataCollectionId>> incrementalSnapshotChangeEventSource)	Enable support for incremental snapshotting.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

topicNamingStrategy

private final TopicNamingStrategy<T extends DataCollectionId> topicNamingStrategy

schema

private final DatabaseSchema<T extends DataCollectionId> schema

historizedSchema

private final HistorizedDatabaseSchema<T extends DataCollectionId> historizedSchema

queue

private final ChangeEventQueue<DataChangeEvent> queue

filter

private final DataCollectionFilters.DataCollectionFilter<T extends DataCollectionId> filter

changeEventCreator

private final ChangeEventCreator changeEventCreator

heartbeat

private final Heartbeat heartbeat

eventListener

private DataChangeEventListener<P extends Partition> eventListener

emitTombstonesOnDelete

private final boolean emitTombstonesOnDelete

inconsistentSchemaHandler

private final EventDispatcher.InconsistentSchemaHandler<P extends Partition,T extends DataCollectionId> inconsistentSchemaHandler

transactionMonitor

private final TransactionMonitor transactionMonitor

connectorConfig

private final CommonConnectorConfig connectorConfig

skippedOperations

private final EnumSet<Envelope.Operation> skippedOperations

neverSkip

private final boolean neverSkip

schemaChangeKeySchema

private final org.apache.kafka.connect.data.Schema schemaChangeKeySchema

schemaChangeValueSchema

private final org.apache.kafka.connect.data.Schema schemaChangeValueSchema

tableChangesSerializer

private final ConnectTableChangeSerializer tableChangesSerializer

signal

private final Signal<P extends Partition> signal

incrementalSnapshotChangeEventSource

private IncrementalSnapshotChangeEventSource<P extends Partition,T extends DataCollectionId> incrementalSnapshotChangeEventSource

streamingReceiver

private final EventDispatcher<P extends Partition,T extends DataCollectionId>.StreamingChangeRecordReceiver streamingReceiver

Change event receiver for events dispatched from a streaming change event source.

Constructor Details

EventDispatcher

public EventDispatcher(CommonConnectorConfig connectorConfig,
 TopicNamingStrategy<T> topicNamingStrategy,
 DatabaseSchema<T> schema,
 ChangeEventQueue<DataChangeEvent> queue,
 DataCollectionFilters.DataCollectionFilter<T> filter,
 ChangeEventCreator changeEventCreator,
 EventMetadataProvider metadataProvider,
 SchemaNameAdjuster schemaNameAdjuster)

EventDispatcher

public EventDispatcher(CommonConnectorConfig connectorConfig,
 TopicNamingStrategy<T> topicNamingStrategy,
 DatabaseSchema<T> schema,
 ChangeEventQueue<DataChangeEvent> queue,
 DataCollectionFilters.DataCollectionFilter<T> filter,
 ChangeEventCreator changeEventCreator,
 EventMetadataProvider metadataProvider,
 Heartbeat heartbeat,
 SchemaNameAdjuster schemaNameAdjuster)

EventDispatcher

public EventDispatcher(CommonConnectorConfig connectorConfig,
 TopicNamingStrategy<T> topicNamingStrategy,
 DatabaseSchema<T> schema,
 ChangeEventQueue<DataChangeEvent> queue,
 DataCollectionFilters.DataCollectionFilter<T> filter,
 ChangeEventCreator changeEventCreator,
 EventDispatcher.InconsistentSchemaHandler<P,T> inconsistentSchemaHandler,
 EventMetadataProvider metadataProvider,
 Heartbeat heartbeat,
 SchemaNameAdjuster schemaNameAdjuster)

EventDispatcher

public EventDispatcher(CommonConnectorConfig connectorConfig,
 TopicNamingStrategy<T> topicNamingStrategy,
 DatabaseSchema<T> schema,
 ChangeEventQueue<DataChangeEvent> queue,
 DataCollectionFilters.DataCollectionFilter<T> filter,
 ChangeEventCreator changeEventCreator,
 EventDispatcher.InconsistentSchemaHandler<P,T> inconsistentSchemaHandler,
 Heartbeat heartbeat,
 SchemaNameAdjuster schemaNameAdjuster,
 TransactionMonitor transactionMonitor)

Method Details

dispatchSnapshotEvent

public void dispatchSnapshotEvent(P partition,
 T dataCollectionId,
 ChangeRecordEmitter<P> changeRecordEmitter,
 EventDispatcher.SnapshotReceiver<P> receiver)
                           throws InterruptedException

Throws:

InterruptedException

getSnapshotChangeEventReceiver

public EventDispatcher.SnapshotReceiver<P> getSnapshotChangeEventReceiver()

getIncrementalSnapshotChangeEventReceiver

public EventDispatcher.SnapshotReceiver<P> getIncrementalSnapshotChangeEventReceiver(DataChangeEventListener<P> dataListener)

dispatchDataChangeEvent

public boolean dispatchDataChangeEvent(P partition,
 T dataCollectionId,
 ChangeRecordEmitter<P> changeRecordEmitter)
                                throws InterruptedException

Dispatches one or more DataChangeEvents. If the given data collection is included in the currently captured set of collections, the given emitter will be invoked, so it can emit one or more events (in the common case, one event will be emitted, but e.g. in case of PK updates, it may be a deletion and a creation event). The receiving coordinator creates SourceRecords for all emitted events and passes them to this dispatcher's ChangeEventCreator for converting them into data change events.

Returns:

true if an event was dispatched (i.e. sent to the message broker), false otherwise.

Throws:

InterruptedException

dispatchFilteredEvent

public void dispatchFilteredEvent(P partition,
 OffsetContext offset)
                           throws InterruptedException

Throws:

InterruptedException

dispatchTransactionCommittedEvent

public void dispatchTransactionCommittedEvent(P partition,
 OffsetContext offset,
 Instant timestamp)
                                       throws InterruptedException

Throws:

InterruptedException

dispatchTransactionStartedEvent

public void dispatchTransactionStartedEvent(P partition,
 String transactionId,
 OffsetContext offset,
 Instant timestamp)
                                     throws InterruptedException

Throws:

InterruptedException

dispatchConnectorEvent

public void dispatchConnectorEvent(P partition,
 ConnectorEvent event)

errorOnMissingSchema

public Optional<DataCollectionSchema> errorOnMissingSchema(P partition,
 T dataCollectionId,
 ChangeRecordEmitter<P> changeRecordEmitter)

ignoreMissingSchema

public Optional<DataCollectionSchema> ignoreMissingSchema(T dataCollectionId,
 ChangeRecordEmitter<P> changeRecordEmitter)

dispatchSchemaChangeEvent

public void dispatchSchemaChangeEvent(P partition,
 T dataCollectionId,
 SchemaChangeEventEmitter schemaChangeEventEmitter)
                               throws InterruptedException

Throws:

InterruptedException

dispatchSchemaChangeEvent

public void dispatchSchemaChangeEvent(Collection<T> dataCollectionIds,
 SchemaChangeEventEmitter schemaChangeEventEmitter)
                               throws InterruptedException

Throws:

InterruptedException

alwaysDispatchHeartbeatEvent

public void alwaysDispatchHeartbeatEvent(P partition,
 OffsetContext offset)
                                  throws InterruptedException

Throws:

InterruptedException

dispatchHeartbeatEvent

public void dispatchHeartbeatEvent(P partition,
 OffsetContext offset)
                            throws InterruptedException

Throws:

InterruptedException

heartbeatsEnabled

public boolean heartbeatsEnabled()

enqueueHeartbeat

private void enqueueHeartbeat(org.apache.kafka.connect.source.SourceRecord record)
                       throws InterruptedException

Throws:

InterruptedException

enqueueTransactionMessage

private void enqueueTransactionMessage(org.apache.kafka.connect.source.SourceRecord record)
                                throws InterruptedException

Throws:

InterruptedException

enqueueSchemaChangeMessage

private void enqueueSchemaChangeMessage(org.apache.kafka.connect.source.SourceRecord record)
                                 throws InterruptedException

Throws:

InterruptedException

dispatchServerHeartbeatEvent

public void dispatchServerHeartbeatEvent(P partition,
 OffsetContext offset)
                                  throws InterruptedException

Throws:

InterruptedException

setEventListener

public void setEventListener(DataChangeEventListener<P> eventListener)

Provide a listener that is invoked for every incoming event to be processed.

Parameters:

eventListener -

setIncrementalSnapshotChangeEventSource

public void setIncrementalSnapshotChangeEventSource(Optional<IncrementalSnapshotChangeEventSource<P,? extends DataCollectionId>> incrementalSnapshotChangeEventSource)

Enable support for incremental snapshotting.

getSchema

public DatabaseSchema<T> getSchema()

getHistorizedSchema

public HistorizedDatabaseSchema<T> getHistorizedSchema()

getIncrementalSnapshotChangeEventSource

public IncrementalSnapshotChangeEventSource<P,T> getIncrementalSnapshotChangeEventSource()

close

public void close()

Specified by:

close in interface AutoCloseable