io.debezium.pipeline

Class EventDispatcher<T extends DataCollectionId>

java.lang.Object

io.debezium.pipeline.EventDispatcher<T>

public class EventDispatcher<T extends DataCollectionId>
extends Object

Central dispatcher for data change and schema change events. The former will be routed to the change event queue, the latter will be routed to the DatabaseSchema. But based on the applying whitelist/blacklist configuration, events may be not be dispatched at all.

This router is also in charge of emitting heartbeat messages, exposing of metrics via JMX etc.

Author:

Gunnar Morling

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private class	EventDispatcher.BufferingSnapshotChangeRecordReceiver
static interface	EventDispatcher.InconsistentSchemaHandler<T extends DataCollectionId> Reaction to an incoming change event for which schema is not found
private class	EventDispatcher.SchemaChangeEventReceiver
static interface	EventDispatcher.SnapshotReceiver Change record receiver used during snapshotting.
private class	EventDispatcher.StreamingChangeRecordReceiver

Field Summary

Fields

Modifier and Type	Field and Description
private ChangeEventCreator	changeEventCreator
private CommonConnectorConfig	connectorConfig
private boolean	emitTombstonesOnDelete
private DataChangeEventListener	eventListener
private DataCollectionFilters.DataCollectionFilter<T>	filter
private Heartbeat	heartbeat
private HistorizedDatabaseSchema<T>	historizedSchema
private EventDispatcher.InconsistentSchemaHandler<T>	inconsistentSchemaHandler
private static org.slf4j.Logger	LOGGER
private ChangeEventQueue<DataChangeEvent>	queue
private DatabaseSchema<T>	schema
private org.apache.kafka.connect.data.Schema	schemaChangeKeySchema
private org.apache.kafka.connect.data.Schema	schemaChangeValueSchema
private EventDispatcher.StreamingChangeRecordReceiver	streamingReceiver Change event receiver for events dispatched from a streaming change event source.
private TableChanges.TableChangesSerializer<List<org.apache.kafka.connect.data.Struct>>	tableChangesSerializer
private TopicSelector<T>	topicSelector
private TransactionMonitor	transactionMonitor

Constructor Summary

Constructors

Constructor and Description
EventDispatcher(CommonConnectorConfig connectorConfig, TopicSelector<T> topicSelector, DatabaseSchema<T> schema, ChangeEventQueue<DataChangeEvent> queue, DataCollectionFilters.DataCollectionFilter<T> filter, ChangeEventCreator changeEventCreator, EventDispatcher.InconsistentSchemaHandler<T> inconsistentSchemaHandler, EventMetadataProvider metadataProvider, Heartbeat customHeartbeat, SchemaNameAdjuster schemaNameAdjuster)
EventDispatcher(CommonConnectorConfig connectorConfig, TopicSelector<T> topicSelector, DatabaseSchema<T> schema, ChangeEventQueue<DataChangeEvent> queue, DataCollectionFilters.DataCollectionFilter<T> filter, ChangeEventCreator changeEventCreator, EventMetadataProvider metadataProvider, Heartbeat heartbeat, SchemaNameAdjuster schemaNameAdjuster)
EventDispatcher(CommonConnectorConfig connectorConfig, TopicSelector<T> topicSelector, DatabaseSchema<T> schema, ChangeEventQueue<DataChangeEvent> queue, DataCollectionFilters.DataCollectionFilter<T> filter, ChangeEventCreator changeEventCreator, EventMetadataProvider metadataProvider, SchemaNameAdjuster schemaNameAdjuster)

Method Summary

Modifier and Type	Method and Description
void	alwaysDispatchHeartbeatEvent(OffsetContext offset)
void	dispatchConnectorEvent(ConnectorEvent event)
boolean	dispatchDataChangeEvent(T dataCollectionId, ChangeRecordEmitter changeRecordEmitter) Dispatches one or more DataChangeEvents.
void	dispatchHeartbeatEvent(OffsetContext offset)
void	dispatchSchemaChangeEvent(List<T> dataCollectionIds, SchemaChangeEventEmitter schemaChangeEventEmitter)
void	dispatchSchemaChangeEvent(T dataCollectionId, SchemaChangeEventEmitter schemaChangeEventEmitter)
void	dispatchSnapshotEvent(T dataCollectionId, ChangeRecordEmitter changeRecordEmitter, EventDispatcher.SnapshotReceiver receiver)
void	dispatchTransactionCommittedEvent(OffsetContext offset)
void	dispatchTransactionStartedEvent(String transactionId, OffsetContext offset)
private void	enqueueHeartbeat(org.apache.kafka.connect.source.SourceRecord record)
private void	enqueueSchemaChangeMessage(org.apache.kafka.connect.source.SourceRecord record)
private void	enqueueTransactionMessage(org.apache.kafka.connect.source.SourceRecord record)
Optional<DataCollectionSchema>	errorOnMissingSchema(T dataCollectionId, ChangeRecordEmitter changeRecordEmitter)
EventDispatcher.SnapshotReceiver	getSnapshotChangeEventReceiver()
boolean	heartbeatsEnabled()
Optional<DataCollectionSchema>	ignoreMissingSchema(T dataCollectionId, ChangeRecordEmitter changeRecordEmitter)
void	setEventListener(DataChangeEventListener eventListener) Provide a listener that is invoked for every incoming event to be processed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

topicSelector

private final TopicSelector<T extends DataCollectionId> topicSelector

schema

private final DatabaseSchema<T extends DataCollectionId> schema

historizedSchema

private final HistorizedDatabaseSchema<T extends DataCollectionId> historizedSchema

queue

private final ChangeEventQueue<DataChangeEvent> queue

filter

private final DataCollectionFilters.DataCollectionFilter<T extends DataCollectionId> filter

changeEventCreator

private final ChangeEventCreator changeEventCreator

heartbeat

private final Heartbeat heartbeat

eventListener

private DataChangeEventListener eventListener

emitTombstonesOnDelete

private final boolean emitTombstonesOnDelete

inconsistentSchemaHandler

private final EventDispatcher.InconsistentSchemaHandler<T extends DataCollectionId> inconsistentSchemaHandler

transactionMonitor

private final TransactionMonitor transactionMonitor

connectorConfig

private final CommonConnectorConfig connectorConfig

schemaChangeKeySchema

private final org.apache.kafka.connect.data.Schema schemaChangeKeySchema

schemaChangeValueSchema

private final org.apache.kafka.connect.data.Schema schemaChangeValueSchema

tableChangesSerializer

private final TableChanges.TableChangesSerializer<List<org.apache.kafka.connect.data.Struct>> tableChangesSerializer

streamingReceiver

private final EventDispatcher.StreamingChangeRecordReceiver streamingReceiver

Change event receiver for events dispatched from a streaming change event source.

Constructor Detail

EventDispatcher

public EventDispatcher(CommonConnectorConfig connectorConfig,
                       TopicSelector<T> topicSelector,
                       DatabaseSchema<T> schema,
                       ChangeEventQueue<DataChangeEvent> queue,
                       DataCollectionFilters.DataCollectionFilter<T> filter,
                       ChangeEventCreator changeEventCreator,
                       EventMetadataProvider metadataProvider,
                       SchemaNameAdjuster schemaNameAdjuster)

EventDispatcher

public EventDispatcher(CommonConnectorConfig connectorConfig,
                       TopicSelector<T> topicSelector,
                       DatabaseSchema<T> schema,
                       ChangeEventQueue<DataChangeEvent> queue,
                       DataCollectionFilters.DataCollectionFilter<T> filter,
                       ChangeEventCreator changeEventCreator,
                       EventMetadataProvider metadataProvider,
                       Heartbeat heartbeat,
                       SchemaNameAdjuster schemaNameAdjuster)

EventDispatcher

public EventDispatcher(CommonConnectorConfig connectorConfig,
                       TopicSelector<T> topicSelector,
                       DatabaseSchema<T> schema,
                       ChangeEventQueue<DataChangeEvent> queue,
                       DataCollectionFilters.DataCollectionFilter<T> filter,
                       ChangeEventCreator changeEventCreator,
                       EventDispatcher.InconsistentSchemaHandler<T> inconsistentSchemaHandler,
                       EventMetadataProvider metadataProvider,
                       Heartbeat customHeartbeat,
                       SchemaNameAdjuster schemaNameAdjuster)

Method Detail

dispatchSnapshotEvent

public void dispatchSnapshotEvent(T dataCollectionId,
                                  ChangeRecordEmitter changeRecordEmitter,
                                  EventDispatcher.SnapshotReceiver receiver)
                           throws InterruptedException

Throws:

InterruptedException

getSnapshotChangeEventReceiver

public EventDispatcher.SnapshotReceiver getSnapshotChangeEventReceiver()

dispatchDataChangeEvent

public boolean dispatchDataChangeEvent(T dataCollectionId,
                                       ChangeRecordEmitter changeRecordEmitter)
                                throws InterruptedException

Dispatches one or more DataChangeEvents. If the given data collection is included in the currently captured set of collections, the given emitter will be invoked, so it can emit one or more events (in the common case, one event will be emitted, but e.g. in case of PK updates, it may be a deletion and a creation event). The receiving coordinator creates SourceRecords for all emitted events and passes them to this dispatcher's ChangeEventCreator for converting them into data change events.

Returns:

true if an event was dispatched (i.e. sent to the message broker), false otherwise.

Throws:

InterruptedException

dispatchTransactionCommittedEvent

public void dispatchTransactionCommittedEvent(OffsetContext offset)
                                       throws InterruptedException

Throws:

InterruptedException

dispatchTransactionStartedEvent

public void dispatchTransactionStartedEvent(String transactionId,
                                            OffsetContext offset)
                                     throws InterruptedException

Throws:

InterruptedException

dispatchConnectorEvent

public void dispatchConnectorEvent(ConnectorEvent event)

errorOnMissingSchema

public Optional<DataCollectionSchema> errorOnMissingSchema(T dataCollectionId,
                                                           ChangeRecordEmitter changeRecordEmitter)

ignoreMissingSchema

public Optional<DataCollectionSchema> ignoreMissingSchema(T dataCollectionId,
                                                          ChangeRecordEmitter changeRecordEmitter)

dispatchSchemaChangeEvent

public void dispatchSchemaChangeEvent(T dataCollectionId,
                                      SchemaChangeEventEmitter schemaChangeEventEmitter)
                               throws InterruptedException

Throws:

InterruptedException

dispatchSchemaChangeEvent

public void dispatchSchemaChangeEvent(List<T> dataCollectionIds,
                                      SchemaChangeEventEmitter schemaChangeEventEmitter)
                               throws InterruptedException

Throws:

InterruptedException

alwaysDispatchHeartbeatEvent

public void alwaysDispatchHeartbeatEvent(OffsetContext offset)
                                  throws InterruptedException

Throws:

InterruptedException

dispatchHeartbeatEvent

public void dispatchHeartbeatEvent(OffsetContext offset)
                            throws InterruptedException

Throws:

InterruptedException

heartbeatsEnabled

public boolean heartbeatsEnabled()

enqueueHeartbeat

private void enqueueHeartbeat(org.apache.kafka.connect.source.SourceRecord record)
                       throws InterruptedException

Throws:

InterruptedException

enqueueTransactionMessage

private void enqueueTransactionMessage(org.apache.kafka.connect.source.SourceRecord record)
                                throws InterruptedException

Throws:

InterruptedException

enqueueSchemaChangeMessage

private void enqueueSchemaChangeMessage(org.apache.kafka.connect.source.SourceRecord record)
                                 throws InterruptedException

Throws:

InterruptedException

setEventListener

public void setEventListener(DataChangeEventListener eventListener)

Provide a listener that is invoked for every incoming event to be processed.

Parameters:

eventListener -