io.debezium.relational

Class RelationalSnapshotChangeEventSource

java.lang.Object

io.debezium.relational.RelationalSnapshotChangeEventSource

All Implemented Interfaces:

ChangeEventSource, SnapshotChangeEventSource

public abstract class RelationalSnapshotChangeEventSource
extends Object
implements SnapshotChangeEventSource

Base class for SnapshotChangeEventSource for relational databases with or without a schema history.

A transaction is managed by this base class, sub-classes shouldn't rollback or commit this transaction. They are free to use nested transactions or savepoints, though.

Author:

Gunnar Morling

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RelationalSnapshotChangeEventSource.SnapshotContext Mutable context which is populated in the course of snapshotting.
static class	RelationalSnapshotChangeEventSource.SnapshottingTask A configuration describing the task to be performed during snapshotting.

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field and Description
protected Clock	clock
private RelationalDatabaseConnectorConfig	connectorConfig
private EventDispatcher<TableId>	dispatcher
private JdbcConnection	jdbcConnection
private static Duration	LOG_INTERVAL Interval for showing a log statement with the progress while scanning a single table.
private static org.slf4j.Logger	LOGGER
private OffsetContext	previousOffset
private HistorizedRelationalDatabaseSchema	schema
private SnapshotProgressListener	snapshotProgressListener

Constructor Summary

Constructors

Constructor and Description
RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, OffsetContext previousOffset, JdbcConnection jdbcConnection, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)
RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, OffsetContext previousOffset, JdbcConnection jdbcConnection, HistorizedRelationalDatabaseSchema schema, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)

Method Summary

Modifier and Type	Method and Description
protected abstract void	complete(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext) Completes the snapshot, doing any required clean-up (resource disposal etc.).
protected void	connectionCreated(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext) Executes steps which have to be taken just after the database connection is created.
private void	createDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)
private void	createDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext, EventDispatcher.SnapshotReceiver snapshotReceiver, Table table) Dispatches the data change events for the records of a single table.
private void	createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)
private void	delaySnapshotIfNeeded(ChangeEventSource.ChangeEventSourceContext context) Delays snapshot execution as per the CommonConnectorConfig.SNAPSHOT_DELAY_MS parameter.
private void	determineCapturedTables(RelationalSnapshotChangeEventSource.SnapshotContext ctx)
protected abstract void	determineSnapshotOffset(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext) Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object.
private Optional<String>	determineSnapshotSelect(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext, TableId tableId) Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.
SnapshotResult	execute(ChangeEventSource.ChangeEventSourceContext context) Executes this source.
protected abstract Set<TableId>	getAllTableIds(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext) Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.
protected ChangeRecordEmitter	getChangeRecordEmitter(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext, TableId tableId, Object[] row) Returns a ChangeRecordEmitter producing the change records for the given table row.
protected Clock	getClock()
private Column[]	getColumnsForResultSet(Table table, ResultSet rs)
protected Object	getColumnValue(ResultSet rs, int columnIndex, Column column)
protected abstract SchemaChangeEvent	getCreateTableEvent(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext, Table table) Creates a SchemaChangeEvent representing the creation of the given table.
protected abstract Optional<String>	getSnapshotSelect(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext, TableId tableId) Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshotted
protected abstract RelationalSnapshotChangeEventSource.SnapshottingTask	getSnapshottingTask(OffsetContext previousOffset) Returns the snapshotting task based on the previous offset (if available) and the connector's snapshotting mode.
private Threads.Timer	getTableScanLogTimer()
protected abstract void	lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext) Locks all tables to be captured, so that no concurrent schema changes can be applied to them.
protected abstract RelationalSnapshotChangeEventSource.SnapshotContext	prepare(ChangeEventSource.ChangeEventSourceContext changeEventSourceContext) Prepares the taking of a snapshot and returns an initial RelationalSnapshotChangeEventSource.SnapshotContext.
private Statement	readTableStatement()
protected abstract void	readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext) Reads the structure of all the captured tables, writing it to RelationalSnapshotChangeEventSource.SnapshotContext.tables.
protected abstract void	releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext) Releases all locks established in order to create a consistent schema snapshot.
private void	rollbackTransaction(Connection connection)
private Set<TableId>	sort(Set<TableId> capturedTables)
private Stream<TableId>	toTableIds(Set<TableId> tableIds, Pattern pattern)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

LOG_INTERVAL

private static final Duration LOG_INTERVAL

Interval for showing a log statement with the progress while scanning a single table.

connectorConfig

private final RelationalDatabaseConnectorConfig connectorConfig

previousOffset

private final OffsetContext previousOffset

jdbcConnection

private final JdbcConnection jdbcConnection

schema

private final HistorizedRelationalDatabaseSchema schema

dispatcher

private final EventDispatcher<TableId> dispatcher

clock

protected final Clock clock

snapshotProgressListener

private final SnapshotProgressListener snapshotProgressListener

Constructor Detail

RelationalSnapshotChangeEventSource

public RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig,
                                           OffsetContext previousOffset,
                                           JdbcConnection jdbcConnection,
                                           HistorizedRelationalDatabaseSchema schema,
                                           EventDispatcher<TableId> dispatcher,
                                           Clock clock,
                                           SnapshotProgressListener snapshotProgressListener)

RelationalSnapshotChangeEventSource

public RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig,
                                           OffsetContext previousOffset,
                                           JdbcConnection jdbcConnection,
                                           EventDispatcher<TableId> dispatcher,
                                           Clock clock,
                                           SnapshotProgressListener snapshotProgressListener)

Method Detail

execute

public SnapshotResult execute(ChangeEventSource.ChangeEventSourceContext context)
                       throws InterruptedException

Description copied from interface: SnapshotChangeEventSource

Executes this source. Implementations should regularly check via the given context if they should stop. If that's the case, they should abort their processing and perform any clean-up needed, such as rolling back pending transactions, releasing locks etc.

Specified by:

execute in interface SnapshotChangeEventSource

Parameters:

context - contextual information for this source's execution

Returns:

an indicator to the position at which the snapshot was taken

Throws:

InterruptedException - in case the snapshot was aborted before completion

getSnapshottingTask

protected abstract RelationalSnapshotChangeEventSource.SnapshottingTask getSnapshottingTask(OffsetContext previousOffset)

Returns the snapshotting task based on the previous offset (if available) and the connector's snapshotting mode.

delaySnapshotIfNeeded

private void delaySnapshotIfNeeded(ChangeEventSource.ChangeEventSourceContext context)
                            throws InterruptedException

Delays snapshot execution as per the CommonConnectorConfig.SNAPSHOT_DELAY_MS parameter.

Throws:

InterruptedException

prepare

protected abstract RelationalSnapshotChangeEventSource.SnapshotContext prepare(ChangeEventSource.ChangeEventSourceContext changeEventSourceContext)
                                                                        throws Exception

Prepares the taking of a snapshot and returns an initial RelationalSnapshotChangeEventSource.SnapshotContext.

Throws:

Exception

connectionCreated

protected void connectionCreated(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)
                          throws Exception

Executes steps which have to be taken just after the database connection is created.

Throws:

Exception

toTableIds

private Stream<TableId> toTableIds(Set<TableId> tableIds,
                                   Pattern pattern)

sort

private Set<TableId> sort(Set<TableId> capturedTables)
                   throws Exception

Throws:

Exception

determineCapturedTables

private void determineCapturedTables(RelationalSnapshotChangeEventSource.SnapshotContext ctx)
                              throws Exception

Throws:

Exception

getAllTableIds

protected abstract Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)
                                        throws Exception

Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.

Throws:

Exception

lockTablesForSchemaSnapshot

protected abstract void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                                    RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)
                                             throws Exception

Locks all tables to be captured, so that no concurrent schema changes can be applied to them.

Throws:

Exception

determineSnapshotOffset

protected abstract void determineSnapshotOffset(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)
                                         throws Exception

Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object. Subsequently, the DB's schema (and data) will be be read at this position. Once the snapshot is completed, a StreamingChangeEventSource will be set up with this initial position to continue with stream reading from there.

Throws:

Exception

readTableStructure

protected abstract void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                           RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)
                                    throws Exception

Reads the structure of all the captured tables, writing it to RelationalSnapshotChangeEventSource.SnapshotContext.tables.

Throws:

Exception

releaseSchemaSnapshotLocks

protected abstract void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)
                                            throws Exception

Releases all locks established in order to create a consistent schema snapshot.

Throws:

Exception

createSchemaChangeEventsForTables

private void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                               RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)
                                        throws Exception

Throws:

Exception

getCreateTableEvent

protected abstract SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext,
                                                         Table table)
                                                  throws Exception

Creates a SchemaChangeEvent representing the creation of the given table.

Throws:

Exception

createDataEvents

private void createDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext,
                              RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)
                       throws InterruptedException

Throws:

InterruptedException

createDataEventsForTable

private void createDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                      RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext,
                                      EventDispatcher.SnapshotReceiver snapshotReceiver,
                                      Table table)
                               throws InterruptedException

Dispatches the data change events for the records of a single table.

Throws:

InterruptedException

getTableScanLogTimer

private Threads.Timer getTableScanLogTimer()

getChangeRecordEmitter

protected ChangeRecordEmitter getChangeRecordEmitter(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext,
                                                     TableId tableId,
                                                     Object[] row)

Returns a ChangeRecordEmitter producing the change records for the given table row.

determineSnapshotSelect

private Optional<String> determineSnapshotSelect(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext,
                                                 TableId tableId)

Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.

Parameters:

tableId - the table to generate a query for

Returns:

a valid query string or empty if table will not be snapshotted

getSnapshotSelect

protected abstract Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext,
                                                      TableId tableId)

Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshotted

getColumnsForResultSet

private Column[] getColumnsForResultSet(Table table,
                                        ResultSet rs)
                                 throws SQLException

Throws:

SQLException

getColumnValue

protected Object getColumnValue(ResultSet rs,
                                int columnIndex,
                                Column column)
                         throws SQLException

Throws:

SQLException

readTableStatement

private Statement readTableStatement()
                              throws SQLException

Throws:

SQLException

complete

protected abstract void complete(RelationalSnapshotChangeEventSource.SnapshotContext snapshotContext)

Completes the snapshot, doing any required clean-up (resource disposal etc.).

Parameters:

snapshotContext - snapshot context

rollbackTransaction

private void rollbackTransaction(Connection connection)

getClock

protected Clock getClock()