Package io.debezium.relational

Class RelationalSnapshotChangeEventSource

java.lang.Object

io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

io.debezium.relational.RelationalSnapshotChangeEventSource

All Implemented Interfaces:

ChangeEventSource, SnapshotChangeEventSource

public abstract class RelationalSnapshotChangeEventSource
extends AbstractSnapshotChangeEventSource

Base class for SnapshotChangeEventSource for relational databases with or without a schema history.

A transaction is managed by this base class, sub-classes shouldn't rollback or commit this transaction. They are free to use nested transactions or savepoints, though.

Author:

Gunnar Morling

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	RelationalSnapshotChangeEventSource.RelationalSnapshotContext	Mutable context which is populated in the course of snapshotting.

Nested classes/interfaces inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

AbstractSnapshotChangeEventSource.SnapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field	Description
protected Clock	clock
private RelationalDatabaseConnectorConfig	connectorConfig
protected EventDispatcher<TableId>	dispatcher
private JdbcConnection	jdbcConnection
static Duration	LOG_INTERVAL	Interval for showing a log statement with the progress while scanning a single table.
private static org.slf4j.Logger	LOGGER
private OffsetContext	previousOffset
private HistorizedRelationalDatabaseSchema	schema
private SnapshotProgressListener	snapshotProgressListener

Constructor Summary

Constructors

Constructor	Description
RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, OffsetContext previousOffset, JdbcConnection jdbcConnection, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)
RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, OffsetContext previousOffset, JdbcConnection jdbcConnection, HistorizedRelationalDatabaseSchema schema, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)

Method Summary

Modifier and Type	Method	Description
protected void	connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)	Executes steps which have to be taken just after the database connection is created.
private void	createDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
private void	createDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, EventDispatcher.SnapshotReceiver snapshotReceiver, Table table, int tableOrder, int tableCount)	Dispatches the data change events for the records of a single table.
protected void	createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
Connection	createSnapshotConnection()
private void	determineCapturedTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext ctx)
protected abstract void	determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)	Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object.
private Optional<String>	determineSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, TableId tableId)	Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.
SnapshotResult	doExecute(ChangeEventSource.ChangeEventSourceContext context, AbstractSnapshotChangeEventSource.SnapshotContext snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)	Executes this source.
protected String	enhanceOverriddenSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, String overriddenSelect, TableId tableId)	This method is overridden for Oracle to implement "as of SCN" predicate
protected abstract Set<TableId>	getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)	Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.
protected ChangeRecordEmitter	getChangeRecordEmitter(AbstractSnapshotChangeEventSource.SnapshotContext snapshotContext, TableId tableId, Object[] row)	Returns a ChangeRecordEmitter producing the change records for the given table row.
protected Clock	getClock()
protected Object	getColumnValue(ResultSet rs, int columnIndex, Column column, Table table)
protected abstract SchemaChangeEvent	getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, Table table)	Creates a SchemaChangeEvent representing the creation of the given table.
protected abstract Optional<String>	getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, TableId tableId)	Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshotted
private Threads.Timer	getTableScanLogTimer()
protected void	lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
protected abstract void	lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)	Locks all tables to be captured, so that no concurrent schema changes can be applied to them.
protected void	postSnapshot()
protected Statement	readTableStatement(OptionalLong tableSize)	Allow per-connector query creation to override for best database performance depending on the table size.
protected abstract void	readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)	Reads the structure of all the captured tables, writing it to RelationalSnapshotChangeEventSource.RelationalSnapshotContext.tables.
protected void	releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)	Releases all locks established in order to create a consistent data snapshot.
protected abstract void	releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)	Releases all locks established in order to create a consistent schema snapshot.
private void	rollbackTransaction(Connection connection)
protected OptionalLong	rowCountForTable(TableId tableId)	If connector is able to provide statistics-based number of records per table.
private Set<TableId>	sort(Set<TableId> capturedTables)
private Stream<TableId>	toTableIds(Set<TableId> tableIds, Pattern pattern)
protected void	tryStartingSnapshot(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)

Methods inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

complete, delaySnapshotIfNeeded, determineDataCollectionsToBeSnapshotted, execute, getSnapshottingTask, prepare

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

LOG_INTERVAL

public static final Duration LOG_INTERVAL

Interval for showing a log statement with the progress while scanning a single table.

connectorConfig

private final RelationalDatabaseConnectorConfig connectorConfig

previousOffset

private final OffsetContext previousOffset

jdbcConnection

private final JdbcConnection jdbcConnection

schema

private final HistorizedRelationalDatabaseSchema schema

dispatcher

protected final EventDispatcher<TableId> dispatcher

clock

protected final Clock clock

snapshotProgressListener

private final SnapshotProgressListener snapshotProgressListener

Constructor Detail

RelationalSnapshotChangeEventSource

public RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig,
                                           OffsetContext previousOffset,
                                           JdbcConnection jdbcConnection,
                                           HistorizedRelationalDatabaseSchema schema,
                                           EventDispatcher<TableId> dispatcher,
                                           Clock clock,
                                           SnapshotProgressListener snapshotProgressListener)

RelationalSnapshotChangeEventSource

public RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig,
                                           OffsetContext previousOffset,
                                           JdbcConnection jdbcConnection,
                                           EventDispatcher<TableId> dispatcher,
                                           Clock clock,
                                           SnapshotProgressListener snapshotProgressListener)

Method Detail

doExecute

public SnapshotResult doExecute(ChangeEventSource.ChangeEventSourceContext context,
                                AbstractSnapshotChangeEventSource.SnapshotContext snapshotContext,
                                AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
                         throws Exception

Description copied from class: AbstractSnapshotChangeEventSource

Executes this source. Implementations should regularly check via the given context if they should stop. If that's the case, they should abort their processing and perform any clean-up needed, such as rolling back pending transactions, releasing locks, etc.

Specified by:

doExecute in class AbstractSnapshotChangeEventSource

Parameters:

context - contextual information for this source's execution

snapshotContext - mutable context information populated throughout the snapshot process

snapshottingTask - immutable information about what tasks should be performed during snapshot

Returns:

an indicator to the position at which the snapshot was taken

Throws:

Exception

createSnapshotConnection

public Connection createSnapshotConnection()
                                    throws SQLException

Throws:

SQLException

connectionCreated

protected void connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                          throws Exception

Executes steps which have to be taken just after the database connection is created.

Throws:

Exception

toTableIds

private Stream<TableId> toTableIds(Set<TableId> tableIds,
                                   Pattern pattern)

sort

private Set<TableId> sort(Set<TableId> capturedTables)
                   throws Exception

Throws:

Exception

determineCapturedTables

private void determineCapturedTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext ctx)
                              throws Exception

Throws:

Exception

getAllTableIds

protected abstract Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                                        throws Exception

Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.

Throws:

Exception

lockTablesForSchemaSnapshot

protected abstract void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                                    RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                                             throws Exception

Locks all tables to be captured, so that no concurrent schema changes can be applied to them.

Throws:

Exception

determineSnapshotOffset

protected abstract void determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                                         throws Exception

Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object. Subsequently, the DB's schema (and data) will be be read at this position. Once the snapshot is completed, a StreamingChangeEventSource will be set up with this initial position to continue with stream reading from there.

Throws:

Exception

readTableStructure

protected abstract void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                           RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                                    throws Exception

Reads the structure of all the captured tables, writing it to RelationalSnapshotChangeEventSource.RelationalSnapshotContext.tables.

Throws:

Exception

releaseSchemaSnapshotLocks

protected abstract void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                                            throws Exception

Releases all locks established in order to create a consistent schema snapshot.

Throws:

Exception

releaseDataSnapshotLocks

protected void releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                                 throws Exception

Releases all locks established in order to create a consistent data snapshot.

Throws:

Exception

createSchemaChangeEventsForTables

protected void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                                 RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                                                 AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
                                          throws Exception

Throws:

Exception

getCreateTableEvent

protected abstract SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                                                         Table table)
                                                  throws Exception

Creates a SchemaChangeEvent representing the creation of the given table.

Throws:

Exception

createDataEvents

private void createDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext,
                              RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
                       throws Exception

Throws:

Exception

tryStartingSnapshot

protected void tryStartingSnapshot(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)

createDataEventsForTable

private void createDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext,
                                      RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                                      EventDispatcher.SnapshotReceiver snapshotReceiver,
                                      Table table,
                                      int tableOrder,
                                      int tableCount)
                               throws InterruptedException

Dispatches the data change events for the records of a single table.

Throws:

InterruptedException

lastSnapshotRecord

protected void lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)

rowCountForTable

protected OptionalLong rowCountForTable(TableId tableId)

If connector is able to provide statistics-based number of records per table.

getTableScanLogTimer

private Threads.Timer getTableScanLogTimer()

getChangeRecordEmitter

protected ChangeRecordEmitter getChangeRecordEmitter(AbstractSnapshotChangeEventSource.SnapshotContext snapshotContext,
                                                     TableId tableId,
                                                     Object[] row)

Returns a ChangeRecordEmitter producing the change records for the given table row.

determineSnapshotSelect

private Optional<String> determineSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                                                 TableId tableId)

Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.

Parameters:

tableId - the table to generate a query for

Returns:

a valid query string or empty if table will not be snapshotted

enhanceOverriddenSelect

protected String enhanceOverriddenSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                                         String overriddenSelect,
                                         TableId tableId)

This method is overridden for Oracle to implement "as of SCN" predicate

Parameters:

snapshotContext - snapshot context, used for getting offset SCN

overriddenSelect - conditional snapshot select

Returns:

enhanced select statement. By default it just returns original select statements.

getSnapshotSelect

protected abstract Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext,
                                                      TableId tableId)

Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshotted

getColumnValue

protected Object getColumnValue(ResultSet rs,
                                int columnIndex,
                                Column column,
                                Table table)
                         throws SQLException

Throws:

SQLException

readTableStatement

protected Statement readTableStatement(OptionalLong tableSize)
                                throws SQLException

Allow per-connector query creation to override for best database performance depending on the table size.

Throws:

SQLException

rollbackTransaction

private void rollbackTransaction(Connection connection)

getClock

protected Clock getClock()

postSnapshot

protected void postSnapshot()
                     throws InterruptedException

Throws:

InterruptedException