Package io.debezium.relational

Class RelationalSnapshotChangeEventSource<P extends Partition,O extends OffsetContext>

java.lang.Object

io.debezium.pipeline.source.AbstractSnapshotChangeEventSource<P,O>

io.debezium.relational.RelationalSnapshotChangeEventSource<P,O>

All Implemented Interfaces:

ChangeEventSource, SnapshotChangeEventSource<P,O>, AutoCloseable

public abstract class RelationalSnapshotChangeEventSource<P extends Partition,O extends OffsetContext>
extends AbstractSnapshotChangeEventSource<P,O>

Base class for SnapshotChangeEventSource for relational databases with or without a schema history.

A transaction is managed by this base class, sub-classes shouldn't rollback or commit this transaction. They are free to use nested transactions or savepoints, though.

Author:

Gunnar Morling

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P extends Partition,O extends OffsetContext>	Mutable context which is populated in the course of snapshotting.

Nested classes/interfaces inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

AbstractSnapshotChangeEventSource.SnapshotContext<P extends Partition,O extends OffsetContext>, AbstractSnapshotChangeEventSource.SnapshottingTask

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field	Description
protected final Clock	clock
private final RelationalDatabaseConnectorConfig	connectorConfig
protected final EventDispatcher<P,TableId>	dispatcher
private final JdbcConnection	jdbcConnection
private final MainConnectionProvidingConnectionFactory<? extends JdbcConnection>	jdbcConnectionFactory
private static final org.slf4j.Logger	LOGGER
private final RelationalDatabaseSchema	schema
static final Pattern	SELECT_ALL_PATTERN
private final SnapshotProgressListener<P>	snapshotProgressListener

Fields inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

LOG_INTERVAL

Constructor Summary

Constructors

Constructor	Description
RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, MainConnectionProvidingConnectionFactory<? extends JdbcConnection> jdbcConnectionFactory, RelationalDatabaseSchema schema, EventDispatcher<P,TableId> dispatcher, Clock clock, SnapshotProgressListener<P> snapshotProgressListener)

Method Summary

Modifier and Type	Method	Description
protected boolean	additionalColumnFilter(P partition, TableId tableId, String columnName)	Additional filter handling for preparing column names for snapshot select
private Set<TableId>	addSignalingCollectionAndSort(Set<TableId> capturedTables)
protected void	connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)	Executes steps which have to be taken just after the database connection is created.
protected abstract O	copyOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
private Queue<JdbcConnection>	createConnectionPool(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> ctx)
private void	createDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, Queue<JdbcConnection> connectionPool)
private Callable<Void>	createDataEventsForTableCallable(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, EventDispatcher.SnapshotReceiver<P> snapshotReceiver, Table table, boolean firstTable, boolean lastTable, int tableOrder, int tableCount, String selectStatement, OptionalLong rowCount, Queue<JdbcConnection> connectionPool, Queue<O> offsets)
protected void	createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
Connection	createSnapshotConnection()
private void	determineCapturedTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> ctx)
protected abstract void	determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, O previousOffset)	Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object.
private Optional<String>	determineSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, TableId tableId)	Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.
private void	doCreateDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext, P partition, O offset, EventDispatcher.SnapshotReceiver<P> snapshotReceiver, Table table, boolean firstTable, boolean lastTable, int tableOrder, int tableCount, String selectStatement, OptionalLong rowCount, JdbcConnection jdbcConnection)
SnapshotResult<O>	doExecute(ChangeEventSource.ChangeEventSourceContext context, O previousOffset, AbstractSnapshotChangeEventSource.SnapshotContext<P,O> snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)	Executes this source.
protected String	enhanceOverriddenSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, String overriddenSelect, TableId tableId)	This method is overridden for Oracle to implement "as of SCN" predicate
protected abstract Set<TableId>	getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)	Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.
protected ChangeRecordEmitter<P>	getChangeRecordEmitter(P partition, O offset, TableId tableId, Object[] row, Instant timestamp)	Returns a ChangeRecordEmitter producing the change records for the given table row.
protected Clock	getClock()
protected abstract SchemaChangeEvent	getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, Table table)	Creates a SchemaChangeEvent representing the creation of the given table.
protected List<String>	getPreparedColumnNames(P partition, Table table)	Prepares a list of columns to be used in the snapshot select.
protected Optional<String>	getSnapshotConnectionFirstSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, TableId tableId)
protected abstract Optional<String>	getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, TableId tableId, List<String> columns)	Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshotted
protected String	getSnapshotSelectOverridesByTable(TableId tableId)
protected Instant	getSnapshotSourceTimestamp(JdbcConnection jdbcConnection, O offset, TableId tableId)	For the given table gets source.ts_ms value from the database for snapshot data! For Postgresql its globally static for all tables since postgresql snapshot process setting auto commit off.
private Threads.Timer	getTableScanLogTimer()
protected Collection<TableId>	getTablesForSchemaChange(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
protected void	lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
protected abstract void	lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)	Locks all tables to be captured, so that no concurrent schema changes can be applied to them.
protected void	postSnapshot()
protected Statement	readTableStatement(JdbcConnection jdbcConnection, OptionalLong tableSize)	Allow per-connector query creation to override for best database performance depending on the table size.
protected abstract void	readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, O offsetContext)	Reads the structure of all the captured tables, writing it to RelationalSnapshotChangeEventSource.RelationalSnapshotContext.tables.
protected void	releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)	Releases all locks established in order to create a consistent data snapshot.
protected abstract void	releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)	Releases all locks established in order to create a consistent schema snapshot.
private void	rollbackTransaction(Connection connection)
protected OptionalLong	rowCountForTable(TableId tableId)	If connector is able to provide statistics-based number of records per table.
private void	setSnapshotMarker(OffsetContext offset, boolean firstTable, boolean lastTable, boolean firstRecordInTable, boolean lastRecordInTable)
private Stream<TableId>	toTableIds(Set<TableId> tableIds, Pattern pattern)
protected void	tryStartingSnapshot(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)

Methods inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

aborted, close, completed, delaySnapshotIfNeeded, determineDataCollectionsToBeSnapshotted, execute, getSnapshottingTask, prepare

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

SELECT_ALL_PATTERN

public static final Pattern SELECT_ALL_PATTERN

connectorConfig

private final RelationalDatabaseConnectorConfig connectorConfig

jdbcConnection

private final JdbcConnection jdbcConnection

jdbcConnectionFactory

private final MainConnectionProvidingConnectionFactory<? extends JdbcConnection> jdbcConnectionFactory

schema

private final RelationalDatabaseSchema schema

dispatcher

protected final EventDispatcher<P extends Partition,TableId> dispatcher

clock

protected final Clock clock

snapshotProgressListener

private final SnapshotProgressListener<P extends Partition> snapshotProgressListener

Constructor Details

RelationalSnapshotChangeEventSource

public RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig,
 MainConnectionProvidingConnectionFactory<? extends JdbcConnection> jdbcConnectionFactory,
 RelationalDatabaseSchema schema,
 EventDispatcher<P,TableId> dispatcher,
 Clock clock,
 SnapshotProgressListener<P> snapshotProgressListener)

Method Details

doExecute

public SnapshotResult<O> doExecute(ChangeEventSource.ChangeEventSourceContext context,
 O previousOffset,
 AbstractSnapshotChangeEventSource.SnapshotContext<P,O> snapshotContext,
 AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
                            throws Exception

Description copied from class: AbstractSnapshotChangeEventSource

Executes this source. Implementations should regularly check via the given context if they should stop. If that's the case, they should abort their processing and perform any clean-up needed, such as rolling back pending transactions, releasing locks, etc.

Specified by:

doExecute in class AbstractSnapshotChangeEventSource<P extends Partition,O extends OffsetContext>

Parameters:

context - contextual information for this source's execution

previousOffset - previous offset restored from Kafka

snapshotContext - mutable context information populated throughout the snapshot process

snapshottingTask - immutable information about what tasks should be performed during snapshot

Returns:

an indicator to the position at which the snapshot was taken

Throws:

Exception

createConnectionPool

private Queue<JdbcConnection> createConnectionPool(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> ctx)
                                            throws SQLException

Throws:

SQLException

createSnapshotConnection

public Connection createSnapshotConnection()
                                    throws SQLException

Throws:

SQLException

connectionCreated

protected void connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
                          throws Exception

Executes steps which have to be taken just after the database connection is created.

Throws:

Exception

toTableIds

private Stream<TableId> toTableIds(Set<TableId> tableIds,
 Pattern pattern)

addSignalingCollectionAndSort

private Set<TableId> addSignalingCollectionAndSort(Set<TableId> capturedTables)
                                            throws Exception

Throws:

Exception

determineCapturedTables

private void determineCapturedTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> ctx)
                              throws Exception

Throws:

Exception

getAllTableIds

protected abstract Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
                                        throws Exception

Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.

Throws:

Exception

lockTablesForSchemaSnapshot

protected abstract void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
                                             throws Exception

Locks all tables to be captured, so that no concurrent schema changes can be applied to them.

Throws:

Exception

determineSnapshotOffset

protected abstract void determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 O previousOffset)
                                         throws Exception

Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object. Subsequently, the DB's schema (and data) will be be read at this position. Once the snapshot is completed, a StreamingChangeEventSource will be set up with this initial position to continue with stream reading from there.

Throws:

Exception

readTableStructure

protected abstract void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 O offsetContext)
                                    throws Exception

Reads the structure of all the captured tables, writing it to RelationalSnapshotChangeEventSource.RelationalSnapshotContext.tables.

Throws:

Exception

releaseSchemaSnapshotLocks

protected abstract void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
                                            throws Exception

Releases all locks established in order to create a consistent schema snapshot.

Throws:

Exception

releaseDataSnapshotLocks

protected void releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
                                 throws Exception

Releases all locks established in order to create a consistent data snapshot.

Throws:

Exception

createSchemaChangeEventsForTables

protected void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)
                                          throws Exception

Throws:

Exception

getTablesForSchemaChange

protected Collection<TableId> getTablesForSchemaChange(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)

getCreateTableEvent

protected abstract SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 Table table)
                                                  throws Exception

Creates a SchemaChangeEvent representing the creation of the given table.

Throws:

Exception

createDataEvents

private void createDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 Queue<JdbcConnection> connectionPool)
                       throws Exception

Throws:

Exception

copyOffset

protected abstract O copyOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)

tryStartingSnapshot

protected void tryStartingSnapshot(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)

getSnapshotSourceTimestamp

protected Instant getSnapshotSourceTimestamp(JdbcConnection jdbcConnection,
 O offset,
 TableId tableId)

For the given table gets source.ts_ms value from the database for snapshot data! For Postgresql its globally static for all tables since postgresql snapshot process setting auto commit off. For Mysql its static per table and might be ~second behind of the select statements start ts.

createDataEventsForTableCallable

private Callable<Void> createDataEventsForTableCallable(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 EventDispatcher.SnapshotReceiver<P> snapshotReceiver,
 Table table,
 boolean firstTable,
 boolean lastTable,
 int tableOrder,
 int tableCount,
 String selectStatement,
 OptionalLong rowCount,
 Queue<JdbcConnection> connectionPool,
 Queue<O> offsets)

doCreateDataEventsForTable

private void doCreateDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext,
 P partition,
 O offset,
 EventDispatcher.SnapshotReceiver<P> snapshotReceiver,
 Table table,
 boolean firstTable,
 boolean lastTable,
 int tableOrder,
 int tableCount,
 String selectStatement,
 OptionalLong rowCount,
 JdbcConnection jdbcConnection)
                                 throws InterruptedException

Throws:

InterruptedException

setSnapshotMarker

private void setSnapshotMarker(OffsetContext offset,
 boolean firstTable,
 boolean lastTable,
 boolean firstRecordInTable,
 boolean lastRecordInTable)

lastSnapshotRecord

protected void lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)

rowCountForTable

protected OptionalLong rowCountForTable(TableId tableId)

If connector is able to provide statistics-based number of records per table.

getTableScanLogTimer

private Threads.Timer getTableScanLogTimer()

getChangeRecordEmitter

protected ChangeRecordEmitter<P> getChangeRecordEmitter(P partition,
 O offset,
 TableId tableId,
 Object[] row,
 Instant timestamp)

Returns a ChangeRecordEmitter producing the change records for the given table row.

determineSnapshotSelect

private Optional<String> determineSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 TableId tableId)

Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.

Parameters:

tableId - the table to generate a query for

Returns:

a valid query string or empty if table will not be snapshotted

getSnapshotSelectOverridesByTable

protected String getSnapshotSelectOverridesByTable(TableId tableId)

getPreparedColumnNames

protected List<String> getPreparedColumnNames(P partition,
 Table table)

Prepares a list of columns to be used in the snapshot select. The selected columns are based on the column include/exclude filters and if all columns are excluded, the list will contain all the primary key columns.

Returns:

list of snapshot select columns

additionalColumnFilter

protected boolean additionalColumnFilter(P partition,
 TableId tableId,
 String columnName)

Additional filter handling for preparing column names for snapshot select

enhanceOverriddenSelect

protected String enhanceOverriddenSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 String overriddenSelect,
 TableId tableId)

This method is overridden for Oracle to implement "as of SCN" predicate

Parameters:

snapshotContext - snapshot context, used for getting offset SCN

overriddenSelect - conditional snapshot select

Returns:

enhanced select statement. By default it just returns original select statements.

getSnapshotSelect

protected abstract Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 TableId tableId,
 List<String> columns)

Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshotted

getSnapshotConnectionFirstSelect

protected Optional<String> getSnapshotConnectionFirstSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 TableId tableId)

readTableStatement

protected Statement readTableStatement(JdbcConnection jdbcConnection,
 OptionalLong tableSize)
                                throws SQLException

Allow per-connector query creation to override for best database performance depending on the table size.

Throws:

SQLException

rollbackTransaction

private void rollbackTransaction(Connection connection)

getClock

protected Clock getClock()

postSnapshot

protected void postSnapshot()
                     throws InterruptedException

Throws:

InterruptedException