Package io.debezium.relational
Class RelationalSnapshotChangeEventSource
- java.lang.Object
-
- io.debezium.pipeline.source.AbstractSnapshotChangeEventSource
-
- io.debezium.relational.RelationalSnapshotChangeEventSource
-
- All Implemented Interfaces:
ChangeEventSource,SnapshotChangeEventSource
public abstract class RelationalSnapshotChangeEventSource extends AbstractSnapshotChangeEventSource
Base class forSnapshotChangeEventSourcefor relational databases with or without a schema history.A transaction is managed by this base class, sub-classes shouldn't rollback or commit this transaction. They are free to use nested transactions or savepoints, though.
- Author:
- Gunnar Morling
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classRelationalSnapshotChangeEventSource.RelationalSnapshotContextMutable context which is populated in the course of snapshotting.-
Nested classes/interfaces inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource
AbstractSnapshotChangeEventSource.SnapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask
-
Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource
ChangeEventSource.ChangeEventSourceContext
-
-
Field Summary
Fields Modifier and Type Field Description protected Clockclockprivate RelationalDatabaseConnectorConfigconnectorConfigprotected EventDispatcher<TableId>dispatcherprivate JdbcConnectionjdbcConnectionstatic DurationLOG_INTERVALInterval for showing a log statement with the progress while scanning a single table.private static org.slf4j.LoggerLOGGERprivate OffsetContextpreviousOffsetprivate HistorizedRelationalDatabaseSchemaschemaprivate SnapshotProgressListenersnapshotProgressListener
-
Constructor Summary
Constructors Constructor Description RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, OffsetContext previousOffset, JdbcConnection jdbcConnection, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, OffsetContext previousOffset, JdbcConnection jdbcConnection, HistorizedRelationalDatabaseSchema schema, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description protected voidconnectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)Executes steps which have to be taken just after the database connection is created.private voidcreateDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)private voidcreateDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, EventDispatcher.SnapshotReceiver snapshotReceiver, Table table, int tableOrder, int tableCount)Dispatches the data change events for the records of a single table.protected voidcreateSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)ConnectioncreateSnapshotConnection()private voiddetermineCapturedTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext ctx)protected abstract voiddetermineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object.private Optional<String>determineSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, TableId tableId)Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.SnapshotResultdoExecute(ChangeEventSource.ChangeEventSourceContext context, AbstractSnapshotChangeEventSource.SnapshotContext snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask)Executes this source.protected StringenhanceOverriddenSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, String overriddenSelect, TableId tableId)This method is overridden for Oracle to implement "as of SCN" predicateprotected abstract Set<TableId>getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.protected ChangeRecordEmittergetChangeRecordEmitter(AbstractSnapshotChangeEventSource.SnapshotContext snapshotContext, TableId tableId, Object[] row)Returns aChangeRecordEmitterproducing the change records for the given table row.protected ClockgetClock()protected ObjectgetColumnValue(ResultSet rs, int columnIndex, Column column, Table table)protected abstract SchemaChangeEventgetCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, Table table)Creates aSchemaChangeEventrepresenting the creation of the given table.protected abstract Optional<String>getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, TableId tableId)Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshottedprivate Threads.TimergetTableScanLogTimer()protected voidlastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)protected abstract voidlockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)Locks all tables to be captured, so that no concurrent schema changes can be applied to them.protected voidpostSnapshot()protected StatementreadTableStatement(OptionalLong tableSize)Allow per-connector query creation to override for best database performance depending on the table size.protected abstract voidreadTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)Reads the structure of all the captured tables, writing it toRelationalSnapshotChangeEventSource.RelationalSnapshotContext.tables.protected voidreleaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)Releases all locks established in order to create a consistent data snapshot.protected abstract voidreleaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)Releases all locks established in order to create a consistent schema snapshot.private voidrollbackTransaction(Connection connection)protected OptionalLongrowCountForTable(TableId tableId)If connector is able to provide statistics-based number of records per table.private Set<TableId>sort(Set<TableId> capturedTables)private Stream<TableId>toTableIds(Set<TableId> tableIds, Pattern pattern)protected voidtryStartingSnapshot(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)-
Methods inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource
complete, delaySnapshotIfNeeded, determineDataCollectionsToBeSnapshotted, execute, getSnapshottingTask, prepare
-
-
-
-
Field Detail
-
LOGGER
private static final org.slf4j.Logger LOGGER
-
LOG_INTERVAL
public static final Duration LOG_INTERVAL
Interval for showing a log statement with the progress while scanning a single table.
-
connectorConfig
private final RelationalDatabaseConnectorConfig connectorConfig
-
previousOffset
private final OffsetContext previousOffset
-
jdbcConnection
private final JdbcConnection jdbcConnection
-
schema
private final HistorizedRelationalDatabaseSchema schema
-
dispatcher
protected final EventDispatcher<TableId> dispatcher
-
clock
protected final Clock clock
-
snapshotProgressListener
private final SnapshotProgressListener snapshotProgressListener
-
-
Constructor Detail
-
RelationalSnapshotChangeEventSource
public RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, OffsetContext previousOffset, JdbcConnection jdbcConnection, HistorizedRelationalDatabaseSchema schema, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)
-
RelationalSnapshotChangeEventSource
public RelationalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig connectorConfig, OffsetContext previousOffset, JdbcConnection jdbcConnection, EventDispatcher<TableId> dispatcher, Clock clock, SnapshotProgressListener snapshotProgressListener)
-
-
Method Detail
-
doExecute
public SnapshotResult doExecute(ChangeEventSource.ChangeEventSourceContext context, AbstractSnapshotChangeEventSource.SnapshotContext snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask) throws Exception
Description copied from class:AbstractSnapshotChangeEventSourceExecutes this source. Implementations should regularly check via the given context if they should stop. If that's the case, they should abort their processing and perform any clean-up needed, such as rolling back pending transactions, releasing locks, etc.- Specified by:
doExecutein classAbstractSnapshotChangeEventSource- Parameters:
context- contextual information for this source's executionsnapshotContext- mutable context information populated throughout the snapshot processsnapshottingTask- immutable information about what tasks should be performed during snapshot- Returns:
- an indicator to the position at which the snapshot was taken
- Throws:
Exception
-
createSnapshotConnection
public Connection createSnapshotConnection() throws SQLException
- Throws:
SQLException
-
connectionCreated
protected void connectionCreated(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext) throws Exception
Executes steps which have to be taken just after the database connection is created.- Throws:
Exception
-
determineCapturedTables
private void determineCapturedTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext ctx) throws Exception
- Throws:
Exception
-
getAllTableIds
protected abstract Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext) throws Exception
Returns all candidate tables; the current filter configuration will be applied to the result set, resulting in the effective set of captured tables.- Throws:
Exception
-
lockTablesForSchemaSnapshot
protected abstract void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext) throws Exception
Locks all tables to be captured, so that no concurrent schema changes can be applied to them.- Throws:
Exception
-
determineSnapshotOffset
protected abstract void determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext) throws Exception
Determines the current offset (MySQL binlog position, Oracle SCN etc.), storing it into the passed context object. Subsequently, the DB's schema (and data) will be be read at this position. Once the snapshot is completed, aStreamingChangeEventSourcewill be set up with this initial position to continue with stream reading from there.- Throws:
Exception
-
readTableStructure
protected abstract void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext) throws Exception
Reads the structure of all the captured tables, writing it toRelationalSnapshotChangeEventSource.RelationalSnapshotContext.tables.- Throws:
Exception
-
releaseSchemaSnapshotLocks
protected abstract void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext) throws Exception
Releases all locks established in order to create a consistent schema snapshot.- Throws:
Exception
-
releaseDataSnapshotLocks
protected void releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext) throws Exception
Releases all locks established in order to create a consistent data snapshot.- Throws:
Exception
-
createSchemaChangeEventsForTables
protected void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, AbstractSnapshotChangeEventSource.SnapshottingTask snapshottingTask) throws Exception
- Throws:
Exception
-
getCreateTableEvent
protected abstract SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, Table table) throws Exception
Creates aSchemaChangeEventrepresenting the creation of the given table.- Throws:
Exception
-
createDataEvents
private void createDataEvents(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext) throws Exception
- Throws:
Exception
-
tryStartingSnapshot
protected void tryStartingSnapshot(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
-
createDataEventsForTable
private void createDataEventsForTable(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, EventDispatcher.SnapshotReceiver snapshotReceiver, Table table, int tableOrder, int tableCount) throws InterruptedException
Dispatches the data change events for the records of a single table.- Throws:
InterruptedException
-
lastSnapshotRecord
protected void lastSnapshotRecord(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext)
-
rowCountForTable
protected OptionalLong rowCountForTable(TableId tableId)
If connector is able to provide statistics-based number of records per table.
-
getTableScanLogTimer
private Threads.Timer getTableScanLogTimer()
-
getChangeRecordEmitter
protected ChangeRecordEmitter getChangeRecordEmitter(AbstractSnapshotChangeEventSource.SnapshotContext snapshotContext, TableId tableId, Object[] row)
Returns aChangeRecordEmitterproducing the change records for the given table row.
-
determineSnapshotSelect
private Optional<String> determineSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, TableId tableId)
Returns a valid query string for the specified table, either given by the user via snapshot select overrides or defaulting to a statement provided by the DB-specific change event source.- Parameters:
tableId- the table to generate a query for- Returns:
- a valid query string or empty if table will not be snapshotted
-
enhanceOverriddenSelect
protected String enhanceOverriddenSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, String overriddenSelect, TableId tableId)
This method is overridden for Oracle to implement "as of SCN" predicate- Parameters:
snapshotContext- snapshot context, used for getting offset SCNoverriddenSelect- conditional snapshot select- Returns:
- enhanced select statement. By default it just returns original select statements.
-
getSnapshotSelect
protected abstract Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext snapshotContext, TableId tableId)
Returns the SELECT statement to be used for scanning the given table or empty value if the table will be streamed from but not snapshotted
-
getColumnValue
protected Object getColumnValue(ResultSet rs, int columnIndex, Column column, Table table) throws SQLException
- Throws:
SQLException
-
readTableStatement
protected Statement readTableStatement(OptionalLong tableSize) throws SQLException
Allow per-connector query creation to override for best database performance depending on the table size.- Throws:
SQLException
-
rollbackTransaction
private void rollbackTransaction(Connection connection)
-
getClock
protected Clock getClock()
-
postSnapshot
protected void postSnapshot() throws InterruptedException- Throws:
InterruptedException
-
-