Package io.debezium.connector.binlog

Class BinlogSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

java.lang.Object

io.debezium.pipeline.source.AbstractSnapshotChangeEventSource<P,O>

io.debezium.relational.RelationalSnapshotChangeEventSource<P,O>

io.debezium.connector.binlog.BinlogSnapshotChangeEventSource<P,O>

All Implemented Interfaces:

ChangeEventSource, SnapshotChangeEventSource<P,O>, AutoCloseable

public abstract class BinlogSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>
extends RelationalSnapshotChangeEventSource<P,O>

An abstract implementation of SnapshotChangeEventSource for binlog-based connectors.

Author:

Chris Cranford

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	BinlogSnapshotChangeEventSource.BinlogSnapshotContext<P extends BinlogPartition,O extends BinlogOffsetContext>	Mutable context which is populated in the course of snapshotting.

Nested classes/interfaces inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P extends Partition,O extends OffsetContext>

Nested classes/interfaces inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

AbstractSnapshotChangeEventSource.SnapshotContext<P extends Partition,O extends OffsetContext>

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field	Description
private final BinlogConnectorConnection	connection
private final BinlogConnectorConfig	connectorConfig
private final BinlogDatabaseSchema<P,O,?,?>	databaseSchema
private Set<TableId>	delayedSchemaSnapshotTables
private final RelationalTableFilters	filters
private long	globalLockAcquiredAt
private final BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>>	lastEventProcessor
private static final org.slf4j.Logger	LOGGER
private final BinlogSnapshotChangeEventSourceMetrics<P>	metrics
private final Runnable	preSnapshotAction
private static final org.slf4j.Logger	ROW_ESTIMATE_LOGGER
private final Set<SchemaChangeEvent>	schemaEvents
private long	tableLockAcquiredAt

Fields inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

clock, connectionPool, dispatcher, MATCH_ALL_PATTERN, SELECT_ALL_PATTERN, snapshotterService

Fields inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

LOG_INTERVAL, notificationService

Constructor Summary

Constructors

Constructor

Description

BinlogSnapshotChangeEventSource(BinlogConnectorConfig connectorConfig, MainConnectionProvidingConnectionFactory<BinlogConnectorConnection> connectionFactory, BinlogDatabaseSchema<P,O,?,?> schema, EventDispatcher<P,TableId> dispatcher, Clock clock, BinlogSnapshotChangeEventSourceMetrics<P> metrics, BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor, Runnable preSnapshotAction, NotificationService<P,O> notificationService, SnapshotterService snapshotterService)

Method Summary

Modifier and Type	Method	Description
private void	addSchemaEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, String database, String ddl)
private Callable<Map<TableId,String>>	createDdlForTableCallable(TableId tableId, Queue<JdbcConnection> connectionPool)
protected void	createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, SnapshottingTask snapshottingTask)
private void	createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, Collection<TableId> tablesToRead, boolean firstPhase)
private void	createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, Collection<TableId> tablesToRead, boolean firstPhase, ExecutorService executorService)
private Statement	createStatementWithLargeResultSet(BinlogConnectorConnection connection)	Create a JDBC statement that can be used for large result sets.
protected void	determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> ctx, O previousOffset)
protected Set<TableId>	getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> ctx)
protected SchemaChangeEvent	getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, Table table)
protected abstract O	getInitialOffsetContext(BinlogConnectorConfig connectorConfig)
protected Optional<String>	getSnapshotConnectionFirstSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, TableId tableId)
protected Optional<String>	getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, TableId tableId, List<String> columns)	Generate a valid MySQL query string for the specified table and columns
private Optional<String>	getSnapshotSelect(TableId tableId, List<String> columns)
private void	globalLock()
private void	globalUnlock()
private boolean	isGloballyLocked()
private boolean	isTablesLocked()
protected void	lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
protected void	postSnapshot()
protected AbstractSnapshotChangeEventSource.SnapshotContext<P,O>	prepare(P partition, boolean onDemand)
protected void	preSnapshot()
private String	quote(TableId id)
private String	quote(String dbOrTableName)
protected Statement	readTableStatement(JdbcConnection jdbcConnection, OptionalLong rowCount)
protected void	readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext, RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext, O offsetContext, SnapshottingTask snapshottingTask)
protected void	releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
protected void	releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
protected OptionalLong	rowCountForTable(TableId tableId)
protected abstract void	setOffsetContextBinlogPositionAndGtidDetailsForSnapshot(O offsetContext, BinlogConnectorConnection connection, SnapshotterService snapshotterService)
private void	tableLock(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
private void	tableUnlock()
private boolean	twoPhaseSchemaSnapshot()

Methods inherited from class io.debezium.relational.RelationalSnapshotChangeEventSource

additionalColumnFilter, connectionCreated, connectionPoolConnectionCreated, copyOffset, createDataEventsForTableCallable, createSnapshotConnection, doCreateDataEventsForTable, doExecute, enhanceOverriddenSelect, getBlockingSnapshottingTask, getChangeRecordEmitter, getClock, getPreparedColumnNames, getSignalDataCollectionPattern, getSnapshotSelectOverridesByTable, getSnapshotSourceTimestamp, getSnapshottingTask, getTablesForSchemaChange, lastSnapshotRecord, resultSetForDataEvents, tryStartingSnapshot

Methods inherited from class io.debezium.pipeline.source.AbstractSnapshotChangeEventSource

aborted, close, completed, delaySnapshotIfNeeded, determineDataCollectionsToBeSnapshotted, execute, getDataCollectionPattern, getOffsets

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

ROW_ESTIMATE_LOGGER

private static final org.slf4j.Logger ROW_ESTIMATE_LOGGER

connectorConfig

private final BinlogConnectorConfig connectorConfig

connection

private final BinlogConnectorConnection connection

filters

private final RelationalTableFilters filters

metrics

private final BinlogSnapshotChangeEventSourceMetrics<P extends BinlogPartition> metrics

databaseSchema

private final BinlogDatabaseSchema<P extends BinlogPartition,O extends BinlogOffsetContext<?>,?,?> databaseSchema

schemaEvents

private final Set<SchemaChangeEvent> schemaEvents

lastEventProcessor

private final BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor

preSnapshotAction

private final Runnable preSnapshotAction

delayedSchemaSnapshotTables

private Set<TableId> delayedSchemaSnapshotTables

globalLockAcquiredAt

private long globalLockAcquiredAt

tableLockAcquiredAt

private long tableLockAcquiredAt

Constructor Details

BinlogSnapshotChangeEventSource

public BinlogSnapshotChangeEventSource(BinlogConnectorConfig connectorConfig,
 MainConnectionProvidingConnectionFactory<BinlogConnectorConnection> connectionFactory,
 BinlogDatabaseSchema<P,O,?,?> schema,
 EventDispatcher<P,TableId> dispatcher,
 Clock clock,
 BinlogSnapshotChangeEventSourceMetrics<P> metrics,
 BlockingConsumer<Function<org.apache.kafka.connect.source.SourceRecord,org.apache.kafka.connect.source.SourceRecord>> lastEventProcessor,
 Runnable preSnapshotAction,
 NotificationService<P,O> notificationService,
 SnapshotterService snapshotterService)

Method Details

prepare

protected AbstractSnapshotChangeEventSource.SnapshotContext<P,O> prepare(P partition,
 boolean onDemand)

Specified by:

prepare in class AbstractSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

getAllTableIds

protected Set<TableId> getAllTableIds(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> ctx)
                               throws Exception

Specified by:

getAllTableIds in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Throws:

Exception

lockTablesForSchemaSnapshot

protected void lockTablesForSchemaSnapshot(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
                                    throws SQLException

Specified by:

lockTablesForSchemaSnapshot in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Throws:

SQLException

releaseSchemaSnapshotLocks

protected void releaseSchemaSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
                                   throws SQLException

Specified by:

releaseSchemaSnapshotLocks in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Throws:

SQLException

releaseDataSnapshotLocks

protected void releaseDataSnapshotLocks(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
                                 throws Exception

Overrides:

releaseDataSnapshotLocks in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Throws:

Exception

determineSnapshotOffset

protected void determineSnapshotOffset(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> ctx,
 O previousOffset)
                                throws Exception

Specified by:

determineSnapshotOffset in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Throws:

Exception

getInitialOffsetContext

protected abstract O getInitialOffsetContext(BinlogConnectorConfig connectorConfig)

setOffsetContextBinlogPositionAndGtidDetailsForSnapshot

protected abstract void setOffsetContextBinlogPositionAndGtidDetailsForSnapshot(O offsetContext,
 BinlogConnectorConnection connection,
 SnapshotterService snapshotterService)
                                                                         throws Exception

Throws:

Exception

addSchemaEvent

private void addSchemaEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 String database,
 String ddl)

readTableStructure

protected void readTableStructure(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 O offsetContext,
 SnapshottingTask snapshottingTask)
                           throws Exception

Specified by:

readTableStructure in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Throws:

Exception

createSchemaEventsForTables

private void createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 Collection<TableId> tablesToRead,
 boolean firstPhase)
                                  throws Exception

Throws:

Exception

createSchemaEventsForTables

private void createSchemaEventsForTables(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 Collection<TableId> tablesToRead,
 boolean firstPhase,
 ExecutorService executorService)
                                  throws Exception

Throws:

Exception

createDdlForTableCallable

private Callable<Map<TableId,String>> createDdlForTableCallable(TableId tableId,
 Queue<JdbcConnection> connectionPool)

twoPhaseSchemaSnapshot

private boolean twoPhaseSchemaSnapshot()

getCreateTableEvent

protected SchemaChangeEvent getCreateTableEvent(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 Table table)

Specified by:

getCreateTableEvent in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

getSnapshotSelect

protected Optional<String> getSnapshotSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 TableId tableId,
 List<String> columns)

Generate a valid MySQL query string for the specified table and columns

Specified by:

getSnapshotSelect in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Parameters:

tableId - the table to generate a query for

Returns:

a valid query string

getSnapshotSelect

private Optional<String> getSnapshotSelect(TableId tableId,
 List<String> columns)

getSnapshotConnectionFirstSelect

protected Optional<String> getSnapshotConnectionFirstSelect(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 TableId tableId)

Overrides:

getSnapshotConnectionFirstSelect in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

isGloballyLocked

private boolean isGloballyLocked()

isTablesLocked

private boolean isTablesLocked()

globalLock

private void globalLock()
                 throws SQLException

Throws:

SQLException

globalUnlock

private void globalUnlock()
                   throws SQLException

Throws:

SQLException

tableLock

private void tableLock(RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext)
                throws SQLException

Throws:

SQLException

tableUnlock

private void tableUnlock()
                  throws SQLException

Throws:

SQLException

quote

private String quote(String dbOrTableName)

quote

private String quote(TableId id)

rowCountForTable

protected OptionalLong rowCountForTable(TableId tableId)

Overrides:

rowCountForTable in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

readTableStatement

protected Statement readTableStatement(JdbcConnection jdbcConnection,
 OptionalLong rowCount)
                                throws SQLException

Overrides:

readTableStatement in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Throws:

SQLException

createStatementWithLargeResultSet

private Statement createStatementWithLargeResultSet(BinlogConnectorConnection connection)
                                             throws SQLException

Create a JDBC statement that can be used for large result sets.

By default, the MySQL Connector/J driver retrieves all rows for ResultSets and stores them in memory. In most cases this is the most efficient way to operate and, due to the design of the MySQL network protocol, is easier to implement. However, when ResultSets that have a large number of rows or large values, the driver may not be able to allocate heap space in the JVM and may result in an OutOfMemoryError. See DBZ-94 for details.

This method handles such cases using the recommended technique for MySQL by creating the JDBC Statement with forward-only cursor and read-only concurrency flags, and with a minimum value fetch size hint.

Returns:

the statement; never null

Throws:

SQLException - if there is a problem creating the statement

createSchemaChangeEventsForTables

protected void createSchemaChangeEventsForTables(ChangeEventSource.ChangeEventSourceContext sourceContext,
 RelationalSnapshotChangeEventSource.RelationalSnapshotContext<P,O> snapshotContext,
 SnapshottingTask snapshottingTask)
                                          throws Exception

Overrides:

createSchemaChangeEventsForTables in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Throws:

Exception

postSnapshot

protected void postSnapshot()
                     throws InterruptedException

Overrides:

postSnapshot in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Throws:

InterruptedException

preSnapshot

protected void preSnapshot()
                    throws InterruptedException

Overrides:

preSnapshot in class RelationalSnapshotChangeEventSource<P extends BinlogPartition,O extends BinlogOffsetContext<?>>

Throws:

InterruptedException