Package io.debezium.connector.sqlserver

Class SqlServerStreamingChangeEventSource

java.lang.Object

io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource

All Implemented Interfaces:

ChangeEventSource, StreamingChangeEventSource<SqlServerPartition,SqlServerOffsetContext>

public class SqlServerStreamingChangeEventSource
extends Object
implements StreamingChangeEventSource<SqlServerPartition,SqlServerOffsetContext>

A StreamingChangeEventSource based on SQL Server change data capture functionality. A main loop polls database DDL change and change data tables and turns them into change events.

The connector uses CDC functionality of SQL Server that is implemented as as a process that monitors source table and write changes from the table into the change table.

The main loop keeps a pointer to the LSN of changes that were already processed. It queries all change tables and get result set of changes. It always finds the smallest LSN across all tables and the change is converted into the event message and sent downstream. The process repeats until all result sets are empty. The LSN is marked and the procedure repeats.

The schema changes detection follows the procedure recommended by SQL Server CDC documentation. The database operator should create one more capture process (and table) when a table schema is updated. The code detects presence of two change tables for a single source table. It decides which table is the new one depending on LSNs stored in them. The loop streams changes from the older table till there are events in new table with the LSN larger than in the old one. Then the change table is switched and streaming is executed from the new one.

Author:

Jiri Pechanec

Nested Class Summary

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field	Description
private final Map<SqlServerPartition,Set<SqlServerChangeTable>>	changeTablesWithKnownStopLsn
private boolean	checkAgent
private final Clock	clock
private final SqlServerConnectorConfig	connectorConfig
private final SqlServerConnection	dataConnection	Connection used for reading CDC tables.
private static final Duration	DEFAULT_INTERVAL_BETWEEN_COMMITS
private final EventDispatcher<SqlServerPartition,TableId>	dispatcher
private SqlServerOffsetContext	effectiveOffset
private final ErrorHandler	errorHandler
private static final int	INTERVAL_BETWEEN_COMMITS_BASED_ON_POLL_FACTOR
private static final org.slf4j.Logger	LOGGER
private final SqlServerConnection	metadataConnection	A separate connection for retrieving details of the schema changes; without it, adaptive buffering will not work.
private static final Pattern	MISSING_CDC_FUNCTION_CHANGES_ERROR
private final NotificationService<SqlServerPartition,SqlServerOffsetContext>	notificationService
private final ElapsedTimeStrategy	pauseBetweenCommits
private final Duration	pollInterval
private final SqlServerDatabaseSchema	schema
private final Map<SqlServerPartition,SqlServerStreamingExecutionContext>	streamingExecutionContexts

Constructor Summary

Constructors

Constructor	Description
SqlServerStreamingChangeEventSource(SqlServerConnectorConfig connectorConfig, SqlServerConnection dataConnection, SqlServerConnection metadataConnection, EventDispatcher<SqlServerPartition,TableId> dispatcher, ErrorHandler errorHandler, Clock clock, SqlServerDatabaseSchema schema, NotificationService<SqlServerPartition,SqlServerOffsetContext> notificationService)

Method Summary

Modifier and Type	Method	Description
private void	collectChangeTablesWithKnownStopLsn(SqlServerPartition partition, SqlServerChangeTable[] tables)
void	commitOffset(Map<String,?> sourcePartition, Map<String,?> offset)
private void	commitTransaction()
void	execute(ChangeEventSource.ChangeEventSourceContext context, SqlServerPartition partition, SqlServerOffsetContext offsetContext)
boolean	executeIteration(ChangeEventSource.ChangeEventSourceContext context, SqlServerPartition partition, SqlServerOffsetContext offsetContext)
private SqlServerChangeTable[]	getChangeTablesToQuery(SqlServerPartition partition, SqlServerOffsetContext offsetContext, Lsn toLsn)
SqlServerOffsetContext	getOffsetContext()
private Lsn	getToLsn(SqlServerConnection connection, String databaseName, TxLogPosition lastProcessedPosition, int maxTransactionsPerIteration)
void	init(SqlServerOffsetContext offsetContext)
private void	migrateTable(SqlServerPartition partition, Queue<SqlServerChangeTable> schemaChangeCheckpoints, SqlServerOffsetContext offsetContext)
private SqlServerChangeTable[]	processErrorFromChangeTableQuery(String databaseName, SQLException exception, SqlServerChangeTable[] currentChangeTables)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

MISSING_CDC_FUNCTION_CHANGES_ERROR

private static final Pattern MISSING_CDC_FUNCTION_CHANGES_ERROR

LOGGER

private static final org.slf4j.Logger LOGGER

DEFAULT_INTERVAL_BETWEEN_COMMITS

private static final Duration DEFAULT_INTERVAL_BETWEEN_COMMITS

INTERVAL_BETWEEN_COMMITS_BASED_ON_POLL_FACTOR

private static final int INTERVAL_BETWEEN_COMMITS_BASED_ON_POLL_FACTOR

See Also:

• Constant Field Values

dataConnection

private final SqlServerConnection dataConnection

Connection used for reading CDC tables.

metadataConnection

private final SqlServerConnection metadataConnection

A separate connection for retrieving details of the schema changes; without it, adaptive buffering will not work.

dispatcher

private final EventDispatcher<SqlServerPartition,TableId> dispatcher

errorHandler

private final ErrorHandler errorHandler

clock

private final Clock clock

schema

private final SqlServerDatabaseSchema schema

pollInterval

private final Duration pollInterval

connectorConfig

private final SqlServerConnectorConfig connectorConfig

pauseBetweenCommits

private final ElapsedTimeStrategy pauseBetweenCommits

streamingExecutionContexts

private final Map<SqlServerPartition,SqlServerStreamingExecutionContext> streamingExecutionContexts

changeTablesWithKnownStopLsn

private final Map<SqlServerPartition,Set<SqlServerChangeTable>> changeTablesWithKnownStopLsn

checkAgent

private boolean checkAgent

effectiveOffset

private SqlServerOffsetContext effectiveOffset

notificationService

private final NotificationService<SqlServerPartition,SqlServerOffsetContext> notificationService

Constructor Details

SqlServerStreamingChangeEventSource

public SqlServerStreamingChangeEventSource(SqlServerConnectorConfig connectorConfig,
 SqlServerConnection dataConnection,
 SqlServerConnection metadataConnection,
 EventDispatcher<SqlServerPartition,TableId> dispatcher,
 ErrorHandler errorHandler,
 Clock clock,
 SqlServerDatabaseSchema schema,
 NotificationService<SqlServerPartition,SqlServerOffsetContext> notificationService)

Method Details

init

public void init(SqlServerOffsetContext offsetContext)
          throws InterruptedException

Specified by:

init in interface StreamingChangeEventSource<SqlServerPartition,SqlServerOffsetContext>

Throws:

InterruptedException

execute

public void execute(ChangeEventSource.ChangeEventSourceContext context,
 SqlServerPartition partition,
 SqlServerOffsetContext offsetContext)
             throws InterruptedException

Specified by:

execute in interface StreamingChangeEventSource<SqlServerPartition,SqlServerOffsetContext>

Throws:

InterruptedException

executeIteration

public boolean executeIteration(ChangeEventSource.ChangeEventSourceContext context,
 SqlServerPartition partition,
 SqlServerOffsetContext offsetContext)
                         throws InterruptedException

Specified by:

executeIteration in interface StreamingChangeEventSource<SqlServerPartition,SqlServerOffsetContext>

Throws:

InterruptedException

getOffsetContext

public SqlServerOffsetContext getOffsetContext()

Specified by:

getOffsetContext in interface StreamingChangeEventSource<SqlServerPartition,SqlServerOffsetContext>

collectChangeTablesWithKnownStopLsn

private void collectChangeTablesWithKnownStopLsn(SqlServerPartition partition,
 SqlServerChangeTable[] tables)

commitTransaction

private void commitTransaction()
                        throws SQLException

Throws:

SQLException

migrateTable

private void migrateTable(SqlServerPartition partition,
 Queue<SqlServerChangeTable> schemaChangeCheckpoints,
 SqlServerOffsetContext offsetContext)
                   throws InterruptedException,
SQLException

Throws:

InterruptedException

SQLException

processErrorFromChangeTableQuery

private SqlServerChangeTable[] processErrorFromChangeTableQuery(String databaseName,
 SQLException exception,
 SqlServerChangeTable[] currentChangeTables)
                                                         throws Exception

Throws:

Exception

getChangeTablesToQuery

private SqlServerChangeTable[] getChangeTablesToQuery(SqlServerPartition partition,
 SqlServerOffsetContext offsetContext,
 Lsn toLsn)
                                               throws SQLException,
InterruptedException

Throws:

SQLException

InterruptedException

getToLsn

private Lsn getToLsn(SqlServerConnection connection,
 String databaseName,
 TxLogPosition lastProcessedPosition,
 int maxTransactionsPerIteration)
              throws SQLException

Returns:

the log sequence number up until which the connector should query changes from the database.

Throws:

SQLException

commitOffset

public void commitOffset(Map<String,?> sourcePartition,
 Map<String,?> offset)

Specified by:

commitOffset in interface StreamingChangeEventSource<SqlServerPartition,SqlServerOffsetContext>