Package io.debezium.connector.db2

Class Db2StreamingChangeEventSource

java.lang.Object

io.debezium.connector.db2.Db2StreamingChangeEventSource

All Implemented Interfaces:

io.debezium.pipeline.source.spi.ChangeEventSource, io.debezium.pipeline.source.spi.StreamingChangeEventSource<Db2Partition,Db2OffsetContext>

public class Db2StreamingChangeEventSource
extends Object
implements io.debezium.pipeline.source.spi.StreamingChangeEventSource<Db2Partition,Db2OffsetContext>

A StreamingChangeEventSource based on DB2 change data capture functionality. A main loop polls database DDL change and change data tables and turns them into change events.

The connector uses CDC functionality of DB2 that is implemented as as a process that monitors source table and write changes from the table into the change table.

The main loop keeps a pointer to the LSN of changes that were already processed. It queries all change tables and get result set of changes. It always finds the smallest LSN across all tables and the change is converted into the event message and sent downstream. The process repeats until all result sets are empty. The LSN is marked and the procedure repeats.

The schema changes detection follows the procedure recommended by DB2 CDC documentation. The database operator should create one more capture process (and table) when a table schema is updated. The code detects presence of two change tables for a single source table. It decides which table is the new one depending on LSNs stored in them. The loop streams changes from the older table till there are events in new table with the LSN larger than in the old one. Then the change table is switched and streaming is executed from the new one.

Author:

Jiri Pechanec, Peter Urbanetz

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	Db2StreamingChangeEventSource.ChangeTablePointer	The logical representation of a position for the change in the transaction log.

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

io.debezium.pipeline.source.spi.ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field	Description
private final io.debezium.util.Clock	clock
private static final int	COL_COMMIT_LSN
private static final int	COL_DATA
private static final int	COL_OPERATION
private static final int	COL_ROW_LSN
private final Db2ConnectorConfig	connectorConfig
private final Db2Connection	dataConnection	Connection used for reading CDC tables.
private final io.debezium.pipeline.EventDispatcher<Db2Partition,io.debezium.relational.TableId>	dispatcher
private Db2OffsetContext	effectiveOffsetContext
private final io.debezium.pipeline.ErrorHandler	errorHandler
private static final org.slf4j.Logger	LOGGER
private final Db2Connection	metadataConnection	A separate connection for retrieving timestamps; without it, adaptive buffering will not work.
private static final Pattern	MISSING_CDC_FUNCTION_CHANGES_ERROR
private final Duration	pollInterval
private final Db2DatabaseSchema	schema

Constructor Summary

Constructors

Constructor	Description
Db2StreamingChangeEventSource(Db2ConnectorConfig connectorConfig, Db2Connection dataConnection, Db2Connection metadataConnection, io.debezium.pipeline.EventDispatcher<Db2Partition,io.debezium.relational.TableId> dispatcher, io.debezium.pipeline.ErrorHandler errorHandler, io.debezium.util.Clock clock, Db2DatabaseSchema schema)

Method Summary

Modifier and Type	Method	Description
void	execute(io.debezium.pipeline.source.spi.ChangeEventSource.ChangeEventSourceContext context, Db2Partition partition, Db2OffsetContext offsetContext)
private Db2ChangeTable[]	getCdcTablesToQuery(Db2Partition partition, Db2OffsetContext offsetContext)
Db2OffsetContext	getOffsetContext()
void	init(Db2OffsetContext offsetContext)
private void	migrateTable(Db2Partition partition, Db2OffsetContext offsetContext, Queue<Db2ChangeTable> schemaChangeCheckpoints)
private Db2ChangeTable[]	processErrorFromChangeTableQuery(SQLException exception, Db2ChangeTable[] currentChangeTables)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.debezium.pipeline.source.spi.StreamingChangeEventSource

commitOffset, executeIteration

Field Details

COL_COMMIT_LSN

private static final int COL_COMMIT_LSN

See Also:

• Constant Field Values

COL_ROW_LSN

private static final int COL_ROW_LSN

See Also:

• Constant Field Values

COL_OPERATION

private static final int COL_OPERATION

See Also:

• Constant Field Values

COL_DATA

private static final int COL_DATA

See Also:

• Constant Field Values

MISSING_CDC_FUNCTION_CHANGES_ERROR

private static final Pattern MISSING_CDC_FUNCTION_CHANGES_ERROR

LOGGER

private static final org.slf4j.Logger LOGGER

dataConnection

private final Db2Connection dataConnection

Connection used for reading CDC tables.

metadataConnection

private final Db2Connection metadataConnection

A separate connection for retrieving timestamps; without it, adaptive buffering will not work.

dispatcher

private final io.debezium.pipeline.EventDispatcher<Db2Partition,io.debezium.relational.TableId> dispatcher

errorHandler

private final io.debezium.pipeline.ErrorHandler errorHandler

clock

private final io.debezium.util.Clock clock

schema

private final Db2DatabaseSchema schema

pollInterval

private final Duration pollInterval

connectorConfig

private final Db2ConnectorConfig connectorConfig

effectiveOffsetContext

private Db2OffsetContext effectiveOffsetContext

Constructor Details

Db2StreamingChangeEventSource

public Db2StreamingChangeEventSource(Db2ConnectorConfig connectorConfig,
 Db2Connection dataConnection,
 Db2Connection metadataConnection,
 io.debezium.pipeline.EventDispatcher<Db2Partition,io.debezium.relational.TableId> dispatcher,
 io.debezium.pipeline.ErrorHandler errorHandler,
 io.debezium.util.Clock clock,
 Db2DatabaseSchema schema)

Method Details

init

public void init(Db2OffsetContext offsetContext)

Specified by:

init in interface io.debezium.pipeline.source.spi.StreamingChangeEventSource<Db2Partition,Db2OffsetContext>

execute

public void execute(io.debezium.pipeline.source.spi.ChangeEventSource.ChangeEventSourceContext context,
 Db2Partition partition,
 Db2OffsetContext offsetContext)
             throws InterruptedException

Specified by:

execute in interface io.debezium.pipeline.source.spi.StreamingChangeEventSource<Db2Partition,Db2OffsetContext>

Throws:

InterruptedException

getOffsetContext

public Db2OffsetContext getOffsetContext()

Specified by:

getOffsetContext in interface io.debezium.pipeline.source.spi.StreamingChangeEventSource<Db2Partition,Db2OffsetContext>

migrateTable

private void migrateTable(Db2Partition partition,
 Db2OffsetContext offsetContext,
 Queue<Db2ChangeTable> schemaChangeCheckpoints)
                   throws InterruptedException,
SQLException

Throws:

InterruptedException

SQLException

processErrorFromChangeTableQuery

private Db2ChangeTable[] processErrorFromChangeTableQuery(SQLException exception,
 Db2ChangeTable[] currentChangeTables)
                                                   throws Exception

Throws:

Exception

getCdcTablesToQuery

private Db2ChangeTable[] getCdcTablesToQuery(Db2Partition partition,
 Db2OffsetContext offsetContext)
                                      throws SQLException,
InterruptedException

Throws:

SQLException

InterruptedException