Package io.debezium.connector.cassandra

Class SnapshotProcessor

java.lang.Object

io.debezium.connector.cassandra.AbstractProcessor

io.debezium.connector.cassandra.SnapshotProcessor

public class SnapshotProcessor
extends AbstractProcessor

This reader is responsible for initial bootstrapping of a table, which entails converting each row into a change event and enqueueing that event to the ChangeEventQueue. IMPORTANT: Currently, only when a snapshot is completed will the OffsetWriter record the table in the offset.properties file (with filename "" and position -1). This means if the SnapshotProcessor is terminated midway, upon restart it will skip all the tables that are already recorded in offset.properties

Field Summary

Fields

Modifier and Type	Field	Description
private static String	CASSANDRA_NOW_UNIXTIMESTAMP
private CassandraClient	cassandraClient
private static Set<com.datastax.driver.core.DataType.Name>	collectionTypes
private com.datastax.driver.core.ConsistencyLevel	consistencyLevel
private static String	EXECUTION_TIME_ALIAS
private boolean	initial
private static org.slf4j.Logger	LOGGER
private SnapshotProcessorMetrics	metrics
private static String	NAME
private OffsetWriter	offsetWriter
private List<io.debezium.connector.base.ChangeEventQueue<Event>>	queues
private RecordMaker	recordMaker
private SchemaHolder	schemaHolder
private CassandraConnectorConfig.SnapshotMode	snapshotMode
private Set<String>	startedTableNames

Constructor Summary

Constructors

Constructor	Description
SnapshotProcessor(CassandraConnectorContext context)

Method Summary

Modifier and Type	Method	Description
private static long	calculateDeletionTs(Object executionTime, Object ttl)	it is not possible to query deletion time via cql, so instead calculate it from execution time (in milliseconds) + ttl (in seconds)
void	destroy()	Override destroy to clean up resources after stopping the processor
private static RowData	extractRowData(com.datastax.driver.core.Row row, List<com.datastax.driver.core.ColumnMetadata> columns, Set<String> partitionKeyNames, Set<String> clusteringKeyNames, Object executionTime)	This function extracts the relevant row data from Row and updates the maximum writetime for each row.
private static com.datastax.driver.core.querybuilder.BuiltStatement	generateSnapshotStatement(com.datastax.driver.core.TableMetadata tableMetadata)	Build the SELECT query statement for execution.
private Set<com.datastax.driver.core.TableMetadata>	getTablesToSnapshot()	Return a set of TableMetadata for tables that have not been snapshotted but have CDC enabled.
private static CellData.ColumnType	getType(String name, Set<String> partitionKeyNames, Set<String> clusteringKeyNames)
void	initialize()	Override initialize to initialize resources before starting the processor
void	process()	The actual work the processor is doing.
private void	processResultSet(com.datastax.driver.core.TableMetadata tableMetadata, com.datastax.driver.core.ResultSet resultSet)	Process the result set from the query.
private static Object	readCol(com.datastax.driver.core.Row row, String col, com.datastax.driver.core.ColumnMetadata cm)
private static Object	readColTtl(com.datastax.driver.core.Row row, String col)
private static Object	readExecutionTime(com.datastax.driver.core.Row row)
(package private) void	snapshot()	Fetch for all new tables that have not yet been snapshotted, and then iterate through the tables to snapshot each one of them.
private static String	tableName(com.datastax.driver.core.TableMetadata tm)
private void	takeTableSnapshot(com.datastax.driver.core.TableMetadata tableMetadata)	Runs a SELECT query on a given table and process each row in the result set by converting the row into a record and enqueue it to ChangeRecord
private static String	ttlAlias(String colName)
private static String	withQuotes(String s)

Methods inherited from class io.debezium.connector.cassandra.AbstractProcessor

getName, isRunning, start, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

NAME

private static final String NAME

See Also:

Constant Field Values

CASSANDRA_NOW_UNIXTIMESTAMP

private static final String CASSANDRA_NOW_UNIXTIMESTAMP

See Also:

Constant Field Values

EXECUTION_TIME_ALIAS

private static final String EXECUTION_TIME_ALIAS

See Also:

Constant Field Values

collectionTypes

private static final Set<com.datastax.driver.core.DataType.Name> collectionTypes

cassandraClient

private final CassandraClient cassandraClient

queues

private final List<io.debezium.connector.base.ChangeEventQueue<Event>> queues

offsetWriter

private final OffsetWriter offsetWriter

schemaHolder

private final SchemaHolder schemaHolder

recordMaker

private final RecordMaker recordMaker

snapshotMode

private final CassandraConnectorConfig.SnapshotMode snapshotMode

consistencyLevel

private final com.datastax.driver.core.ConsistencyLevel consistencyLevel

startedTableNames

private final Set<String> startedTableNames

metrics

private final SnapshotProcessorMetrics metrics

initial

private boolean initial

Constructor Detail

SnapshotProcessor

public SnapshotProcessor(CassandraConnectorContext context)

Method Detail

initialize

public void initialize()

Description copied from class: AbstractProcessor

Override initialize to initialize resources before starting the processor

Overrides:

initialize in class AbstractProcessor

destroy

public void destroy()

Description copied from class: AbstractProcessor

Override destroy to clean up resources after stopping the processor

Overrides:

destroy in class AbstractProcessor

process

public void process()
             throws IOException

Description copied from class: AbstractProcessor

The actual work the processor is doing. This method will be executed in a while loop until processor stops or encounters exception.

Specified by:

process in class AbstractProcessor

Throws:

IOException

snapshot

void snapshot()
       throws IOException

Fetch for all new tables that have not yet been snapshotted, and then iterate through the tables to snapshot each one of them.

Throws:

IOException

getTablesToSnapshot

private Set<com.datastax.driver.core.TableMetadata> getTablesToSnapshot()

Return a set of TableMetadata for tables that have not been snapshotted but have CDC enabled.

takeTableSnapshot

private void takeTableSnapshot(com.datastax.driver.core.TableMetadata tableMetadata)
                        throws IOException

Runs a SELECT query on a given table and process each row in the result set by converting the row into a record and enqueue it to ChangeRecord

Throws:

IOException

generateSnapshotStatement

private static com.datastax.driver.core.querybuilder.BuiltStatement generateSnapshotStatement(com.datastax.driver.core.TableMetadata tableMetadata)

Build the SELECT query statement for execution. For every non-primary-key column, the TTL, WRITETIME, and execution time are also queried. For example, a table t with columns a, b, and c, where A is the partition key, B is the clustering key, and C is a regular column, looks like the following:

     SELECT now() as execution_time, a, b, c, TTL(c) as c_ttl, WRITETIME(c) as c_writetime FROM t;
 

processResultSet

private void processResultSet(com.datastax.driver.core.TableMetadata tableMetadata,
                              com.datastax.driver.core.ResultSet resultSet)
                       throws IOException

Process the result set from the query. Each row is converted into a ChangeRecord and enqueued to the ChangeEventQueue.

Throws:

IOException

extractRowData

private static RowData extractRowData(com.datastax.driver.core.Row row,
                                      List<com.datastax.driver.core.ColumnMetadata> columns,
                                      Set<String> partitionKeyNames,
                                      Set<String> clusteringKeyNames,
                                      Object executionTime)

This function extracts the relevant row data from Row and updates the maximum writetime for each row.

getType

private static CellData.ColumnType getType(String name,
                                           Set<String> partitionKeyNames,
                                           Set<String> clusteringKeyNames)

readExecutionTime

private static Object readExecutionTime(com.datastax.driver.core.Row row)

readCol

private static Object readCol(com.datastax.driver.core.Row row,
                              String col,
                              com.datastax.driver.core.ColumnMetadata cm)

readColTtl

private static Object readColTtl(com.datastax.driver.core.Row row,
                                 String col)

calculateDeletionTs

private static long calculateDeletionTs(Object executionTime,
                                        Object ttl)

it is not possible to query deletion time via cql, so instead calculate it from execution time (in milliseconds) + ttl (in seconds)

ttlAlias

private static String ttlAlias(String colName)

withQuotes

private static String withQuotes(String s)

tableName

private static String tableName(com.datastax.driver.core.TableMetadata tm)