Package io.debezium.connector.oracle.logminer.processor.infinispan

Class AbstractInfinispanLogMinerEventProcessor

java.lang.Object

io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor<InfinispanTransaction>

io.debezium.connector.oracle.logminer.processor.infinispan.AbstractInfinispanLogMinerEventProcessor

All Implemented Interfaces:

CacheProvider, LogMinerEventProcessor, AutoCloseable

Direct Known Subclasses:

EmbeddedInfinispanLogMinerEventProcessor, RemoteInfinispanLogMinerEventProcessor

public abstract class AbstractInfinispanLogMinerEventProcessor
extends AbstractLogMinerEventProcessor<InfinispanTransaction>
implements CacheProvider

An implementation of LogMinerEventProcessor that uses Infinispan to persist the transaction cache across restarts on disk.

Author:

Chris Cranford

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	AbstractInfinispanLogMinerEventProcessor.EventKeySortComparator	A comparator that guarantees that the sort order applied to event keys is such that they are treated as numerical values, sorted as numeric values rather than strings which would allow "100" to come before "9".

Nested classes/interfaces inherited from class io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor

AbstractLogMinerEventProcessor.Counters

Field Summary

Fields

Modifier and Type	Field	Description
private final EventDispatcher<OraclePartition,TableId>	dispatcher
private InMemoryPendingTransactionsCache	inMemoryPendingTransactionsCache
private static AbstractInfinispanLogMinerEventProcessor	instance
private final OracleConnection	jdbcConnection
private static final org.slf4j.Logger	LOGGER
private final LogMinerStreamingChangeEventSourceMetrics	metrics
private final OracleOffsetContext	offsetContext
private final OraclePartition	partition

Fields inherited from class io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor

counters, sqlQuery

Fields inherited from interface io.debezium.connector.oracle.logminer.processor.infinispan.CacheProvider

EVENTS_CACHE_NAME, PROCESSED_TRANSACTIONS_CACHE_NAME, SCHEMA_CHANGES_CACHE_NAME, TRANSACTIONS_CACHE_NAME

Constructor Summary

Constructors

Constructor	Description
AbstractInfinispanLogMinerEventProcessor(ChangeEventSource.ChangeEventSourceContext context, OracleConnectorConfig connectorConfig, OracleConnection jdbcConnection, EventDispatcher<OraclePartition,TableId> dispatcher, OraclePartition partition, OracleOffsetContext offsetContext, OracleDatabaseSchema schema, LogMinerStreamingChangeEventSourceMetrics metrics)

Method Summary

Modifier and Type	Method	Description
protected void	addToTransaction(String transactionId, LogMinerEventRow row, Supplier<LogMinerEvent> eventSupplier)	Add a transaction to the transaction map if allowed.
protected Scn	calculateNewStartScn(Scn endScn, Scn maxCommittedScn)	Calculates the new starting system change number based on the current processing range.
protected void	cleanupAfterTransactionRemovedFromCache(InfinispanTransaction transaction, boolean isAbandoned)	Removes the items associated with the transaction (e.g.
protected PreparedStatement	createQueryStatement()	Create the JDBC query that will be used to fetch the mining result set.
protected InfinispanTransaction	createTransaction(LogMinerEventRow row)	Creates a new transaction based on the supplied START event.
void	displayCacheStatistics()	Displays cache statistics
protected void	finalizeTransactionCommit(String transactionId, Scn commitScn)	Finalizes the commit of a transaction.
protected void	finalizeTransactionRollback(String transactionId, Scn rollbackScn)	Finalizes the rollback the specified transaction
protected InfinispanTransaction	getAndRemoveTransactionFromCache(String transactionId)	Gets a transaction instance from the transaction cache while also removing its cache entry.
protected int	getTransactionEventCount(InfinispanTransaction transaction)	Returns the number of events associated with the specified transaction.
protected Iterator<LogMinerEvent>	getTransactionEventIterator(InfinispanTransaction transaction)	Get an iterator over the events that are part of the specified transaction.
private List<String>	getTransactionKeysWithPrefix(String prefix)
protected void	handleSchemaChange(LogMinerEventRow row)	Handle processing a LogMinerEventRow for a DDL event.
protected boolean	hasSchemaChangeBeenSeen(LogMinerEventRow row)	Checks whether the LogMinerEvent row for a schema change can be emitted.
protected boolean	isRecentlyProcessed(String transactionId)	Check whether a transaction has been recently processed through either a commit or rollback.
static void	logCacheStats()	Can be used for reporting in Debezium Embedded mode
protected void	processRow(OraclePartition partition, LogMinerEventRow row)	Processes a single LogMinerEventRow.
protected abstract void	purgeCache(Scn minCacheScn)	Purge the necessary caches with all entries that occurred prior to the specified change number.
protected void	reCreateInMemoryCache()
private void	removeEventsWithTransaction(InfinispanTransaction transaction)
protected void	removeEventWithRowId(LogMinerEventRow row)	Removes a specific transaction event by database row identifier.
protected <K, V> void	removeIf(org.infinispan.commons.util.CloseableIterator<Map.Entry<K,V>> iterator, Predicate<Map.Entry<K,V>> filter)	Helper method to remove entries that match the given predicate from a closeable iterator.
protected void	resetTransactionToStart(InfinispanTransaction transaction)

Methods inherited from class io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor

abandonTransactionOverEventThreshold, abandonTransactions, getAbandonedTransactionsCache, getConfig, getFirstActiveTransactionKey, getLastProcessedScn, getLastProcessedScnChangeTime, getLastScnToAbandon, getOldestTransactionInCache, getQueryString, getSchema, getTransactionCache, getTransactionCacheMinimumScn, getTransactionIdPrefix, handleCommit, handleCommitNotFoundInBuffer, handleDataEvent, handleLobWrite, handleMissingScn, handleRollback, handleRollbackNotFoundInBuffer, handleSelectLobLocator, handleStart, handleUnsupportedEvent, isTransactionIdWithNoSequence, isTransactionOverEventThreshold, isTransactionUserExcluded, isTrxIdRawValue, process, processResults, warnPotentiallyStuckScn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.AutoCloseable

close

Methods inherited from interface io.debezium.connector.oracle.logminer.processor.infinispan.CacheProvider

getEventCache, getProcessedTransactionsCache, getSchemaChangesCache, getTransactionCache

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

jdbcConnection

private final OracleConnection jdbcConnection

metrics

private final LogMinerStreamingChangeEventSourceMetrics metrics

partition

private final OraclePartition partition

offsetContext

private final OracleOffsetContext offsetContext

dispatcher

private final EventDispatcher<OraclePartition,TableId> dispatcher

inMemoryPendingTransactionsCache

private InMemoryPendingTransactionsCache inMemoryPendingTransactionsCache

instance

private static AbstractInfinispanLogMinerEventProcessor instance

Constructor Details

AbstractInfinispanLogMinerEventProcessor

public AbstractInfinispanLogMinerEventProcessor(ChangeEventSource.ChangeEventSourceContext context,
 OracleConnectorConfig connectorConfig,
 OracleConnection jdbcConnection,
 EventDispatcher<OraclePartition,TableId> dispatcher,
 OraclePartition partition,
 OracleOffsetContext offsetContext,
 OracleDatabaseSchema schema,
 LogMinerStreamingChangeEventSourceMetrics metrics)

Method Details

reCreateInMemoryCache

protected void reCreateInMemoryCache()

logCacheStats

public static void logCacheStats()

Can be used for reporting in Debezium Embedded mode

displayCacheStatistics

public void displayCacheStatistics()

Description copied from interface: CacheProvider

Displays cache statistics

Specified by:

displayCacheStatistics in interface CacheProvider

isRecentlyProcessed

protected boolean isRecentlyProcessed(String transactionId)

Description copied from class: AbstractLogMinerEventProcessor

Check whether a transaction has been recently processed through either a commit or rollback.

Overrides:

isRecentlyProcessed in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

transactionId - the unique transaction id

Returns:

true if the transaction has been recently processed, false otherwise

createTransaction

protected InfinispanTransaction createTransaction(LogMinerEventRow row)

Description copied from class: AbstractLogMinerEventProcessor

Creates a new transaction based on the supplied START event.

Specified by:

createTransaction in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

row - the event row, must not be null

Returns:

the implementation-specific Transaction instance

removeEventWithRowId

protected void removeEventWithRowId(LogMinerEventRow row)

Description copied from class: AbstractLogMinerEventProcessor

Removes a specific transaction event by database row identifier.

Specified by:

removeEventWithRowId in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

row - the event row that contains the row identifier, must not be null

getTransactionKeysWithPrefix

private List<String> getTransactionKeysWithPrefix(String prefix)

processRow

protected void processRow(OraclePartition partition,
 LogMinerEventRow row)
                   throws SQLException,
InterruptedException

Description copied from class: AbstractLogMinerEventProcessor

Processes a single LogMinerEventRow.

Overrides:

processRow in class AbstractLogMinerEventProcessor<InfinispanTransaction>

row - the event row, must not be null

Throws:

SQLException - if a database exception occurred

InterruptedException - if the dispatcher was interrupted sending an event

hasSchemaChangeBeenSeen

protected boolean hasSchemaChangeBeenSeen(LogMinerEventRow row)

Description copied from class: AbstractLogMinerEventProcessor

Checks whether the LogMinerEvent row for a schema change can be emitted.

Overrides:

hasSchemaChangeBeenSeen in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

row - the result set row

Returns:

true if the schema change has been seen, false otherwise.

getAndRemoveTransactionFromCache

protected InfinispanTransaction getAndRemoveTransactionFromCache(String transactionId)

Description copied from class: AbstractLogMinerEventProcessor

Gets a transaction instance from the transaction cache while also removing its cache entry.

Specified by:

getAndRemoveTransactionFromCache in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

transactionId - the transaction's unique identifier, should not be null

Returns:

the transaction instance if found, null if the transaction wasn't found

cleanupAfterTransactionRemovedFromCache

protected void cleanupAfterTransactionRemovedFromCache(InfinispanTransaction transaction,
 boolean isAbandoned)

Description copied from class: AbstractLogMinerEventProcessor

Removes the items associated with the transaction (e.g. events if they are stored independently).

Overrides:

cleanupAfterTransactionRemovedFromCache in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

transaction - the transaction instance, should never be null

isAbandoned - whether the removal is because transaction is being abandoned

getTransactionEventIterator

protected Iterator<LogMinerEvent> getTransactionEventIterator(InfinispanTransaction transaction)

Description copied from class: AbstractLogMinerEventProcessor

Get an iterator over the events that are part of the specified transaction.

Specified by:

getTransactionEventIterator in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

transaction - the transaction instance, should never be null

Returns:

an iterator over the transaction's events, never null

finalizeTransactionCommit

protected void finalizeTransactionCommit(String transactionId,
 Scn commitScn)

Description copied from class: AbstractLogMinerEventProcessor

Finalizes the commit of a transaction.

Specified by:

finalizeTransactionCommit in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

transactionId - the transaction's unique identifier, should not be null

commitScn - the transaction's system change number, should not be null

finalizeTransactionRollback

protected void finalizeTransactionRollback(String transactionId,
 Scn rollbackScn)

Description copied from class: AbstractLogMinerEventProcessor

Finalizes the rollback the specified transaction

Specified by:

finalizeTransactionRollback in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

transactionId - the unique transaction identifier, never null

rollbackScn - the rollback transaction's system change number, never null

resetTransactionToStart

protected void resetTransactionToStart(InfinispanTransaction transaction)

Overrides:

resetTransactionToStart in class AbstractLogMinerEventProcessor<InfinispanTransaction>

handleSchemaChange

protected void handleSchemaChange(LogMinerEventRow row)
                           throws InterruptedException

Description copied from class: AbstractLogMinerEventProcessor

Handle processing a LogMinerEventRow for a DDL event.

Overrides:

handleSchemaChange in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

row - the result set row

Throws:

InterruptedException - if the event dispatcher is interrupted sending the event

addToTransaction

protected void addToTransaction(String transactionId,
 LogMinerEventRow row,
 Supplier<LogMinerEvent> eventSupplier)

Description copied from class: AbstractLogMinerEventProcessor

Add a transaction to the transaction map if allowed.

Specified by:

addToTransaction in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

transactionId - the unqiue transaction id

row - the LogMiner event row

eventSupplier - the supplier of the event to create if the event is allowed to be added

getTransactionEventCount

protected int getTransactionEventCount(InfinispanTransaction transaction)

Description copied from class: AbstractLogMinerEventProcessor

Returns the number of events associated with the specified transaction.

Specified by:

getTransactionEventCount in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

transaction - the transaction, must not be null

Returns:

the number of events in the transaction

createQueryStatement

protected PreparedStatement createQueryStatement()
                                          throws SQLException

Description copied from class: AbstractLogMinerEventProcessor

Create the JDBC query that will be used to fetch the mining result set.

Specified by:

createQueryStatement in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Returns:

a prepared query statement, never null

Throws:

SQLException - if a database exception occurred creating the statement

calculateNewStartScn

protected Scn calculateNewStartScn(Scn endScn,
 Scn maxCommittedScn)
                            throws InterruptedException

Description copied from class: AbstractLogMinerEventProcessor

Calculates the new starting system change number based on the current processing range.

Specified by:

calculateNewStartScn in class AbstractLogMinerEventProcessor<InfinispanTransaction>

Parameters:

endScn - the end system change number for the previously mined range, never null

maxCommittedScn - the maximum committed system change number, never null

Returns:

the system change number to start then next mining iteration from, never null

Throws:

InterruptedException - if the current thread is interrupted

purgeCache

protected abstract void purgeCache(Scn minCacheScn)

Purge the necessary caches with all entries that occurred prior to the specified change number. NOTE: This method is abstract despite the code used by both all implementations being identical. This is because the method needed entrySet() is made available on two different concrete interfaces between the embedded and remote cache implementations, and therefore we need to access this method from the concrete implementation classes (RemoteCache and Cache) rather than from the common class used by CacheProvider (BasicCache).

Parameters:

minCacheScn - the minimum system change number to keep entries until

removeIf

protected <K,
V> void removeIf(org.infinispan.commons.util.CloseableIterator<Map.Entry<K,V>> iterator,
 Predicate<Map.Entry<K,V>> filter)

Helper method to remove entries that match the given predicate from a closeable iterator. This method guarantees that the underlying resources are released at the end of the operation.

Type Parameters:

K - the key type

V - the value type

Parameters:

iterator - the iterator

filter - the predicate

removeEventsWithTransaction

private void removeEventsWithTransaction(InfinispanTransaction transaction)