Package io.debezium.connector.oracle.logminer.processor.memory

Class MemoryLogMinerEventProcessor

java.lang.Object

io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor<MemoryTransaction>

io.debezium.connector.oracle.logminer.processor.memory.MemoryLogMinerEventProcessor

All Implemented Interfaces:

LogMinerEventProcessor, AutoCloseable

public class MemoryLogMinerEventProcessor
extends AbstractLogMinerEventProcessor<MemoryTransaction>

A LogMinerEventProcessor that uses the JVM heap to store events as they're being processed and emitted from Oracle LogMiner.

Author:

Chris Cranford

Nested Class Summary

Nested classes/interfaces inherited from class io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor

AbstractLogMinerEventProcessor.Counters

Field Summary

Fields

Modifier and Type	Field	Description
private final Set<String>	abandonedTransactionsCache
private final EventDispatcher<OraclePartition,TableId>	dispatcher
private final OracleConnection	jdbcConnection
private static final org.slf4j.Logger	LOGGER
private final OracleStreamingChangeEventSourceMetrics	metrics
private final OracleOffsetContext	offsetContext
private final OraclePartition	partition
private final Map<String,Scn>	recentlyProcessedTransactionsCache	Cache of processed transactions (committed or rolled back), keyed based on the transaction's unique identifier.
private final Set<Scn>	schemaChangesCache
private final Map<String,MemoryTransaction>	transactionCache	Cache of transactions, keyed based on the transaction's unique identifier

Fields inherited from class io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor

counters, sqlQuery

Constructor Summary

Constructors

Constructor	Description
MemoryLogMinerEventProcessor(ChangeEventSource.ChangeEventSourceContext context, OracleConnectorConfig connectorConfig, OracleConnection jdbcConnection, EventDispatcher<OraclePartition,TableId> dispatcher, OraclePartition partition, OracleOffsetContext offsetContext, OracleDatabaseSchema schema, OracleStreamingChangeEventSourceMetrics metrics)

Method Summary

Modifier and Type	Method	Description
protected void	abandonTransactionOverEventThreshold(MemoryTransaction transaction)
void	abandonTransactions(Duration retention)	A callback for the event processor to abandon long running transactions.
protected void	addToTransaction(String transactionId, LogMinerEventRow row, Supplier<LogMinerEvent> eventSupplier)	Add a transaction to the transaction map if allowed.
protected Scn	calculateNewStartScn(Scn endScn, Scn maxCommittedScn)	Calculates the new starting system change number based on the current processing range.
void	close()
protected PreparedStatement	createQueryStatement()	Create the JDBC query that will be used to fetch the mining result set.
protected MemoryTransaction	createTransaction(LogMinerEventRow row)	Creates a new transaction based on the supplied START event.
protected void	finalizeTransactionCommit(String transactionId, Scn commitScn)	Finalizes the commit of a transaction.
protected void	finalizeTransactionRollback(String transactionId, Scn rollbackScn)	Finalizes the rollback the specified transaction
protected MemoryTransaction	getAndRemoveTransactionFromCache(String transactionId)	Gets a transaction instance from the transaction cache while also removing its cache entry.
protected Optional<Scn>	getLastScnToAbandon(OracleConnection connection, Duration retention)	Calculates the SCN as a watermark to abandon for long running transactions.
private Scn	getLastScnToAbandonFallbackByTransactionChangeTime(Duration retention)	Calculates the last system change number to abandon by directly examining the transaction buffer cache and comparing the transaction start time to the most recent last processed change time and comparing the difference to the configured transaction retention policy.
protected Map<String,MemoryTransaction>	getTransactionCache()	Returns the TransactionCache implementation.
protected Scn	getTransactionCacheMinimumScn()	Gets the minimum system change number stored in the transaction cache.
protected int	getTransactionEventCount(MemoryTransaction transaction)	Returns the number of events associated with the specified transaction.
protected Iterator<LogMinerEvent>	getTransactionEventIterator(MemoryTransaction transaction)	Get an iterator over the events that are part of the specified transaction.
protected void	handleCommitNotFoundInBuffer(LogMinerEventRow row)	Allow for post-processing of a transaction commit in the stream that was not found in the transaction buffer, perhaps because it aged out due to retention policies.
protected void	handleRollbackNotFoundInBuffer(LogMinerEventRow row)	Allow for post-processing of a transaction rollback in the stream that was not found in the transaction buffer, perhaps because it aged out due to retention policies.
protected void	handleSchemaChange(LogMinerEventRow row)	Handle processing a LogMinerEventRow for a DDL event.
protected boolean	hasSchemaChangeBeenSeen(LogMinerEventRow row)	Checks whether the LogMinerEvent row for a schema change can be emitted.
protected boolean	isRecentlyProcessed(String transactionId)	Check whether a transaction has been recently processed through either a commit or rollback.
protected void	removeEventWithRowId(LogMinerEventRow row)	Removes a specific transaction event by database row identifier.
protected void	removeTransactionAndEventsFromCache(MemoryTransaction transaction)	Removes the transaction and all its associated event entries from the connector's caches.

Methods inherited from class io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor

getConfig, getLastProcessedScn, getLastProcessedScnChangeTime, getQueryString, getSchema, getTransactionIdPrefix, handleCommit, handleDataEvent, handleLobWrite, handleMissingScn, handleRollback, handleSelectLobLocator, handleStart, handleUnsupportedEvent, isTransactionIdWithNoSequence, isTransactionOverEventThreshold, isTransactionUserExcluded, isTrxIdRawValue, process, processResults, processRow, resetTransactionToStart, warnPotentiallyStuckScn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

jdbcConnection

private final OracleConnection jdbcConnection

dispatcher

private final EventDispatcher<OraclePartition,TableId> dispatcher

partition

private final OraclePartition partition

offsetContext

private final OracleOffsetContext offsetContext

metrics

private final OracleStreamingChangeEventSourceMetrics metrics

transactionCache

private final Map<String,MemoryTransaction> transactionCache

Cache of transactions, keyed based on the transaction's unique identifier

recentlyProcessedTransactionsCache

private final Map<String,Scn> recentlyProcessedTransactionsCache

Cache of processed transactions (committed or rolled back), keyed based on the transaction's unique identifier.

schemaChangesCache

private final Set<Scn> schemaChangesCache

abandonedTransactionsCache

private final Set<String> abandonedTransactionsCache

Constructor Details

MemoryLogMinerEventProcessor

public MemoryLogMinerEventProcessor(ChangeEventSource.ChangeEventSourceContext context,
 OracleConnectorConfig connectorConfig,
 OracleConnection jdbcConnection,
 EventDispatcher<OraclePartition,TableId> dispatcher,
 OraclePartition partition,
 OracleOffsetContext offsetContext,
 OracleDatabaseSchema schema,
 OracleStreamingChangeEventSourceMetrics metrics)

Method Details

getTransactionCache

protected Map<String,MemoryTransaction> getTransactionCache()

Description copied from class: AbstractLogMinerEventProcessor

Returns the TransactionCache implementation.

Specified by:

getTransactionCache in class AbstractLogMinerEventProcessor<MemoryTransaction>

Returns:

the transaction cache, never null

createTransaction

protected MemoryTransaction createTransaction(LogMinerEventRow row)

Description copied from class: AbstractLogMinerEventProcessor

Creates a new transaction based on the supplied START event.

Specified by:

createTransaction in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

row - the event row, must not be null

Returns:

the implementation-specific Transaction instance

removeEventWithRowId

protected void removeEventWithRowId(LogMinerEventRow row)

Description copied from class: AbstractLogMinerEventProcessor

Removes a specific transaction event by database row identifier.

Specified by:

removeEventWithRowId in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

row - the event row that contains the row identifier, must not be null

close

public void close()
           throws Exception

Throws:

Exception

abandonTransactions

public void abandonTransactions(Duration retention)
                         throws InterruptedException

Description copied from interface: LogMinerEventProcessor

A callback for the event processor to abandon long running transactions.

Parameters:

retention - the maximum duration in which long running transactions are allowed.

Throws:

InterruptedException

isRecentlyProcessed

protected boolean isRecentlyProcessed(String transactionId)

Description copied from class: AbstractLogMinerEventProcessor

Check whether a transaction has been recently processed through either a commit or rollback.

Overrides:

isRecentlyProcessed in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

transactionId - the unique transaction id

Returns:

true if the transaction has been recently processed, false otherwise

hasSchemaChangeBeenSeen

protected boolean hasSchemaChangeBeenSeen(LogMinerEventRow row)

Description copied from class: AbstractLogMinerEventProcessor

Checks whether the LogMinerEvent row for a schema change can be emitted.

Overrides:

hasSchemaChangeBeenSeen in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

row - the result set row

Returns:

true if the schema change has been seen, false otherwise.

getAndRemoveTransactionFromCache

protected MemoryTransaction getAndRemoveTransactionFromCache(String transactionId)

Description copied from class: AbstractLogMinerEventProcessor

Gets a transaction instance from the transaction cache while also removing its cache entry.

Specified by:

getAndRemoveTransactionFromCache in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

transactionId - the transaction's unique identifier, should not be null

Returns:

the transaction instance if found, null if the transaction wasn't found

removeTransactionAndEventsFromCache

protected void removeTransactionAndEventsFromCache(MemoryTransaction transaction)

Description copied from class: AbstractLogMinerEventProcessor

Removes the transaction and all its associated event entries from the connector's caches.

Specified by:

removeTransactionAndEventsFromCache in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

transaction - the transaction instance, should never be null

getTransactionEventIterator

protected Iterator<LogMinerEvent> getTransactionEventIterator(MemoryTransaction transaction)

Description copied from class: AbstractLogMinerEventProcessor

Get an iterator over the events that are part of the specified transaction.

Specified by:

getTransactionEventIterator in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

transaction - the transaction instance, should never be null

Returns:

an iterator over the transaction's events, never null

finalizeTransactionCommit

protected void finalizeTransactionCommit(String transactionId,
 Scn commitScn)

Description copied from class: AbstractLogMinerEventProcessor

Finalizes the commit of a transaction.

Specified by:

finalizeTransactionCommit in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

transactionId - the transaction's unique identifier, should not be null

commitScn - the transaction's system change number, should not be null

finalizeTransactionRollback

protected void finalizeTransactionRollback(String transactionId,
 Scn rollbackScn)

Description copied from class: AbstractLogMinerEventProcessor

Finalizes the rollback the specified transaction

Specified by:

finalizeTransactionRollback in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

transactionId - the unique transaction identifier, never null

rollbackScn - the rollback transaction's system change number, never null

handleSchemaChange

protected void handleSchemaChange(LogMinerEventRow row)
                           throws InterruptedException

Description copied from class: AbstractLogMinerEventProcessor

Handle processing a LogMinerEventRow for a DDL event.

Overrides:

handleSchemaChange in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

row - the result set row

Throws:

InterruptedException - if the event dispatcher is interrupted sending the event

handleCommitNotFoundInBuffer

protected void handleCommitNotFoundInBuffer(LogMinerEventRow row)

Description copied from class: AbstractLogMinerEventProcessor

Allow for post-processing of a transaction commit in the stream that was not found in the transaction buffer, perhaps because it aged out due to retention policies.

Overrides:

handleCommitNotFoundInBuffer in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

row - the result set row

handleRollbackNotFoundInBuffer

protected void handleRollbackNotFoundInBuffer(LogMinerEventRow row)

Description copied from class: AbstractLogMinerEventProcessor

Allow for post-processing of a transaction rollback in the stream that was not found in the transaction buffer, perhaps because it aged out due to retention policies.

Overrides:

handleRollbackNotFoundInBuffer in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

row - the result set row

addToTransaction

protected void addToTransaction(String transactionId,
 LogMinerEventRow row,
 Supplier<LogMinerEvent> eventSupplier)

Description copied from class: AbstractLogMinerEventProcessor

Add a transaction to the transaction map if allowed.

Specified by:

addToTransaction in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

transactionId - the unqiue transaction id

row - the LogMiner event row

eventSupplier - the supplier of the event to create if the event is allowed to be added

getTransactionEventCount

protected int getTransactionEventCount(MemoryTransaction transaction)

Description copied from class: AbstractLogMinerEventProcessor

Returns the number of events associated with the specified transaction.

Specified by:

getTransactionEventCount in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

transaction - the transaction, must not be null

Returns:

the number of events in the transaction

createQueryStatement

protected PreparedStatement createQueryStatement()
                                          throws SQLException

Description copied from class: AbstractLogMinerEventProcessor

Create the JDBC query that will be used to fetch the mining result set.

Specified by:

createQueryStatement in class AbstractLogMinerEventProcessor<MemoryTransaction>

Returns:

a prepared query statement, never null

Throws:

SQLException - if a database exception occurred creating the statement

calculateNewStartScn

protected Scn calculateNewStartScn(Scn endScn,
 Scn maxCommittedScn)
                            throws InterruptedException

Description copied from class: AbstractLogMinerEventProcessor

Calculates the new starting system change number based on the current processing range.

Specified by:

calculateNewStartScn in class AbstractLogMinerEventProcessor<MemoryTransaction>

Parameters:

endScn - the end system change number for the previously mined range, never null

maxCommittedScn - the maximum committed system change number, never null

Returns:

the system change number to start then next mining iteration from, never null

Throws:

InterruptedException - if the current thread is interrupted

getLastScnToAbandon

protected Optional<Scn> getLastScnToAbandon(OracleConnection connection,
 Duration retention)

Calculates the SCN as a watermark to abandon for long running transactions. The criteria is do not let the offset SCN expire from archives older the specified retention hours.

Parameters:

connection - database connection, should not be null

retention - duration to tolerate long running transactions before being abandoned, must not be null

Returns:

an optional system change number as the watermark for transaction buffer abandonment

abandonTransactionOverEventThreshold

protected void abandonTransactionOverEventThreshold(MemoryTransaction transaction)

Overrides:

abandonTransactionOverEventThreshold in class AbstractLogMinerEventProcessor<MemoryTransaction>

getTransactionCacheMinimumScn

protected Scn getTransactionCacheMinimumScn()

Description copied from class: AbstractLogMinerEventProcessor

Gets the minimum system change number stored in the transaction cache.

Specified by:

getTransactionCacheMinimumScn in class AbstractLogMinerEventProcessor<MemoryTransaction>

Returns:

the minimum system change number, never null but could be Scn.NULL.

getLastScnToAbandonFallbackByTransactionChangeTime

private Scn getLastScnToAbandonFallbackByTransactionChangeTime(Duration retention)

Calculates the last system change number to abandon by directly examining the transaction buffer cache and comparing the transaction start time to the most recent last processed change time and comparing the difference to the configured transaction retention policy.

Parameters:

retention - duration to tolerate long-running transactions before being abandoned, must not be null

Returns:

the system change number to consider for transaction abandonment, never null