Package io.debezium.connector.oracle.logminer

Class LogMinerStreamingChangeEventSource

java.lang.Object
io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource
All Implemented Interfaces:
ChangeEventSource, StreamingChangeEventSource<OraclePartition,OracleOffsetContext>
public class LogMinerStreamingChangeEventSource extends Object implements StreamingChangeEventSource<OraclePartition,OracleOffsetContext>
A StreamingChangeEventSource based on Oracle's LogMiner utility. The event handler loop is executed in a separate executor.

  • Field Details

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • MAXIMUM_NAME_LENGTH

      private static final int MAXIMUM_NAME_LENGTH
      See Also:

    • ALL_COLUMN_LOGGING

      private static final String ALL_COLUMN_LOGGING
      See Also:

    • MINING_START_RETRIES

      private static final int MINING_START_RETRIES
      See Also:

    • SMALL_REDO_LOG_WARNING

      private static final Long SMALL_REDO_LOG_WARNING

    • jdbcConnection

      private final OracleConnection jdbcConnection

    • dispatcher

      private final EventDispatcher<OraclePartition,TableId> dispatcher

    • clock

      private final Clock clock

    • schema

      private final OracleDatabaseSchema schema

    • jdbcConfiguration

      private final JdbcConfiguration jdbcConfiguration

    • strategy

      private final OracleConnectorConfig.LogMiningStrategy strategy

    • errorHandler

      private final ErrorHandler errorHandler

    • isContinuousMining

      private final boolean isContinuousMining

    • streamingMetrics

      private final LogMinerStreamingChangeEventSourceMetrics streamingMetrics

    • connectorConfig

      private final OracleConnectorConfig connectorConfig

    • archiveLogRetention

      private final Duration archiveLogRetention

    • archiveLogOnlyMode

      private final boolean archiveLogOnlyMode

    • archiveDestinationName

      private final String archiveDestinationName

    • logFileQueryMaxRetries

      private final int logFileQueryMaxRetries

    • initialDelay

      private final Duration initialDelay

    • maxDelay

      private final Duration maxDelay

    • startScn

      private Scn startScn

    • endScn

      private Scn endScn

    • snapshotScn

      private Scn snapshotScn

    • currentLogFiles

      private List<LogFile> currentLogFiles

    • currentRedoLogSequences

      private List<BigInteger> currentRedoLogSequences

    • effectiveOffset

      private OracleOffsetContext effectiveOffset

    • currentBatchSize

      private int currentBatchSize

    • currentSleepTime

      private long currentSleepTime

  • Constructor Details

  • Method Details

    • init

      public void init(OracleOffsetContext offsetContext) throws InterruptedException
      Specified by:
      init in interface StreamingChangeEventSource<OraclePartition,OracleOffsetContext>
      Throws:
      InterruptedException

    • emptyContext

      private OracleOffsetContext emptyContext()

    • execute

      public void execute(ChangeEventSource.ChangeEventSourceContext context, OraclePartition partition, OracleOffsetContext offsetContext)
      This is the loop to get changes from LogMiner
      Specified by:
      execute in interface StreamingChangeEventSource<OraclePartition,OracleOffsetContext>
      Parameters:
      context - change event source context

    • prepareConnection

      private void prepareConnection(boolean closeAndReconnect) throws SQLException
      Throws:
      SQLException

    • logOnlineRedoLogSizes

      private void logOnlineRedoLogSizes(OracleConnectorConfig config) throws SQLException
      Throws:
      SQLException

    • computeStartScnForFirstMiningSession

      private void computeStartScnForFirstMiningSession(OracleOffsetContext offsetContext, Scn firstScn)
      Computes the start SCN for the first mining session. Normally, this would be the snapshot SCN, but if there were pending transactions at the time the snapshot was taken, we'd miss the events in those transactions that have an SCN smaller than the snapshot SCN.
      Parameters:
      offsetContext - the offset context
      firstScn - the oldest SCN still available in the REDO logs

    • captureSessionMemoryStatistics

      private void captureSessionMemoryStatistics(OracleConnection connection) throws SQLException
      Throws:
      SQLException

    • createProcessor

      private LogMinerEventProcessor createProcessor(ChangeEventSource.ChangeEventSourceContext context, OraclePartition partition, OracleOffsetContext offsetContext)

    • getFirstScnInLogs

      private Scn getFirstScnInLogs(OracleConnection connection) throws SQLException
      Gets the first system change number in both archive and redo logs.
      Parameters:
      connection - database connection, should not be null
      Returns:
      the oldest system change number
      Throws:
      SQLException - if a database exception occurred
      io.debezium.DebeziumException - if the oldest system change number cannot be found due to no logs available

    • initializeRedoLogsForMining

      private void initializeRedoLogsForMining(OracleConnection connection, boolean postEndMiningSession, Scn startScn) throws SQLException
      Throws:
      SQLException

    • getCurrentLogFileSequences

      private List<BigInteger> getCurrentLogFileSequences(List<LogFile> logFiles)
      Get the current log file sequences from the supplied list of log files.
      Parameters:
      logFiles - list of log files
      Returns:
      list of sequences for the logs that are marked "current" in the database.

    • getMaxArchiveLogScn

      private Scn getMaxArchiveLogScn(List<LogFile> logFiles)
      Get the maximum archive log SCN
      Parameters:
      logFiles - the current logs that are part of the mining session
      Returns:
      the maximum system change number from the archive logs
      Throws:
      io.debezium.DebeziumException - if no logs are provided or if the provided logs has no archive log types

    • buildDataDictionary

      private void buildDataDictionary(OracleConnection connection) throws SQLException
      Requests Oracle to build the data dictionary. During the build step, Oracle will perform an additional series of redo log switches. Additionally, this call may introduce a delay in delivering incremental changes since the dictionary will need to have statistics gathered, analyzed, and prepared by LogMiner before any redo entries can be mined. This should only be used in conjunction with the mining strategy OracleConnectorConfig.LogMiningStrategy.CATALOG_IN_REDO.
      Parameters:
      connection - database connection
      Throws:
      SQLException - if a database exception occurred

    • hasLogSwitchOccurred

      private boolean hasLogSwitchOccurred() throws SQLException
      Checks whether a database log switch has occurred and updates metrics if so.
      Returns:
      true if a log switch was detected, otherwise false
      Throws:
      SQLException - if a database exception occurred

    • updateRedoLogMetrics

      private void updateRedoLogMetrics() throws SQLException
      Updates the redo log names and statues in the streaming metrics.
      Throws:
      SQLException - if a database exception occurred

    • getCurrentRedoLogFiles

      private Set<String> getCurrentRedoLogFiles(OracleConnection connection) throws SQLException
      Get a list of all the CURRENT redo log file names. For Oracle RAC clusters, multiple filenames will be returned, one for each node that participates in the cluster.
      Parameters:
      connection - database connection, should not be null
      Returns:
      unique set of all current redo log file names, with full paths, never null
      Throws:
      SQLException - if a database exception occurred

    • getCurrentRedoLogSequences

      private List<BigInteger> getCurrentRedoLogSequences() throws SQLException
      Get the current redo log sequence(s). In an Oracle RAC environment, there are multiple current redo logs and therefore this method returns multiple values, each relating to a single RAC node in the Oracle cluster.
      Returns:
      list of sequence numbers
      Throws:
      SQLException - if a database exception occurred

    • pauseBetweenMiningSessions

      private void pauseBetweenMiningSessions() throws InterruptedException
      Throws:
      InterruptedException

    • setNlsSessionParameters

      private void setNlsSessionParameters(OracleConnection connection) throws SQLException
      Sets the NLS parameters for the mining session.
      Parameters:
      connection - database connection, should not be null
      Throws:
      SQLException - if a database exception occurred

    • getDatabaseSystemTime

      private OffsetDateTime getDatabaseSystemTime(OracleConnection connection) throws SQLException
      Get the database system time in the database system's time zone.
      Parameters:
      connection - database connection, should not be null
      Returns:
      the database system time
      Throws:
      SQLException - if a database exception occurred

    • startMiningSession

      public boolean startMiningSession(OracleConnection connection, Scn startScn, Scn endScn, int attempts) throws SQLException
      Starts a new Oracle LogMiner session. When this is called, LogMiner prepares all the necessary state for an upcoming LogMiner view query. If the mining statement defines using DDL tracking, the data dictionary will be mined as a part of this call to prepare DDL tracking state for the upcoming LogMiner view query.
      Parameters:
      connection - database connection, should not be null
      startScn - mining session's starting system change number (exclusive), should not be null
      endScn - mining session's ending system change number (inclusive), can be null
      attempts - the number of mining start attempts
      Returns:
      true if the session was started successfully, false if it should be retried
      Throws:
      SQLException - if mining session failed to start

    • endMiningSession

      public void endMiningSession(OracleConnection connection, OracleOffsetContext offsetContext) throws SQLException
      End the current Oracle LogMiner session, if one is in progress. If the current session does not have an active mining session, a log message is recorded and the method is a no-op.
      Parameters:
      connection - database connection, should not be null
      offsetContext - connector offset context, should not be null
      Throws:
      SQLException - if the current mining session cannot be ended gracefully

    • updateBatchSize

      private void updateBatchSize(boolean increment)

    • updateSleepTime

      private void updateSleepTime(boolean increment)

    • calculateEndScn

      private Scn calculateEndScn(OracleConnection connection, Scn startScn, Scn prevEndScn) throws SQLException
      Calculates the mining session's end system change number. This calculation is based upon a sliding window algorithm to where if the connector is falling behind, the mining session's end point will be calculated based on the batch size and either be increased up to the maximum batch size or reduced to as low as the minimum batch size. Additionally, this method calculates and maintains a sliding algorithm for the sleep time between the mining sessions, increasing the pause up to the maximum sleep time if the connector is not behind or is mining too quick and reducing the pause down to the mimum sleep time if the connector has fallen behind and needs to catch-up faster.
      Parameters:
      connection - database connection, should not be null
      startScn - upcoming mining session's starting change number, should not be null
      prevEndScn - last mining session's ending system change number, can be null
      Returns:
      the ending system change number to be used for the upcoming mining session, never null
      Throws:
      SQLException - if the current max system change number cannot be obtained from the database

    • calculateDeviatedEndScn

      private Optional<Scn> calculateDeviatedEndScn(Scn lowerboundsScn, Scn upperboundsScn, Duration deviation)
      Calculates the deviated end scn based on the scn range and deviation.
      Parameters:
      lowerboundsScn - the mining range's lower bounds
      upperboundsScn - the mining range's upper bounds
      deviation - the time deviation
      Returns:
      an optional that contains the deviated scn or empty if the operation should be performed again

    • getDeviatedMaxScn

      private Optional<Scn> getDeviatedMaxScn(Scn upperboundsScn, Duration deviation)
      Uses the provided Upperbound SCN and deviation to calculate an SCN that happened in the past at a time based on Oracle's TIMESTAMP_TO_SCN and SCN_TO_TIMESTAMP functions.
      Parameters:
      upperboundsScn - the upper bound system change number, should not be null
      deviation - the time deviation to be applied, should not be null
      Returns:
      the newly calculated Scn

    • checkDatabaseAndTableState

      private void checkDatabaseAndTableState(OracleConnection connection, String pdbName, OracleDatabaseSchema schema) throws SQLException
      Checks and validates the database's supplemental logging configuration as well as the lengths of the table and column names that are part of the database schema.
      Parameters:
      connection - database connection, should not be null
      pdbName - pluggable database name, can be null when not using pluggable databases
      schema - connector's database schema, should not be null
      Throws:
      SQLException - if a database exception occurred

    • checkArchiveLogDestination

      private void checkArchiveLogDestination(OracleConnection connection, String destinationName) throws SQLException
      Throws:
      SQLException

    • checkTableColumnNameLengths

      private void checkTableColumnNameLengths(Table table)
      Examines the table and column names and logs a warning if any name exceeds MAXIMUM_NAME_LENGTH.
      Parameters:
      table - the table, should not be null

    • isDatabaseAllSupplementalLoggingEnabled

      private boolean isDatabaseAllSupplementalLoggingEnabled(OracleConnection connection) throws SQLException
      Returns whether the database is configured with ALL supplemental logging.
      Parameters:
      connection - database connection, must not be null
      Returns:
      true if all supplemental logging is enabled, false otherwise
      Throws:
      SQLException - if a database exception occurred

    • isDatabaseMinSupplementalLoggingEnabled

      private boolean isDatabaseMinSupplementalLoggingEnabled(OracleConnection connection) throws SQLException
      Returns whether the database is configured with MIN supplemental logging.
      Parameters:
      connection - database connection, must not be null
      Returns:
      true if min supplemental logging is enabled, false otherwise
      Throws:
      SQLException - if a database exception occurred

    • isTableAllColumnsSupplementalLoggingEnabled

      private boolean isTableAllColumnsSupplementalLoggingEnabled(OracleConnection connection, TableId tableId) throws SQLException
      Return whether the table is configured with ALL COLUMN supplemental logging.
      Parameters:
      connection - database connection, must not be null
      tableId - table identifier, must not be null
      Returns:
      true if all column supplemental logging is enabled, false otherwise
      Throws:
      SQLException - if a database exception occurred

    • resolveFlushStrategy

      private LogWriterFlushStrategy resolveFlushStrategy()
      Resolves the Oracle LGWR buffer flushing strategy.
      Returns:
      the strategy to be used to flush Oracle's LGWR process, never null.

    • waitForStartScnInArchiveLogs

      private boolean waitForStartScnInArchiveLogs(ChangeEventSource.ChangeEventSourceContext context, Scn startScn) throws SQLException, InterruptedException
      Waits for the starting system change number to exist in the archive logs before returning.
      Parameters:
      context - the change event source context
      startScn - the starting system change number
      Returns:
      true if the code should continue, false if the code should end.
      Throws:
      SQLException - if a database exception occurred
      InterruptedException - if the pause between checks is interrupted

    • isStartScnInArchiveLogs

      private boolean isStartScnInArchiveLogs(Scn startScn) throws SQLException
      Returns whether the starting system change number is in the archive logs.
      Parameters:
      startScn - the starting system change number
      Returns:
      true if the starting system change number is in the archive logs; false otherwise.
      Throws:
      SQLException - if a database exception occurred

    • commitOffset

      public void commitOffset(Map<String,?> partition, Map<String,?> offset)
      Specified by:
      commitOffset in interface StreamingChangeEventSource<OraclePartition,OracleOffsetContext>

    • getOffsetContext

      public OracleOffsetContext getOffsetContext()
      Specified by:
      getOffsetContext in interface StreamingChangeEventSource<OraclePartition,OracleOffsetContext>