io.debezium.connector.oracle.logminer

Class LogMinerHelper

java.lang.Object

io.debezium.connector.oracle.logminer.LogMinerHelper

public class LogMinerHelper
extends Object

This class contains methods to configure and manage LogMiner utility

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LogMinerHelper.DATATYPE

Field Summary

Fields

Modifier and Type	Field and Description
private static String	ALL_COLUMN_LOGGING
private static String	CURRENT
private static org.slf4j.Logger	LOGGER
private static Map<String,OracleConnection>	racFlushConnections
private static String	TOTAL
private static String	UNKNOWN

Constructor Summary

Constructors

Constructor and Description
LogMinerHelper()

Method Summary

Modifier and Type	Method and Description
(package private) static void	buildDataDictionary(OracleConnection connection) This builds data dictionary objects in redo log files.
(package private) static void	checkSupplementalLogging(OracleConnection connection, String pdbName, OracleDatabaseSchema schema) This method validates the supplemental logging configuration for the source database.
private static OracleConnection	createFlushConnection(JdbcConfiguration config, String host)
(package private) static void	createFlushTable(OracleConnection connection)
static void	endMining(OracleConnection connection) This call completes LogMiner session.
private static void	executeCallableStatement(OracleConnection connection, String statement)
(package private) static void	flushLogWriter(OracleConnection connection, JdbcConfiguration config, boolean isRac, Set<String> racHosts) It is critical to flush LogWriter(s) buffer
private static void	flushRacLogWriters(Scn currentScn, JdbcConfiguration config, Set<String> racHosts) Oracle RAC has one LogWriter per node (instance), we have to flush them all We cannot use a query like from gv_instance view to get all the nodes, because not all nodes could be load balanced.
static List<LogFile>	getArchivedLogFilesForOffsetScn(OracleConnection connection, Scn offsetScn, Duration archiveLogRetention) This method returns all archived log files for one day, containing given offset scn
(package private) static Set<String>	getCurrentRedoLogFiles(OracleConnection connection, LogMinerMetrics metrics) This method query the database to get CURRENT online redo log file(s).
static Scn	getCurrentScn(OracleConnection connection) This method returns current SCN from the database
(package private) static Scn	getEndScn(OracleConnection connection, Scn startScn, LogMinerMetrics metrics, int defaultBatchSize) This method returns next SCN for mining and also updates MBean metrics We use a configurable limit, because the larger mining range, the slower query from LogMiner content view.
(package private) static Scn	getFirstOnlineLogScn(OracleConnection connection, Duration archiveLogRetention) This method fetches the oldest SCN from online redo log files
static Optional<Long>	getLastScnToAbandon(OracleConnection connection, Long offsetScn, Duration transactionRetention) This method calculates SCN as a watermark to abandon long lasting transactions.
static Map<String,String>	getMap(OracleConnection connection, String query, String nullReplacement)
static List<LogFile>	getOnlineLogFilesForOffsetScn(OracleConnection connection, Scn offsetScn) This method returns all online log files, starting from one which contains offset SCN and ending with one containing largest SCN 18446744073709551615 on Ora 19c is the max value of the nextScn in the current redo
private static Map<String,String>	getRedoLogStatus(OracleConnection connection) This fetches online redo log statuses
private static Scn	getScnFromString(String value)
static Object	getSingleResult(OracleConnection connection, String query, LogMinerHelper.DATATYPE type)
private static int	getSwitchCount(OracleConnection connection) This fetches REDO LOG switch count for the last day
(package private) static Duration	getTimeDifference(OracleConnection connection) Calculate time difference between database and connector timers.
(package private) static void	instantiateFlushConnections(JdbcConfiguration config, Set<String> hosts)
(package private) static boolean	isTableSupplementalLogDataAll(OracleConnection connection, TableId tableId)
private static void	logDatabaseState(OracleConnection connection) Helper method that will dump the state of various critical tables used by the LogMiner implementation to derive state about which logs are to be mined and processed by the Oracle LogMiner session.
(package private) static void	logError(TransactionalBufferMetrics metrics, String format, Object... args)
private static void	logQueryResults(OracleConnection connection, String query) Helper method that dumps the result set of an arbitrary SQL query to the connector's logs.
(package private) static void	logWarn(TransactionalBufferMetrics metrics, String format, Object... args)
static void	removeLogFilesFromMining(OracleConnection conn) This method removes all added log files from mining
(package private) static void	setNlsSessionParameters(JdbcConnection connection) Sets NLS parameters for mining session.
static void	setRedoLogFilesForMining(OracleConnection connection, Scn lastProcessedScn, Duration archiveLogRetention) This method substitutes CONTINUOUS_MINE functionality
(package private) static void	startLogMining(OracleConnection connection, Scn startScn, Scn endScn, OracleConnectorConfig.LogMiningStrategy strategy, boolean isContinuousMining, LogMinerMetrics metrics) This method builds mining view to query changes from.
private static void	updateRedoLogMetrics(OracleConnection connection, LogMinerMetrics metrics, Set<String> fileNames) This is to update MBean metrics associated with REDO LOG groups

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CURRENT

private static final String CURRENT

See Also:

Constant Field Values

UNKNOWN

private static final String UNKNOWN

See Also:

Constant Field Values

TOTAL

private static final String TOTAL

See Also:

Constant Field Values

ALL_COLUMN_LOGGING

private static final String ALL_COLUMN_LOGGING

See Also:

Constant Field Values

LOGGER

private static final org.slf4j.Logger LOGGER

racFlushConnections

private static Map<String,OracleConnection> racFlushConnections

Constructor Detail

LogMinerHelper

public LogMinerHelper()

Method Detail

instantiateFlushConnections

static void instantiateFlushConnections(JdbcConfiguration config,
                                        Set<String> hosts)

buildDataDictionary

static void buildDataDictionary(OracleConnection connection)
                         throws SQLException

This builds data dictionary objects in redo log files. During this build, Oracle does an additional REDO LOG switch. This call may take time, which leads to delay in delivering incremental changes. With this option the lag between source database and dispatching event fluctuates.

Parameters:

connection - connection to the database as LogMiner user (connection to the container)

Throws:

SQLException - any exception

getCurrentScn

public static Scn getCurrentScn(OracleConnection connection)
                         throws SQLException

This method returns current SCN from the database

Parameters:

connection - container level database connection

Returns:

current SCN

Throws:

SQLException - if anything unexpected happens

createFlushTable

static void createFlushTable(OracleConnection connection)
                      throws SQLException

Throws:

SQLException

getEndScn

static Scn getEndScn(OracleConnection connection,
                     Scn startScn,
                     LogMinerMetrics metrics,
                     int defaultBatchSize)
              throws SQLException

This method returns next SCN for mining and also updates MBean metrics We use a configurable limit, because the larger mining range, the slower query from LogMiner content view. In addition capturing unlimited number of changes can blow up Java heap. Gradual querying helps to catch up faster after long delays in mining.

Parameters:

connection - container level database connection

startScn - start SCN

metrics - MBean accessible metrics

Returns:

next SCN to mine up to

Throws:

SQLException - if anything unexpected happens

flushLogWriter

static void flushLogWriter(OracleConnection connection,
                           JdbcConfiguration config,
                           boolean isRac,
                           Set<String> racHosts)
                    throws SQLException

It is critical to flush LogWriter(s) buffer

Parameters:

connection - container level database connection

config - configuration

isRac - true if this is the RAC system

racHosts - set of RAC host

Throws:

SQLException - exception

getTimeDifference

static Duration getTimeDifference(OracleConnection connection)
                           throws SQLException

Calculate time difference between database and connector timers. It could be negative if DB time is ahead.

Parameters:

connection - connection

Returns:

the time difference as a Duration

Throws:

SQLException

startLogMining

static void startLogMining(OracleConnection connection,
                           Scn startScn,
                           Scn endScn,
                           OracleConnectorConfig.LogMiningStrategy strategy,
                           boolean isContinuousMining,
                           LogMinerMetrics metrics)
                    throws SQLException

This method builds mining view to query changes from. This view is built for online redo log files. It starts log mining session. It uses data dictionary objects, incorporated in previous steps. It tracks DDL changes and mines committed data only.

Parameters:

connection - container level database connection

startScn - the SCN to mine from

endScn - the SCN to mine to

strategy - this is about dictionary location

isContinuousMining - works < 19 version only

metrics - log miner metrics

Throws:

SQLException - if anything unexpected happens

getCurrentRedoLogFiles

static Set<String> getCurrentRedoLogFiles(OracleConnection connection,
                                          LogMinerMetrics metrics)
                                   throws SQLException

This method query the database to get CURRENT online redo log file(s). Multiple is applicable for RAC systems.

Parameters:

connection - connection to reuse

metrics - MBean accessible metrics

Returns:

full redo log file name(s), including path

Throws:

SQLException - if anything unexpected happens

getFirstOnlineLogScn

static Scn getFirstOnlineLogScn(OracleConnection connection,
                                Duration archiveLogRetention)
                         throws SQLException

This method fetches the oldest SCN from online redo log files

Parameters:

connection - container level database connection

archiveLogRetention - duration that archive logs are mined

Returns:

oldest SCN from online redo log

Throws:

SQLException - if anything unexpected happens

setNlsSessionParameters

static void setNlsSessionParameters(JdbcConnection connection)
                             throws SQLException

Sets NLS parameters for mining session.

Parameters:

connection - session level database connection

Throws:

SQLException - if anything unexpected happens

updateRedoLogMetrics

private static void updateRedoLogMetrics(OracleConnection connection,
                                         LogMinerMetrics metrics,
                                         Set<String> fileNames)

This is to update MBean metrics associated with REDO LOG groups

Parameters:

connection - connection

fileNames - name of current REDO LOG files

metrics - current metrics

getRedoLogStatus

private static Map<String,String> getRedoLogStatus(OracleConnection connection)
                                            throws SQLException

This fetches online redo log statuses

Parameters:

connection - privileged connection

Returns:

REDO LOG statuses Map, where key is the REDO name and value is the status

Throws:

SQLException - if anything unexpected happens

getSwitchCount

private static int getSwitchCount(OracleConnection connection)

This fetches REDO LOG switch count for the last day

Parameters:

connection - privileged connection

Returns:

counter

flushRacLogWriters

private static void flushRacLogWriters(Scn currentScn,
                                       JdbcConfiguration config,
                                       Set<String> racHosts)

Oracle RAC has one LogWriter per node (instance), we have to flush them all We cannot use a query like from gv_instance view to get all the nodes, because not all nodes could be load balanced. We also cannot rely on connection factory, because it may return connection to the same instance multiple times Instead we are asking node ip list from configuration

createFlushConnection

private static OracleConnection createFlushConnection(JdbcConfiguration config,
                                                      String host)
                                               throws SQLException

Throws:

SQLException

checkSupplementalLogging

static void checkSupplementalLogging(OracleConnection connection,
                                     String pdbName,
                                     OracleDatabaseSchema schema)
                              throws SQLException

This method validates the supplemental logging configuration for the source database.

Parameters:

connection - oracle connection on LogMiner level

pdbName - pdb name

schema - oracle schema

Throws:

SQLException - if anything unexpected happens

isTableSupplementalLogDataAll

static boolean isTableSupplementalLogDataAll(OracleConnection connection,
                                             TableId tableId)
                                      throws SQLException

Throws:

SQLException

endMining

public static void endMining(OracleConnection connection)

This call completes LogMiner session. Complete gracefully.

Parameters:

connection - container level database connection

setRedoLogFilesForMining

public static void setRedoLogFilesForMining(OracleConnection connection,
                                            Scn lastProcessedScn,
                                            Duration archiveLogRetention)
                                     throws SQLException

This method substitutes CONTINUOUS_MINE functionality

Parameters:

connection - connection

lastProcessedScn - current offset

archiveLogRetention - the duration that archive logs will be mined

Throws:

SQLException - if anything unexpected happens

getLastScnToAbandon

public static Optional<Long> getLastScnToAbandon(OracleConnection connection,
                                                 Long offsetScn,
                                                 Duration transactionRetention)

This method calculates SCN as a watermark to abandon long lasting transactions. The criteria is don't let offset scn go out of archives older given number of hours

Parameters:

connection - connection

offsetScn - current offset scn

transactionRetention - duration to tolerate long running transactions

Returns:

optional SCN as a watermark for abandonment

logWarn

static void logWarn(TransactionalBufferMetrics metrics,
                    String format,
                    Object... args)

logError

static void logError(TransactionalBufferMetrics metrics,
                     String format,
                     Object... args)

getOnlineLogFilesForOffsetScn

public static List<LogFile> getOnlineLogFilesForOffsetScn(OracleConnection connection,
                                                          Scn offsetScn)
                                                   throws SQLException

This method returns all online log files, starting from one which contains offset SCN and ending with one containing largest SCN 18446744073709551615 on Ora 19c is the max value of the nextScn in the current redo

Throws:

SQLException

getScnFromString

private static Scn getScnFromString(String value)

logDatabaseState

private static void logDatabaseState(OracleConnection connection)

Helper method that will dump the state of various critical tables used by the LogMiner implementation to derive state about which logs are to be mined and processed by the Oracle LogMiner session.

Parameters:

connection - the database connection

logQueryResults

private static void logQueryResults(OracleConnection connection,
                                    String query)
                             throws SQLException

Helper method that dumps the result set of an arbitrary SQL query to the connector's logs.

Parameters:

connection - the database connection

query - the query to execute

Throws:

SQLException - thrown if an exception occurs performing a SQL operation

getArchivedLogFilesForOffsetScn

public static List<LogFile> getArchivedLogFilesForOffsetScn(OracleConnection connection,
                                                            Scn offsetScn,
                                                            Duration archiveLogRetention)
                                                     throws SQLException

This method returns all archived log files for one day, containing given offset scn

Parameters:

connection - connection

offsetScn - offset scn

archiveLogRetention - duration that archive logs will be mined

Returns:

list of archive logs

Throws:

SQLException - if something happens

removeLogFilesFromMining

public static void removeLogFilesFromMining(OracleConnection conn)
                                     throws SQLException

This method removes all added log files from mining

Parameters:

conn - connection

Throws:

SQLException - something happened

executeCallableStatement

private static void executeCallableStatement(OracleConnection connection,
                                             String statement)
                                      throws SQLException

Throws:

SQLException

getMap

public static Map<String,String> getMap(OracleConnection connection,
                                        String query,
                                        String nullReplacement)
                                 throws SQLException

Throws:

SQLException

getSingleResult

public static Object getSingleResult(OracleConnection connection,
                                     String query,
                                     LogMinerHelper.DATATYPE type)
                              throws SQLException

Throws:

SQLException