io.debezium.connector.oracle.logminer

Class LogMinerHelper

java.lang.Object

io.debezium.connector.oracle.logminer.LogMinerHelper

public class LogMinerHelper
extends Object

This class contains methods to configure and manage Log Miner utility

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LogMinerHelper.DATATYPE

Field Summary

Fields

Modifier and Type	Field and Description
private static org.slf4j.Logger	LOGGER
static BigInteger	MAX_SCN_BI
static String	MAX_SCN_S
private static Map<String,OracleConnection>	racFlushConnections
private static String	TOTAL
private static String	UNKNOWN

Constructor Summary

Constructors

Constructor and Description
LogMinerHelper()

Method Summary

Modifier and Type	Method and Description
(package private) static void	buildDataDictionary(Connection connection) This builds data dictionary objects in redo log files.
(package private) static void	checkSupplementalLogging(OracleConnection connection, String pdbName, OracleDatabaseSchema schema) This method validates the supplemental logging configuration for the source database.
private static OracleConnection	createFlushConnection(io.debezium.jdbc.JdbcConfiguration config, String host)
(package private) static void	createFlushTable(Connection connection)
(package private) static void	createLogMiningHistoryObjects(Connection connection, String historyTableName)
(package private) static void	deleteOutdatedHistory(Connection connection, long retention)
static void	endMining(Connection connection) This call completes log miner session.
private static void	executeCallableStatement(Connection connection, String statement)
(package private) static void	flushLogWriter(Connection connection, io.debezium.jdbc.JdbcConfiguration config, boolean isRac, Set<String> racHosts) It is critical to flush LogWriter(s) buffer
private static void	flushRacLogWriters(long currentScn, io.debezium.jdbc.JdbcConfiguration config, Set<String> racHosts) Oracle RAC has one LogWriter per node (instance), we have to flush them all We cannot use a query like from gv_instance view to get all the nodes, because not all nodes could be load balanced.
static Map<String,BigInteger>	getArchivedLogFilesForOffsetScn(Connection connection, Long offsetScn, Duration archiveLogRetention) This method returns all archived log files for one day, containing given offset scn
(package private) static Set<String>	getCurrentRedoLogFiles(Connection connection, LogMinerMetrics metrics) This method query the database to get CURRENT online redo log file(s).
static long	getCurrentScn(Connection connection) This method returns current SCN from the database
(package private) static long	getEndScn(Connection connection, long startScn, LogMinerMetrics metrics) This method returns next SCN for mining and also updates MBean metrics We use a configurable limit, because the larger mining range, the slower query from Log Miner content view.
(package private) static long	getFirstOnlineLogScn(Connection connection, Duration archiveLogRetention) This method fetches the oldest SCN from online redo log files
static Optional<Long>	getLastScnToAbandon(Connection connection, Long offsetScn, int hoursToKeepTransaction) This method calculates SCN as a watermark to abandon long lasting transactions.
static Map<String,String>	getMap(Connection connection, String query, String nullReplacement)
static Map<String,BigInteger>	getOnlineLogFilesForOffsetScn(Connection connection, Long offsetScn) This method returns all online log files, starting from one which contains offset SCN and ending with one containing largest SCN 18446744073709551615 on Ora 19c is the max value of the nextScn in the current redo todo replace all Long with BigInteger for SCN
private static int	getRedoLogGroupSize(Connection connection) get size of online REDO groups
private static Map<String,String>	getRedoLogStatus(Connection connection) This fetches online redo log statuses
static Object	getSingleResult(Connection connection, String query, LogMinerHelper.DATATYPE type)
private static int	getSwitchCount(Connection connection) This fetches REDO LOG switch count for the last day
(package private) static Duration	getTimeDifference(Connection connection) Calculate time difference between database and connector timers.
(package private) static void	instantiateFlushConnections(io.debezium.jdbc.JdbcConfiguration config, Set<String> hosts)
(package private) static void	logError(TransactionalBufferMetrics metrics, String format, Object... args)
(package private) static void	logWarn(TransactionalBufferMetrics metrics, String format, Object... args)
static void	removeLogFilesFromMining(Connection conn) This method removes all added log files from mining
(package private) static void	setNlsSessionParameters(io.debezium.jdbc.JdbcConnection connection) Sets NLS parameters for mining session.
static void	setRedoLogFilesForMining(Connection connection, Long lastProcessedScn, Duration archiveLogRetention) This method substitutes CONTINUOUS_MINE functionality
(package private) static void	startLogMining(Connection connection, Long startScn, Long endScn, OracleConnectorConfig.LogMiningStrategy strategy, boolean isContinuousMining) This method builds mining view to query changes from.
private static void	updateRedoLogMetrics(Connection connection, LogMinerMetrics metrics, Set<String> fileNames) This is to update MBean metrics associated with REDO LOG groups

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

UNKNOWN

private static final String UNKNOWN

See Also:

Constant Field Values

TOTAL

private static final String TOTAL

See Also:

Constant Field Values

LOGGER

private static final org.slf4j.Logger LOGGER

MAX_SCN_S

public static final String MAX_SCN_S

See Also:

Constant Field Values

MAX_SCN_BI

public static final BigInteger MAX_SCN_BI

racFlushConnections

private static Map<String,OracleConnection> racFlushConnections

Constructor Detail

LogMinerHelper

public LogMinerHelper()

Method Detail

instantiateFlushConnections

static void instantiateFlushConnections(io.debezium.jdbc.JdbcConfiguration config,
                                        Set<String> hosts)

buildDataDictionary

static void buildDataDictionary(Connection connection)
                         throws SQLException

This builds data dictionary objects in redo log files. During this build, Oracle does an additional REDO LOG switch. This call may take time, which leads to delay in delivering incremental changes. With this option the lag between source database and dispatching event fluctuates.

Parameters:

connection - connection to the database as log miner user (connection to the container)

Throws:

SQLException - any exception

getCurrentScn

public static long getCurrentScn(Connection connection)
                          throws SQLException

This method returns current SCN from the database

Parameters:

connection - container level database connection

Returns:

current SCN

Throws:

SQLException - if anything unexpected happens

createFlushTable

static void createFlushTable(Connection connection)
                      throws SQLException

Throws:

SQLException

createLogMiningHistoryObjects

static void createLogMiningHistoryObjects(Connection connection,
                                          String historyTableName)
                                   throws SQLException

Throws:

SQLException

deleteOutdatedHistory

static void deleteOutdatedHistory(Connection connection,
                                  long retention)
                           throws SQLException

Throws:

SQLException

getEndScn

static long getEndScn(Connection connection,
                      long startScn,
                      LogMinerMetrics metrics)
               throws SQLException

This method returns next SCN for mining and also updates MBean metrics We use a configurable limit, because the larger mining range, the slower query from Log Miner content view. In addition capturing unlimited number of changes can blow up Java heap. Gradual querying helps to catch up faster after long delays in mining.

Parameters:

connection - container level database connection

startScn - start SCN

metrics - MBean accessible metrics

Returns:

next SCN to mine up to

Throws:

SQLException - if anything unexpected happens

flushLogWriter

static void flushLogWriter(Connection connection,
                           io.debezium.jdbc.JdbcConfiguration config,
                           boolean isRac,
                           Set<String> racHosts)
                    throws SQLException

It is critical to flush LogWriter(s) buffer

Parameters:

connection - container level database connection

config - configuration

isRac - true if this is the RAC system

racHosts - set of RAC host

Throws:

SQLException - exception

getTimeDifference

static Duration getTimeDifference(Connection connection)
                           throws SQLException

Calculate time difference between database and connector timers. It could be negative if DB time is ahead.

Parameters:

connection - connection

Returns:

the time difference as a Duration

Throws:

SQLException

startLogMining

static void startLogMining(Connection connection,
                           Long startScn,
                           Long endScn,
                           OracleConnectorConfig.LogMiningStrategy strategy,
                           boolean isContinuousMining)
                    throws SQLException

This method builds mining view to query changes from. This view is built for online redo log files. It starts log mining session. It uses data dictionary objects, incorporated in previous steps. It tracks DDL changes and mines committed data only.

Parameters:

connection - container level database connection

startScn - the SCN to mine from

endScn - the SCN to mine to

strategy - this is about dictionary location

isContinuousMining - works < 19 version only

Throws:

SQLException - if anything unexpected happens

getCurrentRedoLogFiles

static Set<String> getCurrentRedoLogFiles(Connection connection,
                                          LogMinerMetrics metrics)
                                   throws SQLException

This method query the database to get CURRENT online redo log file(s). Multiple is applicable for RAC systems.

Parameters:

connection - connection to reuse

metrics - MBean accessible metrics

Returns:

full redo log file name(s), including path

Throws:

SQLException - if anything unexpected happens

getFirstOnlineLogScn

static long getFirstOnlineLogScn(Connection connection,
                                 Duration archiveLogRetention)
                          throws SQLException

This method fetches the oldest SCN from online redo log files

Parameters:

connection - container level database connection

archiveLogRetention - duration that archive logs are mined

Returns:

oldest SCN from online redo log

Throws:

SQLException - if anything unexpected happens

setNlsSessionParameters

static void setNlsSessionParameters(io.debezium.jdbc.JdbcConnection connection)
                             throws SQLException

Sets NLS parameters for mining session.

Parameters:

connection - session level database connection

Throws:

SQLException - if anything unexpected happens

updateRedoLogMetrics

private static void updateRedoLogMetrics(Connection connection,
                                         LogMinerMetrics metrics,
                                         Set<String> fileNames)

This is to update MBean metrics associated with REDO LOG groups

Parameters:

connection - connection

fileNames - name of current REDO LOG files

metrics - current metrics

getRedoLogStatus

private static Map<String,String> getRedoLogStatus(Connection connection)
                                            throws SQLException

This fetches online redo log statuses

Parameters:

connection - privileged connection

Returns:

REDO LOG statuses Map, where key is the REDO name and value is the status

Throws:

SQLException - if anything unexpected happens

getSwitchCount

private static int getSwitchCount(Connection connection)

This fetches REDO LOG switch count for the last day

Parameters:

connection - privileged connection

Returns:

counter

flushRacLogWriters

private static void flushRacLogWriters(long currentScn,
                                       io.debezium.jdbc.JdbcConfiguration config,
                                       Set<String> racHosts)

Oracle RAC has one LogWriter per node (instance), we have to flush them all We cannot use a query like from gv_instance view to get all the nodes, because not all nodes could be load balanced. We also cannot rely on connection factory, because it may return connection to the same instance multiple times Instead we are asking node ip list from configuration

createFlushConnection

private static OracleConnection createFlushConnection(io.debezium.jdbc.JdbcConfiguration config,
                                                      String host)
                                               throws SQLException

Throws:

SQLException

checkSupplementalLogging

static void checkSupplementalLogging(OracleConnection connection,
                                     String pdbName,
                                     OracleDatabaseSchema schema)
                              throws SQLException

This method validates the supplemental logging configuration for the source database.

Parameters:

connection - oracle connection on logminer level

pdbName - pdb name

schema - oracle schema

Throws:

SQLException - if anything unexpected happens

endMining

public static void endMining(Connection connection)

This call completes log miner session. Complete gracefully.

Parameters:

connection - container level database connection

setRedoLogFilesForMining

public static void setRedoLogFilesForMining(Connection connection,
                                            Long lastProcessedScn,
                                            Duration archiveLogRetention)
                                     throws SQLException

This method substitutes CONTINUOUS_MINE functionality

Parameters:

connection - connection

lastProcessedScn - current offset

archiveLogRetention - the duration that archive logs will be mined

Throws:

SQLException - if anything unexpected happens

getLastScnToAbandon

public static Optional<Long> getLastScnToAbandon(Connection connection,
                                                 Long offsetScn,
                                                 int hoursToKeepTransaction)

This method calculates SCN as a watermark to abandon long lasting transactions. The criteria is don't let offset scn go out of archives older given number of hours

Parameters:

connection - connection

offsetScn - current offset scn

hoursToKeepTransaction - hours to tolerate long transaction

Returns:

optional SCN as a watermark for abandonment

logWarn

static void logWarn(TransactionalBufferMetrics metrics,
                    String format,
                    Object... args)

logError

static void logError(TransactionalBufferMetrics metrics,
                     String format,
                     Object... args)

getRedoLogGroupSize

private static int getRedoLogGroupSize(Connection connection)
                                throws SQLException

get size of online REDO groups

Parameters:

connection - connection

Returns:

size

Throws:

SQLException

getOnlineLogFilesForOffsetScn

public static Map<String,BigInteger> getOnlineLogFilesForOffsetScn(Connection connection,
                                                                   Long offsetScn)
                                                            throws SQLException

This method returns all online log files, starting from one which contains offset SCN and ending with one containing largest SCN 18446744073709551615 on Ora 19c is the max value of the nextScn in the current redo todo replace all Long with BigInteger for SCN

Throws:

SQLException

getArchivedLogFilesForOffsetScn

public static Map<String,BigInteger> getArchivedLogFilesForOffsetScn(Connection connection,
                                                                     Long offsetScn,
                                                                     Duration archiveLogRetention)
                                                              throws SQLException

This method returns all archived log files for one day, containing given offset scn

Parameters:

connection - connection

offsetScn - offset scn

archiveLogRetention - duration that archive logs will be mined

Returns:

Map of archived files

Throws:

SQLException - if something happens

removeLogFilesFromMining

public static void removeLogFilesFromMining(Connection conn)
                                     throws SQLException

This method removes all added log files from mining

Parameters:

conn - connection

Throws:

SQLException - something happened

executeCallableStatement

private static void executeCallableStatement(Connection connection,
                                             String statement)
                                      throws SQLException

Throws:

SQLException

getMap

public static Map<String,String> getMap(Connection connection,
                                        String query,
                                        String nullReplacement)
                                 throws SQLException

Throws:

SQLException

getSingleResult

public static Object getSingleResult(Connection connection,
                                     String query,
                                     LogMinerHelper.DATATYPE type)
                              throws SQLException

Throws:

SQLException