Package io.debezium.connector.oracle.logminer

Class LogMinerHelper

  • public class LogMinerHelper
extends Object
    This class contains methods to configure and manage LogMiner utility

    • Constructor Detail

      • LogMinerHelper

        public LogMinerHelper()

    • Method Detail

      • buildDataDictionary

        static void buildDataDictionary​(OracleConnection connection)
                         throws SQLException
        This builds data dictionary objects in redo log files. During this build, Oracle does an additional REDO LOG switch. This call may take time, which leads to delay in delivering incremental changes. With this option the lag between source database and dispatching event fluctuates.
        Parameters:
        connection - connection to the database as LogMiner user (connection to the container)
        Throws:
        SQLException - any exception

      • getEndScn

        static Scn getEndScn​(OracleConnection connection,
                     Scn startScn,
                     Scn prevEndScn,
                     OracleStreamingChangeEventSourceMetrics streamingMetrics,
                     int defaultBatchSize,
                     boolean lobEnabled,
                     boolean archiveLogOnlyMode,
                     String archiveLogDestinationName)
              throws SQLException
        This method returns next SCN for mining and also updates streaming metrics. We use a configurable limit, because the larger mining range, the slower query from LogMiner content view. In addition capturing unlimited number of changes can blow up Java heap. Gradual querying helps to catch up faster after long delays in mining.
        Parameters:
        connection - container level database connection
        startScn - start SCN
        prevEndScn - previous end SCN
        streamingMetrics - the streaming metrics
        lobEnabled - specifies whether LOB support is enabled
        archiveLogOnlyMode - specifies whether archive log only mode is enabled
        archiveLogDestinationName - the archive log destination name
        Returns:
        next SCN to mine up to
        Throws:
        SQLException - if anything unexpected happens

      • flushLogWriter

        static void flushLogWriter​(OracleConnection connection,
                           JdbcConfiguration config,
                           boolean isRac,
                           Set<String> racHosts)
                    throws SQLException
        It is critical to flush LogWriter(s) buffer
        Parameters:
        connection - container level database connection
        config - configuration
        isRac - true if this is the RAC system
        racHosts - set of RAC host
        Throws:
        SQLException - exception

      • getSystime

        static OffsetDateTime getSystime​(OracleConnection connection)
                          throws SQLException
        Get the database time in the time zone of the system this database is running on
        Parameters:
        connection - connection
        Returns:
        the database system time
        Throws:
        SQLException

      • startLogMining

        static void startLogMining​(OracleConnection connection,
                           Scn startScn,
                           Scn endScn,
                           OracleConnectorConfig.LogMiningStrategy strategy,
                           boolean isContinuousMining,
                           OracleStreamingChangeEventSourceMetrics streamingMetrics)
                    throws SQLException
        This method builds mining view to query changes from. This view is built for online redo log files. It starts log mining session. It uses data dictionary objects, incorporated in previous steps. It tracks DDL changes and mines committed data only.
        Parameters:
        connection - container level database connection
        startScn - the SCN to mine from
        endScn - the SCN to mine to
        strategy - this is about dictionary location
        isContinuousMining - works < 19 version only
        streamingMetrics - the streaming metrics
        Throws:
        SQLException - if anything unexpected happens

      • getCurrentRedoLogFiles

        static Set<String> getCurrentRedoLogFiles​(OracleConnection connection)
                                   throws SQLException
        This method query the database to get CURRENT online redo log file(s). Multiple is applicable for RAC systems.
        Parameters:
        connection - connection to reuse
        Returns:
        full redo log file name(s), including path
        Throws:
        SQLException - if anything unexpected happens

      • getFirstOnlineLogScn

        static Scn getFirstOnlineLogScn​(OracleConnection connection,
                                Duration archiveLogRetention,
                                String archiveDestinationName)
                         throws SQLException
        This method fetches the oldest SCN from online redo log files
        Parameters:
        connection - container level database connection
        archiveLogRetention - duration that archive logs are mined
        archiveDestinationName - configured archive destination name to use, may be null
        Returns:
        oldest SCN from online redo log
        Throws:
        SQLException - if anything unexpected happens

      • setNlsSessionParameters

        static void setNlsSessionParameters​(JdbcConnection connection)
                             throws SQLException
        Sets NLS parameters for mining session.
        Parameters:
        connection - session level database connection
        Throws:
        SQLException - if anything unexpected happens

      • getRedoLogStatus

        private static Map<String,​String> getRedoLogStatus​(OracleConnection connection)
                                                  throws SQLException
        This fetches online redo log statuses
        Parameters:
        connection - privileged connection
        Returns:
        REDO LOG statuses Map, where key is the REDO name and value is the status
        Throws:
        SQLException - if anything unexpected happens

      • getSwitchCount

        private static int getSwitchCount​(OracleConnection connection,
                                  String archiveDestinationName)
        This fetches REDO LOG switch count for the last day
        Parameters:
        connection - privileged connection
        archiveDestinationName - configured archive destination name, may be null
        Returns:
        counter

      • flushRacLogWriters

        private static void flushRacLogWriters​(Scn currentScn,
                                       JdbcConfiguration config,
                                       Set<String> racHosts)
        Oracle RAC has one LogWriter per node (instance), we have to flush them all We cannot use a query like from gv_instance view to get all the nodes, because not all nodes could be load balanced. We also cannot rely on connection factory, because it may return connection to the same instance multiple times Instead we are asking node ip list from configuration

      • checkSupplementalLogging

        static void checkSupplementalLogging​(OracleConnection connection,
                                     String pdbName,
                                     OracleDatabaseSchema schema)
                              throws SQLException
        This method validates the supplemental logging configuration for the source database.
        Parameters:
        connection - oracle connection on LogMiner level
        pdbName - pdb name
        schema - oracle schema
        Throws:
        SQLException - if anything unexpected happens

      • endMining

        public static void endMining​(OracleConnection connection)
        This call completes LogMiner session. Complete gracefully.
        Parameters:
        connection - container level database connection

      • setLogFilesForMining

        public static void setLogFilesForMining​(OracleConnection connection,
                                        Scn lastProcessedScn,
                                        Duration archiveLogRetention,
                                        boolean archiveLogOnlyMode,
                                        String archiveDestinationName)
                                 throws SQLException
        This method substitutes CONTINUOUS_MINE functionality
        Parameters:
        connection - connection
        lastProcessedScn - current offset
        archiveLogRetention - the duration that archive logs will be mined
        archiveLogOnlyMode - true to mine only archive lgos, false to mine all available logs
        archiveDestinationName - configured archive log destination name to use, may be null
        Throws:
        SQLException - if anything unexpected happens

      • getLastScnToAbandon

        public static Optional<Scn> getLastScnToAbandon​(OracleConnection connection,
                                                Scn offsetScn,
                                                Duration transactionRetention)
        This method calculates SCN as a watermark to abandon long lasting transactions. The criteria is don't let offset scn go out of archives older given number of hours
        Parameters:
        connection - connection
        offsetScn - current offset scn
        transactionRetention - duration to tolerate long running transactions
        Returns:
        optional SCN as a watermark for abandonment

      • getLogFilesForOffsetScn

        public static List<LogFile> getLogFilesForOffsetScn​(OracleConnection connection,
                                                    Scn offsetScn,
                                                    Duration archiveLogRetention,
                                                    boolean archiveLogOnlyMode,
                                                    String archiveDestinationName)
                                             throws SQLException
        Get all log files that should be mined.
        Parameters:
        connection - database connection
        offsetScn - offset system change number
        archiveLogRetention - duration that archive logs should be mined
        archiveLogOnlyMode - true to mine only archive logs, false to mine all available logs
        archiveDestinationName - archive destination to use, may be null
        Returns:
        list of log files
        Throws:
        SQLException - if a database exception occurs

      • getScnFromString

        private static Scn getScnFromString​(String value)

      • logDatabaseState

        private static void logDatabaseState​(OracleConnection connection)
        Helper method that will dump the state of various critical tables used by the LogMiner implementation to derive state about which logs are to be mined and processed by the Oracle LogMiner session.
        Parameters:
        connection - the database connection

      • logQueryResults

        private static void logQueryResults​(OracleConnection connection,
                                    String query)
                             throws SQLException
        Helper method that dumps the result set of an arbitrary SQL query to the connector's logs.
        Parameters:
        connection - the database connection
        query - the query to execute
        Throws:
        SQLException - thrown if an exception occurs performing a SQL operation

      • removeLogFilesFromMining

        public static void removeLogFilesFromMining​(OracleConnection conn)
                                     throws SQLException
        This method removes all added log files from mining
        Parameters:
        conn - connection
        Throws:
        SQLException - something happened

      • getColumnIndexByName

        public static int getColumnIndexByName​(String columnName,
                                       Table table)
        Returns a 0-based index offset for the column name in the relational table.
        Parameters:
        columnName - the column name, should not be null.
        table - the relational table, should not be null.
        Returns:
        the 0-based index offset for the column name