Package io.debezium.connector.mongodb

Class SourceInfo

  • @NotThreadSafe
public final class SourceInfo
extends BaseSourceInfo
    Information about the source of information, which includes the partitions and offsets within those partitions. The MongoDB connector considers each MongoDB database a separate "partitions" (in Kafka Connect terminology), and each partition has an "offset" (if Kafka Connect terminology) that defines the position within that partition/database where the connector finds a particular record. As the connector uses the Kafka Connect framework to process each record in a partition, Kafka Connect keeps track of the most recent offset for that partition.

    The source partition information identifies the particular MongoDB replica set and the connector's logical name of the MongoDB server. A JSON-like representation of the source partition for a database named "customers" hosted in a MongoDB replica set named "myMongoServer" is as follows: 

     {
     "server_id" : "myMongoServer",
     "replicaSetName" : "rs0"
 }

    The source offset information describes the position within a particular partition of each record. Since each event in MongoDB's oplog is identified by a BSONTimestamp that tracks the time and the order of the event for that particular time (e.g., multiple events that occur at the same time will have unique orders), the offset includes the BSONTimetamp representation. (The event's h field is the unique ID for the operation, so this is also included in the offset.) And, if an initial sync is in progress, the offset will include the initsync field.

    Here's a JSON-like representation of an example timestamp: 

     {
         "sec" = 1422998530,
         "ord" = 0,
         "h" = 398278732173914406,
         "initsync" = true
 }
    Author:
    Randall Hauch

    • Method Detail

      • replicaSetNameForPartition

        public static String replicaSetNameForPartition​(Map<String,​?> partition)
        Get the replica set name for the given partition.
        Parameters:
        partition - the partition map
        Returns:
        the replica set name (when the partition is valid), or null if the partition is null or has no replica set name entry

      • serverId

        public String serverId()
        Returns:
        server id

      • partition

        public Map<String,​String> partition​(String replicaSetName)
        Get the Kafka Connect detail about the source "partition" for the given database in the replica set. If the database is not known, this method records the new partition.
        Parameters:
        replicaSetName - the name of the replica set name for which the partition is to be obtained; may not be null
        Returns:
        the source partition information; never null

      • lastOffsetTimestamp

        public org.bson.BsonTimestamp lastOffsetTimestamp​(String replicaSetName)
        Get the MongoDB timestamp of the last offset position for the replica set.
        Parameters:
        replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
        Returns:
        the timestamp of the last offset, or the beginning of time if there is none

      • lastOffsetTxOrder

        public OptionalLong lastOffsetTxOrder​(String replicaSetName)
        Get the MongoDB transaction order of the last offset position for the replica set.
        Parameters:
        replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
        Returns:
        the tx order of the transaction in progress or 0 in case of non-transactional event

      • lastOffset

        public Map<String,​?> lastOffset​(String replicaSetName)
        Get the Kafka Connect detail about the source "offset" for the named database, which describes the given position in the database where we have last read. If the database has not yet been seen, this records the starting position for that database. However, if there is a position for the database, the offset representation is returned.
        Parameters:
        replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
        Returns:
        a copy of the current offset for the database; never null

      • collectionEvent

        public void collectionEvent​(String replicaSetName,
                            CollectionId collectionId)
        Get a Struct representation of the source partition and offset information where we have last read. The Struct complies with the AbstractSourceInfo.schema() for the MongoDB connector.
        Parameters:
        replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
        collectionId - the event's collection identifier; may not be null
        See Also:
        AbstractSourceInfo.schema()

      • opLogEvent

        public void opLogEvent​(String replicaSetName,
                       org.bson.Document oplogEvent,
                       org.bson.Document masterEvent,
                       long orderInTx)
        Get a Struct representation of the source partition and offset information. The Struct complies with the AbstractSourceInfo.schema() for the MongoDB connector.
        Parameters:
        replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
        oplogEvent - the replica set oplog event that was last read; may be null if the position is the start of the oplog
        masterEvent - the replica set oplog event that contains event metadata; same as oplogEvent for non-transactional changes
        orderInTx - order in transaction batch, 0 for non-transactional events
        See Also:
        AbstractSourceInfo.schema()

      • opLogEvent

        public void opLogEvent​(String replicaSetName,
                       org.bson.Document oplogEvent)
        Get a Struct representation of the source partition and offset information. The Struct complies with the AbstractSourceInfo.schema() for the MongoDB connector.
        Parameters:
        replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
        oplogEvent - the replica set oplog event that was last read; may be null if the position is the start of the oplog
        See Also:
        AbstractSourceInfo.schema()

      • extractEventTimestamp

        protected static org.bson.BsonTimestamp extractEventTimestamp​(org.bson.Document oplogEvent)
        Utility to extract the timestamp value from the event.
        Parameters:
        oplogEvent - the event
        Returns:
        the timestamp, or null if the event is null or there is no ts field

      • extractSessionTxnId

        protected static String extractSessionTxnId​(org.bson.Document oplogEvent)
        Utility to extract the unique transaction id value from the event.
        Parameters:
        oplogEvent - the event
        Returns:
        the session transaction id or null

      • hasOffset

        public boolean hasOffset​(String replicaSetName)
        Determine whether we have previously recorded a MongoDB timestamp for the replica set.
        Parameters:
        replicaSetName - the name of the replica set name; may not be null
        Returns:
        true if an offset has been recorded for the replica set, or false if the replica set has not yet been seen

      • setOffsetFor

        public boolean setOffsetFor​(String replicaSetName,
                            Map<String,​?> sourceOffset)
        Set the source offset, as read from Kafka Connect, for the given replica set. This method does nothing if the supplied map is null.
        Parameters:
        replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
        sourceOffset - the previously-recorded Kafka Connect source offset; may be null
        Returns:
        true if the offset was recorded, or false if the source offset is null
        Throws:
        org.apache.kafka.connect.errors.ConnectException - if any offset parameter values are missing, invalid, or of the wrong type

      • setOffsetFor

        public boolean setOffsetFor​(Map<String,​String> partition,
                            Map<String,​?> sourceOffset)
        Set the source offset, as read from Kafka Connect, for the given replica set. This method does nothing if the supplied map is null.
        Parameters:
        partition - the partition information; may not be null
        sourceOffset - the previously-recorded Kafka Connect source offset; may be null
        Returns:
        true if the offset was recorded, or false if the source offset is null
        Throws:
        org.apache.kafka.connect.errors.ConnectException - if any offset parameter values are missing, invalid, or of the wrong type

      • startInitialSync

        public void startInitialSync​(String replicaSetName)
        Record that an initial sync has started for the given replica set.
        Parameters:
        replicaSetName - the name of the replica set; never null

      • stopInitialSync

        public void stopInitialSync​(String replicaSetName)
        Record that an initial sync has stopped for the given replica set.
        Parameters:
        replicaSetName - the name of the replica set; never null

      • isInitialSyncOngoing

        public boolean isInitialSyncOngoing​(String replicaSetName)
        Determine if the initial sync for the given replica set is still ongoing.
        Parameters:
        replicaSetName - the name of the replica set; never null
        Returns:
        true if the initial sync for this replica is still ongoing or was not completed before restarting, or false if there is currently no initial sync operation for this replica set

      • isSnapshotRunning

        public boolean isSnapshotRunning()
        Returns whether any replica sets are still running a snapshot.

      • intOffsetValue

        private static int intOffsetValue​(Map<String,​?> values,
                                  String key)

      • longOffsetValue

        private static long longOffsetValue​(Map<String,​?> values,
                                    String key)

      • booleanOffsetValue

        private static boolean booleanOffsetValue​(Map<String,​?> values,
                                          String key)

      • replicaSetName

        String replicaSetName()

      • transactionPosition

        protected OptionalLong transactionPosition()