Package io.debezium.connector.mongodb

Class SourceInfo

java.lang.Object
io.debezium.connector.AbstractSourceInfo
io.debezium.connector.common.BaseSourceInfo
io.debezium.connector.mongodb.SourceInfo
@NotThreadSafe public final class SourceInfo extends BaseSourceInfo
Information about the source of information, which includes the partitions and offsets within those partitions. The MongoDB connector considers each MongoDB database a separate "partitions" (in Kafka Connect terminology), and each partition has an "offset" (if Kafka Connect terminology) that defines the position within that partition/database where the connector finds a particular record. As the connector uses the Kafka Connect framework to process each record in a partition, Kafka Connect keeps track of the most recent offset for that partition.

The source partition information identifies the particular MongoDB replica set and the connector's logical name of the MongoDB server. A JSON-like representation of the source partition for a database named "customers" hosted in a MongoDB replica set named "myMongoServer" is as follows: 

 {
     "server_id" : "myMongoServer",
     "replicaSetName" : "rs0"
 }

The source offset information describes the position within a particular partition of each record. Since each event in MongoDB's oplog is identified by a BSONTimestamp that tracks the time and the order of the event for that particular time (e.g., multiple events that occur at the same time will have unique orders), the offset includes the BSONTimetamp representation. (The event's h field is the unique ID for the operation, so this is also included in the offset.) And, if an initial sync is in progress, the offset will include the initsync field.

Here's a JSON-like representation of an example timestamp: 

 {
         "sec" = 1422998530,
         "ord" = 0,
         "h" = 398278732173914406,
         "initsync" = true
 }
Author:
Randall Hauch

  • Field Details

  • Constructor Details

  • Method Details

    • replicaSetNameForPartition

      public static String replicaSetNameForPartition(Map<String,?> partition)
      Get the replica set name for the given partition.
      Parameters:
      partition - the partition map
      Returns:
      the replica set name (when the partition is valid), or null if the partition is null or has no replica set name entry

    • collectionId

      CollectionId collectionId()

    • position

      SourceInfo.Position position()

    • serverId

      public String serverId()
      Returns:
      server id

    • partition

      public Map<String,String> partition(String replicaSetName)
      Get the Kafka Connect detail about the source "partition" for the given database in the replica set. If the database is not known, this method records the new partition.
      Parameters:
      replicaSetName - the name of the replica set name for which the partition is to be obtained; may not be null
      Returns:
      the source partition information; never null

    • lastOffsetTimestamp

      public org.bson.BsonTimestamp lastOffsetTimestamp(String replicaSetName)
      Get the MongoDB timestamp of the last offset position for the replica set.
      Parameters:
      replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
      Returns:
      the timestamp of the last offset, or the beginning of time if there is none

    • lastOffsetTxOrder

      public OptionalLong lastOffsetTxOrder(String replicaSetName)
      Get the MongoDB transaction order of the last offset position for the replica set.
      Parameters:
      replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
      Returns:
      the tx order of the transaction in progress or 0 in case of non-transactional event

    • lastResumeToken

      public String lastResumeToken(String replicaSetName)

    • lastPosition

      public SourceInfo.Position lastPosition(String replicaSetName)

    • lastOffset

      public Map<String,?> lastOffset(String replicaSetName)
      Get the Kafka Connect detail about the source "offset" for the named database, which describes the given position in the database where we have last read. If the database has not yet been seen, this records the starting position for that database. However, if there is a position for the database, the offset representation is returned.
      Parameters:
      replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
      Returns:
      a copy of the current offset for the database; never null

    • addSessionTxnIdToOffset

      private Map<String,?> addSessionTxnIdToOffset(SourceInfo.Position position, Map<String,Object> offset)

    • collectionEvent

      public void collectionEvent(String replicaSetName, CollectionId collectionId)
      Get a Struct representation of the source partition and offset information where we have last read. The Struct complies with the AbstractSourceInfo.schema() for the MongoDB connector.
      Parameters:
      replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
      collectionId - the event's collection identifier; may not be null
      See Also:

    • opLogEvent

      public void opLogEvent(String replicaSetName, org.bson.Document oplogEvent, org.bson.Document masterEvent, long orderInTx)
      Get a Struct representation of the source partition and offset information. The Struct complies with the AbstractSourceInfo.schema() for the MongoDB connector.
      Parameters:
      replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
      oplogEvent - the replica set oplog event that was last read; may be null if the position is the start of the oplog
      masterEvent - the replica set oplog event that contains event metadata; same as oplogEvent for non-transactional changes
      orderInTx - order in transaction batch, 0 for non-transactional events
      See Also:

    • changeStreamEvent

      public void changeStreamEvent(String replicaSetName, com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.Document> changeStreamEvent, long orderInTx)

    • opLogEvent

      public void opLogEvent(String replicaSetName, org.bson.Document oplogEvent)
      Get a Struct representation of the source partition and offset information. The Struct complies with the AbstractSourceInfo.schema() for the MongoDB connector.
      Parameters:
      replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
      oplogEvent - the replica set oplog event that was last read; may be null if the position is the start of the oplog
      See Also:

    • extractEventTimestamp

      protected static org.bson.BsonTimestamp extractEventTimestamp(org.bson.Document oplogEvent)
      Utility to extract the timestamp value from the event.
      Parameters:
      oplogEvent - the event
      Returns:
      the timestamp, or null if the event is null or there is no ts field

    • extractSessionTxnId

      protected static String extractSessionTxnId(org.bson.Document oplogEvent)
      Utility to extract the unique transaction id value from the event.
      Parameters:
      oplogEvent - the event
      Returns:
      the session transaction id or null

    • onEvent

      private void onEvent(String replicaSetName, CollectionId collectionId, SourceInfo.Position position)

    • hasOffset

      public boolean hasOffset(String replicaSetName)
      Determine whether we have previously recorded a MongoDB timestamp for the replica set.
      Parameters:
      replicaSetName - the name of the replica set name; may not be null
      Returns:
      true if an offset has been recorded for the replica set, or false if the replica set has not yet been seen

    • setOffsetFor

      public boolean setOffsetFor(String replicaSetName, Map<String,?> sourceOffset)
      Set the source offset, as read from Kafka Connect, for the given replica set. This method does nothing if the supplied map is null.
      Parameters:
      replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
      sourceOffset - the previously-recorded Kafka Connect source offset; may be null
      Returns:
      true if the offset was recorded, or false if the source offset is null
      Throws:
      org.apache.kafka.connect.errors.ConnectException - if any offset parameter values are missing, invalid, or of the wrong type

    • setOffsetFor

      public boolean setOffsetFor(Map<String,String> partition, Map<String,?> sourceOffset)
      Set the source offset, as read from Kafka Connect, for the given replica set. This method does nothing if the supplied map is null.
      Parameters:
      partition - the partition information; may not be null
      sourceOffset - the previously-recorded Kafka Connect source offset; may be null
      Returns:
      true if the offset was recorded, or false if the source offset is null
      Throws:
      org.apache.kafka.connect.errors.ConnectException - if any offset parameter values are missing, invalid, or of the wrong type

    • startInitialSync

      public void startInitialSync(String replicaSetName)
      Record that an initial sync has started for the given replica set.
      Parameters:
      replicaSetName - the name of the replica set; never null

    • stopInitialSync

      public void stopInitialSync(String replicaSetName)
      Record that an initial sync has stopped for the given replica set.
      Parameters:
      replicaSetName - the name of the replica set; never null

    • isInitialSyncOngoing

      public boolean isInitialSyncOngoing(String replicaSetName)
      Determine if the initial sync for the given replica set is still ongoing.
      Parameters:
      replicaSetName - the name of the replica set; never null
      Returns:
      true if the initial sync for this replica is still ongoing or was not completed before restarting, or false if there is currently no initial sync operation for this replica set

    • isSnapshotRunning

      public boolean isSnapshotRunning()
      Returns whether any replica sets are still running a snapshot.

    • intOffsetValue

      private static int intOffsetValue(Map<String,?> values, String key)

    • longOffsetValue

      private static long longOffsetValue(Map<String,?> values, String key)

    • stringOffsetValue

      private static String stringOffsetValue(Map<String,?> values, String key)

    • booleanOffsetValue

      private static boolean booleanOffsetValue(Map<String,?> values, String key)

    • timestamp

      protected Instant timestamp()
      Specified by:
      timestamp in class AbstractSourceInfo

    • snapshot

      protected SnapshotRecord snapshot()
      Overrides:
      snapshot in class BaseSourceInfo

    • database

      protected String database()
      Specified by:
      database in class AbstractSourceInfo

    • replicaSetName

      String replicaSetName()

    • transactionPosition

      protected OptionalLong transactionPosition()