Package io.debezium.connector.mongodb

Class SourceInfo

java.lang.Object
io.debezium.connector.AbstractSourceInfo
io.debezium.connector.common.BaseSourceInfo
io.debezium.connector.mongodb.SourceInfo
@NotThreadSafe public final class SourceInfo extends BaseSourceInfo
Information about the source of information, which includes the partitions and offsets within those partitions. The MongoDB connector considers each MongoDB database a separate "partitions" (in Kafka Connect terminology), and each partition has an "offset" (if Kafka Connect terminology) that defines the position within that partition/database where the connector finds a particular record. As the connector uses the Kafka Connect framework to process each record in a partition, Kafka Connect keeps track of the most recent offset for that partition.

The source partition information identifies the particular MongoDB replica set and the connector's logical name of the MongoDB server. A JSON-like representation of the source partition for a database named "customers" hosted in a MongoDB replica set named "myMongoServer" is as follows: 

 {
     "server_id" : "myMongoServer",
     "replicaSetName" : "rs0"
 }

The source offset information describes the position within a particular partition of each record. Since each event in MongoDB's oplog is identified by a BSONTimestamp that tracks the time and the order of the event for that particular time (e.g., multiple events that occur at the same time will have unique orders), the offset includes the BSONTimetamp representation. (The event's h field is the unique ID for the operation, so this is also included in the offset.) And, if an initial sync is in progress, the offset will include the initsync field.

Here's a JSON-like representation of an example timestamp: 

 {
         "sec" = 1422998530,
         "ord" = 0,
         "h" = 398278732173914406,
         "initsync" = true
 }
Author:
Randall Hauch

  • Field Details

  • Constructor Details

  • Method Details

    • collectionId

      CollectionId collectionId()

    • position

      SourceInfo.Position position()

    • serverId

      public String serverId()
      Returns:
      server id

    • partition

      public Map<String,String> partition(String replicaSetName)
      Get the Kafka Connect detail about the source "partition" for the given database in the replica set. If the database is not known, this method records the new partition.
      Parameters:
      replicaSetName - the name of the replica set name for which the partition is to be obtained; may not be null
      Returns:
      the source partition information; never null

    • lastResumeToken

      public String lastResumeToken(String replicaSetName)

    • lastOffset

      public Map<String,?> lastOffset(String replicaSetName)
      Get the Kafka Connect detail about the source "offset" for the named database, which describes the given position in the database where we have last read. If the database has not yet been seen, this records the starting position for that database. However, if there is a position for the database, the offset representation is returned.
      Parameters:
      replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
      Returns:
      a copy of the current offset for the database; never null

    • addSessionTxnIdToOffset

      private Map<String,?> addSessionTxnIdToOffset(SourceInfo.Position position, Map<String,Object> offset)

    • collectionEvent

      public void collectionEvent(String replicaSetName, CollectionId collectionId, long wallTime)
      Get a Struct representation of the source partition and offset information where we have last read. The Struct complies with the AbstractSourceInfo.schema() for the MongoDB connector.
      Parameters:
      replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
      collectionId - the event's collection identifier; may not be null
      See Also:

    • initEvent

      public void initEvent(String replicaSetName, com.mongodb.client.MongoChangeStreamCursor<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> cursor)

    • noEvent

      public void noEvent(String replicaSetName, com.mongodb.client.MongoChangeStreamCursor<?> cursor)

    • changeStreamEvent

      public void changeStreamEvent(String replicaSetName, com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument> changeStreamEvent)

    • onEvent

      private void onEvent(String replicaSetName, CollectionId collectionId, SourceInfo.Position position, long wallTime)

    • hasOffset

      public boolean hasOffset(String replicaSetName)
      Determine whether we have previously recorded a MongoDB timestamp for the replica set.
      Parameters:
      replicaSetName - the name of the replica set name; may not be null
      Returns:
      true if an offset has been recorded for the replica set, or false if the replica set has not yet been seen

    • setOffsetFor

      public boolean setOffsetFor(String replicaSetName, Map<String,?> sourceOffset)
      Set the source offset, as read from Kafka Connect, for the given replica set. This method does nothing if the supplied map is null.
      Parameters:
      replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null
      sourceOffset - the previously-recorded Kafka Connect source offset; may be null
      Returns:
      true if the offset was recorded, or false if the source offset is null
      Throws:
      io.debezium.DebeziumException - if any offset parameter values are missing, invalid, or of the wrong type

    • setOffsetFor

      public boolean setOffsetFor(Map<String,String> partition, Map<String,?> sourceOffset)
      Set the source offset, as read from Kafka Connect, for the given replica set. This method does nothing if the supplied map is null.
      Parameters:
      partition - the partition information; may not be null
      sourceOffset - the previously-recorded Kafka Connect source offset; may be null
      Returns:
      true if the offset was recorded, or false if the source offset is null
      Throws:
      io.debezium.DebeziumException - if any offset parameter values are missing, invalid, or of the wrong type

    • startInitialSync

      public void startInitialSync(String replicaSetName)
      Record that an initial sync has started for the given replica set.
      Parameters:
      replicaSetName - the name of the replica set; never null

    • stopInitialSync

      public void stopInitialSync(String replicaSetName)
      Record that an initial sync has stopped for the given replica set.
      Parameters:
      replicaSetName - the name of the replica set; never null

    • isInitialSyncOngoing

      public boolean isInitialSyncOngoing(String replicaSetName)
      Determine if the initial sync for the given replica set is still ongoing.
      Parameters:
      replicaSetName - the name of the replica set; never null
      Returns:
      true if the initial sync for this replica is still ongoing or was not completed before restarting, or false if there is currently no initial sync operation for this replica set

    • isSnapshotRunning

      public boolean isSnapshotRunning()
      Returns whether any replica sets are still running a snapshot.

    • intOffsetValue

      private static int intOffsetValue(Map<String,?> values, String key)

    • longOffsetValue

      private static long longOffsetValue(Map<String,?> values, String key)

    • stringOffsetValue

      private static String stringOffsetValue(Map<String,?> values, String key)

    • booleanOffsetValue

      private static boolean booleanOffsetValue(Map<String,?> values, String key)

    • timestamp

      protected Instant timestamp()
      Specified by:
      timestamp in class AbstractSourceInfo

    • snapshot

      public SnapshotRecord snapshot()
      Overrides:
      snapshot in class BaseSourceInfo

    • database

      protected String database()
      Specified by:
      database in class AbstractSourceInfo

    • replicaSetName

      String replicaSetName()

    • wallTime

      long wallTime()

    • toString

      public String toString()
      Overrides:
      toString in class Object