io.debezium.connector.mongodb

Class SourceInfo

java.lang.Object

io.debezium.connector.AbstractSourceInfo

io.debezium.connector.common.BaseSourceInfo

io.debezium.connector.mongodb.SourceInfo

@NotThreadSafe
public final class SourceInfo
extends BaseSourceInfo

Information about the source of information, which includes the partitions and offsets within those partitions. The MongoDB connector considers each MongoDB database a separate "partitions" (in Kafka Connect terminology), and each partition has an "offset" (if Kafka Connect terminology) that defines the position within that partition/database where the connector finds a particular record. As the connector uses the Kafka Connect framework to process each record in a partition, Kafka Connect keeps track of the most recent offset for that partition.

The source partition information identifies the particular MongoDB replica set and the connector's logical name of the MongoDB server. A JSON-like representation of the source partition for a database named "customers" hosted in a MongoDB replica set named "myMongoServer" is as follows:

 {
     "server_id" : "myMongoServer",
     "replicaSetName" : "rs0"
 }
 

The source offset information describes the position within a particular partition of each record. Since each event in MongoDB's oplog is identified by a BSONTimestamp that tracks the time and the order of the event for that particular time (e.g., multiple events that occur at the same time will have unique orders), the offset includes the BSONTimetamp representation. (The event's h field is the unique ID for the operation, so this is also included in the offset.) And, if an initial sync is in progress, the offset will include the initsync field.

Here's a JSON-like representation of an example timestamp:

 {
         "sec" = 1422998530,
         "ord" = 0,
         "h" = 398278732173914406,
         "initsync" = true
 }
 

Author:

Randall Hauch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	SourceInfo.Position

Field Summary

Fields

Modifier and Type	Field and Description
static String	COLLECTION
private CollectionId	collectionId Id of collection the current event applies to.
private static SourceInfo.Position	INITIAL_POSITION
static String	INITIAL_SYNC
private static org.bson.BsonTimestamp	INITIAL_TIMESTAMP
private Set<String>	initialSyncReplicaSets
static String	NAMESPACE
static String	OPERATION_ID
static String	ORDER
private SourceInfo.Position	position
private ConcurrentMap<String,SourceInfo.Position>	positionsByReplicaSetName
static String	REPLICA_SET_NAME
private String	replicaSetName
static int	SCHEMA_VERSION
static String	SERVER_ID_KEY
static String	SESSION_TXN_ID
private ConcurrentMap<String,Map<String,String>>	sourcePartitionsByReplicaSetName
static String	TIMESTAMP
static String	TX_ORD

Fields inherited from class io.debezium.connector.common.BaseSourceInfo

snapshotRecord

Fields inherited from class io.debezium.connector.AbstractSourceInfo

COLLECTION_NAME_KEY, DATABASE_NAME_KEY, DEBEZIUM_CONNECTOR_KEY, DEBEZIUM_VERSION_KEY, SCHEMA_NAME_KEY, SERVER_NAME_KEY, SNAPSHOT_KEY, TABLE_NAME_KEY, TIMESTAMP_KEY

Constructor Summary

Constructors

Constructor and Description
SourceInfo(MongoDbConnectorConfig connectorConfig)

Method Summary

Modifier and Type	Method and Description
private static boolean	booleanOffsetValue(Map<String,?> values, String key)
void	collectionEvent(String replicaSetName, CollectionId collectionId) Get a Struct representation of the source partition and offset information where we have last read.
(package private) CollectionId	collectionId()
protected String	database()
protected static org.bson.BsonTimestamp	extractEventTimestamp(org.bson.Document oplogEvent) Utility to extract the timestamp value from the event.
protected static String	extractSessionTxnId(org.bson.Document oplogEvent) Utility to extract the unique transaction id value from the event.
boolean	hasOffset(String replicaSetName) Determine whether we have previously recorded a MongoDB timestamp for the replica set.
private static int	intOffsetValue(Map<String,?> values, String key)
boolean	isInitialSyncOngoing(String replicaSetName) Determine if the initial sync for the given replica set is still ongoing.
boolean	isSnapshotRunning() Returns whether any replica sets are still running a snapshot.
Map<String,?>	lastOffset(String replicaSetName) Get the Kafka Connect detail about the source "offset" for the named database, which describes the given position in the database where we have last read.
org.bson.BsonTimestamp	lastOffsetTimestamp(String replicaSetName) Get the MongoDB timestamp of the last offset position for the replica set.
OptionalLong	lastOffsetTxOrder(String replicaSetName) Get the MongoDB transaction order of the last offset position for the replica set.
private static long	longOffsetValue(Map<String,?> values, String key)
private void	onEvent(String replicaSetName, CollectionId collectionId, SourceInfo.Position position)
void	opLogEvent(String replicaSetName, org.bson.Document oplogEvent) Get a Struct representation of the source partition and offset information.
void	opLogEvent(String replicaSetName, org.bson.Document oplogEvent, org.bson.Document masterEvent, long orderInTx) Get a Struct representation of the source partition and offset information.
Map<String,String>	partition(String replicaSetName) Get the Kafka Connect detail about the source "partition" for the given database in the replica set.
(package private) SourceInfo.Position	position()
(package private) String	replicaSetName()
static String	replicaSetNameForPartition(Map<String,?> partition) Get the replica set name for the given partition.
boolean	setOffsetFor(Map<String,String> partition, Map<String,?> sourceOffset) Set the source offset, as read from Kafka Connect, for the given replica set.
boolean	setOffsetFor(String replicaSetName, Map<String,?> sourceOffset) Set the source offset, as read from Kafka Connect, for the given replica set.
protected SnapshotRecord	snapshot()
void	startInitialSync(String replicaSetName) Record that an initial sync has started for the given replica set.
void	stopInitialSync(String replicaSetName) Record that an initial sync has stopped for the given replica set.
private static String	stringOffsetValue(Map<String,?> values, String key)
protected Instant	timestamp()
protected OptionalLong	transactionPosition()

Methods inherited from class io.debezium.connector.common.BaseSourceInfo

isSnapshot, setSnapshot

Methods inherited from class io.debezium.connector.AbstractSourceInfo

schema, serverName, struct, structMaker

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SCHEMA_VERSION

public static final int SCHEMA_VERSION

See Also:

Constant Field Values

SERVER_ID_KEY

public static final String SERVER_ID_KEY

See Also:

Constant Field Values

REPLICA_SET_NAME

public static final String REPLICA_SET_NAME

See Also:

Constant Field Values

NAMESPACE

public static final String NAMESPACE

See Also:

Constant Field Values

TIMESTAMP

public static final String TIMESTAMP

See Also:

Constant Field Values

ORDER

public static final String ORDER

See Also:

Constant Field Values

OPERATION_ID

public static final String OPERATION_ID

See Also:

Constant Field Values

TX_ORD

public static final String TX_ORD

See Also:

Constant Field Values

SESSION_TXN_ID

public static final String SESSION_TXN_ID

See Also:

Constant Field Values

INITIAL_SYNC

public static final String INITIAL_SYNC

See Also:

Constant Field Values

COLLECTION

public static final String COLLECTION

See Also:

Constant Field Values

INITIAL_TIMESTAMP

private static final org.bson.BsonTimestamp INITIAL_TIMESTAMP

INITIAL_POSITION

private static final SourceInfo.Position INITIAL_POSITION

sourcePartitionsByReplicaSetName

private final ConcurrentMap<String,Map<String,String>> sourcePartitionsByReplicaSetName

positionsByReplicaSetName

private final ConcurrentMap<String,SourceInfo.Position> positionsByReplicaSetName

initialSyncReplicaSets

private final Set<String> initialSyncReplicaSets

replicaSetName

private String replicaSetName

collectionId

private CollectionId collectionId

Id of collection the current event applies to. May be null after noop events, after which the recorded offset may be retrieved but not the source struct.

position

private SourceInfo.Position position

Constructor Detail

SourceInfo

public SourceInfo(MongoDbConnectorConfig connectorConfig)

Method Detail

replicaSetNameForPartition

public static String replicaSetNameForPartition(Map<String,?> partition)

Get the replica set name for the given partition.

Parameters:

partition - the partition map

Returns:

the replica set name (when the partition is valid), or null if the partition is null or has no replica set name entry

collectionId

CollectionId collectionId()

position

SourceInfo.Position position()

partition

public Map<String,String> partition(String replicaSetName)

Get the Kafka Connect detail about the source "partition" for the given database in the replica set. If the database is not known, this method records the new partition.

Parameters:

replicaSetName - the name of the replica set name for which the partition is to be obtained; may not be null

Returns:

the source partition information; never null

lastOffsetTimestamp

public org.bson.BsonTimestamp lastOffsetTimestamp(String replicaSetName)

Get the MongoDB timestamp of the last offset position for the replica set.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

Returns:

the timestamp of the last offset, or the beginning of time if there is none

lastOffsetTxOrder

public OptionalLong lastOffsetTxOrder(String replicaSetName)

Get the MongoDB transaction order of the last offset position for the replica set.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

Returns:

the tx order of the transaction in progress or 0 in case of non-transactional event

lastOffset

public Map<String,?> lastOffset(String replicaSetName)

Get the Kafka Connect detail about the source "offset" for the named database, which describes the given position in the database where we have last read. If the database has not yet been seen, this records the starting position for that database. However, if there is a position for the database, the offset representation is returned.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

Returns:

a copy of the current offset for the database; never null

collectionEvent

public void collectionEvent(String replicaSetName,
                            CollectionId collectionId)

Get a Struct representation of the source partition and offset information where we have last read. The Struct complies with the AbstractSourceInfo.schema() for the MongoDB connector.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

collectionId - the event's collection identifier; may not be null

See Also:

AbstractSourceInfo.schema()

opLogEvent

public void opLogEvent(String replicaSetName,
                       org.bson.Document oplogEvent,
                       org.bson.Document masterEvent,
                       long orderInTx)

Get a Struct representation of the source partition and offset information. The Struct complies with the AbstractSourceInfo.schema() for the MongoDB connector.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

oplogEvent - the replica set oplog event that was last read; may be null if the position is the start of the oplog

masterEvent - the replica set oplog event that contains event metadata; same as oplogEvent for non-transactional changes

orderInTx - order in transaction batch, 0 for non-transactional events

See Also:

AbstractSourceInfo.schema()

opLogEvent

public void opLogEvent(String replicaSetName,
                       org.bson.Document oplogEvent)

Get a Struct representation of the source partition and offset information. The Struct complies with the AbstractSourceInfo.schema() for the MongoDB connector.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

oplogEvent - the replica set oplog event that was last read; may be null if the position is the start of the oplog

See Also:

AbstractSourceInfo.schema()

extractEventTimestamp

protected static org.bson.BsonTimestamp extractEventTimestamp(org.bson.Document oplogEvent)

Utility to extract the timestamp value from the event.

Parameters:

oplogEvent - the event

Returns:

the timestamp, or null if the event is null or there is no ts field

extractSessionTxnId

protected static String extractSessionTxnId(org.bson.Document oplogEvent)

Utility to extract the unique transaction id value from the event.

Parameters:

oplogEvent - the event

Returns:

the session transaction id or null

onEvent

private void onEvent(String replicaSetName,
                     CollectionId collectionId,
                     SourceInfo.Position position)

hasOffset

public boolean hasOffset(String replicaSetName)

Determine whether we have previously recorded a MongoDB timestamp for the replica set.

Parameters:

replicaSetName - the name of the replica set name; may not be null

Returns:

true if an offset has been recorded for the replica set, or false if the replica set has not yet been seen

setOffsetFor

public boolean setOffsetFor(String replicaSetName,
                            Map<String,?> sourceOffset)

Set the source offset, as read from Kafka Connect, for the given replica set. This method does nothing if the supplied map is null.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

sourceOffset - the previously-recorded Kafka Connect source offset; may be null

Returns:

true if the offset was recorded, or false if the source offset is null

Throws:

org.apache.kafka.connect.errors.ConnectException - if any offset parameter values are missing, invalid, or of the wrong type

setOffsetFor

public boolean setOffsetFor(Map<String,String> partition,
                            Map<String,?> sourceOffset)

Set the source offset, as read from Kafka Connect, for the given replica set. This method does nothing if the supplied map is null.

Parameters:

partition - the partition information; may not be null

sourceOffset - the previously-recorded Kafka Connect source offset; may be null

Returns:

true if the offset was recorded, or false if the source offset is null

Throws:

org.apache.kafka.connect.errors.ConnectException - if any offset parameter values are missing, invalid, or of the wrong type

startInitialSync

public void startInitialSync(String replicaSetName)

Record that an initial sync has started for the given replica set.

Parameters:

replicaSetName - the name of the replica set; never null

stopInitialSync

public void stopInitialSync(String replicaSetName)

Record that an initial sync has stopped for the given replica set.

Parameters:

replicaSetName - the name of the replica set; never null

isInitialSyncOngoing

public boolean isInitialSyncOngoing(String replicaSetName)

Determine if the initial sync for the given replica set is still ongoing.

Parameters:

replicaSetName - the name of the replica set; never null

Returns:

true if the initial sync for this replica is still ongoing or was not completed before restarting, or false if there is currently no initial sync operation for this replica set

isSnapshotRunning

public boolean isSnapshotRunning()

Returns whether any replica sets are still running a snapshot.

intOffsetValue

private static int intOffsetValue(Map<String,?> values,
                                  String key)

longOffsetValue

private static long longOffsetValue(Map<String,?> values,
                                    String key)

stringOffsetValue

private static String stringOffsetValue(Map<String,?> values,
                                        String key)

booleanOffsetValue

private static boolean booleanOffsetValue(Map<String,?> values,
                                          String key)

timestamp

protected Instant timestamp()

Specified by:

timestamp in class AbstractSourceInfo

snapshot

protected SnapshotRecord snapshot()

Overrides:

snapshot in class BaseSourceInfo

database

protected String database()

Specified by:

database in class AbstractSourceInfo

replicaSetName

String replicaSetName()

transactionPosition

protected OptionalLong transactionPosition()