io.debezium.connector.mongodb

Class SourceInfo

java.lang.Object

io.debezium.connector.AbstractSourceInfo

io.debezium.connector.mongodb.SourceInfo

@NotThreadSafe
public final class SourceInfo
extends AbstractSourceInfo

Information about the source of information, which includes the partitions and offsets within those partitions. The MongoDB connector considers each MongoDB database a separate "partitions" (in Kafka Connect terminology), and each partition has an "offset" (if Kafka Connect terminology) that defines the position within that partition/database where the connector finds a particular record. As the connector uses the Kafka Connect framework to process each record in a partition, Kafka Connect keeps track of the most recent offset for that partition.

The source partition information identifies the particular MongoDB replica set and the connector's logical name of the MongoDB server. A JSON-like representation of the source partition for a database named "customers" hosted in a MongoDB replica set named "myMongoServer" is as follows:

 {
     "server_id" : "myMongoServer",
     "replicaSetName" : "rs0"
 }
 

The source offset information describes the position within a particular partition of each record. Since each event in MongoDB's oplog is identified by a BSONTimestamp that tracks the time and the order of the event for that particular time (e.g., multiple events that occur at the same time will have unique orders), the offset includes the BSONTimetamp representation. (The event's h field is the unique ID for the operation, so this is also included in the offset.) And, if an initial sync is in progress, the offset will include the initsync field.

Here's a JSON-like representation of an example timestamp:

 {
         "sec" = 1422998530,
         "ord" = 0,
         "h" = 398278732173914406,
         "initsync" = true
 }
 

Author:

Randall Hauch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	SourceInfo.Position

Field Summary

Fields

Modifier and Type	Field and Description
private static SourceInfo.Position	INITIAL_POSITION
static String	INITIAL_SYNC
private static org.bson.BsonTimestamp	INITIAL_TIMESTAMP
private Set<String>	initialSyncReplicaSets
static String	NAMESPACE
static String	OPERATION_ID
static String	ORDER
private ConcurrentMap<String,SourceInfo.Position>	positionsByReplicaSetName
static String	REPLICA_SET_NAME
static int	SCHEMA_VERSION
static String	SERVER_ID_KEY
static String	SERVER_NAME
private String	serverName
private org.apache.kafka.connect.data.Schema	SOURCE_SCHEMA A Schema definition for a Struct used to store the partition(String) and lastOffset(java.lang.String) information.
private ConcurrentMap<String,Map<String,String>>	sourcePartitionsByReplicaSetName
static String	TIMESTAMP

Fields inherited from class io.debezium.connector.AbstractSourceInfo

DEBEZIUM_VERSION_KEY

Constructor Summary

Constructors

Constructor and Description
SourceInfo(String serverName)

Method Summary

Modifier and Type	Method and Description
protected static org.bson.BsonTimestamp	extractEventTimestamp(org.bson.Document oplogEvent) Utility to extract the timestamp value from the event.
boolean	hasOffset(String replicaSetName) Determine whether we have previously recorded a MongoDB timestamp for the replica set.
private static int	intOffsetValue(Map<String,?> values, String key)
boolean	isInitialSyncOngoing(String replicaSetName) Determine if the initial sync for the given replica set is still ongoing.
Map<String,?>	lastOffset(String replicaSetName) Get the Kafka Connect detail about the source "offset" for the named database, which describes the given position in the database where we have last read.
org.apache.kafka.connect.data.Struct	lastOffsetStruct(String replicaSetName, CollectionId collectionId) Get a Struct representation of the source partition and offset information where we have last read.
org.bson.BsonTimestamp	lastOffsetTimestamp(String replicaSetName) Get the MongoDB timestamp of the last offset position for the replica set.
private static long	longOffsetValue(Map<String,?> values, String key)
private org.apache.kafka.connect.data.Struct	offsetStructFor(String replicaSetName, String namespace, SourceInfo.Position position, boolean isInitialSync)
org.apache.kafka.connect.data.Struct	offsetStructForEvent(String replicaSetName, org.bson.Document oplogEvent) Get a Struct representation of the source partition and offset information.
Map<String,String>	partition(String replicaSetName) Get the Kafka Connect detail about the source "partition" for the given database in the replica set.
static String	replicaSetNameForPartition(Map<String,?> partition) Get the replica set name for the given partition.
org.apache.kafka.connect.data.Schema	schema() Get a Schema representation of the source partition and offset information.
boolean	setOffsetFor(Map<String,String> partition, Map<String,?> sourceOffset) Set the source offset, as read from Kafka Connect, for the given replica set.
boolean	setOffsetFor(String replicaSetName, Map<String,?> sourceOffset) Set the source offset, as read from Kafka Connect, for the given replica set.
void	startInitialSync(String replicaSetName) Record that an initial sync has started for the given replica set.
void	stopInitialSync(String replicaSetName) Record that an initial sync has stopped for the given replica set.

Methods inherited from class io.debezium.connector.AbstractSourceInfo

schemaBuilder, struct

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SCHEMA_VERSION

public static final int SCHEMA_VERSION

See Also:

Constant Field Values

SERVER_ID_KEY

public static final String SERVER_ID_KEY

See Also:

Constant Field Values

SERVER_NAME

public static final String SERVER_NAME

See Also:

Constant Field Values

REPLICA_SET_NAME

public static final String REPLICA_SET_NAME

See Also:

Constant Field Values

NAMESPACE

public static final String NAMESPACE

See Also:

Constant Field Values

TIMESTAMP

public static final String TIMESTAMP

See Also:

Constant Field Values

ORDER

public static final String ORDER

See Also:

Constant Field Values

OPERATION_ID

public static final String OPERATION_ID

See Also:

Constant Field Values

INITIAL_SYNC

public static final String INITIAL_SYNC

See Also:

Constant Field Values

INITIAL_TIMESTAMP

private static final org.bson.BsonTimestamp INITIAL_TIMESTAMP

INITIAL_POSITION

private static final SourceInfo.Position INITIAL_POSITION

SOURCE_SCHEMA

private final org.apache.kafka.connect.data.Schema SOURCE_SCHEMA

A Schema definition for a Struct used to store the partition(String) and lastOffset(java.lang.String) information.

sourcePartitionsByReplicaSetName

private final ConcurrentMap<String,Map<String,String>> sourcePartitionsByReplicaSetName

positionsByReplicaSetName

private final ConcurrentMap<String,SourceInfo.Position> positionsByReplicaSetName

initialSyncReplicaSets

private final Set<String> initialSyncReplicaSets

serverName

private final String serverName

Constructor Detail

SourceInfo

public SourceInfo(String serverName)

Method Detail

replicaSetNameForPartition

public static String replicaSetNameForPartition(Map<String,?> partition)

Get the replica set name for the given partition.

Parameters:

partition - the partition map

Returns:

the replica set name (when the partition is valid), or null if the partition is null or has no replica set name entry

schema

public org.apache.kafka.connect.data.Schema schema()

Get a Schema representation of the source partition and offset information.

Specified by:

schema in class AbstractSourceInfo

Returns:

the source partition and offset Schema; never null

See Also:

offsetStructForEvent(String, Document)

partition

public Map<String,String> partition(String replicaSetName)

Get the Kafka Connect detail about the source "partition" for the given database in the replica set. If the database is not known, this method records the new partition.

Parameters:

replicaSetName - the name of the replica set name for which the partition is to be obtained; may not be null

Returns:

the source partition information; never null

lastOffsetTimestamp

public org.bson.BsonTimestamp lastOffsetTimestamp(String replicaSetName)

Get the MongoDB timestamp of the last offset position for the replica set.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

Returns:

the timestamp of the last offset, or the beginning of time if there is none

lastOffset

public Map<String,?> lastOffset(String replicaSetName)

Get the Kafka Connect detail about the source "offset" for the named database, which describes the given position in the database where we have last read. If the database has not yet been seen, this records the starting position for that database. However, if there is a position for the database, the offset representation is returned.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

Returns:

a copy of the current offset for the database; never null

lastOffsetStruct

public org.apache.kafka.connect.data.Struct lastOffsetStruct(String replicaSetName,
                                                             CollectionId collectionId)

Get a Struct representation of the source partition and offset information where we have last read. The Struct complies with the schema() for the MongoDB connector.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

collectionId - the event's collection identifier; may not be null

Returns:

the source partition and offset Struct; never null

See Also:

schema()

offsetStructForEvent

public org.apache.kafka.connect.data.Struct offsetStructForEvent(String replicaSetName,
                                                                 org.bson.Document oplogEvent)

Get a Struct representation of the source partition and offset information. The Struct complies with the schema() for the MongoDB connector.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

oplogEvent - the replica set oplog event that was last read; may be null if the position is the start of the oplog

Returns:

the source partition and offset Struct; never null

See Also:

schema()

extractEventTimestamp

protected static org.bson.BsonTimestamp extractEventTimestamp(org.bson.Document oplogEvent)

Utility to extract the timestamp value from the event.

Parameters:

oplogEvent - the event

Returns:

the timestamp, or null if the event is null or there is no ts field

offsetStructFor

private org.apache.kafka.connect.data.Struct offsetStructFor(String replicaSetName,
                                                             String namespace,
                                                             SourceInfo.Position position,
                                                             boolean isInitialSync)

hasOffset

public boolean hasOffset(String replicaSetName)

Determine whether we have previously recorded a MongoDB timestamp for the replica set.

Parameters:

replicaSetName - the name of the replica set name; may not be null

Returns:

true if an offset has been recorded for the replica set, or false if the replica set has not yet been seen

setOffsetFor

public boolean setOffsetFor(String replicaSetName,
                            Map<String,?> sourceOffset)

Set the source offset, as read from Kafka Connect, for the given replica set. This method does nothing if the supplied map is null.

Parameters:

replicaSetName - the name of the replica set name for which the new offset is to be obtained; may not be null

sourceOffset - the previously-recorded Kafka Connect source offset; may be null

Returns:

true if the offset was recorded, or false if the source offset is null

Throws:

org.apache.kafka.connect.errors.ConnectException - if any offset parameter values are missing, invalid, or of the wrong type

setOffsetFor

public boolean setOffsetFor(Map<String,String> partition,
                            Map<String,?> sourceOffset)

Set the source offset, as read from Kafka Connect, for the given replica set. This method does nothing if the supplied map is null.

Parameters:

partition - the partition information; may not be null

sourceOffset - the previously-recorded Kafka Connect source offset; may be null

Returns:

true if the offset was recorded, or false if the source offset is null

Throws:

org.apache.kafka.connect.errors.ConnectException - if any offset parameter values are missing, invalid, or of the wrong type

startInitialSync

public void startInitialSync(String replicaSetName)

Record that an initial sync has started for the given replica set.

Parameters:

replicaSetName - the name of the replica set; never null

stopInitialSync

public void stopInitialSync(String replicaSetName)

Record that an initial sync has stopped for the given replica set.

Parameters:

replicaSetName - the name of the replica set; never null

isInitialSyncOngoing

public boolean isInitialSyncOngoing(String replicaSetName)

Determine if the initial sync for the given replica set is still ongoing.

Parameters:

replicaSetName - the name of the replica set; never null

Returns:

true if the initial sync for this replica is still ongoing or was not completed before restarting, or false if there is currently no initial sync operation for this replica set

intOffsetValue

private static int intOffsetValue(Map<String,?> values,
                                  String key)

longOffsetValue

private static long longOffsetValue(Map<String,?> values,
                                    String key)