io.debezium.connector.mongodb

Class Replicator

java.lang.Object

io.debezium.connector.mongodb.Replicator

@ThreadSafe
public class Replicator
extends Object

A component that replicates the content of a replica set, starting with an initial sync or continuing to read the oplog where it last left off.

Initial Sync

If no offsets have been recorded for this replica set, replication begins with an MongoDB initial sync of the replica set.

The logic used in this component to perform an initial sync is similar to that of the official initial sync functionality used by secondary MongoDB nodes, although our logic can be simpler since we are not storing state nor building indexes:

1. Read the primary node's current oplog time. This is our start time.
2. Clone all of the databases and collections from the primary node, using multiple threads. This steps is completed only after all collections are successfully copied.
3. Start reading the primary node's oplog from start time and applying the changes.

It is important to understand that step 2 is not performing a consistent snapshot. That means that once we start copying a collection, clients can make changes and we may or may not see those changes in our copy. However, this is not a problem because the MongoDB replication process -- and our logic -- relies upon the fact that every change recorded in the MongoDB oplog is idempotent. So, as long as we read the oplog from the same point in time (or earlier) than we started our copy operation, and apply all of the changes in the same order, then the state of all documents described by this connector will be the same.

Restart

If prior runs of the replicator have recorded offsets in the source information, then when the replicator starts it will simply start reading the primary's oplog starting at the same point it last left off.

Handling problems

This replicator does each of its tasks using a connection to the primary. If the replicator is not able to establish a connection to the primary (e.g., there is no primary, or the replicator cannot communicate with the primary), the replicator will continue to try to establish a connection, using an exponential back-off strategy to prevent saturating the system. After a configurable number of failed attempts, the replicator will fail by throwing a ConnectException.

Author:

Randall Hauch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	Replicator.BufferableRecorder A BlockingConsumer implementation that will buffer the last source record only during an initial sync, so that when the initial sync is complete the last record's offset can be updated to reflect the completion of the initial sync before it is flushed.

Field Summary

Fields

Modifier and Type	Field and Description
private static String	AUTHORIZATION_FAILURE_MESSAGE
private Replicator.BufferableRecorder	bufferedRecorder
private Clock	clock
private MongoDbTaskContext	context
private ExecutorService	copyThreads
private org.slf4j.Logger	logger
private Consumer<Throwable>	onFailure
private ConnectionContext.MongoPrimary	primaryClient
private RecordMakers	recordMakers
private ReplicaSet	replicaSet
private String	rsName
private AtomicBoolean	running
private SourceInfo	source

Constructor Summary

Constructors

Constructor and Description
Replicator(MongoDbTaskContext context, ReplicaSet replicaSet, BlockingConsumer<org.apache.kafka.connect.source.SourceRecord> recorder, Consumer<Throwable> onFailure)

Method Summary

Modifier and Type	Method and Description
protected long	copyCollection(CollectionId collectionId, long timestamp) Copy the collection, sending to the recorder a record for each document.
protected long	copyCollection(com.mongodb.MongoClient primary, CollectionId collectionId, long timestamp) Copy the collection, sending to the recorder a record for each document.
protected boolean	establishConnectionToPrimary() Establish a connection to the primary.
protected boolean	handleOplogEvent(com.mongodb.ServerAddress primaryAddress, org.bson.Document event) Handle a single oplog event.
protected boolean	isInitialSyncExpected() Determine if an initial sync should be performed.
protected boolean	performInitialSync() Perform the initial sync of the collections in the replica set.
protected void	readOplog() Repeatedly obtain a connection to the replica set's current primary and use that primary to read the oplog.
protected void	readOplog(com.mongodb.MongoClient primary) Use the given primary to read the oplog.
protected void	recordCurrentOplogPosition() Obtain the current position of the oplog, and record it in the source.
void	run() Perform the replication logic.
void	stop() Stop the replication from running.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

private final org.slf4j.Logger logger

AUTHORIZATION_FAILURE_MESSAGE

private static final String AUTHORIZATION_FAILURE_MESSAGE

See Also:

Constant Field Values

context

private final MongoDbTaskContext context

copyThreads

private final ExecutorService copyThreads

replicaSet

private final ReplicaSet replicaSet

rsName

private final String rsName

running

private final AtomicBoolean running

source

private final SourceInfo source

recordMakers

private final RecordMakers recordMakers

bufferedRecorder

private final Replicator.BufferableRecorder bufferedRecorder

clock

private final Clock clock

primaryClient

private ConnectionContext.MongoPrimary primaryClient

onFailure

private final Consumer<Throwable> onFailure

Constructor Detail

Replicator

public Replicator(MongoDbTaskContext context,
                  ReplicaSet replicaSet,
                  BlockingConsumer<org.apache.kafka.connect.source.SourceRecord> recorder,
                  Consumer<Throwable> onFailure)

Parameters:

context - the replication context; may not be null

replicaSet - the replica set to be replicated; may not be null

recorder - the recorder for source record produced by this replicator; may not be null

onFailure - listener of exceptions thrown by replicator task

Method Detail

stop

public void stop()

Stop the replication from running.

This method does nothing if the snapshot is not running

run

public void run()

Perform the replication logic. This can be run once.

establishConnectionToPrimary

protected boolean establishConnectionToPrimary()

Establish a connection to the primary.

Returns:

true if a connection was established, or false otherwise

recordCurrentOplogPosition

protected void recordCurrentOplogPosition()

Obtain the current position of the oplog, and record it in the source.

isInitialSyncExpected

protected boolean isInitialSyncExpected()

Determine if an initial sync should be performed. An initial sync is expected if the source has no previously recorded offsets for this replica set, or if a snapshot should always be performed.

Returns:

true if the initial sync should be performed, or false otherwise

performInitialSync

protected boolean performInitialSync()

Perform the initial sync of the collections in the replica set.

Returns:

true if the initial sync was completed, or false if it was stopped for any reason

copyCollection

protected long copyCollection(CollectionId collectionId,
                              long timestamp)
                       throws InterruptedException

Copy the collection, sending to the recorder a record for each document.

Parameters:

collectionId - the identifier of the collection to be copied; may not be null

timestamp - the timestamp in milliseconds at which the copy operation was started

Returns:

number of documents that were copied

Throws:

InterruptedException - if the thread was interrupted while the copy operation was running

copyCollection

protected long copyCollection(com.mongodb.MongoClient primary,
                              CollectionId collectionId,
                              long timestamp)
                       throws InterruptedException

Copy the collection, sending to the recorder a record for each document.

Parameters:

primary - the connection to the replica set's primary node; may not be null

collectionId - the identifier of the collection to be copied; may not be null

timestamp - the timestamp in milliseconds at which the copy operation was started

Returns:

number of documents that were copied

Throws:

InterruptedException - if the thread was interrupted while the copy operation was running

readOplog

protected void readOplog()

Repeatedly obtain a connection to the replica set's current primary and use that primary to read the oplog. This method will continue to run even if there are errors or problems. The method will return when a sufficient number of errors occur or if the replicator should stop reading the oplog. The latter occurs when a new primary is elected (as identified by an oplog event), of if the current thread doing the reading is interrupted.

readOplog

protected void readOplog(com.mongodb.MongoClient primary)

Use the given primary to read the oplog.

Parameters:

primary - the connection to the replica set's primary node; may not be null

handleOplogEvent

protected boolean handleOplogEvent(com.mongodb.ServerAddress primaryAddress,
                                   org.bson.Document event)

Handle a single oplog event.

Parameters:

primaryAddress - the address of the primary server from which the event was obtained; may not be null

event - the oplog event; may not be null

Returns:

true if additional events should be processed, or false if the caller should stop processing events