Replicator (Debezium Connector for MongoDB 0.3.4 API)

java.lang.Object
- io.debezium.connector.mongodb.Replicator

```
@ThreadSafe
public class Replicator
extends Object
```
A component that replicates the content of a replica set, starting with an initial sync or continuing to read the oplog where it last left off.
Initial Sync
If no offsets have been recorded for this replica set, replication begins with an MongoDB initial sync of the replica set.
The logic used in this component to perform an initial sync is similar to that of the official initial sync functionality used by secondary MongoDB nodes, although our logic can be simpler since we are not storing state nor building indexes:
1. Read the primary node's current oplog time. This is our start time.
2. Clone all of the databases and collections from the primary node, using multiple threads. This steps is completed only after all collections are successfully copied.
3. Start reading the primary node's oplog from start time and applying the changes.
It is important to understand that step 2 is not performing a consistent snapshot. That means that once we start copying a collection, clients can make changes and we may or may not see those changes in our copy. However, this is not a problem because the MongoDB replication process -- and our logic -- relies upon the fact that every change recorded in the MongoDB oplog is idempotent. So, as long as we read the oplog from the same point in time (or earlier) than we started our copy operation, and apply all of the changes in the same order, then the state of all documents described by this connector will be the same.
Restart
If prior runs of the replicator have recorded offsets in the source information, then when the replicator starts it will simply start reading the primary's oplog starting at the same point it last left off.
Handling problems

This replicator does each of its tasks using a connection to the primary. If the replicator is not able to establish a connection to the primary (e.g., there is no primary, or the replicator cannot communicate with the primary), the replicator will continue to try to establish a connection, using an exponential back-off strategy to prevent saturating the system. After a configurable number of failed attempts, the replicator will fail by throwing a ConnectException.
Author:

Randall Hauch

Field Summary

Fields
Modifier and Type	Field and Description
`private Clock`	`clock`
`private Predicate<CollectionId>`	`collectionFilter`
`private ReplicationContext`	`context`
`private ExecutorService`	`copyThreads`
`private org.slf4j.Logger`	`logger`
`private ConnectionContext.MongoPrimary`	`primaryClient`
`private RecordMakers`	`recordMakers`
`private ReplicaSet`	`replicaSet`
`private String`	`rsName`
`private AtomicBoolean`	`running`
`private SourceInfo`	`source`

Constructor Summary

Constructors
Constructor and Description
`Replicator(ReplicationContext context, ReplicaSet replicaSet, BlockingConsumer<org.apache.kafka.connect.source.SourceRecord> recorder)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected long`	`copyCollection(CollectionId collectionId, long timestamp)` Copy the collection, sending to the recorder a record for each document.
`protected long`	`copyCollection(com.mongodb.MongoClient primary, CollectionId collectionId, long timestamp)` Copy the collection, sending to the recorder a record for each document.
`protected boolean`	`establishConnectionToPrimary()` Establish a connection to the primary.
`protected boolean`	`handleOplogEvent(com.mongodb.ServerAddress primaryAddress, org.bson.Document event)` Handle a single oplog event.
`protected boolean`	`isInitialSyncExpected()` Determine if an initial sync should be performed.
`protected boolean`	`performInitialSync()` Perform the initial sync of the collections in the replica set.
`protected void`	`readOplog()` Repeatedly obtain a connection to the replica set's current primary and use that primary to read the oplog.
`protected void`	`readOplog(com.mongodb.MongoClient primary)` Use the given primary to read the oplog.
`protected void`	`recordCurrentOplogPosition()` Obtain the current position of the oplog, and record it in the source.
`void`	`run()` Perform the replication logic.
`void`	`stop()` Stop the replication from running.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - logger
```
private final org.slf4j.Logger logger
```
  - context
```
private final ReplicationContext context
```
  - copyThreads
```
private final ExecutorService copyThreads
```
  - replicaSet
```
private final ReplicaSet replicaSet
```
  - rsName
```
private final String rsName
```
  - running
```
private final AtomicBoolean running
```
  - source
```
private final SourceInfo source
```
  - recordMakers
```
private final RecordMakers recordMakers
```
  - collectionFilter
```
private final Predicate<CollectionId> collectionFilter
```
  - clock
```
private final Clock clock
```
  - primaryClient
```
private ConnectionContext.MongoPrimary primaryClient
```
- Constructor Detail
  - Replicator
```
public Replicator(ReplicationContext context,
                  ReplicaSet replicaSet,
                  BlockingConsumer<org.apache.kafka.connect.source.SourceRecord> recorder)
```
    Parameters:
    
    context - the replication context; may not be null
    
    replicaSet - the replica set to be replicated; may not be null
    
    recorder - the recorder for source record produced by this replicator; may not be null
- Method Detail
  - stop
```
public void stop()
```
    Stop the replication from running.
    This method does nothing if the snapshot is not running
  - run
```
public void run()
```
    Perform the replication logic. This can be run once.
  - establishConnectionToPrimary
```
protected boolean establishConnectionToPrimary()
```
    Establish a connection to the primary.
    
    Returns:
    
    true if a connection was established, or false otherwise
  - recordCurrentOplogPosition
```
protected void recordCurrentOplogPosition()
```
    Obtain the current position of the oplog, and record it in the source.
  - isInitialSyncExpected
```
protected boolean isInitialSyncExpected()
```
    Determine if an initial sync should be performed. An initial sync is expected if the source has no previously recorded offsets for this replica set, or if a snapshot should always be performed.
    
    Returns:
    
    true if the initial sync should be performed, or false otherwise
  - performInitialSync
```
protected boolean performInitialSync()
```
    Perform the initial sync of the collections in the replica set.
    
    Returns:
    
    true if the initial sync was completed, or false if it was stopped for any reason
  - copyCollection
```
protected long copyCollection(CollectionId collectionId,
                              long timestamp)
                       throws InterruptedException
```
    Copy the collection, sending to the recorder a record for each document.
    
    Parameters:
    
    collectionId - the identifier of the collection to be copied; may not be null
    
    timestamp - the timestamp in milliseconds at which the copy operation was started
    
    Returns:
    
    number of documents that were copied
    
    Throws:
    
    InterruptedException - if the thread was interrupted while the copy operation was running
  - copyCollection
```
protected long copyCollection(com.mongodb.MongoClient primary,
                              CollectionId collectionId,
                              long timestamp)
                       throws InterruptedException
```
    Copy the collection, sending to the recorder a record for each document.
    
    Parameters:
    
    primary - the connection to the replica set's primary node; may not be null
    
    collectionId - the identifier of the collection to be copied; may not be null
    
    timestamp - the timestamp in milliseconds at which the copy operation was started
    
    Returns:
    
    number of documents that were copied
    
    Throws:
    
    InterruptedException - if the thread was interrupted while the copy operation was running
  - readOplog
```
protected void readOplog()
```
    Repeatedly obtain a connection to the replica set's current primary and use that primary to read the oplog. This method will continue to run even if there are errors or problems. The method will return when a sufficient number of errors occur or if the replicator should stop reading the oplog. The latter occurs when a new primary is elected (as identified by an oplog event), of if the current thread doing the reading is interrupted.
  - readOplog
```
protected void readOplog(com.mongodb.MongoClient primary)
```
    Use the given primary to read the oplog.
    
    Parameters:
    
    primary - the connection to the replica set's primary node; may not be null
  - handleOplogEvent
```
protected boolean handleOplogEvent(com.mongodb.ServerAddress primaryAddress,
                                   org.bson.Document event)
```
    Handle a single oplog event.
    
    Parameters:
    
    primaryAddress - the address of the primary server from which the event was obtained; may not be null
    
    event - the oplog event; may not be null
    
    Returns:
    
    true if additional events should be processed, or false if the caller should stop processing events

Class Replicator

Initial Sync

Restart

Handling problems

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

logger

context

copyThreads

replicaSet

rsName

running

source

recordMakers

collectionFilter

clock

primaryClient

Constructor Detail

Replicator

Method Detail

stop

run

establishConnectionToPrimary

recordCurrentOplogPosition

isInitialSyncExpected

performInitialSync

copyCollection

copyCollection

readOplog

readOplog

handleOplogEvent