Package io.debezium.connector.mongodb

Class OffsetConsolidationShardedIT

java.lang.Object

io.debezium.embedded.AbstractConnectorTest

io.debezium.connector.mongodb.AbstractShardedMongoConnectorIT

io.debezium.connector.mongodb.OffsetConsolidationShardedIT

All Implemented Interfaces:

Testing

public class OffsetConsolidationShardedIT
extends AbstractShardedMongoConnectorIT

Nested Class Summary

Nested classes/interfaces inherited from class io.debezium.embedded.AbstractConnectorTest

AbstractConnectorTest.SourceRecords

Nested classes/interfaces inherited from interface io.debezium.util.Testing

Testing.Debug, Testing.Files, Testing.InterruptableFunction, Testing.Network, Testing.Print, Testing.Timer

Field Summary

Fields

Modifier and Type	Field	Description
private final SortedSet<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>>	allRouterEvents	Contains all events consumed from router (always at least 4)
Configuration	config
private io.debezium.connector.mongodb.MongoDbConnectorConfig	connectorConfig
private static final Comparator<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>>	EVENT_COMPARATOR	Compares change event by cluster time
private final List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>>	shardEvents0	Guaranteed to contain all consumed from shard0 (always at least 2)
private final List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>>	shardEvents1	Guaranteed to contain all consumed from shard1 (always at least 2)
static final String	TOPIC_PREFIX

Fields inherited from class io.debezium.connector.mongodb.AbstractShardedMongoConnectorIT

DEFAULT_COLLECTION, DEFAULT_DATABASE, DEFAULT_SHARDING_KEY, mongo

Fields inherited from class io.debezium.embedded.AbstractConnectorTest

consumedLines, engine, isEngineRunning, logger, logTestName, OFFSET_STORE_PATH, pollTimeoutInMs, skipTestRule

Constructor Summary

Constructors

Constructor	Description
OffsetConsolidationShardedIT()

Method Summary

Modifier and Type	Method	Description
Set<String>	getExpectedIds(com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument> offsetEvent)
Set<String>	getExpectedIds(Collection<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> events)
com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>	getOffsetEvent(List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> events)	Gets next-to-last event consumed by router change streams.
com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>	getOffsetEvent(List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> shard0, List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> shard1)	Gets the older out of the two next-to-last events consumed by shard specific change streams.
void	insertInitialDocuments(io.debezium.connector.mongodb.ChangeStreamPipeline pipeline)
static <T> T	nextToLastElement(List<T> collection)
private Map<Map<String,?>,Map<String,?>>	prepareReplicaSetOffset(String rsName, List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> events)
void	setupDatabase()
void	shouldConsolidateOffsetsFromRsMode()
void	shouldFailToConsolidateOffsetsFromRsModeWhenInvalidationIsNotAllowed()
void	shouldFailToConsolidateOffsetsFromRsModeWhenOneShardOffsetIsMissing()
void	shouldUseOffsetsFromShardedMode()
private void	storeReplicaSetModeOffsets(List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> shard0, List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> shard1)
private void	storeShardedModeOffsets(List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> events)

Methods inherited from class io.debezium.connector.mongodb.AbstractShardedMongoConnectorIT

afterAll, afterEach, beforeAll, beforeEach, connect, connect, insertDocuments, insertDocumentsInTx, shardedCollection, shardedCollections, shardedDatabase, verifyFromInitialSnapshot, verifyNotFromInitialSnapshot, verifyOperation

Methods inherited from class io.debezium.embedded.AbstractConnectorTest

assertBeginTransaction, assertConfigurationErrors, assertConfigurationErrors, assertConfigurationErrors, assertConnectorIsRunning, assertConnectorNotRunning, assertDelete, assertEndTransaction, assertEngineIsRunning, assertHasNoSourceQuery, assertInsert, assertKey, assertNoConfigurationErrors, assertNoRecordsToConsume, assertOffset, assertOffset, assertOnlyTransactionRecordsToConsume, assertRecordTransactionMetadata, assertSchemaMatchesStruct, assertSchemaMatchesStruct, assertSourceQuery, assertTombstone, assertTombstone, assertUpdate, assertValueField, configValue, consumeAvailableRecords, consumeAvailableRecordsByTopic, consumeDmlRecordsByTopic, consumeDmlRecordsByTopic, consumeDmlRecordsByTopic, consumeRecord, consumeRecords, consumeRecords, consumeRecords, consumeRecordsButSkipUntil, consumeRecordsByTopic, consumeRecordsByTopic, consumeRecordsByTopic, consumeRecordsByTopicUntil, consumeRecordsUntil, createEngine, createEngineBuilder, debug, getConsumer, getMaximumEnqueuedRecordCount, getSnapshotMetricsObjectName, getSnapshotMetricsObjectName, getSnapshotMetricsObjectName, getStreamingMetricsObjectName, getStreamingMetricsObjectName, getStreamingMetricsObjectName, getStreamingMetricsObjectName, getStreamingNamespace, initializeConnectorTestFramework, isStreamingRunning, isStreamingRunning, isStreamingRunning, isStreamingRunning, isTransactionRecord, loggingCompletion, print, readLastCommittedOffset, readLastCommittedOffsets, setConsumeTimeout, skipAvroValidation, start, start, start, start, start, start, start, startAndConsumeTillEnd, startAndConsumeTillEnd, stopConnector, stopConnector, storeOffsets, validate, waitForAvailableRecords, waitForConnectorShutdown, waitForEngineShutdown, waitForSnapshotToBeCompleted, waitForSnapshotToBeCompleted, waitForSnapshotWithCustomMetricsToBeCompleted, waitForStreamingRunning, waitForStreamingRunning, waitForStreamingRunning, waitForStreamingWithCustomMetricsToStart, waitTimeForEngine, waitTimeForRecords, waitTimeForRecordsAfterNulls

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.debezium.util.Testing

once, once, resetBeforeEachTest, time, time

Field Details

TOPIC_PREFIX

public static final String TOPIC_PREFIX

See Also:

• Constant Field Values

EVENT_COMPARATOR

private static final Comparator<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> EVENT_COMPARATOR

Compares change event by cluster time

config

public Configuration config

connectorConfig

private io.debezium.connector.mongodb.MongoDbConnectorConfig connectorConfig

allRouterEvents

private final SortedSet<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> allRouterEvents

Contains all events consumed from router (always at least 4)

shardEvents0

private final List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> shardEvents0

Guaranteed to contain all consumed from shard0 (always at least 2)

shardEvents1

private final List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> shardEvents1

Guaranteed to contain all consumed from shard1 (always at least 2)

Constructor Details

OffsetConsolidationShardedIT

public OffsetConsolidationShardedIT()

Method Details

setupDatabase

public void setupDatabase()

insertInitialDocuments

public void insertInitialDocuments(io.debezium.connector.mongodb.ChangeStreamPipeline pipeline)

storeReplicaSetModeOffsets

private void storeReplicaSetModeOffsets(List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> shard0,
 List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> shard1)
                                 throws InterruptedException

Throws:

InterruptedException

storeShardedModeOffsets

private void storeShardedModeOffsets(List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> events)
                              throws InterruptedException

Throws:

InterruptedException

prepareReplicaSetOffset

private Map<Map<String,?>,Map<String,?>> prepareReplicaSetOffset(String rsName,
 List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> events)

nextToLastElement

public static <T> T nextToLastElement(List<T> collection)

getOffsetEvent

public com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument> getOffsetEvent(List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> shard0,
 List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> shard1)

Gets the older out of the two next-to-last events consumed by shard specific change streams.

Returns:

offset event

getOffsetEvent

public com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument> getOffsetEvent(List<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> events)

Gets next-to-last event consumed by router change streams.

Returns:

offset event

getExpectedIds

public Set<String> getExpectedIds(com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument> offsetEvent)

getExpectedIds

public Set<String> getExpectedIds(Collection<com.mongodb.client.model.changestream.ChangeStreamDocument<org.bson.BsonDocument>> events)

shouldConsolidateOffsetsFromRsMode

public void shouldConsolidateOffsetsFromRsMode()
                                        throws InterruptedException

Throws:

InterruptedException

shouldUseOffsetsFromShardedMode

public void shouldUseOffsetsFromShardedMode()
                                     throws InterruptedException

Throws:

InterruptedException

shouldFailToConsolidateOffsetsFromRsModeWhenInvalidationIsNotAllowed

public void shouldFailToConsolidateOffsetsFromRsModeWhenInvalidationIsNotAllowed()
                                                                          throws InterruptedException

Throws:

InterruptedException

shouldFailToConsolidateOffsetsFromRsModeWhenOneShardOffsetIsMissing

public void shouldFailToConsolidateOffsetsFromRsModeWhenOneShardOffsetIsMissing()
                                                                         throws InterruptedException

Throws:

InterruptedException