io.debezium.connector.base

Class ChangeEventQueue<T>

java.lang.Object

io.debezium.connector.base.ChangeEventQueue<T>

Type Parameters:

T - the type of events in this queue. Usually SourceRecord is used, but in cases where additional metadata must be passed from producers to the consumer, a custom type wrapping source records may be used.

All Implemented Interfaces:

ChangeEventQueueMetrics

public class ChangeEventQueue<T>
extends Object
implements ChangeEventQueueMetrics

A queue which serves as handover point between producer threads (e.g. MySQL's binlog reader thread) and the Kafka Connect polling loop.

The queue is configurable in different aspects, e.g. its maximum size and the time to sleep (block) between two subsequent poll calls. See the ChangeEventQueue.Builder for the different options. The queue applies back-pressure semantics, i.e. if it holds the maximum number of elements, subsequent calls to enqueue(Object) will block until elements have been removed from the queue.

If an exception occurs on the producer side, the producer should make that exception known by calling #producerFailure before stopping its operation. Upon the next call to poll(), that exception will be raised, causing Kafka Connect to stop the connector and mark it as FAILED.

Author:

Gunnar Morling

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ChangeEventQueue.Builder<T>

Field Summary

Fields

Modifier and Type	Field and Description
private static org.slf4j.Logger	LOGGER
private Supplier<LoggingContext.PreviousContext>	loggingContextSupplier
private int	maxBatchSize
private int	maxQueueSize
private Metronome	metronome
private Duration	pollInterval
private RuntimeException	producerException
private BlockingQueue<T>	queue

Constructor Summary

Constructors

Modifier	Constructor and Description
private	ChangeEventQueue(Duration pollInterval, int maxQueueSize, int maxBatchSize, Supplier<LoggingContext.PreviousContext> loggingContextSupplier)

Method Summary

Modifier and Type	Method and Description
void	enqueue(T record) Enqueues a record so that it can be obtained via poll().
List<T>	poll() Returns the next batch of elements from this queue.
void	producerException(RuntimeException producerException)
int	remainingCapacity()
private void	throwProducerExceptionIfPresent()
int	totalCapacity()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

pollInterval

private final Duration pollInterval

maxBatchSize

private final int maxBatchSize

maxQueueSize

private final int maxQueueSize

queue

private final BlockingQueue<T> queue

metronome

private final Metronome metronome

loggingContextSupplier

private final Supplier<LoggingContext.PreviousContext> loggingContextSupplier

producerException

private volatile RuntimeException producerException

Constructor Detail

ChangeEventQueue

private ChangeEventQueue(Duration pollInterval,
                         int maxQueueSize,
                         int maxBatchSize,
                         Supplier<LoggingContext.PreviousContext> loggingContextSupplier)

Method Detail

enqueue

public void enqueue(T record)
             throws InterruptedException

Enqueues a record so that it can be obtained via poll(). This method will block if the queue is full.

Parameters:

record - the record to be enqueued

Throws:

InterruptedException - if this thread has been interrupted

poll

public List<T> poll()
             throws InterruptedException

Returns the next batch of elements from this queue. May be empty in case no elements have arrived in the maximum waiting time.

Throws:

InterruptedException - if this thread has been interrupted while waiting for more elements to arrive

producerException

public void producerException(RuntimeException producerException)

throwProducerExceptionIfPresent

private void throwProducerExceptionIfPresent()

totalCapacity

public int totalCapacity()

Specified by:

totalCapacity in interface ChangeEventQueueMetrics

remainingCapacity

public int remainingCapacity()

Specified by:

remainingCapacity in interface ChangeEventQueueMetrics