Package io.debezium.connector.base

Class ChangeEventQueue<T extends Sizeable>

java.lang.Object

io.debezium.connector.base.ChangeEventQueue<T>

Type Parameters:

T - the type of events in this queue. Usually Sizeable is used, but in cases where additional metadata must be passed from producers to the consumer, a custom type extending source records may be used.

All Implemented Interfaces:

ChangeEventQueueMetrics

@ThreadSafe
public class ChangeEventQueue<T extends Sizeable>
extends Object
implements ChangeEventQueueMetrics

A queue which serves as handover point between producer threads (e.g. MySQL's binlog reader thread) and the Kafka Connect polling loop.

The queue is configurable in different aspects, e.g. its maximum size and the time to sleep (block) between two subsequent poll calls. See the ChangeEventQueue.Builder for the different options. The queue applies back-pressure semantics, i.e. if it holds the maximum number of elements, subsequent calls to enqueue(T) will block until elements have been removed from the queue.

If an exception occurs on the producer side, the producer should make that exception known by calling producerException(RuntimeException) before stopping its operation. Upon the next call to poll(), that exception will be raised, causing Kafka Connect to stop the connector and mark it as FAILED.

Author:

Gunnar Morling

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ChangeEventQueue.Builder<T extends Sizeable>

Field Summary

Fields

Modifier and Type	Field	Description
private AtomicReference<T>	bufferedEvent
private boolean	buffering
private long	currentQueueSizeInBytes
private final Condition	isFull
private final Condition	isNotFull
private final Lock	lock
private static final org.slf4j.Logger	LOGGER
private final Supplier<LoggingContext.PreviousContext>	loggingContextSupplier
private final int	maxBatchSize
private final int	maxQueueSize
private final long	maxQueueSizeInBytes
private final Duration	pollInterval
private RuntimeException	producerException
private final Queue<T>	queue
private final Queue<Long>	sizeInBytesQueue

Constructor Summary

Constructors

Modifier	Constructor	Description
private	ChangeEventQueue(Duration pollInterval, int maxQueueSize, int maxBatchSize, Supplier<LoggingContext.PreviousContext> loggingContextSupplier, long maxQueueSizeInBytes, boolean buffering)

Method Summary

Modifier and Type	Method	Description
long	currentQueueSizeInBytes()
void	disableBuffering()	Disable buffering for the queue
protected void	doEnqueue(T record)
private long	drainRecords(List<T> records, int maxElements)
void	enqueue(T record)	Enqueues a record so that it can be obtained via poll().
void	flushBuffer(Function<T,T> recordModifier)	Applies a function to the event and the buffer and adds it to the queue.
long	maxQueueSizeInBytes()
List<T>	poll()	Returns the next batch of elements from this queue.
void	producerException(RuntimeException producerException)
int	remainingCapacity()
private void	throwProducerExceptionIfPresent()
int	totalCapacity()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

pollInterval

private final Duration pollInterval

maxBatchSize

private final int maxBatchSize

maxQueueSize

private final int maxQueueSize

maxQueueSizeInBytes

private final long maxQueueSizeInBytes

lock

private final Lock lock

isFull

private final Condition isFull

isNotFull

private final Condition isNotFull

queue

private final Queue<T extends Sizeable> queue

loggingContextSupplier

private final Supplier<LoggingContext.PreviousContext> loggingContextSupplier

sizeInBytesQueue

private final Queue<Long> sizeInBytesQueue

currentQueueSizeInBytes

private long currentQueueSizeInBytes

buffering

private boolean buffering

bufferedEvent

private AtomicReference<T extends Sizeable> bufferedEvent

producerException

private volatile RuntimeException producerException

Constructor Details

ChangeEventQueue

private ChangeEventQueue(Duration pollInterval,
 int maxQueueSize,
 int maxBatchSize,
 Supplier<LoggingContext.PreviousContext> loggingContextSupplier,
 long maxQueueSizeInBytes,
 boolean buffering)

Method Details

enqueue

public void enqueue(T record)
             throws InterruptedException

Enqueues a record so that it can be obtained via poll(). This method will block if the queue is full.

Parameters:

record - the record to be enqueued

Throws:

InterruptedException - if this thread has been interrupted

flushBuffer

public void flushBuffer(Function<T,T> recordModifier)
                 throws InterruptedException

Applies a function to the event and the buffer and adds it to the queue. Buffer is emptied.

Parameters:

recordModifier -

Throws:

InterruptedException

disableBuffering

public void disableBuffering()

Disable buffering for the queue

doEnqueue

protected void doEnqueue(T record)
                  throws InterruptedException

Throws:

InterruptedException

poll

public List<T> poll()
             throws InterruptedException

Returns the next batch of elements from this queue. May be empty in case no elements have arrived in the maximum waiting time.

Throws:

InterruptedException - if this thread has been interrupted while waiting for more elements to arrive

drainRecords

private long drainRecords(List<T> records,
 int maxElements)

producerException

public void producerException(RuntimeException producerException)

throwProducerExceptionIfPresent

private void throwProducerExceptionIfPresent()

totalCapacity

public int totalCapacity()

Specified by:

totalCapacity in interface ChangeEventQueueMetrics

remainingCapacity

public int remainingCapacity()

Specified by:

remainingCapacity in interface ChangeEventQueueMetrics

maxQueueSizeInBytes

public long maxQueueSizeInBytes()

Specified by:

maxQueueSizeInBytes in interface ChangeEventQueueMetrics

currentQueueSizeInBytes

public long currentQueueSizeInBytes()

Specified by:

currentQueueSizeInBytes in interface ChangeEventQueueMetrics