Package io.debezium.connector.binlog

Class EventBuffer<T extends BinlogStreamingChangeEventSource<P,O>,P extends BinlogPartition,O extends BinlogOffsetContext>

java.lang.Object

io.debezium.connector.binlog.EventBuffer<T,P,O>

public class EventBuffer<T extends BinlogStreamingChangeEventSource<P,O>,P extends BinlogPartition,O extends BinlogOffsetContext>
extends Object

This class represents a look-ahead buffer that allows Debezium to accumulate binlog events and decide if the last event in transaction is either ROLLBACK or COMMIT. The incoming events are either supposed to be in transaction or out-of-transaction. When out-of-transaction they are sent directly into the destination handler. When in transaction the event goes through the buffering.

The reason for the buffering is that the binlog contains rolled back transactions in some cases. E.g. that's the case when a temporary table is dropped (see DBZ-390). For rolled back transactions we may not propagate any of the contained events, hence the buffering is applied.

The transaction start is identified by a BEGIN event. Transaction is ended either by COMMIT event or by XID an event.

If there are more events that can fit to the buffer then:

• Binlog position is recorded for the first event not fitting into the buffer
• Binlog position is recorded for the commit event
• Buffer content is sent to the final handler
• Binlog position is rewound and all events between the above recorded positions are sent to the final handler

Author:

Jiri Pechanec, Chris Cranford

Field Summary

Fields

Modifier and Type	Field	Description
private final Queue<com.github.shyiko.mysql.binlog.event.Event>	buffer
private final int	capacity
private final ChangeEventSource.ChangeEventSourceContext	changeEventSourceContext
private BinlogStreamingChangeEventSource.BinlogPosition	forwardTillPosition	Contains the position of the last event belonging to the transaction that has not fit into the buffer.
private BinlogStreamingChangeEventSource.BinlogPosition	largeTxNotBufferedPosition	Contains the position of the first event that has not fit into the buffer.
private static final org.slf4j.Logger	LOGGER
private final T	streamingChangeEventSource
private boolean	txStarted

Constructor Summary

Constructors

Constructor	Description
EventBuffer(int capacity, T streamingChangeEventSource, ChangeEventSource.ChangeEventSourceContext changeEventSourceContext)

Method Summary

Modifier and Type	Method	Description
void	add(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event)	An entry point to the buffer that should be used by the streaming change event source to push events.
private void	addToBuffer(com.github.shyiko.mysql.binlog.event.Event event)	Adds an event to the buffer if there is a space available.
private void	beginTransaction(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event)	Starts the transaction
private void	clear()	Cleans-up the buffer after the transaction is either thrown away or streamed into a Kafka topic
private void	completeTransaction(P partition, O offsetContext, boolean wellFormed, com.github.shyiko.mysql.binlog.event.Event event)	Sends all events from the buffer int a final handler.
private void	consumeEvent(P partition, O offsetContext, com.github.shyiko.mysql.binlog.event.Event event)	Consumes the specific event.
private boolean	isInBufferFullMode()
private boolean	isReplayingEventsBeyondBufferCapacity()	Whether we are replaying TX events from binlog that have not fit into the buffer before
private void	rollbackTransaction()	Rollback the transaction
private void	switchToBufferFullMode()	Switches the buffer to operating in full-mode.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

capacity

private final int capacity

buffer

private final Queue<com.github.shyiko.mysql.binlog.event.Event> buffer

streamingChangeEventSource

private final T extends BinlogStreamingChangeEventSource<P,O> streamingChangeEventSource

changeEventSourceContext

private final ChangeEventSource.ChangeEventSourceContext changeEventSourceContext

txStarted

private boolean txStarted

largeTxNotBufferedPosition

private BinlogStreamingChangeEventSource.BinlogPosition largeTxNotBufferedPosition

Contains the position of the first event that has not fit into the buffer.

forwardTillPosition

private BinlogStreamingChangeEventSource.BinlogPosition forwardTillPosition

Contains the position of the last event belonging to the transaction that has not fit into the buffer.

Constructor Details

EventBuffer

public EventBuffer(int capacity,
 T streamingChangeEventSource,
 ChangeEventSource.ChangeEventSourceContext changeEventSourceContext)

Method Details

add

public void add(P partition,
 O offsetContext,
 com.github.shyiko.mysql.binlog.event.Event event)

An entry point to the buffer that should be used by the streaming change event source to push events.

Parameters:

partition - the partition to which the events belong

offsetContext - the offset context

event - event to be stored in the buffer

isReplayingEventsBeyondBufferCapacity

private boolean isReplayingEventsBeyondBufferCapacity()

Whether we are replaying TX events from binlog that have not fit into the buffer before

addToBuffer

private void addToBuffer(com.github.shyiko.mysql.binlog.event.Event event)

Adds an event to the buffer if there is a space available. Records binlog position for the first event that does not fit for later replay.

Parameters:

event - the event

switchToBufferFullMode

private void switchToBufferFullMode()

Switches the buffer to operating in full-mode.

isInBufferFullMode

private boolean isInBufferFullMode()

Returns:

true if the buffer is operating in full mode; false otherwise

consumeEvent

private void consumeEvent(P partition,
 O offsetContext,
 com.github.shyiko.mysql.binlog.event.Event event)

Consumes the specific event.

Parameters:

partition - the partition where the event was committed

offsetContext - the offset context

event - the event

beginTransaction

private void beginTransaction(P partition,
 O offsetContext,
 com.github.shyiko.mysql.binlog.event.Event event)

Starts the transaction

Parameters:

partition - the partition where the transaction was committed

offsetContext - the offset context

event - the event

completeTransaction

private void completeTransaction(P partition,
 O offsetContext,
 boolean wellFormed,
 com.github.shyiko.mysql.binlog.event.Event event)

Sends all events from the buffer int a final handler. For large transactions it executes rewind of binlog reader back to the first event that was not stored in the buffer.

Parameters:

partition - the partition where the transaction was committed

offsetContext - the offset context

wellFormed - true if the transaction is well-formed, false otherwise

event - the event

rollbackTransaction

private void rollbackTransaction()

Rollback the transaction

clear

private void clear()

Cleans-up the buffer after the transaction is either thrown away or streamed into a Kafka topic