io.debezium.connector.mysql

Class MySqlConnectorTask

java.lang.Object

org.apache.kafka.connect.source.SourceTask

io.debezium.connector.common.BaseSourceTask

io.debezium.connector.mysql.MySqlConnectorTask

All Implemented Interfaces:

org.apache.kafka.connect.connector.Task

@NotThreadSafe
public final class MySqlConnectorTask
extends BaseSourceTask

A Kafka Connect source task reads the MySQL binary log and generate the corresponding data change events.

Author:

Randall Hauch

See Also:

MySqlConnector

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	MySqlConnectorTask.ServerIdGenerator

Field Summary

Fields

Modifier and Type	Field and Description
private MySqlJdbcContext	connectionContext
private org.slf4j.Logger	logger
private ChainedReader	readers
private MySqlTaskContext	taskContext

Fields inherited from class org.apache.kafka.connect.source.SourceTask

context

Constructor Summary

Constructors

Constructor and Description
MySqlConnectorTask() Create an instance of the log reader that uses Kafka to store database schema history and the default topic selector of "<serverName>.<databaseName>.<tableName>" for data and "<serverName>" for metadata.

Method Summary

Modifier and Type	Method and Description
protected void	completeReaders() When the task is stopped, the readers may have additional work to perform before they actually stop and before all their records have been consumed via the poll() method.
private static MySqlTaskContext	createAndStartTaskContext(Configuration config, Filters filters)
protected String	earliestBinlogFilename() Determine the earliest binlog filename that is still available in the server.
protected Iterable<Field>	getAllConfigurationFields()
private static Filters	getAllFilters(Configuration config) Get the filters representing all tables represented by the config.
private static Filters	getNewFilters(Map<String,?> offsets, Configuration config) Get the filters representing the tables that have been newly added to the config, but not those that previously existed in the config.
private static Filters	getOldFilters(Map<String,?> offsets, Configuration config) Get the filters representing those tables that previously existed in the config, but not those newly added to the config.
private Map<String,?>	getRestartOffset(Map<String,?> storedOffset) Get the offset to restart the connector from.
protected boolean	isBinlogAvailable() Determine whether the binlog position as set on the SourceInfo is available in the server.
protected boolean	isRowBinlogEnabled() Determine whether the MySQL server has the row-level binlog enabled.
private boolean	newTablesInConfig()
List<org.apache.kafka.connect.source.SourceRecord>	poll()
void	start(Configuration config)
void	stop()
String	version()

Methods inherited from class io.debezium.connector.common.BaseSourceTask

getPreviousOffset, start

Methods inherited from class org.apache.kafka.connect.source.SourceTask

commit, commitRecord, initialize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

logger

private final org.slf4j.Logger logger

taskContext

private volatile MySqlTaskContext taskContext

connectionContext

private volatile MySqlJdbcContext connectionContext

readers

private volatile ChainedReader readers

Constructor Detail

MySqlConnectorTask

public MySqlConnectorTask()

Create an instance of the log reader that uses Kafka to store database schema history and the default topic selector of "<serverName>.<databaseName>.<tableName>" for data and "<serverName>" for metadata.

Method Detail

version

public String version()

start

public void start(Configuration config)

Specified by:

start in class BaseSourceTask

getRestartOffset

private Map<String,?> getRestartOffset(Map<String,?> storedOffset)

Get the offset to restart the connector from. Normally, this is just the stored offset. However, if we were doing a parallel load with new tables, it's possible that the last committed offset is from reading the new tables, which could be beyond where we want to restart from (and restarting there could cause skipped events). To fix this, the new tables binlog reader records extra information in its offset to tell the connector where to restart from. If this extra information is present in the stored offset, that is the offset that is returned.

Parameters:

storedOffset - the stored offset.

Returns:

the offset to restart from.

See Also:

RecordMakers.RecordMakers(MySqlSchema, SourceInfo, TopicSelector, boolean, Map)

createAndStartTaskContext

private static MySqlTaskContext createAndStartTaskContext(Configuration config,
                                                          Filters filters)

newTablesInConfig

private boolean newTablesInConfig()

Returns:

true if new tables appear to have been added to the config, and false otherwise.

getNewFilters

private static Filters getNewFilters(Map<String,?> offsets,
                                     Configuration config)

Get the filters representing the tables that have been newly added to the config, but not those that previously existed in the config.

Returns:

Filters

getOldFilters

private static Filters getOldFilters(Map<String,?> offsets,
                                     Configuration config)

Get the filters representing those tables that previously existed in the config, but not those newly added to the config.

Returns:

Filters

getAllFilters

private static Filters getAllFilters(Configuration config)

Get the filters representing all tables represented by the config.

Returns:

Filters

poll

public List<org.apache.kafka.connect.source.SourceRecord> poll()
                                                        throws InterruptedException

Specified by:

poll in class org.apache.kafka.connect.source.SourceTask

Throws:

InterruptedException

stop

public void stop()

Specified by:

stop in interface org.apache.kafka.connect.connector.Task

Specified by:

stop in class org.apache.kafka.connect.source.SourceTask

getAllConfigurationFields

protected Iterable<Field> getAllConfigurationFields()

Specified by:

getAllConfigurationFields in class BaseSourceTask

completeReaders

protected void completeReaders()

When the task is stopped, the readers may have additional work to perform before they actually stop and before all their records have been consumed via the poll() method. This method signals that all of this has completed.

isBinlogAvailable

protected boolean isBinlogAvailable()

Determine whether the binlog position as set on the SourceInfo is available in the server.

Returns:

true if the server has the binlog coordinates, or false otherwise

earliestBinlogFilename

protected String earliestBinlogFilename()

Determine the earliest binlog filename that is still available in the server.

Returns:

the name of the earliest binlog filename, or null if there are none.

isRowBinlogEnabled

protected boolean isRowBinlogEnabled()

Determine whether the MySQL server has the row-level binlog enabled.

Returns:

true if the server's binlog_format is set to ROW, or false otherwise