io.debezium.kafka

Class KafkaCluster

java.lang.Object

io.debezium.kafka.KafkaCluster

@ThreadSafe
public class KafkaCluster
extends Object

An embeddable cluster of Kafka servers and a single Zookeeper server. This may be useful when creating a complete environment within a single process, but doing so offers limited durability and fault tolerance compared to the normal approach of using an external cluster of Kafka servers and Zookeeper servers with proper replication and fault tolerance.

Author:

Randall Hauch

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	KafkaCluster.InteractiveConsumer<K,V> A simple interactive Kafka consumer for use with the cluster.
static interface	KafkaCluster.InteractiveProducer<K,V> A simple interactive Kafka producer for use with the cluster.
class	KafkaCluster.Usage A set of methods to use a running KafkaCluster.

Field Summary

Fields

Modifier and Type	Field and Description
private File	dataDir
static boolean	DEFAULT_DELETE_DATA_PRIOR_TO_STARTUP
static boolean	DEFAULT_DELETE_DATA_UPON_SHUTDOWN
private boolean	deleteDataPriorToStartup
private boolean	deleteDataUponShutdown
private Properties	kafkaConfig
private ConcurrentMap<Integer,KafkaServer>	kafkaServers
private static org.slf4j.Logger	LOGGER
private AtomicLong	nextKafkaPort
private boolean	running
private int	startingKafkaPort
private ZookeeperServer	zkServer

Constructor Summary

Constructors

Constructor and Description
KafkaCluster() Create a new embedded cluster.

Method Summary

Modifier and Type	Method and Description
KafkaCluster	addBrokers(int count) Add a number of new Kafka broker to the cluster.
String	brokerList() Get the list of brokers.
void	createTopic(String topic, int numPartitions, int replicationFactor) Create the specified topic.
void	createTopics(int numPartitions, int replicationFactor, Set<String> topics) Create the specified topics.
void	createTopics(int numPartitions, int replicationFactor, String... topics) Create the specified topics.
void	createTopics(Set<String> topics) Create the specified topics.
void	createTopics(String... topics) Create the specified topics.
KafkaCluster	deleteDataPriorToStartup(boolean delete) Specify whether the data is to be deleted prior to startup().
KafkaCluster	deleteDataUponShutdown(boolean delete) Specify whether the data is to be deleted upon shutdown().
boolean	isRunning() Determine if the cluster is running.
(package private) void	onEachDirectory(Consumer<File> consumer) Perform the supplied function on each directory used by this cluster.
KafkaCluster	shutdown() Shutdown the embedded Zookeeper server and the Kafka servers in the cluster.
private void	shutdownReliably(KafkaServer server)
KafkaCluster	startup() Start the embedded Zookeeper server and the Kafka servers in the cluster.
KafkaCluster.Usage	useTo() Obtain the interface for using this cluster.
KafkaCluster	usingDirectory(File dataDir) Set the parent directory where the brokers logs and server's logs and snapshots will be kept.
KafkaCluster	withKafkaConfiguration(Properties properties) Set the configuration properties for each of the brokers.
KafkaCluster	withPorts(int zkPort, int firstKafkaPort) Set the port numbers for Zookeeper and the Kafka brokers.
int	zkPort() Get the Zookeeper port.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_DELETE_DATA_UPON_SHUTDOWN

public static final boolean DEFAULT_DELETE_DATA_UPON_SHUTDOWN

See Also:

Constant Field Values

DEFAULT_DELETE_DATA_PRIOR_TO_STARTUP

public static final boolean DEFAULT_DELETE_DATA_PRIOR_TO_STARTUP

See Also:

Constant Field Values

LOGGER

private static final org.slf4j.Logger LOGGER

kafkaServers

private final ConcurrentMap<Integer,KafkaServer> kafkaServers

zkServer

private final ZookeeperServer zkServer

dataDir

private volatile File dataDir

deleteDataUponShutdown

private volatile boolean deleteDataUponShutdown

deleteDataPriorToStartup

private volatile boolean deleteDataPriorToStartup

running

private volatile boolean running

kafkaConfig

private volatile Properties kafkaConfig

startingKafkaPort

private volatile int startingKafkaPort

nextKafkaPort

private final AtomicLong nextKafkaPort

Constructor Detail

KafkaCluster

public KafkaCluster()

Create a new embedded cluster.

Method Detail

deleteDataUponShutdown

public KafkaCluster deleteDataUponShutdown(boolean delete)

Specify whether the data is to be deleted upon shutdown().

Parameters:

delete - true if the data is to be deleted upon shutdown, or false otherwise

Returns:

this instance to allow chaining methods; never null

Throws:

IllegalStateException - if the cluster is running

deleteDataPriorToStartup

public KafkaCluster deleteDataPriorToStartup(boolean delete)

Specify whether the data is to be deleted prior to startup().

Parameters:

delete - true if the data is to be deleted upon shutdown, or false otherwise

Returns:

this instance to allow chaining methods; never null

Throws:

IllegalStateException - if the cluster is running

addBrokers

public KafkaCluster addBrokers(int count)

Add a number of new Kafka broker to the cluster. The broker IDs will be generated.

Parameters:

count - the number of new brokers to add

Returns:

this instance to allow chaining methods; never null

Throws:

IllegalStateException - if the cluster is running

usingDirectory

public KafkaCluster usingDirectory(File dataDir)

Set the parent directory where the brokers logs and server's logs and snapshots will be kept.

Parameters:

dataDir - the parent directory for the server's logs and snapshots; may be null if a temporary directory will be used

Returns:

this instance to allow chaining methods; never null

Throws:

IllegalStateException - if the cluster is running

IllegalArgumentException - if the supplied file is not a directory or not writable

withKafkaConfiguration

public KafkaCluster withKafkaConfiguration(Properties properties)

Set the configuration properties for each of the brokers. This method does nothing if the supplied properties are null or empty.

Parameters:

properties - the Kafka configuration properties

Returns:

this instance to allow chaining methods; never null

Throws:

IllegalStateException - if the cluster is running

withPorts

public KafkaCluster withPorts(int zkPort,
                              int firstKafkaPort)

Set the port numbers for Zookeeper and the Kafka brokers.

Parameters:

zkPort - the port number that Zookeeper should use; may be -1 if an available port should be discovered

firstKafkaPort - the port number for the first Kafka broker (additional brokers will use subsequent port numbers); may be -1 if available ports should be discovered

Returns:

this instance to allow chaining methods; never null

Throws:

IllegalStateException - if the cluster is running

isRunning

public boolean isRunning()

Determine if the cluster is running.

Returns:

true if the cluster is running, or false otherwise

startup

public KafkaCluster startup()
                     throws IOException

Start the embedded Zookeeper server and the Kafka servers in the cluster. This method does nothing if the cluster is already running.

Returns:

this instance to allow chaining methods; never null

Throws:

IOException - if there is an error during startup

shutdown

public KafkaCluster shutdown()

Shutdown the embedded Zookeeper server and the Kafka servers in the cluster. This method does nothing if the cluster is not running.

Returns:

this instance to allow chaining methods; never null

createTopics

public void createTopics(String... topics)

Create the specified topics.

Parameters:

topics - the names of the topics to create

Throws:

IllegalStateException - if the cluster is not running

createTopics

public void createTopics(Set<String> topics)

Create the specified topics.

Parameters:

topics - the names of the topics to create

Throws:

IllegalStateException - if the cluster is not running

createTopics

public void createTopics(int numPartitions,
                         int replicationFactor,
                         String... topics)

Create the specified topics.

Parameters:

numPartitions - the number of partitions for each topic

replicationFactor - the replication factor for each topic

topics - the names of the topics to create

createTopics

public void createTopics(int numPartitions,
                         int replicationFactor,
                         Set<String> topics)

Create the specified topics.

Parameters:

numPartitions - the number of partitions for each topic

replicationFactor - the replication factor for each topic

topics - the names of the topics to create

createTopic

public void createTopic(String topic,
                        int numPartitions,
                        int replicationFactor)

Create the specified topic.

Parameters:

topic - the name of the topic to create

numPartitions - the number of partitions for the topic

replicationFactor - the replication factor for the topic

onEachDirectory

void onEachDirectory(Consumer<File> consumer)

Perform the supplied function on each directory used by this cluster.

Parameters:

consumer - the consumer function; may not be null

brokerList

public String brokerList()

Get the list of brokers.

Returns:

the broker list

zkPort

public int zkPort()

Get the Zookeeper port.

Returns:

the Zookeeper port

shutdownReliably

private void shutdownReliably(KafkaServer server)

useTo

public KafkaCluster.Usage useTo()

Obtain the interface for using this cluster.

Returns:

the usage interface; never null

Throws:

IllegalStateException - if the cluster is not running