Package io.debezium.kafka

Class KafkaCluster

java.lang.Object
io.debezium.kafka.KafkaCluster
@ThreadSafe public class KafkaCluster extends Object
An embeddable cluster of Kafka servers and a single Zookeeper server. This may be useful when creating a complete environment within a single process, but doing so offers limited durability and fault tolerance compared to the normal approach of using an external cluster of Kafka servers and Zookeeper servers with proper replication and fault tolerance.
Author:
Randall Hauch

  • Field Details

    • DEFAULT_DELETE_DATA_UPON_SHUTDOWN

      public static final boolean DEFAULT_DELETE_DATA_UPON_SHUTDOWN
      See Also:

    • DEFAULT_DELETE_DATA_PRIOR_TO_STARTUP

      public static final boolean DEFAULT_DELETE_DATA_PRIOR_TO_STARTUP
      See Also:

    • LOGGER

      private static final org.slf4j.Logger LOGGER

    • kafkaServers

      private final ConcurrentMap<Integer,KafkaServer> kafkaServers

    • zkServer

      private final ZookeeperServer zkServer

    • dataDir

      private volatile File dataDir

    • deleteDataUponShutdown

      private volatile boolean deleteDataUponShutdown

    • deleteDataPriorToStartup

      private volatile boolean deleteDataPriorToStartup

    • running

      private volatile boolean running

    • kafkaConfig

      private volatile Properties kafkaConfig

    • startingKafkaPort

      private volatile int startingKafkaPort

    • nextKafkaPort

      private final AtomicLong nextKafkaPort

  • Constructor Details

    • KafkaCluster

      public KafkaCluster()
      Create a new embedded cluster.

  • Method Details

    • deleteDataUponShutdown

      public KafkaCluster deleteDataUponShutdown(boolean delete)
      Specify whether the data is to be deleted upon shutdown().
      Parameters:
      delete - true if the data is to be deleted upon shutdown, or false otherwise
      Returns:
      this instance to allow chaining methods; never null
      Throws:
      IllegalStateException - if the cluster is running

    • deleteDataPriorToStartup

      public KafkaCluster deleteDataPriorToStartup(boolean delete)
      Specify whether the data is to be deleted prior to startup().
      Parameters:
      delete - true if the data is to be deleted upon shutdown, or false otherwise
      Returns:
      this instance to allow chaining methods; never null
      Throws:
      IllegalStateException - if the cluster is running

    • addBrokers

      public KafkaCluster addBrokers(int count)
      Add a number of new Kafka broker to the cluster. The broker IDs will be generated.
      Parameters:
      count - the number of new brokers to add
      Returns:
      this instance to allow chaining methods; never null
      Throws:
      IllegalStateException - if the cluster is running

    • usingDirectory

      public KafkaCluster usingDirectory(File dataDir)
      Set the parent directory where the brokers logs and server's logs and snapshots will be kept.
      Parameters:
      dataDir - the parent directory for the server's logs and snapshots; may be null if a temporary directory will be used
      Returns:
      this instance to allow chaining methods; never null
      Throws:
      IllegalStateException - if the cluster is running
      IllegalArgumentException - if the supplied file is not a directory or not writable

    • withKafkaConfiguration

      public KafkaCluster withKafkaConfiguration(Properties properties)
      Set the configuration properties for each of the brokers. This method does nothing if the supplied properties are null or empty.
      Parameters:
      properties - the Kafka configuration properties
      Returns:
      this instance to allow chaining methods; never null
      Throws:
      IllegalStateException - if the cluster is running

    • withPorts

      public KafkaCluster withPorts(int zkPort, int firstKafkaPort)
      Set the port numbers for Zookeeper and the Kafka brokers.
      Parameters:
      zkPort - the port number that Zookeeper should use; may be -1 if an available port should be discovered
      firstKafkaPort - the port number for the first Kafka broker (additional brokers will use subsequent port numbers); may be -1 if available ports should be discovered
      Returns:
      this instance to allow chaining methods; never null
      Throws:
      IllegalStateException - if the cluster is running

    • isRunning

      public boolean isRunning()
      Determine if the cluster is running.
      Returns:
      true if the cluster is running, or false otherwise

    • startup

      public KafkaCluster startup() throws IOException
      Start the embedded Zookeeper server and the Kafka servers in the cluster. This method does nothing if the cluster is already running.
      Returns:
      this instance to allow chaining methods; never null
      Throws:
      IOException - if there is an error during startup

    • shutdown

      public KafkaCluster shutdown()
      Shutdown the embedded Zookeeper server and the Kafka servers in the cluster. This method does nothing if the cluster is not running.
      Returns:
      this instance to allow chaining methods; never null

    • createTopics

      public void createTopics(String... topics)
      Create the specified topics.
      Parameters:
      topics - the names of the topics to create
      Throws:
      IllegalStateException - if the cluster is not running

    • createTopics

      public void createTopics(Set<String> topics)
      Create the specified topics.
      Parameters:
      topics - the names of the topics to create
      Throws:
      IllegalStateException - if the cluster is not running

    • createTopics

      public void createTopics(int numPartitions, int replicationFactor, String... topics)
      Create the specified topics.
      Parameters:
      numPartitions - the number of partitions for each topic
      replicationFactor - the replication factor for each topic
      topics - the names of the topics to create

    • createTopics

      public void createTopics(int numPartitions, int replicationFactor, Set<String> topics)
      Create the specified topics.
      Parameters:
      numPartitions - the number of partitions for each topic
      replicationFactor - the replication factor for each topic
      topics - the names of the topics to create

    • createTopic

      public void createTopic(String topic, int numPartitions, int replicationFactor)
      Create the specified topic.
      Parameters:
      topic - the name of the topic to create
      numPartitions - the number of partitions for the topic
      replicationFactor - the replication factor for the topic

    • onEachDirectory

      void onEachDirectory(Consumer<File> consumer)
      Perform the supplied function on each directory used by this cluster.
      Parameters:
      consumer - the consumer function; may not be null

    • brokerList

      public String brokerList()
      Get the list of brokers.
      Returns:
      the broker list

    • zkPort

      public int zkPort()
      Get the Zookeeper port.
      Returns:
      the Zookeeper port

    • shutdownReliably

      private void shutdownReliably(KafkaServer server)

    • useTo

      public KafkaCluster.Usage useTo()
      Obtain the interface for using this cluster.
      Returns:
      the usage interface; never null
      Throws:
      IllegalStateException - if the cluster is not running