Execute an effect for each record and commit the offset after processing

This method is the easiest way of processing messages on a Kafka topic.

Messages on a single partition are processed sequentially, while the processing of multiple partitions happens in parallel.

Offsets are committed after execution of the effect. They are batched when a commit action is in progress to avoid backpressuring the stream. When commits fail due to a org.apache.kafka.clients.consumer.RetriableCommitFailedException they are retried according to commitRetryPolicy

The effect should absorb any failures. Failures should be handled by retries or ignoring the error, which will result in the Kafka message being skipped.

Messages are processed with 'at least once' consistency: it is not guaranteed that every message that is processed by the effect has a corresponding offset commit before stream termination.

Usage example:

val settings: ConsumerSettings = ???
val subscription = Subscription.Topics(Set("my-kafka-topic"))

val consumerIO = Consumer.consumeWith(settings, subscription, Serdes.string, Serdes.string) { record =>
  // Process the received record here
  putStrLn(s"Received record: ${record.key()}: ${record.value()}")
}

Create a zio-kafka Consumer from an org.apache.kafka KafkaConsumer

You are responsible for creating and closing the KafkaConsumer