cascading.local.tap.kafka

Class KafkaTap<K,V>

java.lang.Object

cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>>

cascading.local.tap.kafka.KafkaTap<K,V>

All Implemented Interfaces:

cascading.flow.FlowElement, cascading.flow.planner.ScopedElement, cascading.util.Traceable, java.io.Serializable

Direct Known Subclasses:

KafkaCommittingTap

public class KafkaTap<K,V>
extends cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>>

Class KafkaTap is a Cascading local mode Tap providing read and write access to data stored in Apache Kafka topics.

This Tap is not intended to be used with any of the other Cascading planners unless they specify they are local mode compatible.

The Kafka producer and consumer interfaces are not Input/OutputStream based, subsequently the KafkaTap must be used with a KafkaScheme sub-class which encapsulates the mechanics of reading/writing from/to a Kafka topic.

Further Kafka tends to treat a topic as data vs a location, causing further coupling between the KafkaTap and KafkaScheme. This may be Kafka's actual intent, so this class may need to be revised, or an alternate implementation implemented.

Subsequently, the KafkaTap can be instantiated with a URI instance of the format: kafka://[hostname]<:port>/[topic]<,topic>.

Where hostname and at least one topic is required.

If the first topic is wrapped by slash (/), e.g. /some-.*-topic/, the string within the slashes will be considered a regular-expression. Any subsequent topics will be ignored.

Note on read, the KafkaTap will continue to retrieve data until the pollTimeout is reached, where the default is 10 seconds.

Use the defaultProperties argument to set Kafka Consumer/Producer specific properties.

By default, CONSUME_AUTO_COMMIT_EARLIEST with values:

• ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG is true
• ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG is 1000
• ConsumerConfig.AUTO_OFFSET_RESET_CONFIG is "earliest"

and PRODUCE_ACK_ALL_NO_RETRY with values:

• ProducerConfig.ACKS_CONFIG is "all
• ProducerConfig.RETRIES_CONFIG is 0

Note, calling @{code close()} on the ConsumerRecord Iterator from the Scheme.source(FlowProcess, SourceCall) method will close the Kafka consumer and notify the parent TupleEntrySchemeIterator to stop providing new values.

See Also:

CONSUME_AUTO_COMMIT_EARLIEST, CONSUME_AUTO_COMMIT_LATEST, PRODUCE_ACK_ALL_NO_RETRY, for conveniently merging Property instances, Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static java.util.Properties	CONSUME_AUTO_COMMIT_EARLIEST Field CONSUME_AUTO_COMMIT_EARLIEST
static java.util.Properties	CONSUME_AUTO_COMMIT_LATEST Field CONSUME_AUTO_COMMIT_LATEST
static int	DEFAULT_NUM_PARTITIONS Field DEFAULT_NUM_PARTITIONS
static long	DEFAULT_POLL_TIMEOUT Field DEFAULT_POLL_TIMEOUT
static short	DEFAULT_REPLICATION_FACTOR Field DEFAULT_REPLICATION_FACTOR
static java.util.Properties	PRODUCE_ACK_ALL_NO_RETRY Field PRODUCE_ACK_ALL_NO_RETRY

Constructor Summary

Constructors

Constructor and Description
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, long pollTimeout, int numPartitions, short replicationFactor, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, long pollTimeout, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, long pollTimeout, int numPartitions, short replicationFactor, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, long pollTimeout, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, int numPartitions, short replicationFactor) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, long pollTimeout) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, long pollTimeout, int numPartitions, short replicationFactor) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, int numPartitions, short replicationFactor) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, long pollTimeout) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, long pollTimeout, int numPartitions, short replicationFactor) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, java.lang.String groupID) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, int numPartitions, short replicationFactor, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, long pollTimeout, int numPartitions, short replicationFactor, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, long pollTimeout, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, int numPartitions, short replicationFactor, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, long pollTimeout, int numPartitions, short replicationFactor, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, long pollTimeout, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, java.lang.String... topics) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, java.lang.String groupID, long pollTimeout, int numPartitions, short replicationFactor, java.lang.String... topics)
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, int numPartitions, short replicationFactor) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, long pollTimeout) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, long pollTimeout, int numPartitions, short replicationFactor) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, int numPartitions, short replicationFactor) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, long pollTimeout) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, long pollTimeout, int numPartitions, short replicationFactor) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, java.lang.String groupID) Constructor KafkaTap creates a new KafkaTap instance.
KafkaTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, java.lang.String groupID, long pollTimeout, int numPartitions, short replicationFactor) Constructor KafkaTap creates a new KafkaTap instance.

Method Summary

Modifier and Type	Method and Description
protected void	applyTopics(java.lang.String[] topics)
protected org.apache.kafka.clients.admin.AdminClient	createAdminClient(java.util.Properties conf)
protected org.apache.kafka.clients.consumer.Consumer<K,V>	createKafkaConsumer(java.util.Properties properties)
boolean	createResource(java.util.Properties conf)
boolean	deleteResource(java.util.Properties conf)
java.lang.String	getClientID() Method getClientID returns the clientID of this KafkaTap object.
protected org.apache.kafka.clients.consumer.ConsumerRebalanceListener	getConsumerRebalanceListener(org.apache.kafka.clients.consumer.Consumer<K,V> consumer) Returns a NoOpConsumerRebalanceListener instance.
java.lang.String	getGroupID() Method getGroupID returns the groupID of this KafkaTap object.
java.lang.String	getHostname() Method getHostname returns the hostname of this KafkaTap object.
java.lang.String	getIdentifier()
long	getModifiedTime(java.util.Properties conf)
java.lang.String[]	getTopics() Method getTopics returns the topics of this KafkaTap object.
boolean	isTopicPattern() Method isTopicPattern returns true if the topic is a regular expression.
static java.net.URI	makeURI(java.lang.String hostname, java.lang.String... topics) Method makeURI creates a kafka URI for use with the KafkaTap.
cascading.tuple.TupleEntryIterator	openForRead(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, KafkaConsumerRecordIterator<K,V> consumerRecord)
cascading.tuple.TupleEntryCollector	openForWrite(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, org.apache.kafka.clients.producer.Producer<K,V> producer)
protected void	postConsumerSubscribe(org.apache.kafka.clients.consumer.Consumer<K,V> consumer) Prepare Consumer post to any topic subscription and prior to any Consumer.poll(long) request.
protected void	preConsumerSubscribe(org.apache.kafka.clients.consumer.Consumer<K,V> consumer) Prepare Consumer prior to any topic subscription.
boolean	resourceExists(java.util.Properties conf)

Methods inherited from class cascading.tap.Tap

commitResource, createResource, deleteResource, entryStream, entryStream, entryStreamCopy, entryStreamCopy, equals, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForReadUnchecked, openForWrite, outgoingScopeFor, prepareResourceForRead, prepareResourceForWrite, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, rollbackResource, setScheme, sinkConfInit, sourceConfInit, spliterator, splititerator, taps, toString, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

CONSUME_AUTO_COMMIT_LATEST

public static final java.util.Properties CONSUME_AUTO_COMMIT_LATEST

Field CONSUME_AUTO_COMMIT_LATEST

CONSUME_AUTO_COMMIT_EARLIEST

public static final java.util.Properties CONSUME_AUTO_COMMIT_EARLIEST

Field CONSUME_AUTO_COMMIT_EARLIEST

PRODUCE_ACK_ALL_NO_RETRY

public static final java.util.Properties PRODUCE_ACK_ALL_NO_RETRY

Field PRODUCE_ACK_ALL_NO_RETRY

DEFAULT_POLL_TIMEOUT

public static final long DEFAULT_POLL_TIMEOUT

Field DEFAULT_POLL_TIMEOUT

See Also:

Constant Field Values

DEFAULT_REPLICATION_FACTOR

public static final short DEFAULT_REPLICATION_FACTOR

Field DEFAULT_REPLICATION_FACTOR

See Also:

Constant Field Values

DEFAULT_NUM_PARTITIONS

public static final int DEFAULT_NUM_PARTITIONS

Field DEFAULT_NUM_PARTITIONS

See Also:

Constant Field Values

Constructor Detail

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

identifier - of URI

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                long pollTimeout)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

identifier - of URI

pollTimeout - of long

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                int numPartitions,
                short replicationFactor)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

identifier - of URI

numPartitions - of int

replicationFactor - of short

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                long pollTimeout,
                int numPartitions,
                short replicationFactor)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

identifier - of URI

pollTimeout - of int

numPartitions - of int

replicationFactor - of short

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

identifier - of URI

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                long pollTimeout)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

identifier - of URI

pollTimeout - of int

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                int numPartitions,
                short replicationFactor)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

identifier - of URI

numPartitions - of int

replicationFactor - of short

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                long pollTimeout,
                int numPartitions,
                short replicationFactor)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

identifier - of URI

pollTimeout - of int

numPartitions - of int

replicationFactor - of short

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

identifier - of URI

clientID - of String

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID,
                java.lang.String groupID)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

identifier - of URI

clientID - of String

groupID - of String

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID,
                long pollTimeout)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

identifier - of URI

clientID - of String

pollTimeout - of long

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID,
                int numPartitions,
                short replicationFactor)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

identifier - of URI

clientID - of String

numPartitions - of int

replicationFactor - of short

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID,
                long pollTimeout,
                int numPartitions,
                short replicationFactor)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

identifier - of URI

clientID - of String

pollTimeout - of int

numPartitions - of int

replicationFactor - of short

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

identifier - of URI

clientID - of String

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID,
                java.lang.String groupID)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

identifier - of URI

clientID - of String

groupID - of String

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID,
                long pollTimeout)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

identifier - of URI

clientID - of String

pollTimeout - of int

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID,
                int numPartitions,
                short replicationFactor)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

identifier - of URI

clientID - of String

numPartitions - of int

replicationFactor - of short

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID,
                long pollTimeout,
                int numPartitions,
                short replicationFactor)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

identifier - of URI

clientID - of String

pollTimeout - of int

numPartitions - of int

replicationFactor - of short

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.net.URI identifier,
                java.lang.String clientID,
                java.lang.String groupID,
                long pollTimeout,
                int numPartitions,
                short replicationFactor)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

identifier - of URI

clientID - of String

groupID - of String

pollTimeout - of int

numPartitions - of int

replicationFactor - of short

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                long pollTimeout,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

hostname - of String

pollTimeout - of long

topics - of String...

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                long pollTimeout,
                int numPartitions,
                short replicationFactor,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

hostname - of String

pollTimeout - of int

numPartitions - of int

replicationFactor - of short

topics - of String...

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                int numPartitions,
                short replicationFactor,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

hostname - of String

numPartitions - of int

replicationFactor - of short

topics - of String...

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

hostname - of String

topics - of String...

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                long pollTimeout,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

hostname - of String

pollTimeout - of int

topics - of String...

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                long pollTimeout,
                int numPartitions,
                short replicationFactor,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

hostname - of String

pollTimeout - of int

numPartitions - of int

replicationFactor - of short

topics - of String...

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                java.lang.String clientID,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

hostname - of String

clientID - of String

topics - of String...

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                java.lang.String clientID,
                long pollTimeout,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

hostname - of String

clientID - of String

pollTimeout - of long

topics - of String...

KafkaTap

public KafkaTap(KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                java.lang.String clientID,
                long pollTimeout,
                int numPartitions,
                short replicationFactor,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

scheme - of KafkaScheme

hostname - of String

clientID - of String

pollTimeout - of int

numPartitions - of int

replicationFactor - of short

topics - of String...

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                java.lang.String clientID,
                int numPartitions,
                short replicationFactor,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

hostname - of String

clientID - of String

numPartitions - of int

replicationFactor - of short

topics - of String...

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                java.lang.String clientID,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

hostname - of String

clientID - of String

topics - of String...

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                java.lang.String clientID,
                long pollTimeout,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

hostname - of String

clientID - of String

pollTimeout - of int

topics - of String...

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                java.lang.String clientID,
                long pollTimeout,
                int numPartitions,
                short replicationFactor,
                java.lang.String... topics)

Constructor KafkaTap creates a new KafkaTap instance.

Parameters:

defaultProperties - of Properties

scheme - of KafkaScheme

hostname - of String

clientID - of String

pollTimeout - of int

numPartitions - of int

replicationFactor - of short

topics - of String...

KafkaTap

public KafkaTap(java.util.Properties defaultProperties,
                KafkaScheme<K,V,?,?> scheme,
                java.lang.String hostname,
                java.lang.String clientID,
                java.lang.String groupID,
                long pollTimeout,
                int numPartitions,
                short replicationFactor,
                java.lang.String... topics)

Method Detail

makeURI

public static java.net.URI makeURI(java.lang.String hostname,
                                   java.lang.String... topics)

Method makeURI creates a kafka URI for use with the KafkaTap.

Parameters:

hostname - hostname and optionally port information to connect too

topics - one more topics to connect too

Returns:

a URI instance

applyTopics

protected void applyTopics(java.lang.String[] topics)

getHostname

public java.lang.String getHostname()

Method getHostname returns the hostname of this KafkaTap object.

Returns:

the hostname (type String) of this KafkaTap object.

getClientID

public java.lang.String getClientID()

Method getClientID returns the clientID of this KafkaTap object.

Returns:

the clientID (type String) of this KafkaTap object.

getGroupID

public java.lang.String getGroupID()

Method getGroupID returns the groupID of this KafkaTap object.

Returns:

the groupID (type String) of this KafkaTap object.

getTopics

public java.lang.String[] getTopics()

Method getTopics returns the topics of this KafkaTap object.

Returns:

the topics (type String[]) of this KafkaTap object.

isTopicPattern

public boolean isTopicPattern()

Method isTopicPattern returns true if the topic is a regular expression.

Returns:

true if the topic is a regular expression.

getIdentifier

public java.lang.String getIdentifier()

Specified by:

getIdentifier in class cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>>

createKafkaConsumer

protected org.apache.kafka.clients.consumer.Consumer<K,V> createKafkaConsumer(java.util.Properties properties)

openForRead

public cascading.tuple.TupleEntryIterator openForRead(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                                                      KafkaConsumerRecordIterator<K,V> consumerRecord)
                                               throws java.io.IOException

Specified by:

openForRead in class cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>>

Throws:

java.io.IOException

preConsumerSubscribe

protected void preConsumerSubscribe(org.apache.kafka.clients.consumer.Consumer<K,V> consumer)

Prepare Consumer prior to any topic subscription.

Parameters:

consumer - the current Consumer

postConsumerSubscribe

protected void postConsumerSubscribe(org.apache.kafka.clients.consumer.Consumer<K,V> consumer)

Prepare Consumer post to any topic subscription and prior to any Consumer.poll(long) request.

Parameters:

consumer - the current Consumer

getConsumerRebalanceListener

protected org.apache.kafka.clients.consumer.ConsumerRebalanceListener getConsumerRebalanceListener(org.apache.kafka.clients.consumer.Consumer<K,V> consumer)

Returns a NoOpConsumerRebalanceListener instance.

Override to supply a customer listener.

Parameters:

consumer -

Returns:

a NoOpConsumerRebalanceListener instance.

openForWrite

public cascading.tuple.TupleEntryCollector openForWrite(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                                                        org.apache.kafka.clients.producer.Producer<K,V> producer)
                                                 throws java.io.IOException

Specified by:

openForWrite in class cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>>

Throws:

java.io.IOException

createAdminClient

protected org.apache.kafka.clients.admin.AdminClient createAdminClient(java.util.Properties conf)

createResource

public boolean createResource(java.util.Properties conf)

Specified by:

createResource in class cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>>

deleteResource

public boolean deleteResource(java.util.Properties conf)

Specified by:

deleteResource in class cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>>

resourceExists

public boolean resourceExists(java.util.Properties conf)

Specified by:

resourceExists in class cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>>

getModifiedTime

public long getModifiedTime(java.util.Properties conf)
                     throws java.io.IOException

Specified by:

getModifiedTime in class cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>>

Throws:

java.io.IOException