cascading.local.tap.kafka

Class KafkaCommittingTap<K,V>

java.lang.Object

cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>>

cascading.local.tap.kafka.KafkaTap<K,V>

cascading.local.tap.kafka.KafkaCommittingTap<K,V>

Type Parameters:

K - the type parameter

V - the type parameter

All Implemented Interfaces:

cascading.flow.FlowElement, cascading.flow.planner.ScopedElement, cascading.util.Traceable, java.io.Serializable

public class KafkaCommittingTap<K,V>
extends KafkaTap<K,V>

Class KafkaCommittingTap commits consumer record offsets on completion of successful or failed flows, and revoked partitions from a rebalance.

if the flow failed, the failed record will be the next record to read on that given partition.

See Also:

Serialized Form

Field Summary

Fields inherited from class cascading.local.tap.kafka.KafkaTap

CONSUME_AUTO_COMMIT_EARLIEST, CONSUME_AUTO_COMMIT_LATEST, DEFAULT_NUM_PARTITIONS, DEFAULT_POLL_TIMEOUT, DEFAULT_REPLICATION_FACTOR, PRODUCE_ACK_ALL_NO_RETRY

Constructor Summary

Constructors

Constructor and Description
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, long pollTimeout, int numPartitions, short replicationFactor, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, long pollTimeout, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, long pollTimeout, int numPartitions, short replicationFactor, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, long pollTimeout, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, int numPartitions, short replicationFactor) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, long pollTimeout) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, long pollTimeout, int numPartitions, short replicationFactor) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, int numPartitions, short replicationFactor) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, long pollTimeout) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, long pollTimeout, int numPartitions, short replicationFactor) Instantiates a new Committing kafka tap.
KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, java.lang.String groupID) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, int numPartitions, short replicationFactor, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, long pollTimeout, int numPartitions, short replicationFactor, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, long pollTimeout, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, int numPartitions, short replicationFactor, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, long pollTimeout, int numPartitions, short replicationFactor, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, long pollTimeout, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.lang.String hostname, java.lang.String clientID, java.lang.String groupID, long pollTimeout, int numPartitions, short replicationFactor, java.lang.String... topics) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, int numPartitions, short replicationFactor) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, long pollTimeout) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, long pollTimeout, int numPartitions, short replicationFactor) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, int numPartitions, short replicationFactor) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, long pollTimeout) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, long pollTimeout, int numPartitions, short replicationFactor) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, java.lang.String groupID) Instantiates a new Committing kafka tap.
KafkaCommittingTap(java.util.Properties defaultProperties, KafkaScheme<K,V,?,?> scheme, java.net.URI identifier, java.lang.String clientID, java.lang.String groupID, long pollTimeout, int numPartitions, short replicationFactor) Instantiates a new Committing kafka tap.

Method Summary

Modifier and Type	Method and Description
protected org.apache.kafka.clients.consumer.Consumer<K,V>	createKafkaConsumer(java.util.Properties properties)

Methods inherited from class cascading.local.tap.kafka.KafkaTap

applyTopics, createAdminClient, createResource, deleteResource, getClientID, getConsumerRebalanceListener, getGroupID, getHostname, getIdentifier, getModifiedTime, getTopics, isTopicPattern, makeURI, openForRead, openForWrite, postConsumerSubscribe, preConsumerSubscribe, resourceExists

Methods inherited from class cascading.tap.Tap

commitResource, createResource, deleteResource, entryStream, entryStream, entryStreamCopy, entryStreamCopy, equals, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForReadUnchecked, openForWrite, outgoingScopeFor, prepareResourceForRead, prepareResourceForWrite, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, rollbackResource, setScheme, sinkConfInit, sourceConfInit, spliterator, splititerator, taps, toString, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

identifier - the identifier

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          long pollTimeout)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

identifier - the identifier

pollTimeout - the poll timeout

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          int numPartitions,
                          short replicationFactor)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

identifier - the identifier

numPartitions - the num partitions

replicationFactor - the replication factor

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          long pollTimeout,
                          int numPartitions,
                          short replicationFactor)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

identifier - the identifier

pollTimeout - the poll timeout

numPartitions - the num partitions

replicationFactor - the replication factor

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

identifier - the identifier

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          long pollTimeout)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

identifier - the identifier

pollTimeout - the poll timeout

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          int numPartitions,
                          short replicationFactor)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

identifier - the identifier

numPartitions - the num partitions

replicationFactor - the replication factor

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          long pollTimeout,
                          int numPartitions,
                          short replicationFactor)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

identifier - the identifier

pollTimeout - the poll timeout

numPartitions - the num partitions

replicationFactor - the replication factor

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

identifier - the identifier

clientID - the client id

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID,
                          java.lang.String groupID)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

identifier - the identifier

clientID - the client id

groupID - the group id

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID,
                          long pollTimeout)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

identifier - the identifier

clientID - the client id

pollTimeout - the poll timeout

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID,
                          int numPartitions,
                          short replicationFactor)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

identifier - the identifier

clientID - the client id

numPartitions - the num partitions

replicationFactor - the replication factor

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID,
                          long pollTimeout,
                          int numPartitions,
                          short replicationFactor)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

identifier - the identifier

clientID - the client id

pollTimeout - the poll timeout

numPartitions - the num partitions

replicationFactor - the replication factor

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

identifier - the identifier

clientID - the client id

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID,
                          java.lang.String groupID)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

identifier - the identifier

clientID - the client id

groupID - the group id

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID,
                          long pollTimeout)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

identifier - the identifier

clientID - the client id

pollTimeout - the poll timeout

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID,
                          int numPartitions,
                          short replicationFactor)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

identifier - the identifier

clientID - the client id

numPartitions - the num partitions

replicationFactor - the replication factor

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID,
                          long pollTimeout,
                          int numPartitions,
                          short replicationFactor)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

identifier - the identifier

clientID - the client id

pollTimeout - the poll timeout

numPartitions - the num partitions

replicationFactor - the replication factor

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.net.URI identifier,
                          java.lang.String clientID,
                          java.lang.String groupID,
                          long pollTimeout,
                          int numPartitions,
                          short replicationFactor)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

identifier - the identifier

clientID - the client id

groupID - the group id

pollTimeout - the poll timeout

numPartitions - the num partitions

replicationFactor - the replication factor

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          long pollTimeout,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

hostname - the hostname

pollTimeout - the poll timeout

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          long pollTimeout,
                          int numPartitions,
                          short replicationFactor,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

hostname - the hostname

pollTimeout - the poll timeout

numPartitions - the num partitions

replicationFactor - the replication factor

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          int numPartitions,
                          short replicationFactor,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

hostname - the hostname

numPartitions - the num partitions

replicationFactor - the replication factor

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

hostname - the hostname

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          long pollTimeout,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

hostname - the hostname

pollTimeout - the poll timeout

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          long pollTimeout,
                          int numPartitions,
                          short replicationFactor,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

hostname - the hostname

pollTimeout - the poll timeout

numPartitions - the num partitions

replicationFactor - the replication factor

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          java.lang.String clientID,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

hostname - the hostname

clientID - the client id

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          java.lang.String clientID,
                          long pollTimeout,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

hostname - the hostname

clientID - the client id

pollTimeout - the poll timeout

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          java.lang.String clientID,
                          long pollTimeout,
                          int numPartitions,
                          short replicationFactor,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

scheme - the scheme

hostname - the hostname

clientID - the client id

pollTimeout - the poll timeout

numPartitions - the num partitions

replicationFactor - the replication factor

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          java.lang.String clientID,
                          int numPartitions,
                          short replicationFactor,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

hostname - the hostname

clientID - the client id

numPartitions - the num partitions

replicationFactor - the replication factor

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          java.lang.String clientID,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

hostname - the hostname

clientID - the client id

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          java.lang.String clientID,
                          long pollTimeout,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

hostname - the hostname

clientID - the client id

pollTimeout - the poll timeout

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          java.lang.String clientID,
                          long pollTimeout,
                          int numPartitions,
                          short replicationFactor,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

hostname - the hostname

clientID - the client id

pollTimeout - the poll timeout

numPartitions - the num partitions

replicationFactor - the replication factor

topics - the topics

KafkaCommittingTap

public KafkaCommittingTap(java.util.Properties defaultProperties,
                          KafkaScheme<K,V,?,?> scheme,
                          java.lang.String hostname,
                          java.lang.String clientID,
                          java.lang.String groupID,
                          long pollTimeout,
                          int numPartitions,
                          short replicationFactor,
                          java.lang.String... topics)

Instantiates a new Committing kafka tap.

Parameters:

defaultProperties - the default properties

scheme - the scheme

hostname - the hostname

clientID - the client id

groupID - the group id

pollTimeout - the poll timeout

numPartitions - the num partitions

replicationFactor - the replication factor

topics - the topics

Method Detail

createKafkaConsumer

protected org.apache.kafka.clients.consumer.Consumer<K,V> createKafkaConsumer(java.util.Properties properties)

Overrides:

createKafkaConsumer in class KafkaTap<K,V>