cascading.local.tap.kafka

Class TextKafkaScheme

java.lang.Object

cascading.scheme.Scheme<java.util.Properties,KafkaConsumerRecordIterator<K,V>,org.apache.kafka.clients.producer.Producer<K,V>,SourceContext,SinkContext>

cascading.local.tap.kafka.KafkaScheme<java.lang.String,java.lang.String,cascading.local.tap.kafka.TextKafkaScheme.Context,cascading.local.tap.kafka.TextKafkaScheme.Context>

cascading.local.tap.kafka.TextKafkaScheme

All Implemented Interfaces:

cascading.util.Traceable, java.io.Serializable

public class TextKafkaScheme
extends KafkaScheme<java.lang.String,java.lang.String,cascading.local.tap.kafka.TextKafkaScheme.Context,cascading.local.tap.kafka.TextKafkaScheme.Context>

Class TextKafkaScheme is a sub-class of the KafkaScheme for use with a KafkaTap instance.

It consumes and produces text/string based keys and values.

As a source, it produces seven fields: TOPIC_FIELDS typed String, PARTITION_FIELDS typed int, OFFSET_FIELDS typed long, KEY_FIELDS typed String, and VALUE_FIELDS typed String, TIMESTAMP_FIELDS typed long, TIMESTAMP_TYPE_FIELDS typed String

If alternate source fields are given, any type information will be honored.

As a sink, the first field encountered will be used as the topic key, and the second field encountered will be used as the value.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static cascading.tuple.Fields	DEFAULT_SOURCE_FIELDS Field DEFAULT_SOURCE_FIELDS
static cascading.tuple.Fields	KEY_FIELDS Field KEY_FIELDS
static cascading.tuple.Fields	OFFSET_FIELDS Field OFFSET_FIELDS
static cascading.tuple.Fields	PARTITION_FIELDS Field PARTITION_FIELDS
static cascading.tuple.Fields	TIMESTAMP_FIELDS Field TIMESTAMP_FIELDS
static cascading.tuple.Fields	TIMESTAMP_TYPE_FIELDS Field TIMESTAMP_TYPE_FIELDS
static cascading.tuple.Fields	TOPIC_FIELDS Field TOPIC_FIELDS
static cascading.tuple.Fields	VALUE_FIELDS Field VALUE_FIELDS

Constructor Summary

Constructors

Constructor and Description
TextKafkaScheme() Constructor TextKafkaScheme creates a new TextKafkaScheme instance.
TextKafkaScheme(cascading.tuple.Fields sourceFields) Constructor TextKafkaScheme creates a new TextKafkaScheme instance.

Method Summary

Modifier and Type	Method and Description
void	sink(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, cascading.scheme.SinkCall<cascading.local.tap.kafka.TextKafkaScheme.Context,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>> sinkCall)
void	sinkConfInit(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>> tap, java.util.Properties conf)
void	sinkPrepare(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, cascading.scheme.SinkCall<cascading.local.tap.kafka.TextKafkaScheme.Context,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>> sinkCall)
boolean	source(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, cascading.scheme.SourceCall<cascading.local.tap.kafka.TextKafkaScheme.Context,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>> sourceCall)
void	sourceConfInit(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>> tap, java.util.Properties conf)
void	sourcePrepare(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, cascading.scheme.SourceCall<cascading.local.tap.kafka.TextKafkaScheme.Context,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>> sourceCall)

Methods inherited from class cascading.scheme.Scheme

equals, getNumSinkParts, getSinkFields, getSourceFields, getTrace, hashCode, isSink, isSource, isSymmetrical, presentSinkFields, presentSinkFieldsInternal, presentSourceFields, presentSourceFieldsInternal, retrieveSinkFields, retrieveSourceFields, setNumSinkParts, setSinkFields, setSourceFields, sinkCleanup, sinkWrap, sourceCleanup, sourceRePrepare, sourceWrap, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

TOPIC_FIELDS

public static final cascading.tuple.Fields TOPIC_FIELDS

Field TOPIC_FIELDS

PARTITION_FIELDS

public static final cascading.tuple.Fields PARTITION_FIELDS

Field PARTITION_FIELDS

OFFSET_FIELDS

public static final cascading.tuple.Fields OFFSET_FIELDS

Field OFFSET_FIELDS

KEY_FIELDS

public static final cascading.tuple.Fields KEY_FIELDS

Field KEY_FIELDS

VALUE_FIELDS

public static final cascading.tuple.Fields VALUE_FIELDS

Field VALUE_FIELDS

TIMESTAMP_FIELDS

public static final cascading.tuple.Fields TIMESTAMP_FIELDS

Field TIMESTAMP_FIELDS

TIMESTAMP_TYPE_FIELDS

public static final cascading.tuple.Fields TIMESTAMP_TYPE_FIELDS

Field TIMESTAMP_TYPE_FIELDS

DEFAULT_SOURCE_FIELDS

public static final cascading.tuple.Fields DEFAULT_SOURCE_FIELDS

Field DEFAULT_SOURCE_FIELDS

Constructor Detail

TextKafkaScheme

public TextKafkaScheme()

Constructor TextKafkaScheme creates a new TextKafkaScheme instance.

TextKafkaScheme

public TextKafkaScheme(cascading.tuple.Fields sourceFields)

Constructor TextKafkaScheme creates a new TextKafkaScheme instance.

Parameters:

sourceFields - of Fields

Method Detail

sourceConfInit

public void sourceConfInit(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                           cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>> tap,
                           java.util.Properties conf)

Specified by:

sourceConfInit in class cascading.scheme.Scheme<java.util.Properties,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>,cascading.local.tap.kafka.TextKafkaScheme.Context,cascading.local.tap.kafka.TextKafkaScheme.Context>

sinkConfInit

public void sinkConfInit(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                         cascading.tap.Tap<java.util.Properties,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>> tap,
                         java.util.Properties conf)

Specified by:

sinkConfInit in class cascading.scheme.Scheme<java.util.Properties,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>,cascading.local.tap.kafka.TextKafkaScheme.Context,cascading.local.tap.kafka.TextKafkaScheme.Context>

sourcePrepare

public void sourcePrepare(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                          cascading.scheme.SourceCall<cascading.local.tap.kafka.TextKafkaScheme.Context,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>> sourceCall)
                   throws java.io.IOException

Overrides:

sourcePrepare in class cascading.scheme.Scheme<java.util.Properties,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>,cascading.local.tap.kafka.TextKafkaScheme.Context,cascading.local.tap.kafka.TextKafkaScheme.Context>

Throws:

java.io.IOException

sinkPrepare

public void sinkPrepare(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                        cascading.scheme.SinkCall<cascading.local.tap.kafka.TextKafkaScheme.Context,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>> sinkCall)
                 throws java.io.IOException

Overrides:

sinkPrepare in class cascading.scheme.Scheme<java.util.Properties,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>,cascading.local.tap.kafka.TextKafkaScheme.Context,cascading.local.tap.kafka.TextKafkaScheme.Context>

Throws:

java.io.IOException

source

public boolean source(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                      cascading.scheme.SourceCall<cascading.local.tap.kafka.TextKafkaScheme.Context,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>> sourceCall)
               throws java.io.IOException

Specified by:

source in class cascading.scheme.Scheme<java.util.Properties,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>,cascading.local.tap.kafka.TextKafkaScheme.Context,cascading.local.tap.kafka.TextKafkaScheme.Context>

Throws:

java.io.IOException

sink

public void sink(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                 cascading.scheme.SinkCall<cascading.local.tap.kafka.TextKafkaScheme.Context,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>> sinkCall)
          throws java.io.IOException

Specified by:

sink in class cascading.scheme.Scheme<java.util.Properties,KafkaConsumerRecordIterator<java.lang.String,java.lang.String>,org.apache.kafka.clients.producer.Producer<java.lang.String,java.lang.String>,cascading.local.tap.kafka.TextKafkaScheme.Context,cascading.local.tap.kafka.TextKafkaScheme.Context>

Throws:

java.io.IOException