org.apache.pulsar.io.hdfs3.sink.seq

Class HdfsAbstractSequenceFileSink<K,V,HdfsK,HdfsV>

java.lang.Object

org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

org.apache.pulsar.io.hdfs3.sink.HdfsAbstractSink<K,V>

org.apache.pulsar.io.hdfs3.sink.seq.HdfsAbstractSequenceFileSink<K,V,HdfsK,HdfsV>

Type Parameters:

K - - The incoming Key type

V - - The incoming Value type

HdfsK - - The HDFS Key type

HdfsV - - The HDFS Value type

All Implemented Interfaces:

AutoCloseable, Sink<V>

Direct Known Subclasses:

HdfsSequentialTextSink, HdfsTextSink

public abstract class HdfsAbstractSequenceFileSink<K,V,HdfsK,HdfsV>
extends HdfsAbstractSink<K,V>
implements Sink<V>

HDFS Sink that writes it contents to HDFS as Sequence Files.

Field Summary

Fields

Modifier and Type	Field and Description
protected AtomicLong	counter
protected org.apache.hadoop.fs.FSDataOutputStream	hdfsStream
protected org.apache.hadoop.io.SequenceFile.Writer	writer

Fields inherited from class org.apache.pulsar.io.hdfs3.sink.HdfsAbstractSink

hdfsSinkConfig, syncThread, unackedRecords

Fields inherited from class org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

compressionCodecFactory, connectorConfig, hdfsResources

Constructor Summary

Constructors

Constructor and Description
HdfsAbstractSequenceFileSink()

Method Summary

Modifier and Type	Method and Description
void	close()
abstract KeyValue<HdfsK,HdfsV>	convert(KeyValue<K,V> kv)
protected void	createWriter()
protected List<org.apache.hadoop.io.SequenceFile.Writer.Option>	getOptions()
protected org.apache.hadoop.io.SequenceFile.Writer	getWriter()
void	write(org.apache.pulsar.functions.api.Record<V> record)

Methods inherited from class org.apache.pulsar.io.hdfs3.sink.HdfsAbstractSink

connectToHdfs, extractKeyValue, getHdfsStream, getOutputStreamBuilder, getPath, launchSyncThread, open

Methods inherited from class org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

checkHdfsUriForTimeout, getCompressionCodec, getCompressionCodecFactory, getConfiguration, getEncoding, getFileSystem, getFileSystem, getFileSystemAsUser, getUserGroupInformation, resetHDFSResources

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pulsar.io.core.Sink

open

Field Detail

counter

protected AtomicLong counter

hdfsStream

protected org.apache.hadoop.fs.FSDataOutputStream hdfsStream

writer

protected org.apache.hadoop.io.SequenceFile.Writer writer

Constructor Detail

HdfsAbstractSequenceFileSink

public HdfsAbstractSequenceFileSink()

Method Detail

convert

public abstract KeyValue<HdfsK,HdfsV> convert(KeyValue<K,V> kv)

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Overrides:

close in class HdfsAbstractSink<K,V>

Throws:

Exception

createWriter

protected void createWriter()
                     throws IOException

Specified by:

createWriter in class HdfsAbstractSink<K,V>

Throws:

IOException

write

public void write(org.apache.pulsar.functions.api.Record<V> record)

Specified by:

write in interface Sink<V>

getWriter

protected org.apache.hadoop.io.SequenceFile.Writer getWriter()
                                                      throws IOException

Throws:

IOException

getOptions

protected List<org.apache.hadoop.io.SequenceFile.Writer.Option> getOptions()
                                                                    throws IllegalArgumentException,
                                                                           IOException

Throws:

IllegalArgumentException

IOException