Package org.apache.pulsar.io.hdfs2.sink.seq

Class HdfsAbstractSequenceFileSink<K,V,HdfsK,HdfsV>

java.lang.Object

org.apache.pulsar.io.hdfs2.AbstractHdfsConnector

org.apache.pulsar.io.hdfs2.sink.HdfsAbstractSink<K,V>

org.apache.pulsar.io.hdfs2.sink.seq.HdfsAbstractSequenceFileSink<K,V,HdfsK,HdfsV>

Type Parameters:

K - - The incoming Key type

V - - The incoming Value type

HdfsK - - The HDFS Key type

HdfsV - - The HDFS Value type

All Implemented Interfaces:

java.lang.AutoCloseable, org.apache.pulsar.io.core.Sink<V>

Direct Known Subclasses:

HdfsSequentialTextSink, HdfsTextSink

public abstract class HdfsAbstractSequenceFileSink<K,V,HdfsK,HdfsV>
extends HdfsAbstractSink<K,V>
implements org.apache.pulsar.io.core.Sink<V>

HDFS Sink that writes it contents to HDFS as Sequence Files.

Field Summary

Fields

Modifier and Type	Field	Description
protected java.util.concurrent.atomic.AtomicLong	counter
protected org.apache.hadoop.fs.FSDataOutputStream	hdfsStream
protected org.apache.hadoop.io.SequenceFile.Writer	writer

Fields inherited from class org.apache.pulsar.io.hdfs2.sink.HdfsAbstractSink

hdfsSinkConfig, syncThread, unackedRecords

Fields inherited from class org.apache.pulsar.io.hdfs2.AbstractHdfsConnector

compressionCodecFactory, connectorConfig, hdfsResources

Constructor Summary

Constructors

Constructor	Description
HdfsAbstractSequenceFileSink()

Method Summary

Modifier and Type	Method	Description
void	close()
abstract org.apache.pulsar.io.core.KeyValue<HdfsK,HdfsV>	convert(org.apache.pulsar.io.core.KeyValue<K,V> kv)
protected void	createWriter()
protected java.util.List<org.apache.hadoop.io.SequenceFile.Writer.Option>	getOptions()
protected org.apache.hadoop.io.SequenceFile.Writer	getWriter()
void	write(org.apache.pulsar.functions.api.Record<V> record)

Methods inherited from class org.apache.pulsar.io.hdfs2.sink.HdfsAbstractSink

connectToHdfs, extractKeyValue, getHdfsStream, getPath, launchSyncThread, open

Methods inherited from class org.apache.pulsar.io.hdfs2.AbstractHdfsConnector

checkHdfsUriForTimeout, getCompressionCodec, getCompressionCodecFactory, getConfiguration, getEncoding, getFileSystem, getFileSystem, getFileSystemAsUser, getUserGroupInformation, resetHDFSResources

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pulsar.io.core.Sink

open

Field Detail

counter

protected java.util.concurrent.atomic.AtomicLong counter

hdfsStream

protected org.apache.hadoop.fs.FSDataOutputStream hdfsStream

writer

protected org.apache.hadoop.io.SequenceFile.Writer writer

Constructor Detail

HdfsAbstractSequenceFileSink

public HdfsAbstractSequenceFileSink()

Method Detail

convert

public abstract org.apache.pulsar.io.core.KeyValue<HdfsK,HdfsV> convert(org.apache.pulsar.io.core.KeyValue<K,V> kv)

close

public void close()
           throws java.lang.Exception

Specified by:

close in interface java.lang.AutoCloseable

Overrides:

close in class HdfsAbstractSink<K,V>

Throws:

java.lang.Exception

createWriter

protected void createWriter()
                     throws java.io.IOException

Specified by:

createWriter in class HdfsAbstractSink<K,V>

Throws:

java.io.IOException

write

public void write(org.apache.pulsar.functions.api.Record<V> record)

Specified by:

write in interface org.apache.pulsar.io.core.Sink<K>

getWriter

protected org.apache.hadoop.io.SequenceFile.Writer getWriter()
                                                      throws java.io.IOException

Throws:

java.io.IOException

getOptions

protected java.util.List<org.apache.hadoop.io.SequenceFile.Writer.Option> getOptions()
                                                                              throws java.lang.IllegalArgumentException,
                                                                                     java.io.IOException

Throws:

java.lang.IllegalArgumentException

java.io.IOException