org.apache.pulsar.io.hdfs3.sink

Class HdfsAbstractSink<K,V>

java.lang.Object

org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

org.apache.pulsar.io.hdfs3.sink.HdfsAbstractSink<K,V>

All Implemented Interfaces:

AutoCloseable, Sink<V>

Direct Known Subclasses:

HdfsAbstractSequenceFileSink, HdfsAbstractTextFileSink

public abstract class HdfsAbstractSink<K,V>
extends AbstractHdfsConnector
implements Sink<V>

A Simple abstract class for HDFS sink. Users need to implement extractKeyValue function to use this sink.

Field Summary

Fields

Modifier and Type	Field and Description
protected HdfsSinkConfig	hdfsSinkConfig
protected HdfsSyncThread<V>	syncThread
protected BlockingQueue<org.apache.pulsar.functions.api.Record<V>>	unackedRecords

Fields inherited from class org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

compressionCodecFactory, connectorConfig, hdfsResources

Constructor Summary

Constructors

Constructor and Description
HdfsAbstractSink()

Method Summary

Modifier and Type	Method and Description
void	close()
protected void	connectToHdfs()
protected abstract void	createWriter()
abstract KeyValue<K,V>	extractKeyValue(org.apache.pulsar.functions.api.Record<V> record)
protected org.apache.hadoop.fs.FSDataOutputStream	getHdfsStream()
protected org.apache.hadoop.fs.FSDataOutputStreamBuilder	getOutputStreamBuilder()
protected org.apache.hadoop.fs.Path	getPath()
protected void	launchSyncThread()
void	open(Map<String,Object> config, SinkContext sinkContext)

Methods inherited from class org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

checkHdfsUriForTimeout, getCompressionCodec, getCompressionCodecFactory, getConfiguration, getEncoding, getFileSystem, getFileSystem, getFileSystemAsUser, getUserGroupInformation, resetHDFSResources

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pulsar.io.core.Sink

write

Field Detail

hdfsSinkConfig

protected HdfsSinkConfig hdfsSinkConfig

unackedRecords

protected BlockingQueue<org.apache.pulsar.functions.api.Record<V>> unackedRecords

syncThread

protected HdfsSyncThread<V> syncThread

Constructor Detail

HdfsAbstractSink

public HdfsAbstractSink()

Method Detail

extractKeyValue

public abstract KeyValue<K,V> extractKeyValue(org.apache.pulsar.functions.api.Record<V> record)

createWriter

protected abstract void createWriter()
                              throws IOException

Throws:

IOException

open

public void open(Map<String,Object> config,
                 SinkContext sinkContext)
          throws Exception

Specified by:

open in interface Sink<V>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

connectToHdfs

protected final void connectToHdfs()
                            throws IOException

Throws:

IOException

getOutputStreamBuilder

protected final org.apache.hadoop.fs.FSDataOutputStreamBuilder getOutputStreamBuilder()
                                                                               throws IOException

Throws:

IOException

getHdfsStream

protected org.apache.hadoop.fs.FSDataOutputStream getHdfsStream()
                                                         throws IllegalArgumentException,
                                                                IOException

Throws:

IllegalArgumentException

IOException

getPath

protected final org.apache.hadoop.fs.Path getPath()

launchSyncThread

protected final void launchSyncThread()
                               throws IOException

Throws:

IOException