Package org.apache.pulsar.io.hdfs3.sink

Class HdfsAbstractSink<K,V>

java.lang.Object

org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

org.apache.pulsar.io.hdfs3.sink.HdfsAbstractSink<K,V>

All Implemented Interfaces:

AutoCloseable, Sink<V>

Direct Known Subclasses:

HdfsAbstractSequenceFileSink, HdfsAbstractTextFileSink

public abstract class HdfsAbstractSink<K,V>
extends AbstractHdfsConnector
implements Sink<V>

A Simple abstract class for HDFS sink. Users need to implement extractKeyValue function to use this sink.

Field Summary

Fields

Modifier and Type	Field	Description
protected HdfsSinkConfig	hdfsSinkConfig
protected HdfsSyncThread<V>	syncThread
protected BlockingQueue<org.apache.pulsar.functions.api.Record<V>>	unackedRecords

Fields inherited from class org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

compressionCodecFactory, connectorConfig, hdfsResources

Constructor Summary

Constructors

Constructor	Description
HdfsAbstractSink()

Method Summary

Modifier and Type	Method	Description
void	close()
protected final void	connectToHdfs()
protected abstract void	createWriter()
abstract KeyValue<K,V>	extractKeyValue(org.apache.pulsar.functions.api.Record<V> record)
protected org.apache.hadoop.fs.FSDataOutputStream	getHdfsStream()
protected final org.apache.hadoop.fs.FSDataOutputStreamBuilder	getOutputStreamBuilder()
protected final org.apache.hadoop.fs.Path	getPath()
protected final void	launchSyncThread()
void	open(Map<String,Object> config, SinkContext sinkContext)

Methods inherited from class org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

checkHdfsUriForTimeout, getCompressionCodec, getCompressionCodecFactory, getConfiguration, getEncoding, getFileSystem, getFileSystem, getFileSystemAsUser, getUserGroupInformation, resetHDFSResources

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pulsar.io.core.Sink

write

Field Details

hdfsSinkConfig

protected HdfsSinkConfig hdfsSinkConfig

unackedRecords

protected BlockingQueue<org.apache.pulsar.functions.api.Record<V>> unackedRecords

syncThread

protected HdfsSyncThread<V> syncThread

Constructor Details

HdfsAbstractSink

public HdfsAbstractSink()

Method Details

extractKeyValue

public abstract KeyValue<K,V> extractKeyValue(org.apache.pulsar.functions.api.Record<V> record)

createWriter

protected abstract void createWriter()
                              throws IOException

Throws:

IOException

open

public void open(Map<String,Object> config,
 SinkContext sinkContext)
          throws Exception

Specified by:

open in interface Sink<K>

Throws:

Exception

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

connectToHdfs

protected final void connectToHdfs()
                            throws IOException

Throws:

IOException

getOutputStreamBuilder

protected final org.apache.hadoop.fs.FSDataOutputStreamBuilder getOutputStreamBuilder()
                                                                               throws IOException

Throws:

IOException

getHdfsStream

protected org.apache.hadoop.fs.FSDataOutputStream getHdfsStream()
                                                         throws IllegalArgumentException,
IOException

Throws:

IllegalArgumentException

IOException

getPath

protected final org.apache.hadoop.fs.Path getPath()

launchSyncThread

protected final void launchSyncThread()
                               throws IOException

Throws:

IOException