Package org.apache.pulsar.io.hdfs3.sink.seq

Class HdfsSequentialTextSink

java.lang.Object

org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

org.apache.pulsar.io.hdfs3.sink.HdfsAbstractSink<K,V>

org.apache.pulsar.io.hdfs3.sink.seq.HdfsAbstractSequenceFileSink<Long,String,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

org.apache.pulsar.io.hdfs3.sink.seq.HdfsSequentialTextSink

All Implemented Interfaces:

AutoCloseable, Sink<String>

public class HdfsSequentialTextSink
extends HdfsAbstractSequenceFileSink<Long,String,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

This Sink should be used when the records are originating from a sequential source, and we want to retain the record sequence.This class uses the record's sequence id as the sequence id in the HDFS Sequence File if it is available, if not a sequence id is auto-generated for each new record.

Field Summary

Fields inherited from class org.apache.pulsar.io.hdfs3.sink.seq.HdfsAbstractSequenceFileSink

hdfsStream, writer

Fields inherited from class org.apache.pulsar.io.hdfs3.sink.HdfsAbstractSink

hdfsSinkConfig, syncThread, unackedRecords

Fields inherited from class org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

compressionCodecFactory, connectorConfig, hdfsResources

Constructor Summary

Constructors

Constructor	Description
HdfsSequentialTextSink()

Method Summary

Modifier and Type	Method	Description
KeyValue<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>	convert(KeyValue<Long,String> kv)
KeyValue<Long,String>	extractKeyValue(org.apache.pulsar.functions.api.Record<String> record)
protected List<org.apache.hadoop.io.SequenceFile.Writer.Option>	getOptions()
org.apache.hadoop.io.SequenceFile.Writer	getWriter()

Methods inherited from class org.apache.pulsar.io.hdfs3.sink.seq.HdfsAbstractSequenceFileSink

close, createWriter, write

Methods inherited from class org.apache.pulsar.io.hdfs3.sink.HdfsAbstractSink

connectToHdfs, getHdfsStream, getOutputStreamBuilder, getPath, launchSyncThread, open

Methods inherited from class org.apache.pulsar.io.hdfs3.AbstractHdfsConnector

checkHdfsUriForTimeout, getCompressionCodec, getCompressionCodecFactory, getConfiguration, getEncoding, getFileSystem, getFileSystem, getFileSystemAsUser, getUserGroupInformation, resetHDFSResources

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pulsar.io.core.Sink

open

Constructor Details

HdfsSequentialTextSink

public HdfsSequentialTextSink()

Method Details

getWriter

public org.apache.hadoop.io.SequenceFile.Writer getWriter()
                                                   throws IOException

Overrides:

getWriter in class HdfsAbstractSequenceFileSink<Long,String,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

Throws:

IOException

getOptions

protected List<org.apache.hadoop.io.SequenceFile.Writer.Option> getOptions()
                                                                    throws IllegalArgumentException,
IOException

Overrides:

getOptions in class HdfsAbstractSequenceFileSink<Long,String,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

Throws:

IllegalArgumentException

IOException

extractKeyValue

public KeyValue<Long,String> extractKeyValue(org.apache.pulsar.functions.api.Record<String> record)

Specified by:

extractKeyValue in class HdfsAbstractSink<Long,String>

convert

public KeyValue<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> convert(KeyValue<Long,String> kv)

Specified by:

convert in class HdfsAbstractSequenceFileSink<Long,String,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>