Package org.apache.pulsar.io.hdfs2.sink.seq

Class HdfsSequentialTextSink

java.lang.Object

org.apache.pulsar.io.hdfs2.AbstractHdfsConnector

org.apache.pulsar.io.hdfs2.sink.HdfsAbstractSink<K,V>

org.apache.pulsar.io.hdfs2.sink.seq.HdfsAbstractSequenceFileSink<java.lang.Long,java.lang.String,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

org.apache.pulsar.io.hdfs2.sink.seq.HdfsSequentialTextSink

All Implemented Interfaces:

java.lang.AutoCloseable, org.apache.pulsar.io.core.Sink<java.lang.String>

public class HdfsSequentialTextSink
extends HdfsAbstractSequenceFileSink<java.lang.Long,java.lang.String,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

This Sink should be used when the records are originating from a sequential source, and we want to retain the record sequence.This class uses the record's sequence id as the sequence id in the HDFS Sequence File if it is available, if not a sequence id is auto-generated for each new record.

Field Summary

Fields inherited from class org.apache.pulsar.io.hdfs2.sink.seq.HdfsAbstractSequenceFileSink

hdfsStream, writer

Fields inherited from class org.apache.pulsar.io.hdfs2.sink.HdfsAbstractSink

hdfsSinkConfig, syncThread, unackedRecords

Fields inherited from class org.apache.pulsar.io.hdfs2.AbstractHdfsConnector

compressionCodecFactory, connectorConfig, hdfsResources

Constructor Summary

Constructors

Constructor	Description
HdfsSequentialTextSink()

Method Summary

Modifier and Type	Method	Description
org.apache.pulsar.io.core.KeyValue<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>	convert(org.apache.pulsar.io.core.KeyValue<java.lang.Long,java.lang.String> kv)
org.apache.pulsar.io.core.KeyValue<java.lang.Long,java.lang.String>	extractKeyValue(org.apache.pulsar.functions.api.Record<java.lang.String> record)
protected java.util.List<org.apache.hadoop.io.SequenceFile.Writer.Option>	getOptions()
org.apache.hadoop.io.SequenceFile.Writer	getWriter()

Methods inherited from class org.apache.pulsar.io.hdfs2.sink.seq.HdfsAbstractSequenceFileSink

close, createWriter, write

Methods inherited from class org.apache.pulsar.io.hdfs2.sink.HdfsAbstractSink

connectToHdfs, getHdfsStream, getPath, launchSyncThread, open

Methods inherited from class org.apache.pulsar.io.hdfs2.AbstractHdfsConnector

checkHdfsUriForTimeout, getCompressionCodec, getCompressionCodecFactory, getConfiguration, getEncoding, getFileSystem, getFileSystem, getFileSystemAsUser, getUserGroupInformation, resetHDFSResources

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pulsar.io.core.Sink

open

Constructor Detail

HdfsSequentialTextSink

public HdfsSequentialTextSink()

Method Detail

getWriter

public org.apache.hadoop.io.SequenceFile.Writer getWriter()
                                                   throws java.io.IOException

Overrides:

getWriter in class HdfsAbstractSequenceFileSink<java.lang.Long,java.lang.String,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

Throws:

java.io.IOException

getOptions

protected java.util.List<org.apache.hadoop.io.SequenceFile.Writer.Option> getOptions()
                                                                              throws java.lang.IllegalArgumentException,
                                                                                     java.io.IOException

Overrides:

getOptions in class HdfsAbstractSequenceFileSink<java.lang.Long,java.lang.String,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>

Throws:

java.lang.IllegalArgumentException

java.io.IOException

extractKeyValue

public org.apache.pulsar.io.core.KeyValue<java.lang.Long,java.lang.String> extractKeyValue(org.apache.pulsar.functions.api.Record<java.lang.String> record)

Specified by:

extractKeyValue in class HdfsAbstractSink<java.lang.Long,java.lang.String>

convert

public org.apache.pulsar.io.core.KeyValue<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text> convert(org.apache.pulsar.io.core.KeyValue<java.lang.Long,java.lang.String> kv)

Specified by:

convert in class HdfsAbstractSequenceFileSink<java.lang.Long,java.lang.String,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>