Serialized Form

Package org.apache.pulsar.io.hdfs3

Class org.apache.pulsar.io.hdfs3.AbstractHdfsConfig extends Object implements Serializable

serialVersionUID:

1L

Serialized Fields

hdfsConfigResources

String hdfsConfigResources

A file or comma separated list of files which contains the Hadoop file system configuration, e.g. 'core-site.xml', 'hdfs-site.xml'.

directory

String directory

The HDFS directory from which files should be read from or written to.

encoding

String encoding

The character encoding for the files, e.g. UTF-8, ASCII, etc.

compression

Compression compression

The compression codec used to compress/de-compress the files on HDFS.

kerberosUserPrincipal

String kerberosUserPrincipal

The Kerberos user principal account to use for authentication.

keytab

String keytab

The full pathname to the Kerberos keytab file to use for authentication.

Package org.apache.pulsar.io.hdfs3.sink

Class org.apache.pulsar.io.hdfs3.sink.HdfsSinkConfig extends AbstractHdfsConfig implements Serializable

serialVersionUID:

1L

Serialized Fields

filenamePrefix

String filenamePrefix

The prefix of the files to create inside the HDFS directory, i.e. a value of "topicA" will result in files named topicA-, topicA-, etc being produced

fileExtension

String fileExtension

The extension to add to the files written to HDFS, e.g. '.txt', '.seq', etc.

separator

char separator

The character to use to separate records in a text file. If no value is provided then the content from all of the records will be concatenated together in one continuous byte array.

syncInterval

long syncInterval

The interval (in milliseconds) between calls to flush data to HDFS disk.

maxPendingRecords

int maxPendingRecords

The maximum number of records that we hold in memory before acking. Default is Integer.MAX_VALUE. Setting this value to one, results in every record being sent to disk before the record is acked, while setting it to a higher values allows us to buffer records before flushing them all to disk.