Package org.apache.pulsar.io.hdfs3

Class AbstractHdfsConnector

java.lang.Object
org.apache.pulsar.io.hdfs3.AbstractHdfsConnector
Direct Known Subclasses:
HdfsAbstractSink
public abstract class AbstractHdfsConnector extends Object
A Simple abstract class for HDFS connectors. Provides methods for connecting to HDFS

  • Field Details

    • hdfsResources

      protected final AtomicReference<HdfsResources> hdfsResources

    • connectorConfig

      protected AbstractHdfsConfig connectorConfig

    • compressionCodecFactory

      protected org.apache.hadoop.io.compress.CompressionCodecFactory compressionCodecFactory

  • Constructor Details

    • AbstractHdfsConnector

      public AbstractHdfsConnector()

  • Method Details

    • resetHDFSResources

      protected HdfsResources resetHDFSResources(HdfsSinkConfig hdfsSinkConfig) throws IOException
      Throws:
      IOException

    • checkHdfsUriForTimeout

      protected void checkHdfsUriForTimeout(org.apache.hadoop.conf.Configuration config) throws IOException
      Throws:
      IOException

    • getFileSystem

      protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration config) throws IOException
      This exists in order to allow unit tests to override it so that they don't take several minutes waiting for UDP packets to be received.
      Parameters:
      config - the configuration to use
      Returns:
      the FileSystem that is created for the given Configuration
      Throws:
      IOException - if unable to create the FileSystem

    • getFileSystemAsUser

      protected org.apache.hadoop.fs.FileSystem getFileSystemAsUser(org.apache.hadoop.conf.Configuration config, org.apache.hadoop.security.UserGroupInformation ugi) throws IOException
      Throws:
      IOException

    • getConfiguration

      protected org.apache.hadoop.conf.Configuration getConfiguration()

    • getFileSystem

      protected org.apache.hadoop.fs.FileSystem getFileSystem()

    • getUserGroupInformation

      protected org.apache.hadoop.security.UserGroupInformation getUserGroupInformation()

    • getEncoding

      protected String getEncoding()

    • getCompressionCodec

      protected org.apache.hadoop.io.compress.CompressionCodec getCompressionCodec()

    • getCompressionCodecFactory

      protected org.apache.hadoop.io.compress.CompressionCodecFactory getCompressionCodecFactory()