Package org.openstreetmap.atlas.generator.tools.streaming.resource

Class HDFSWalker

  • public final class HDFSWalker
extends java.lang.Object
    Inspired by Files.walk (https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#walk-java.nio.file.Path-int- java.nio.file.FileVisitOption...-), either retrieve a Stream of HDFS file status objects or convert them to HDFSFile objects.

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static int WALK_ALL  

    • Constructor Summary

      Constructors 
      Constructor Description
      HDFSWalker()  
      HDFSWalker​(int maxDepth)  

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      static HDFSFile convert​(org.apache.hadoop.fs.FileStatus status)  
      static java.util.function.Function<org.apache.hadoop.fs.FileStatus,​org.apache.hadoop.fs.FileStatus> debug​(java.util.function.Consumer<java.lang.String> printer)  
      static java.util.function.Function<org.apache.hadoop.fs.FileStatus,​org.apache.hadoop.fs.FileStatus> size​(java.util.concurrent.atomic.AtomicLong value)  
      HDFSWalker usingConfiguration​(org.apache.hadoop.conf.Configuration configuration)  
      java.util.stream.Stream<org.apache.hadoop.fs.FileStatus> walk​(org.apache.hadoop.fs.Path root)  
      java.util.stream.Stream<HDFSFile> walkFiles​(org.apache.hadoop.fs.Path root)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • HDFSWalker

        public HDFSWalker()

      • HDFSWalker

        public HDFSWalker​(int maxDepth)

    • Method Detail

      • convert

        public static HDFSFile convert​(org.apache.hadoop.fs.FileStatus status)

      • debug

        public static java.util.function.Function<org.apache.hadoop.fs.FileStatus,​org.apache.hadoop.fs.FileStatus> debug​(java.util.function.Consumer<java.lang.String> printer)

      • size

        public static java.util.function.Function<org.apache.hadoop.fs.FileStatus,​org.apache.hadoop.fs.FileStatus> size​(java.util.concurrent.atomic.AtomicLong value)

      • usingConfiguration

        public HDFSWalker usingConfiguration​(org.apache.hadoop.conf.Configuration configuration)

      • walk

        public java.util.stream.Stream<org.apache.hadoop.fs.FileStatus> walk​(org.apache.hadoop.fs.Path root)

      • walkFiles

        public java.util.stream.Stream<HDFSFile> walkFiles​(org.apache.hadoop.fs.Path root)