org.openstreetmap.atlas.generator.tools.streaming.resource

Class HDFSWalker

java.lang.Object

org.openstreetmap.atlas.generator.tools.streaming.resource.HDFSWalker

public final class HDFSWalker
extends java.lang.Object

Inspired by Files.walk (https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#walk-java.nio.file.Path-int- java.nio.file.FileVisitOption...-), either retrieve a Stream of HDFS file status objects or convert them to HDFSFile objects.

Field Summary

Fields

Modifier and Type	Field and Description
static int	WALK_ALL

Constructor Summary

Constructors

Constructor and Description
HDFSWalker()
HDFSWalker(int maxDepth)

Method Summary

Modifier and Type	Method and Description
static HDFSFile	convert(org.apache.hadoop.fs.FileStatus status)
static java.util.function.Function<org.apache.hadoop.fs.FileStatus,org.apache.hadoop.fs.FileStatus>	debug(java.util.function.Consumer<java.lang.String> printer)
static java.util.function.Function<org.apache.hadoop.fs.FileStatus,org.apache.hadoop.fs.FileStatus>	size(java.util.concurrent.atomic.AtomicLong value)
HDFSWalker	usingConfiguration(org.apache.hadoop.conf.Configuration configuration)
java.util.stream.Stream<org.apache.hadoop.fs.FileStatus>	walk(org.apache.hadoop.fs.Path root)
java.util.stream.Stream<HDFSFile>	walkFiles(org.apache.hadoop.fs.Path root)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WALK_ALL

public static final int WALK_ALL

See Also:

Constant Field Values

Constructor Detail

HDFSWalker

public HDFSWalker()

HDFSWalker

public HDFSWalker(int maxDepth)

Method Detail

convert

public static HDFSFile convert(org.apache.hadoop.fs.FileStatus status)

debug

public static java.util.function.Function<org.apache.hadoop.fs.FileStatus,org.apache.hadoop.fs.FileStatus> debug(java.util.function.Consumer<java.lang.String> printer)

size

public static java.util.function.Function<org.apache.hadoop.fs.FileStatus,org.apache.hadoop.fs.FileStatus> size(java.util.concurrent.atomic.AtomicLong value)

usingConfiguration

public HDFSWalker usingConfiguration(org.apache.hadoop.conf.Configuration configuration)

walk

public java.util.stream.Stream<org.apache.hadoop.fs.FileStatus> walk(org.apache.hadoop.fs.Path root)

walkFiles

public java.util.stream.Stream<HDFSFile> walkFiles(org.apache.hadoop.fs.Path root)