org.openstreetmap.atlas.generator.tools.filesystem

Class FileSystemHelper

java.lang.Object

org.openstreetmap.atlas.generator.tools.filesystem.FileSystemHelper

public final class FileSystemHelper
extends java.lang.Object

Utility class that helps generating Resource and WritableResource from a Hadoop Path.

Method Summary

Modifier and Type	Method and Description
static boolean	delete(java.lang.String path, boolean recursive, java.util.Map<java.lang.String,java.lang.String> configuration) Deletes given path using given configuration settings.
static boolean	exists(java.lang.String path) Check if the given path exists.
static boolean	exists(java.lang.String path, java.util.Map<java.lang.String,java.lang.String> configuration) Check if the given path exists.
static java.util.List<org.openstreetmap.atlas.streaming.resource.Resource>	listResourcesRecursively(java.lang.String directory, java.util.Map<java.lang.String,java.lang.String> configuration, org.apache.hadoop.fs.PathFilter filter)
static boolean	mkdir(java.lang.String path, java.util.Map<java.lang.String,java.lang.String> configuration) Creates a new directory for given path using given configuration settings.
static boolean	rename(java.lang.String sourcePath, java.lang.String destinationPath, java.util.Map<java.lang.String,java.lang.String> configuration) Renames a source path to a destination path using the given configuration settings.
static org.openstreetmap.atlas.streaming.resource.Resource	resource(java.lang.String path)
static org.openstreetmap.atlas.streaming.resource.Resource	resource(java.lang.String path, java.util.Map<java.lang.String,java.lang.String> configuration)
static java.util.List<org.openstreetmap.atlas.streaming.resource.Resource>	resources(java.lang.String directory, java.util.Map<java.lang.String,java.lang.String> configuration) List resources, but omit the hadoop "_SUCCESS" file.
static java.util.List<org.openstreetmap.atlas.streaming.resource.Resource>	resources(java.lang.String directory, java.util.Map<java.lang.String,java.lang.String> configuration, org.apache.hadoop.fs.PathFilter filter)
static java.util.stream.Stream<org.apache.hadoop.fs.Path>	streamPathsRecursively(java.lang.String directory, java.util.Map<java.lang.String,java.lang.String> configuration, org.apache.hadoop.fs.PathFilter filter)
static org.openstreetmap.atlas.streaming.resource.WritableResource	writableResource(java.lang.String path)
static org.openstreetmap.atlas.streaming.resource.WritableResource	writableResource(java.lang.String path, java.util.Map<java.lang.String,java.lang.String> configuration)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

delete

public static boolean delete(java.lang.String path,
                             boolean recursive,
                             java.util.Map<java.lang.String,java.lang.String> configuration)

Deletes given path using given configuration settings.

Parameters:

path - Path to delete

recursive - If given path is a directory and this is set, then directory and all child items will be deleted, otherwise throws Exception.

configuration - Configuration settings to use as context

Returns:

true if deletion succeeded

exists

public static boolean exists(java.lang.String path)

Check if the given path exists.

Parameters:

path - The path to check

Returns:

If the given path exists

exists

public static boolean exists(java.lang.String path,
                             java.util.Map<java.lang.String,java.lang.String> configuration)

Check if the given path exists.

Parameters:

path - The path to check

configuration - The configuration map

Returns:

If the given path exists

listResourcesRecursively

public static java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> listResourcesRecursively(java.lang.String directory,
                                                                                                           java.util.Map<java.lang.String,java.lang.String> configuration,
                                                                                                           org.apache.hadoop.fs.PathFilter filter)

Parameters:

directory - The directory from which to recursively load files

configuration - The configuration (containing the filesystem definition)

filter - The path filter. If null, all the files will be returned.

Returns:

a list of Resources

mkdir

public static boolean mkdir(java.lang.String path,
                            java.util.Map<java.lang.String,java.lang.String> configuration)

Creates a new directory for given path using given configuration settings.

Parameters:

path - Path to use for directory creation operation

configuration - Configuration settings to use as context

Returns:

true if create operation succeeded

rename

public static boolean rename(java.lang.String sourcePath,
                             java.lang.String destinationPath,
                             java.util.Map<java.lang.String,java.lang.String> configuration)

Renames a source path to a destination path using the given configuration settings. This assumes that the source and destination path file systems are the same. Hence, the source path is used as a reference while initializing file system.

Parameters:

sourcePath - Path to rename from

destinationPath - Path to rename to

configuration - Configuration settings to use as context

Returns:

true if rename operation succeeded

resource

public static org.openstreetmap.atlas.streaming.resource.Resource resource(java.lang.String path)

Parameters:

path - The path to create the resource from

Returns:

A Resource coming from the default RawLocalFileSystem

resource

public static org.openstreetmap.atlas.streaming.resource.Resource resource(java.lang.String path,
                                                                           java.util.Map<java.lang.String,java.lang.String> configuration)

Parameters:

path - The path to create the resource from

configuration - The configuration defining the FileSystem

Returns:

A Resource coming from the appropriate FileSystem

resources

public static java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> resources(java.lang.String directory,
                                                                                            java.util.Map<java.lang.String,java.lang.String> configuration)

List resources, but omit the hadoop "_SUCCESS" file.

Parameters:

directory - The directory from which to load files

configuration - The configuration defining the FileSystem

Returns:

a list of Resources which contains all the files but the _SUCCESS file

resources

public static java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> resources(java.lang.String directory,
                                                                                            java.util.Map<java.lang.String,java.lang.String> configuration,
                                                                                            org.apache.hadoop.fs.PathFilter filter)

Parameters:

directory - The directory from which to load files

configuration - The configuration defining the FileSystem

filter - The path filter. If null, all the files will be returned.

Returns:

a list of Resources

streamPathsRecursively

public static java.util.stream.Stream<org.apache.hadoop.fs.Path> streamPathsRecursively(java.lang.String directory,
                                                                                        java.util.Map<java.lang.String,java.lang.String> configuration,
                                                                                        org.apache.hadoop.fs.PathFilter filter)

Parameters:

directory - The directory from which to recursively stream paths

configuration - The configuration (containing the filesystem definition)

filter - The path filter. If null, all the paths will be returned.

Returns:

a stream of Paths

writableResource

public static org.openstreetmap.atlas.streaming.resource.WritableResource writableResource(java.lang.String path)

Parameters:

path - The path to create the resource from

Returns:

A WritableResource coming from the default RawLocalFileSystem

writableResource

public static org.openstreetmap.atlas.streaming.resource.WritableResource writableResource(java.lang.String path,
                                                                                           java.util.Map<java.lang.String,java.lang.String> configuration)

Parameters:

path - The path to create the resource from

configuration - The configuration defining the FileSystem

Returns:

A WritableResource coming from the appropriate FileSystem