org.openstreetmap.atlas.generator.tools.spark.utilities

Class SparkFileHelper

java.lang.Object

org.openstreetmap.atlas.generator.tools.spark.utilities.SparkFileHelper

All Implemented Interfaces:

java.io.Serializable

public class SparkFileHelper
extends java.lang.Object
implements java.io.Serializable

Helper class for listing and validating the presence of Atlas files

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DIRECTORY_SEPARATOR
static java.lang.String	EMPTY_STRING
static java.lang.String	EXTENSION_SEPARATOR
static java.lang.String	TEMPORARY_FOLDER_NAME

Constructor Summary

Constructors

Constructor and Description
SparkFileHelper(java.util.Map<java.lang.String,java.lang.String> sparkContext) Constructs a helper with given context

Method Summary

Modifier and Type	Method and Description
java.util.List<org.openstreetmap.atlas.streaming.resource.Resource>	collectAtlasFiles(java.lang.String directory, java.lang.String country, boolean recursive) Returns a list of Atlas Resources for the given country in the supplied directory
java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource>	collectSourceFile(java.lang.String uri, org.apache.hadoop.fs.PathFilter... filters) Returns an Atlas Resource for the given location URI string.
java.util.List<org.openstreetmap.atlas.streaming.resource.Resource>	collectSourceFiles(java.lang.String directory, boolean recursive, org.apache.hadoop.fs.PathFilter... filters) Returns an list of Atlas Resources from the given location URI string.
static java.lang.String	combine(java.lang.String basePath, java.lang.String... paths) Combines given paths
void	commit(SparkFilePath path) Renames the SparkFilePath.temporaryPath to the SparkFilePath.targetPath, taking care to avoid producing nested directories.
void	commitByCopy(SparkFilePath path) Copies the SparkFilePath.temporaryPath to the SparkFilePath.targetPath, taking care to avoid producing nested directories.
void	deleteDirectory(java.lang.String path) Deletes given directory and all it's child items
boolean	directoryContainsExpectedCountryAtlases(java.lang.String directory, org.openstreetmap.atlas.utilities.collections.StringList expectedCountries, boolean recursive) Verifies that the input directory containing Atlas files contains all the expected countries
boolean	exists(java.lang.String path)
static java.lang.String	extensionStartingWithSeparator(java.lang.String path) Removes EXTENSION_SEPARATOR from the beginning of given path
boolean	isDirectory(java.lang.String path)
java.util.List<org.openstreetmap.atlas.streaming.resource.Resource>	list(java.lang.String path)
void	mkdir(java.lang.String path) Creates a directory via given path
static java.lang.String	parentPath(java.lang.String path) Returns parent of given path
static java.lang.String	pathNotEndingWithSeparator(java.lang.String path) Removes PATH_SEPARATOR from the end of given path
static java.lang.String	pathNotStartingWithSeparator(java.lang.String path) Removes PATH_SEPARATOR from the beginning of given path.
void	rename(java.lang.String sourcePath, java.lang.String destinationPath) Renames a source path to destination path
void	save(java.util.List<SparkFileOutput> outputs) Executes SparkFileOutput#getSaveFunc() for given SparkFileOutputs
void	save(SparkFileOutput... outputs) Executes SparkFileOutput#getSaveFunc() for given SparkFileOutputs
void	write(java.lang.String directory, java.lang.String filename, byte[] content) Writes given content into given directory with given filename
void	write(java.lang.String directory, java.lang.String filename, java.lang.String content) Writes given content into given directory with given filename

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEMPORARY_FOLDER_NAME

public static final java.lang.String TEMPORARY_FOLDER_NAME

See Also:

Constant Field Values

EMPTY_STRING

public static final java.lang.String EMPTY_STRING

See Also:

Constant Field Values

DIRECTORY_SEPARATOR

public static final java.lang.String DIRECTORY_SEPARATOR

See Also:

Constant Field Values

EXTENSION_SEPARATOR

public static final java.lang.String EXTENSION_SEPARATOR

See Also:

Constant Field Values

Constructor Detail

SparkFileHelper

public SparkFileHelper(java.util.Map<java.lang.String,java.lang.String> sparkContext)

Constructs a helper with given context

Parameters:

sparkContext - Spark context as key-value pairs to use as context

Method Detail

combine

public static java.lang.String combine(java.lang.String basePath,
                                       java.lang.String... paths)

Combines given paths

Parameters:

basePath - a base path

paths - a list of paths

Returns:

the combined path

extensionStartingWithSeparator

public static java.lang.String extensionStartingWithSeparator(java.lang.String path)

Removes EXTENSION_SEPARATOR from the beginning of given path

Parameters:

path - Path to remove EXTENSION_SEPARATOR from the beginning

Returns:

a path not starting with EXTENSION_SEPARATOR

parentPath

public static java.lang.String parentPath(java.lang.String path)

Returns parent of given path

Parameters:

path - a path

Returns:

the parent path

pathNotEndingWithSeparator

public static java.lang.String pathNotEndingWithSeparator(java.lang.String path)

Removes PATH_SEPARATOR from the end of given path

Parameters:

path - a path

Returns:

a path not ending with PATH_SEPARATOR

pathNotStartingWithSeparator

public static java.lang.String pathNotStartingWithSeparator(java.lang.String path)

Removes PATH_SEPARATOR from the beginning of given path.

Parameters:

path - a path

Returns:

a path not beginning with PATH_SEPARATOR

collectAtlasFiles

public java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> collectAtlasFiles(java.lang.String directory,
                                                                                             java.lang.String country,
                                                                                             boolean recursive)

Returns a list of Atlas Resources for the given country in the supplied directory

Parameters:

directory - the directory from which to collect the Atlas files

country - the country, whose Atlas files we're interested in

recursive - true to search the given directory and all sub-directories, false to only search the root directory

Returns:

a list of Atlas Resources for the given country

collectSourceFile

public java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource> collectSourceFile(java.lang.String uri,
                                                                                                 org.apache.hadoop.fs.PathFilter... filters)

Returns an Atlas Resource for the given location URI string. The resource is resolve and returned if the URI points to single resource, not a resource directory, that conforms to a path defined by one of the provided PathFilters. The PathFilters provide a way to find well known data types that can be used either directly as or transformed to an Atlas. With no filters, the file is collected.

Parameters:

uri - the location of the Atlas datasource

filters - PathFilters used to find datasource types

Returns:

an Atlas Resource

collectSourceFiles

public java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> collectSourceFiles(java.lang.String directory,
                                                                                              boolean recursive,
                                                                                              org.apache.hadoop.fs.PathFilter... filters)

Returns an list of Atlas Resources from the given location URI string. Resources are resolve and returned that conform to any one of the PathFilters. The PathFilters provide a way to find well known data types that can be used either directly as or transformed to an Atlas.

Parameters:

directory - a location of the Atlas datasource

recursive - true, to search the given directory and all sub-directories. false , to only search the root directory

filters - PathFilters used to find datasource types

Returns:

an Atlas Resource

commit

public void commit(SparkFilePath path)

Renames the SparkFilePath.temporaryPath to the SparkFilePath.targetPath, taking care to avoid producing nested directories.

Parameters:

path - SparkFilePath to commit

commitByCopy

public void commitByCopy(SparkFilePath path)

Copies the SparkFilePath.temporaryPath to the SparkFilePath.targetPath, taking care to avoid producing nested directories.

Parameters:

path - SparkFilePath to commit

deleteDirectory

public void deleteDirectory(java.lang.String path)

Deletes given directory and all it's child items

Parameters:

path - a path

directoryContainsExpectedCountryAtlases

public boolean directoryContainsExpectedCountryAtlases(java.lang.String directory,
                                                       org.openstreetmap.atlas.utilities.collections.StringList expectedCountries,
                                                       boolean recursive)

Verifies that the input directory containing Atlas files contains all the expected countries

Parameters:

directory - the directory which contains the Atlas files

expectedCountries - the expected StringList of country ISO3 codes

recursive - true to search the given directory and all sub-directories, false to only search the root directory

Returns:

true if all expected country ISO3 codes are in the given directory, false otherwise

exists

public boolean exists(java.lang.String path)

Parameters:

path - Path to check if it exists or not

Returns:

true if given path exists, otherwise false

isDirectory

public boolean isDirectory(java.lang.String path)

Parameters:

path - Path to check if it is a directory or not

Returns:

true if given path is directory, otherwise false.

list

public java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> list(java.lang.String path)

Parameters:

path - Path to directory to list Resources under.

Returns:

List of Resources under given path.

mkdir

public void mkdir(java.lang.String path)

Creates a directory via given path

Parameters:

path - a path

rename

public void rename(java.lang.String sourcePath,
                   java.lang.String destinationPath)

Renames a source path to destination path

Parameters:

sourcePath - source path

destinationPath - destination path

save

public void save(java.util.List<SparkFileOutput> outputs)

Executes SparkFileOutput#getSaveFunc() for given SparkFileOutputs

Parameters:

outputs - SparkFileOutputs to execute

save

public void save(SparkFileOutput... outputs)

Executes SparkFileOutput#getSaveFunc() for given SparkFileOutputs

Parameters:

outputs - SparkFileOutputs to execute

write

public void write(java.lang.String directory,
                  java.lang.String filename,
                  byte[] content)

Writes given content into given directory with given filename

Parameters:

directory - a directory path to write files into

filename - the name of the file

content - file content in byte array form

write

public void write(java.lang.String directory,
                  java.lang.String filename,
                  java.lang.String content)

Writes given content into given directory with given filename

Parameters:

directory - a directory path to write files into

filename - the name of the file

content - file content