public class SparkFileHelper
extends java.lang.Object
implements java.io.Serializable
Atlas files| Modifier and Type | Field and Description |
|---|---|
static java.lang.String |
DIRECTORY_SEPARATOR |
static java.lang.String |
EMPTY_STRING |
static java.lang.String |
EXTENSION_SEPARATOR |
static java.lang.String |
TEMPORARY_FOLDER_NAME |
| Constructor and Description |
|---|
SparkFileHelper(java.util.Map<java.lang.String,java.lang.String> sparkContext)
Constructs a helper with given context
|
| Modifier and Type | Method and Description |
|---|---|
java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> |
collectAtlasFiles(java.lang.String directory,
java.lang.String country,
boolean recursive)
Returns a list of Atlas
Resources for the given country in the supplied directory |
java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource> |
collectSourceFile(java.lang.String uri,
org.apache.hadoop.fs.PathFilter... filters)
Returns an Atlas
Resource for the given location URI string. |
java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> |
collectSourceFiles(java.lang.String directory,
boolean recursive,
org.apache.hadoop.fs.PathFilter... filters)
Returns an list of Atlas
Resources from the given location URI string. |
static java.lang.String |
combine(java.lang.String basePath,
java.lang.String... paths)
Combines given paths
|
void |
commit(SparkFilePath path)
Renames the
SparkFilePath.temporaryPath to the SparkFilePath.targetPath,
taking care to avoid producing nested directories. |
void |
commitByCopy(SparkFilePath path)
Copies the
SparkFilePath.temporaryPath to the SparkFilePath.targetPath,
taking care to avoid producing nested directories. |
void |
deleteDirectory(java.lang.String path)
Deletes given directory and all it's child items
|
boolean |
directoryContainsExpectedCountryAtlases(java.lang.String directory,
org.openstreetmap.atlas.utilities.collections.StringList expectedCountries,
boolean recursive)
Verifies that the input directory containing
Atlas files contains all the expected
countries |
boolean |
exists(java.lang.String path) |
static java.lang.String |
extensionStartingWithSeparator(java.lang.String path)
Removes
EXTENSION_SEPARATOR from the beginning of given path |
boolean |
isDirectory(java.lang.String path) |
java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> |
list(java.lang.String path) |
void |
mkdir(java.lang.String path)
Creates a directory via given path
|
static java.lang.String |
parentPath(java.lang.String path)
Returns parent of given path
|
static java.lang.String |
pathNotEndingWithSeparator(java.lang.String path)
Removes
PATH_SEPARATOR from the end of given path |
static java.lang.String |
pathNotStartingWithSeparator(java.lang.String path)
Removes
PATH_SEPARATOR from the beginning of given path. |
void |
rename(java.lang.String sourcePath,
java.lang.String destinationPath)
Renames a source path to destination path
|
void |
save(java.util.List<SparkFileOutput> outputs)
Executes
SparkFileOutput#getSaveFunc() for given SparkFileOutputs |
void |
save(SparkFileOutput... outputs)
Executes
SparkFileOutput#getSaveFunc() for given SparkFileOutputs |
void |
write(java.lang.String directory,
java.lang.String filename,
byte[] content)
Writes given content into given directory with given filename
|
void |
write(java.lang.String directory,
java.lang.String filename,
java.lang.String content)
Writes given content into given directory with given filename
|
public static final java.lang.String TEMPORARY_FOLDER_NAME
public static final java.lang.String EMPTY_STRING
public static final java.lang.String DIRECTORY_SEPARATOR
public static final java.lang.String EXTENSION_SEPARATOR
public SparkFileHelper(java.util.Map<java.lang.String,java.lang.String> sparkContext)
sparkContext - Spark context as key-value pairs to use as contextpublic static java.lang.String combine(java.lang.String basePath,
java.lang.String... paths)
basePath - a base pathpaths - a list of pathspublic static java.lang.String extensionStartingWithSeparator(java.lang.String path)
EXTENSION_SEPARATOR from the beginning of given pathpath - Path to remove EXTENSION_SEPARATOR from the beginningEXTENSION_SEPARATORpublic static java.lang.String parentPath(java.lang.String path)
path - a pathpublic static java.lang.String pathNotEndingWithSeparator(java.lang.String path)
PATH_SEPARATOR from the end of given pathpath - a pathPATH_SEPARATORpublic static java.lang.String pathNotStartingWithSeparator(java.lang.String path)
PATH_SEPARATOR from the beginning of given path.path - a pathPATH_SEPARATORpublic java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> collectAtlasFiles(java.lang.String directory,
java.lang.String country,
boolean recursive)
Resources for the given country in the supplied directorydirectory - the directory from which to collect the Atlas filescountry - the country, whose Atlas files we're interested inrecursive - true to search the given directory and all sub-directories, false
to only search the root directoryResources for the given countrypublic java.util.Optional<org.openstreetmap.atlas.streaming.resource.Resource> collectSourceFile(java.lang.String uri,
org.apache.hadoop.fs.PathFilter... filters)
Resource for the given location URI string. The resource is resolve
and returned if the URI points to single resource, not a resource directory, that conforms to
a path defined by one of the provided PathFilters. The PathFilters provide a
way to find well known data types that can be used either directly as or transformed to an
Atlas. With no filters, the file is collected.uri - the location of the Atlas datasourcefilters - PathFilters used to find datasource typesAtlas Resourcepublic java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> collectSourceFiles(java.lang.String directory,
boolean recursive,
org.apache.hadoop.fs.PathFilter... filters)
Resources from the given location URI string. Resources are
resolve and returned that conform to any one of the PathFilters. The
PathFilters provide a way to find well known data types that can be used either
directly as or transformed to an Atlas.directory - a location of the Atlas datasourcerecursive - true, to search the given directory and all sub-directories. false
, to only search the root directoryfilters - PathFilters used to find datasource typesResourcepublic void commit(SparkFilePath path)
SparkFilePath.temporaryPath to the SparkFilePath.targetPath,
taking care to avoid producing nested directories.path - SparkFilePath to commitpublic void commitByCopy(SparkFilePath path)
SparkFilePath.temporaryPath to the SparkFilePath.targetPath,
taking care to avoid producing nested directories.path - SparkFilePath to commitpublic void deleteDirectory(java.lang.String path)
path - a pathpublic boolean directoryContainsExpectedCountryAtlases(java.lang.String directory,
org.openstreetmap.atlas.utilities.collections.StringList expectedCountries,
boolean recursive)
Atlas files contains all the expected
countriesdirectory - the directory which contains the Atlas filesexpectedCountries - the expected StringList of country ISO3 codesrecursive - true to search the given directory and all sub-directories, false
to only search the root directorytrue if all expected country ISO3 codes are in the given directory,
false otherwisepublic boolean exists(java.lang.String path)
path - Path to check if it exists or notpublic boolean isDirectory(java.lang.String path)
path - Path to check if it is a directory or notpublic java.util.List<org.openstreetmap.atlas.streaming.resource.Resource> list(java.lang.String path)
path - Path to directory to list Resources under.Resources under given path.public void mkdir(java.lang.String path)
path - a pathpublic void rename(java.lang.String sourcePath,
java.lang.String destinationPath)
sourcePath - source pathdestinationPath - destination pathpublic void save(java.util.List<SparkFileOutput> outputs)
SparkFileOutput#getSaveFunc() for given SparkFileOutputsoutputs - SparkFileOutputs to executepublic void save(SparkFileOutput... outputs)
SparkFileOutput#getSaveFunc() for given SparkFileOutputsoutputs - SparkFileOutputs to executepublic void write(java.lang.String directory,
java.lang.String filename,
byte[] content)
directory - a directory path to write files intofilename - the name of the filecontent - file content in byte array formpublic void write(java.lang.String directory,
java.lang.String filename,
java.lang.String content)
directory - a directory path to write files intofilename - the name of the filecontent - file content