public class IntegrityCheckSparkJob
extends org.openstreetmap.atlas.generator.tools.spark.SparkJob
SparkJob. The job parallelizes on the number of countries.
Each slave will process a country and run all enabled checks.org.openstreetmap.atlas.utilities.runtime.Command.Flag, org.openstreetmap.atlas.utilities.runtime.Command.Optionality, org.openstreetmap.atlas.utilities.runtime.Command.Switch<T>, org.openstreetmap.atlas.utilities.runtime.Command.SwitchList| Modifier and Type | Field and Description |
|---|---|
protected static org.openstreetmap.atlas.utilities.runtime.Command.Switch<java.lang.String> |
ATLAS_FOLDER
Deprecated.
|
ADDITIONAL_SPARK_OPTIONS, COMPRESS_OUTPUT, EXITED_FILE, FAILED_FILE, INPUT, MASTER, OUTPUT, SAVING_SEPARATOR, SENSITIVE_CONFIGURATION_PATTERN, SPARK_CONTEXT_PROVIDER, SPARK_OPTIONS, STARTED_FILE, STARTED_FOLDER, SUCCESS_FILE| Constructor and Description |
|---|
IntegrityCheckSparkJob() |
| Modifier and Type | Method and Description |
|---|---|
protected AtlasDataSource |
getAtlasDataSource(java.util.Map<java.lang.String,java.lang.String> sparkContext,
org.openstreetmap.atlas.utilities.configuration.Configuration checksConfiguration,
org.openstreetmap.atlas.geography.Rectangle pbfBoundary)
Gets the
AtlasDataSource object to load the Atlas from |
java.lang.String |
getName() |
static void |
main(java.lang.String[] args)
Main entry point for the Spark job
|
protected java.util.List<java.lang.String> |
outputToClean(org.openstreetmap.atlas.utilities.runtime.CommandMap command)
Defines all the folders to clean before a run
|
void |
start(org.openstreetmap.atlas.utilities.runtime.CommandMap commandMap) |
protected org.openstreetmap.atlas.utilities.runtime.Command.SwitchList |
switches() |
configuration, configurationMap, getAlternateParallelFolderOutput, getAlternateSubFolderOutput, getContext, input, onRun, output, resource, resource, splitAndSaveAsHadoopFile@Deprecated protected static final org.openstreetmap.atlas.utilities.runtime.Command.Switch<java.lang.String> ATLAS_FOLDER
public static void main(java.lang.String[] args)
args - arguments for the Spark jobpublic java.lang.String getName()
getName in class org.openstreetmap.atlas.generator.tools.spark.SparkJobpublic void start(org.openstreetmap.atlas.utilities.runtime.CommandMap commandMap)
start in class org.openstreetmap.atlas.generator.tools.spark.SparkJobprotected AtlasDataSource getAtlasDataSource(java.util.Map<java.lang.String,java.lang.String> sparkContext, org.openstreetmap.atlas.utilities.configuration.Configuration checksConfiguration, org.openstreetmap.atlas.geography.Rectangle pbfBoundary)
AtlasDataSource object to load the Atlas fromsparkContext - The Spark contextchecksConfiguration - configuration for all the checkspbfBoundary - The pbf boundary of type RectangleAtlasDataSourceprotected java.util.List<java.lang.String> outputToClean(org.openstreetmap.atlas.utilities.runtime.CommandMap command)
outputToClean in class org.openstreetmap.atlas.generator.tools.spark.SparkJobcommand - the command parameters sent to the main classprotected org.openstreetmap.atlas.utilities.runtime.Command.SwitchList switches()
switches in class org.openstreetmap.atlas.generator.tools.spark.SparkJob