Package org.openstreetmap.atlas.checks.distributed

Class IntegrityCheckSparkJob

java.lang.Object

org.openstreetmap.atlas.utilities.runtime.Command

org.openstreetmap.atlas.generator.tools.spark.SparkJob

org.openstreetmap.atlas.checks.distributed.IntegrityChecksCommandArguments

org.openstreetmap.atlas.checks.distributed.IntegrityCheckSparkJob

All Implemented Interfaces:

java.io.Serializable

public class IntegrityCheckSparkJob
extends IntegrityChecksCommandArguments

Executes integrity checks as a SparkJob. The job parallelizes on the number of countries. Each slave will process a country and run all enabled checks.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.openstreetmap.atlas.checks.distributed.IntegrityChecksCommandArguments

IntegrityChecksCommandArguments.OutputFormats

Nested classes/interfaces inherited from class org.openstreetmap.atlas.utilities.runtime.Command

org.openstreetmap.atlas.utilities.runtime.Command.Flag, org.openstreetmap.atlas.utilities.runtime.Command.Optionality, org.openstreetmap.atlas.utilities.runtime.Command.Switch<T extends java.lang.Object>, org.openstreetmap.atlas.utilities.runtime.Command.SwitchList

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	METRICS_FILENAME

Fields inherited from class org.openstreetmap.atlas.checks.distributed.IntegrityChecksCommandArguments

ATLAS_FOLDER, OUTPUT_ATLAS_FOLDER, OUTPUT_FLAG_FOLDER, OUTPUT_GEOJSON_FOLDER, OUTPUT_METRIC_FOLDER, OUTPUT_TIPPECANOE_FOLDER

Fields inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

ADDITIONAL_SPARK_OPTIONS, COMPRESS_OUTPUT, FAILED_FILE, INPUT, MASTER, OUTPUT, SAVING_SEPARATOR, SPARK_CONTEXT_PROVIDER, SPARK_OPTIONS, SUCCESS_FILE

Constructor Summary

Constructors

Constructor	Description
IntegrityCheckSparkJob()

Method Summary

Modifier and Type	Method	Description
java.lang.String	getName()
static void	main(java.lang.String[] args)	Main entry point for the Spark job
protected java.util.List<java.lang.String>	outputToClean(org.openstreetmap.atlas.utilities.runtime.CommandMap command)	Defines all the folders to clean before a run
void	start(org.openstreetmap.atlas.utilities.runtime.CommandMap commandMap)

Methods inherited from class org.openstreetmap.atlas.checks.distributed.IntegrityChecksCommandArguments

countryShardMapFromShardFiles, getAtlasDataSource, objectsToCheck, objectsToCheck, switches

Methods inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

configuration, configurationMap, copyToOutput, getAlternateParallelFolderOutput, getAlternateSubFolderOutput, getContext, input, onRun, output, resource, resource, setContext, splitAndSaveAsHadoopFile

Methods inherited from class org.openstreetmap.atlas.utilities.runtime.Command

commandSummary, getCommandMap, lastRawCommand, run, runWithoutQuitting

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

METRICS_FILENAME

public static final java.lang.String METRICS_FILENAME

See Also:

Constant Field Values

Constructor Detail

IntegrityCheckSparkJob

public IntegrityCheckSparkJob()

Method Detail

main

public static void main(java.lang.String[] args)

Main entry point for the Spark job

Parameters:

args - arguments for the Spark job

getName

public java.lang.String getName()

Specified by:

getName in class org.openstreetmap.atlas.generator.tools.spark.SparkJob

start

public void start(org.openstreetmap.atlas.utilities.runtime.CommandMap commandMap)

Specified by:

start in class org.openstreetmap.atlas.generator.tools.spark.SparkJob

outputToClean

protected java.util.List<java.lang.String> outputToClean(org.openstreetmap.atlas.utilities.runtime.CommandMap command)

Defines all the folders to clean before a run

Overrides:

outputToClean in class org.openstreetmap.atlas.generator.tools.spark.SparkJob

Parameters:

command - the command parameters sent to the main class

Returns:

all the paths to clean