org.openstreetmap.atlas.checks.distributed

Class IntegrityCheckSparkJob

java.lang.Object

org.openstreetmap.atlas.utilities.runtime.Command

org.openstreetmap.atlas.generator.tools.spark.SparkJob

org.openstreetmap.atlas.checks.distributed.IntegrityCheckSparkJob

All Implemented Interfaces:

java.io.Serializable

public class IntegrityCheckSparkJob
extends org.openstreetmap.atlas.generator.tools.spark.SparkJob

Executes integrity checks as a SparkJob. The job parallelizes on the number of countries. Each slave will process a country and run all enabled checks.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.openstreetmap.atlas.utilities.runtime.Command

org.openstreetmap.atlas.utilities.runtime.Command.Flag, org.openstreetmap.atlas.utilities.runtime.Command.Optionality, org.openstreetmap.atlas.utilities.runtime.Command.Switch<T>, org.openstreetmap.atlas.utilities.runtime.Command.SwitchList

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.openstreetmap.atlas.utilities.runtime.Command.Switch<java.lang.String>	ATLAS_FOLDER Deprecated.

Fields inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

ADDITIONAL_SPARK_OPTIONS, COMPRESS_OUTPUT, EXITED_FILE, FAILED_FILE, INPUT, MASTER, OUTPUT, SAVING_SEPARATOR, SENSITIVE_CONFIGURATION_PATTERN, SPARK_CONTEXT_PROVIDER, SPARK_OPTIONS, STARTED_FILE, STARTED_FOLDER, SUCCESS_FILE

Constructor Summary

Constructors

Constructor and Description
IntegrityCheckSparkJob()

Method Summary

Modifier and Type	Method and Description
protected AtlasDataSource	getAtlasDataSource(java.util.Map<java.lang.String,java.lang.String> sparkContext, org.openstreetmap.atlas.utilities.configuration.Configuration checksConfiguration, org.openstreetmap.atlas.geography.Rectangle pbfBoundary) Gets the AtlasDataSource object to load the Atlas from
java.lang.String	getName()
static void	main(java.lang.String[] args) Main entry point for the Spark job
protected java.util.List<java.lang.String>	outputToClean(org.openstreetmap.atlas.utilities.runtime.CommandMap command) Defines all the folders to clean before a run
void	start(org.openstreetmap.atlas.utilities.runtime.CommandMap commandMap)
protected org.openstreetmap.atlas.utilities.runtime.Command.SwitchList	switches()

Methods inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

configuration, configurationMap, getAlternateParallelFolderOutput, getAlternateSubFolderOutput, getContext, input, onRun, output, resource, resource, splitAndSaveAsHadoopFile

Methods inherited from class org.openstreetmap.atlas.utilities.runtime.Command

commandSummary, getCommandMap, lastRawCommand, run, runWithoutQuitting

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ATLAS_FOLDER

@Deprecated
protected static final org.openstreetmap.atlas.utilities.runtime.Command.Switch<java.lang.String> ATLAS_FOLDER

Deprecated.

Constructor Detail

IntegrityCheckSparkJob

public IntegrityCheckSparkJob()

Method Detail

main

public static void main(java.lang.String[] args)

Main entry point for the Spark job

Parameters:

args - arguments for the Spark job

getName

public java.lang.String getName()

Specified by:

getName in class org.openstreetmap.atlas.generator.tools.spark.SparkJob

start

public void start(org.openstreetmap.atlas.utilities.runtime.CommandMap commandMap)

Specified by:

start in class org.openstreetmap.atlas.generator.tools.spark.SparkJob

getAtlasDataSource

protected AtlasDataSource getAtlasDataSource(java.util.Map<java.lang.String,java.lang.String> sparkContext,
                                             org.openstreetmap.atlas.utilities.configuration.Configuration checksConfiguration,
                                             org.openstreetmap.atlas.geography.Rectangle pbfBoundary)

Gets the AtlasDataSource object to load the Atlas from

Parameters:

sparkContext - The Spark context

checksConfiguration - configuration for all the checks

pbfBoundary - The pbf boundary of type Rectangle

Returns:

A AtlasDataSource

outputToClean

protected java.util.List<java.lang.String> outputToClean(org.openstreetmap.atlas.utilities.runtime.CommandMap command)

Defines all the folders to clean before a run

Overrides:

outputToClean in class org.openstreetmap.atlas.generator.tools.spark.SparkJob

Parameters:

command - the command parameters sent to the main class

Returns:

all the paths to clean

switches

protected org.openstreetmap.atlas.utilities.runtime.Command.SwitchList switches()

Overrides:

switches in class org.openstreetmap.atlas.generator.tools.spark.SparkJob