Package org.openstreetmap.atlas.checks.distributed

Class IntegrityCheckSparkJob

  • All Implemented Interfaces:
    java.io.Serializable
    public class IntegrityCheckSparkJob
extends org.openstreetmap.atlas.generator.tools.spark.SparkJob
    Executes integrity checks as a SparkJob. The job parallelizes on the number of countries. Each slave will process a country and run all enabled checks.
    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.openstreetmap.atlas.utilities.runtime.Command

        org.openstreetmap.atlas.utilities.runtime.Command.Flag, org.openstreetmap.atlas.utilities.runtime.Command.Optionality, org.openstreetmap.atlas.utilities.runtime.Command.Switch<T extends java.lang.Object>, org.openstreetmap.atlas.utilities.runtime.Command.SwitchList

    • Field Summary

      • Fields inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

        ADDITIONAL_SPARK_OPTIONS, COMPRESS_OUTPUT, FAILED_FILE, INPUT, MASTER, OUTPUT, SAVING_SEPARATOR, SPARK_CONTEXT_PROVIDER, SPARK_OPTIONS, SUCCESS_FILE

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected AtlasDataSource getAtlasDataSource​(java.util.Map<java.lang.String,​java.lang.String> sparkContext, org.openstreetmap.atlas.utilities.configuration.Configuration checksConfiguration, org.openstreetmap.atlas.geography.Rectangle pbfBoundary)
      Gets the AtlasDataSource object to load the Atlas from
      java.lang.String getName()  
      static void main​(java.lang.String[] args)
      Main entry point for the Spark job
      protected java.util.List<java.lang.String> outputToClean​(org.openstreetmap.atlas.utilities.runtime.CommandMap command)
      Defines all the folders to clean before a run
      void start​(org.openstreetmap.atlas.utilities.runtime.CommandMap commandMap)  
      protected org.openstreetmap.atlas.utilities.runtime.Command.SwitchList switches()  

      • Methods inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

        configuration, configurationMap, copyToOutput, getAlternateParallelFolderOutput, getAlternateSubFolderOutput, getContext, input, onRun, output, resource, resource, setContext, splitAndSaveAsHadoopFile

      • Methods inherited from class org.openstreetmap.atlas.utilities.runtime.Command

        commandSummary, getCommandMap, lastRawCommand, run, runWithoutQuitting

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • IntegrityCheckSparkJob

        public IntegrityCheckSparkJob()

    • Method Detail

      • main

        public static void main​(java.lang.String[] args)
        Main entry point for the Spark job
        Parameters:
        args - arguments for the Spark job

      • getName

        public java.lang.String getName()
        Specified by:
        getName in class org.openstreetmap.atlas.generator.tools.spark.SparkJob

      • start

        public void start​(org.openstreetmap.atlas.utilities.runtime.CommandMap commandMap)
        Specified by:
        start in class org.openstreetmap.atlas.generator.tools.spark.SparkJob

      • getAtlasDataSource

        protected AtlasDataSource getAtlasDataSource​(java.util.Map<java.lang.String,​java.lang.String> sparkContext,
                                             org.openstreetmap.atlas.utilities.configuration.Configuration checksConfiguration,
                                             org.openstreetmap.atlas.geography.Rectangle pbfBoundary)
        Gets the AtlasDataSource object to load the Atlas from
        Parameters:
        sparkContext - The Spark context
        checksConfiguration - configuration for all the checks
        pbfBoundary - The pbf boundary of type Rectangle
        Returns:
        A AtlasDataSource

      • outputToClean

        protected java.util.List<java.lang.String> outputToClean​(org.openstreetmap.atlas.utilities.runtime.CommandMap command)
        Defines all the folders to clean before a run
        Overrides:
        outputToClean in class org.openstreetmap.atlas.generator.tools.spark.SparkJob
        Parameters:
        command - the command parameters sent to the main class
        Returns:
        all the paths to clean

      • switches

        protected org.openstreetmap.atlas.utilities.runtime.Command.SwitchList switches()
        Overrides:
        switches in class org.openstreetmap.atlas.generator.tools.spark.SparkJob