Package org.openstreetmap.atlas.checks.distributed

Class ShardedIntegrityChecksSparkJob

  • All Implemented Interfaces:
    java.io.Serializable
    public class ShardedIntegrityChecksSparkJob
extends IntegrityChecksCommandArguments
    A spark job for generating integrity checks in a sharded fashion. This allows for a lower local memory profile as well as better parallelization.
    This implementation currently only supports Atlas files as an input data source. They must be in the structure: folder/iso_code/iso_z-x-y.atlas
    Also required is a reference for how the Atlases are sharded. This can be provided as a sharding.txt file in the input path, or through the sharding argument.
    See Also:
    Serialized Form

    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.openstreetmap.atlas.utilities.runtime.Command

        org.openstreetmap.atlas.utilities.runtime.Command.Flag, org.openstreetmap.atlas.utilities.runtime.Command.Optionality, org.openstreetmap.atlas.utilities.runtime.Command.Switch<T extends java.lang.Object>, org.openstreetmap.atlas.utilities.runtime.Command.SwitchList

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      java.lang.String getName()  
      static void main​(java.lang.String[] args)  
      void start​(org.openstreetmap.atlas.utilities.runtime.CommandMap commandMap)  
      protected org.openstreetmap.atlas.utilities.runtime.Command.SwitchList switches()  

      • Methods inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

        configuration, configurationMap, copyToOutput, getAlternateParallelFolderOutput, getAlternateSubFolderOutput, getContext, input, onRun, output, outputToClean, resource, resource, setContext, splitAndSaveAsHadoopFile

      • Methods inherited from class org.openstreetmap.atlas.utilities.runtime.Command

        commandSummary, getCommandMap, lastRawCommand, run, runWithoutQuitting

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • ShardedIntegrityChecksSparkJob

        public ShardedIntegrityChecksSparkJob()

    • Method Detail

      • main

        public static void main​(java.lang.String[] args)

      • getName

        public java.lang.String getName()
        Specified by:
        getName in class org.openstreetmap.atlas.generator.tools.spark.SparkJob

      • start

        public void start​(org.openstreetmap.atlas.utilities.runtime.CommandMap commandMap)
        Specified by:
        start in class org.openstreetmap.atlas.generator.tools.spark.SparkJob