Package org.openstreetmap.atlas.checks.distributed

Class IntegrityChecksCommandArguments

  • All Implemented Interfaces:
    java.io.Serializable
    Direct Known Subclasses:
    IntegrityCheckSparkJob, ShardedIntegrityChecksSparkJob
    public abstract class IntegrityChecksCommandArguments
extends org.openstreetmap.atlas.generator.tools.spark.SparkJob
    Handles arguments and base functionality for integrity check sparkjobs generating commands
    See Also:
    Serialized Form

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      protected static class  IntegrityChecksCommandArguments.OutputFormats  

      • Nested classes/interfaces inherited from class org.openstreetmap.atlas.utilities.runtime.Command

        org.openstreetmap.atlas.utilities.runtime.Command.Flag, org.openstreetmap.atlas.utilities.runtime.Command.Optionality, org.openstreetmap.atlas.utilities.runtime.Command.Switch<T extends java.lang.Object>, org.openstreetmap.atlas.utilities.runtime.Command.SwitchList

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected static org.openstreetmap.atlas.utilities.runtime.Command.Switch<java.lang.String> ATLAS_FOLDER
      Deprecated.
      in favor of INPUT from SparkJob
      protected static java.lang.String OUTPUT_ATLAS_FOLDER  
      protected static java.lang.String OUTPUT_FLAG_FOLDER  
      protected static java.lang.String OUTPUT_GEOJSON_FOLDER  
      protected static java.lang.String OUTPUT_METRIC_FOLDER  
      protected static java.lang.String OUTPUT_TIPPECANOE_FOLDER  

      • Fields inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

        ADDITIONAL_SPARK_OPTIONS, CLUSTER, COMPRESS_OUTPUT, FAILED_FILE, INPUT, OUTPUT, SAVING_SEPARATOR, SPARK_CONTEXT_PROVIDER, SPARK_OPTIONS, SUCCESS_FILE

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      static org.openstreetmap.atlas.utilities.maps.MultiMap<java.lang.String,​org.openstreetmap.atlas.geography.sharding.Shard> countryShardMapFromShardFiles​(java.util.Set<java.lang.String> countries, java.lang.String atlasFolder, java.util.Map<java.lang.String,​java.lang.String> sparkContext)
      Creates a map from country name to List of Shard definitions from Atlas files.
      protected AtlasDataSource getAtlasDataSource​(java.util.Map<java.lang.String,​java.lang.String> sparkContext, org.openstreetmap.atlas.utilities.configuration.Configuration checksConfiguration, org.openstreetmap.atlas.geography.Rectangle pbfBoundary)
      Gets the AtlasDataSource object to load the Atlas from
      protected static java.lang.Iterable<org.openstreetmap.atlas.geography.atlas.items.AtlasObject> objectsToCheck​(org.openstreetmap.atlas.geography.atlas.Atlas atlas, Check check)  
      protected static java.lang.Iterable<org.openstreetmap.atlas.geography.atlas.items.AtlasObject> objectsToCheck​(org.openstreetmap.atlas.geography.atlas.Atlas atlas, Check check, java.util.function.Predicate<org.openstreetmap.atlas.geography.atlas.items.AtlasEntity> geoFilter)  
      protected org.openstreetmap.atlas.utilities.runtime.Command.SwitchList switches()  

      • Methods inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

        configuration, configurationMap, copyToOutput, getAlternateParallelFolderOutput, getAlternateSubFolderOutput, getContext, getName, input, onRun, output, outputToClean, resource, resource, setContext, splitAndSaveAsHadoopFile, start

      • Methods inherited from class org.openstreetmap.atlas.utilities.runtime.Command

        commandSummary, getCommandMap, lastRawCommand, run, runWithoutQuitting

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • ATLAS_FOLDER

        @Deprecated(since="6.0.2")
protected static final org.openstreetmap.atlas.utilities.runtime.Command.Switch<java.lang.String> ATLAS_FOLDER
        Deprecated.
        in favor of INPUT from SparkJob

      • OUTPUT_ATLAS_FOLDER

        protected static final java.lang.String OUTPUT_ATLAS_FOLDER
        See Also:
        Constant Field Values

      • OUTPUT_FLAG_FOLDER

        protected static final java.lang.String OUTPUT_FLAG_FOLDER
        See Also:
        Constant Field Values

      • OUTPUT_GEOJSON_FOLDER

        protected static final java.lang.String OUTPUT_GEOJSON_FOLDER
        See Also:
        Constant Field Values

      • OUTPUT_METRIC_FOLDER

        protected static final java.lang.String OUTPUT_METRIC_FOLDER
        See Also:
        Constant Field Values

      • OUTPUT_TIPPECANOE_FOLDER

        protected static final java.lang.String OUTPUT_TIPPECANOE_FOLDER
        See Also:
        Constant Field Values

    • Constructor Detail

      • IntegrityChecksCommandArguments

        public IntegrityChecksCommandArguments()

    • Method Detail

      • countryShardMapFromShardFiles

        public static org.openstreetmap.atlas.utilities.maps.MultiMap<java.lang.String,​org.openstreetmap.atlas.geography.sharding.Shard> countryShardMapFromShardFiles​(java.util.Set<java.lang.String> countries,
                                                                                                                                                                     java.lang.String atlasFolder,
                                                                                                                                                                     java.util.Map<java.lang.String,​java.lang.String> sparkContext)
        Creates a map from country name to List of Shard definitions from Atlas files.
        Parameters:
        countries - Set of countries to find out shards for
        atlasFolder - Path to Atlas folder
        sparkContext - Spark context (or configuration) as a key-value map
        Returns:
        A map from country name to List of Shard definitions

      • objectsToCheck

        protected static java.lang.Iterable<org.openstreetmap.atlas.geography.atlas.items.AtlasObject> objectsToCheck​(org.openstreetmap.atlas.geography.atlas.Atlas atlas,
                                                                                                              Check check)

      • objectsToCheck

        protected static java.lang.Iterable<org.openstreetmap.atlas.geography.atlas.items.AtlasObject> objectsToCheck​(org.openstreetmap.atlas.geography.atlas.Atlas atlas,
                                                                                                              Check check,
                                                                                                              java.util.function.Predicate<org.openstreetmap.atlas.geography.atlas.items.AtlasEntity> geoFilter)

      • getAtlasDataSource

        protected AtlasDataSource getAtlasDataSource​(java.util.Map<java.lang.String,​java.lang.String> sparkContext,
                                             org.openstreetmap.atlas.utilities.configuration.Configuration checksConfiguration,
                                             org.openstreetmap.atlas.geography.Rectangle pbfBoundary)
        Gets the AtlasDataSource object to load the Atlas from
        Parameters:
        sparkContext - The Spark context
        checksConfiguration - configuration for all the checks
        pbfBoundary - The pbf boundary of type Rectangle
        Returns:
        A AtlasDataSource

      • switches

        protected org.openstreetmap.atlas.utilities.runtime.Command.SwitchList switches()
        Overrides:
        switches in class org.openstreetmap.atlas.generator.tools.spark.SparkJob