Package org.openstreetmap.atlas.generator

Class AtlasGeneratorHelper

  • All Implemented Interfaces:
    java.io.Serializable
    public final class AtlasGeneratorHelper
extends java.lang.Object
implements java.io.Serializable
    Utility class for AtlasGenerator.
    See Also:
    Serialized Form

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      protected static java.util.function.Function<org.openstreetmap.atlas.geography.sharding.Shard,​java.util.Optional<org.openstreetmap.atlas.geography.atlas.Atlas>> atlasFetcher​(HadoopAtlasFileCache atlasCache, java.lang.String country, java.util.Set<org.openstreetmap.atlas.geography.sharding.Shard> validShards)  
      static java.util.function.Function<org.openstreetmap.atlas.geography.sharding.Shard,​java.util.Optional<org.openstreetmap.atlas.geography.atlas.Atlas>> atlasFetcher​(HadoopAtlasFileCache subAtlasCache, HadoopAtlasFileCache atlasCache, java.lang.String countryBeingSliced, org.openstreetmap.atlas.geography.sharding.Shard initialShard)  
      static java.util.function.Function<org.openstreetmap.atlas.geography.sharding.Shard,​java.util.Optional<org.openstreetmap.atlas.geography.atlas.Atlas>> atlasFetcher​(HadoopAtlasFileCache lineSlicedSubAtlasCache, HadoopAtlasFileCache lineSlicedAtlasCache, org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap boundaries, java.lang.String countryBeingSliced, org.openstreetmap.atlas.geography.sharding.Shard initialShard)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.delta.AtlasDelta> computeAtlasDelta​(java.util.Map<java.lang.String,​java.lang.String> sparkContext, java.lang.String previousOutputForDelta)  
      protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.statistics.AtlasStatistics> generateAtlasStatistics​(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.sharding.Sharding> sharding)  
      protected static org.apache.spark.api.java.function.PairFunction<org.openstreetmap.atlas.generator.AtlasGenerationTask,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas> generateRawAtlas​(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries, java.util.Map<java.lang.String,​java.lang.String> sparkContext, org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,​java.lang.String>> loadingOptions, PbfContext pbfContext, SlippyTilePersistenceScheme atlasScheme)  
      protected static org.apache.spark.api.java.function.Function2<AtlasGeneratorHelper.NamedAtlasStatistics,​AtlasGeneratorHelper.NamedAtlasStatistics,​AtlasGeneratorHelper.NamedAtlasStatistics> reduceAtlasStatistics()  
      protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas> sectionAtlas​(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries, org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.sharding.Sharding> sharding, java.util.Map<java.lang.String,​java.lang.String> sparkContext, org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,​java.lang.String>> loadingOptions, java.lang.String edgeSubAtlasPath, java.lang.String slicedAtlasPath, SlippyTilePersistenceScheme atlasScheme, java.util.List<org.openstreetmap.atlas.generator.AtlasGenerationTask> tasks)  
      protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas> sliceLines​(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries, org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,​java.lang.String>> loadingOptions)  
      protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas> sliceRelations​(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries, org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,​java.lang.String>> loadingOptions, org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.sharding.Sharding> sharding, java.lang.String lineSlicedSubAtlasPath, java.lang.String lineSlicedAtlasPath, SlippyTilePersistenceScheme atlasScheme, java.util.Map<java.lang.String,​java.lang.String> sparkContext)  
      protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas> subatlas​(java.util.function.Predicate<org.openstreetmap.atlas.tags.Taggable> filter, org.openstreetmap.atlas.geography.atlas.sub.AtlasCutType cutType)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • FINISHED_MESSAGE

        public static final java.lang.String FINISHED_MESSAGE
        See Also:
        Constant Field Values

      • subAtlasFilter

        public static final java.util.function.Predicate<org.openstreetmap.atlas.tags.Taggable> subAtlasFilter

    • Method Detail

      • atlasFetcher

        public static java.util.function.Function<org.openstreetmap.atlas.geography.sharding.Shard,​java.util.Optional<org.openstreetmap.atlas.geography.atlas.Atlas>> atlasFetcher​(HadoopAtlasFileCache lineSlicedSubAtlasCache,
                                                                                                                                                                                 HadoopAtlasFileCache lineSlicedAtlasCache,
                                                                                                                                                                                 org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap boundaries,
                                                                                                                                                                                 java.lang.String countryBeingSliced,
                                                                                                                                                                                 org.openstreetmap.atlas.geography.sharding.Shard initialShard)

      • atlasFetcher

        public static java.util.function.Function<org.openstreetmap.atlas.geography.sharding.Shard,​java.util.Optional<org.openstreetmap.atlas.geography.atlas.Atlas>> atlasFetcher​(HadoopAtlasFileCache subAtlasCache,
                                                                                                                                                                                 HadoopAtlasFileCache atlasCache,
                                                                                                                                                                                 java.lang.String countryBeingSliced,
                                                                                                                                                                                 org.openstreetmap.atlas.geography.sharding.Shard initialShard)

      • atlasFetcher

        protected static java.util.function.Function<org.openstreetmap.atlas.geography.sharding.Shard,​java.util.Optional<org.openstreetmap.atlas.geography.atlas.Atlas>> atlasFetcher​(HadoopAtlasFileCache atlasCache,
                                                                                                                                                                                    java.lang.String country,
                                                                                                                                                                                    java.util.Set<org.openstreetmap.atlas.geography.sharding.Shard> validShards)
        Parameters:
        atlasCache - The cache object for the Atlas files
        country - The country to look for
        validShards - All available shards for given country, to avoid fetching shards that do not exist
        Returns:
        A function that returns an Atlas given a Shard

      • computeAtlasDelta

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.delta.AtlasDelta> computeAtlasDelta​(java.util.Map<java.lang.String,​java.lang.String> sparkContext,
                                                                                                                                                                                                                                                                    java.lang.String previousOutputForDelta)
        Parameters:
        sparkContext - Spark context (or configuration) as a key-value map
        previousOutputForDelta - Previous Atlas generation delta output location
        Returns:
        A Spark PairFlatMapFunction that takes a tuple of a country shard name and atlas file and returns all the AtlasDelta for the country

      • generateAtlasStatistics

        protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.statistics.AtlasStatistics> generateAtlasStatistics​(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.sharding.Sharding> sharding)
        Parameters:
        sharding - The sharding tree
        Returns:
        a Spark PairFunction that processes a shard to Atlas tuple, and constructs a AtlasStatistics for each shard.

      • generateRawAtlas

        protected static org.apache.spark.api.java.function.PairFunction<org.openstreetmap.atlas.generator.AtlasGenerationTask,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas> generateRawAtlas​(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries,
                                                                                                                                                                                                                    java.util.Map<java.lang.String,​java.lang.String> sparkContext,
                                                                                                                                                                                                                    org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,​java.lang.String>> loadingOptions,
                                                                                                                                                                                                                    PbfContext pbfContext,
                                                                                                                                                                                                                    SlippyTilePersistenceScheme atlasScheme)
        Parameters:
        boundaries - The CountryBoundaryMap to use for pbf to atlas generation
        sparkContext - Spark context (or configuration) as a key-value map
        loadingOptions - The basic required properties to create an AtlasLoadingOption
        pbfContext - The context explaining where to find the PBFs
        atlasScheme - The folder structure of the output atlas
        Returns:
        a Spark PairFunction that processes an AtlasGenerationTask, loads the PBF for the task's shard, generates the raw atlas for the shard and outputs a shard name to raw atlas tuple.

      • sectionAtlas

        protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas> sectionAtlas​(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries,
                                                                                                                                                                                                                                             org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.sharding.Sharding> sharding,
                                                                                                                                                                                                                                             java.util.Map<java.lang.String,​java.lang.String> sparkContext,
                                                                                                                                                                                                                                             org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,​java.lang.String>> loadingOptions,
                                                                                                                                                                                                                                             java.lang.String edgeSubAtlasPath,
                                                                                                                                                                                                                                             java.lang.String slicedAtlasPath,
                                                                                                                                                                                                                                             SlippyTilePersistenceScheme atlasScheme,
                                                                                                                                                                                                                                             java.util.List<org.openstreetmap.atlas.generator.AtlasGenerationTask> tasks)
        Parameters:
        boundaries - The CountryBoundaryMap required to create an AtlasLoadingOption
        sharding - The Sharding strategy
        sparkContext - Spark context (or configuration) as a key-value map
        loadingOptions - The basic required properties to create an AtlasLoadingOption
        edgeSubAtlasPath - The path where the edge-only sub atlas files were saved
        slicedAtlasPath - The path where the sliced raw atlas files were saved
        atlasScheme - The folder structure of the output atlas
        tasks - The list of AtlasGenerationTasks used to grab all possible Shards for a country
        Returns:
        a Spark PairFunction that processes a tuple of shard-name and sliced raw atlas, sections the sliced raw atlas and returns the final sectioned (and sliced) raw atlas for that shard name.

      • sliceLines

        protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas> sliceLines​(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries,
                                                                                                                                                                                                                                           org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,​java.lang.String>> loadingOptions)
        Parameters:
        boundaries - The CountryBoundaryMap to use for slicing
        loadingOptions - The basic required properties to create an AtlasLoadingOption
        Returns:
        a Spark PairFunction that processes a tuple of shard-name and raw atlas, slices the raw atlas and returns the sliced raw atlas for that shard name.

      • sliceRelations

        protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas> sliceRelations​(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries,
                                                                                                                                                                                                                                               org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,​java.lang.String>> loadingOptions,
                                                                                                                                                                                                                                               org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.sharding.Sharding> sharding,
                                                                                                                                                                                                                                               java.lang.String lineSlicedSubAtlasPath,
                                                                                                                                                                                                                                               java.lang.String lineSlicedAtlasPath,
                                                                                                                                                                                                                                               SlippyTilePersistenceScheme atlasScheme,
                                                                                                                                                                                                                                               java.util.Map<java.lang.String,​java.lang.String> sparkContext)

      • subatlas

        protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas>,​java.lang.String,​org.openstreetmap.atlas.geography.atlas.Atlas> subatlas​(java.util.function.Predicate<org.openstreetmap.atlas.tags.Taggable> filter,
                                                                                                                                                                                                                                         org.openstreetmap.atlas.geography.atlas.sub.AtlasCutType cutType)