Class AtlasMutatorDriver


  • public final class AtlasMutatorDriver
    extends java.lang.Object
    Helper class to AtlasMutator which handles RDD processing. Those methods are designed to be run on the driver.
    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>> getAtlasGroupsRDD​(java.lang.String name, int size, org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> atlasRDD, java.util.function.Function<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Set<org.openstreetmap.atlas.geography.sharding.CountryShard>> shardExplorer, java.util.Optional<InputDependency> inputDependencyOption)
      Take an AtlasRDD and join it with all the shards needed for expansion for each shard, to get a new RDD that contains all the needed Atlas files for each shard.
      protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>> getAtlasGroupsRDDSimple​(java.lang.String name, org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> atlasRDD)
      Take a simple Atlas RDD and make it look like a group RDD with the identity shard explorer
      protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> getAtlasRDD​(AtlasMutationLevel level, org.apache.spark.api.java.JavaRDD<org.openstreetmap.atlas.geography.sharding.CountryShard> shardsRDD)
      Take a Shard RDD and make it into an AtlasRDD using the level's fetcher.
      protected static java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> iterableToMap​(java.lang.String name, java.lang.Iterable<org.openstreetmap.atlas.utilities.tuples.Tuple<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>> iterable, org.openstreetmap.atlas.geography.sharding.CountryShard originShard, InputDependency inputDependency)  
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Method Detail

      • getAtlasGroupsRDD

        protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>> getAtlasGroupsRDD​(java.lang.String name,
                                                                                                                                                                                                                                                                                        int size,
                                                                                                                                                                                                                                                                                        org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> atlasRDD,
                                                                                                                                                                                                                                                                                        java.util.function.Function<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Set<org.openstreetmap.atlas.geography.sharding.CountryShard>> shardExplorer,
                                                                                                                                                                                                                                                                                        java.util.Optional<InputDependency> inputDependencyOption)
        Take an AtlasRDD and join it with all the shards needed for expansion for each shard, to get a new RDD that contains all the needed Atlas files for each shard.
        Parameters:
        name - The name used to prefix the RDD names in the Spark UI
        size - The pre-computed RDD size
        atlasRDD - The original Atlas RDD. It needs to be cached
        shardExplorer - The function that determines what shards are needed for expansion, given a shard.
        inputDependencyOption - Optionally, an input dependency in case the expansion shards need to be filtered down
        Returns:
        A new RDD that contains all the needed Atlas files for each shard.
      • getAtlasGroupsRDDSimple

        protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>> getAtlasGroupsRDDSimple​(java.lang.String name,
                                                                                                                                                                                                                                                                                              org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> atlasRDD)
        Take a simple Atlas RDD and make it look like a group RDD with the identity shard explorer
        Parameters:
        name - The name used to prefix the RDD names in the Spark UI
        atlasRDD - The original Atlas RDD. It needs to be cached
        Returns:
        A new RDD that contains all the single Atlas files for each shard (in this case it is the same).
      • getAtlasRDD

        protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> getAtlasRDD​(AtlasMutationLevel level,
                                                                                                                                                                                                     org.apache.spark.api.java.JavaRDD<org.openstreetmap.atlas.geography.sharding.CountryShard> shardsRDD)
        Take a Shard RDD and make it into an AtlasRDD using the level's fetcher.
        Parameters:
        level - The level to use to generate the initial Atlas RDD.
        shardsRDD - The original Shard RDD
        Returns:
        A new RDD that contains each Atlas file for each Shard.
      • iterableToMap

        protected static java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> iterableToMap​(java.lang.String name,
                                                                                                                                                                               java.lang.Iterable<org.openstreetmap.atlas.utilities.tuples.Tuple<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>> iterable,
                                                                                                                                                                               org.openstreetmap.atlas.geography.sharding.CountryShard originShard,
                                                                                                                                                                               InputDependency inputDependency)