Package org.openstreetmap.atlas.mutator
Class AtlasMutatorDriver
- java.lang.Object
-
- org.openstreetmap.atlas.mutator.AtlasMutatorDriver
-
public final class AtlasMutatorDriver extends java.lang.ObjectHelper class toAtlasMutatorwhich handles RDD processing. Those methods are designed to be run on the driver.
-
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>>getAtlasGroupsRDD(java.lang.String name, int size, org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> atlasRDD, java.util.function.Function<org.openstreetmap.atlas.geography.sharding.CountryShard,java.util.Set<org.openstreetmap.atlas.geography.sharding.CountryShard>> shardExplorer, java.util.Optional<InputDependency> inputDependencyOption)Take an AtlasRDD and join it with all the shards needed for expansion for each shard, to get a new RDD that contains all the needed Atlas files for each shard.protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>>getAtlasGroupsRDDSimple(java.lang.String name, org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> atlasRDD)Take a simple Atlas RDD and make it look like a group RDD with the identity shard explorerprotected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>getAtlasRDD(AtlasMutationLevel level, org.apache.spark.api.java.JavaRDD<org.openstreetmap.atlas.geography.sharding.CountryShard> shardsRDD)Take a Shard RDD and make it into an AtlasRDD using the level's fetcher.protected static java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>iterableToMap(java.lang.String name, java.lang.Iterable<org.openstreetmap.atlas.utilities.tuples.Tuple<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>> iterable, org.openstreetmap.atlas.geography.sharding.CountryShard originShard, InputDependency inputDependency)
-
-
-
Method Detail
-
getAtlasGroupsRDD
protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>> getAtlasGroupsRDD(java.lang.String name, int size, org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> atlasRDD, java.util.function.Function<org.openstreetmap.atlas.geography.sharding.CountryShard,java.util.Set<org.openstreetmap.atlas.geography.sharding.CountryShard>> shardExplorer, java.util.Optional<InputDependency> inputDependencyOption)Take an AtlasRDD and join it with all the shards needed for expansion for each shard, to get a new RDD that contains all the needed Atlas files for each shard.- Parameters:
name- The name used to prefix the RDD names in the Spark UIsize- The pre-computed RDD sizeatlasRDD- The original Atlas RDD. It needs to be cachedshardExplorer- The function that determines what shards are needed for expansion, given a shard.inputDependencyOption- Optionally, an input dependency in case the expansion shards need to be filtered down- Returns:
- A new RDD that contains all the needed Atlas files for each shard.
-
getAtlasGroupsRDDSimple
protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>> getAtlasGroupsRDDSimple(java.lang.String name, org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> atlasRDD)Take a simple Atlas RDD and make it look like a group RDD with the identity shard explorer- Parameters:
name- The name used to prefix the RDD names in the Spark UIatlasRDD- The original Atlas RDD. It needs to be cached- Returns:
- A new RDD that contains all the single Atlas files for each shard (in this case it is the same).
-
getAtlasRDD
protected static org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> getAtlasRDD(AtlasMutationLevel level, org.apache.spark.api.java.JavaRDD<org.openstreetmap.atlas.geography.sharding.CountryShard> shardsRDD)
Take a Shard RDD and make it into an AtlasRDD using the level's fetcher.- Parameters:
level- The level to use to generate the initial Atlas RDD.shardsRDD- The original Shard RDD- Returns:
- A new RDD that contains each Atlas file for each Shard.
-
iterableToMap
protected static java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> iterableToMap(java.lang.String name, java.lang.Iterable<org.openstreetmap.atlas.utilities.tuples.Tuple<org.openstreetmap.atlas.geography.sharding.CountryShard,org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>> iterable, org.openstreetmap.atlas.geography.sharding.CountryShard originShard, InputDependency inputDependency)
-
-