Package org.openstreetmap.atlas.mutator

Class AtlasMutatorHelper

  • All Implemented Interfaces:
    java.io.Serializable
    public final class AtlasMutatorHelper
extends java.lang.Object
implements java.io.Serializable
    This class contains most of the lambda function executed by Spark slaves when running an AtlasMutator spark job.
    See Also:
    Serialized Form

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      protected static org.apache.spark.api.java.function.Function2<java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> assignedToConcatenatedFeatureChanges​(AtlasMutationLevel level)  
      protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> assignedToShardAppliedFeatureChanges​(AtlasMutationLevel level)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​scala.Tuple2<org.openstreetmap.atlas.geography.atlas.change.Change,​org.apache.spark.api.java.Optional<java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>>>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.utilities.tuples.Tuple<org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas,​org.openstreetmap.atlas.geography.atlas.change.Change>> changeAndShardToAtlasMapToAtlas​(AtlasMutationLevel level)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.change.Change>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.utilities.tuples.Tuple<org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas,​org.openstreetmap.atlas.geography.atlas.change.Change>> changeToAtlas​(AtlasMutationLevel level)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.change.Change>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.utilities.tuples.Tuple<org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas,​org.openstreetmap.atlas.geography.atlas.change.Change>> changeToAtlas​(AtlasMutationLevel level, java.util.function.BiFunction<AtlasMutationLevel,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.dynamic.policy.DynamicAtlasPolicy> levelAndShardToPolicyFunction)  
      protected static <I> org.apache.spark.api.java.JavaPairRDD<java.lang.String,​I> embedCountryNameInKey​(java.lang.String logFileName, org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​I> atlasRDD)  
      protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.change.Change> featureChangeListToChange​(AtlasMutationLevel level)  
      static java.lang.String getAlternateSubFolderOutput​(java.lang.String output, java.lang.String name)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> inputDependencyFilteredAtlas​(AtlasMutationLevel level, InputDependency inputDependency)
      Filter Mutated Atlas to provide input dependency
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> shardFeatureChangesToAssignedShardFeatureChanges​(AtlasMutationLevel level)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> shardToAtlas​(AtlasMutationLevel level)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> shardToAtlasMapToFeatureChanges​(AtlasMutationLevel level)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> shardToFeatureChanges​(AtlasMutationLevel level)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> shardToFeatureChanges​(AtlasMutationLevel level, java.util.function.BiFunction<AtlasMutationLevel,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.dynamic.policy.DynamicAtlasPolicy> levelAndShardToPolicyFunction)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> untouchedShardAndMapToPotentialSourcePackedAtlas​(AtlasMutationLevel level)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> untouchedShardToPotentialSourcePackedAtlas​(AtlasMutationLevel level)  
      protected static org.apache.spark.api.java.function.PairFlatMapFunction<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> untouchedShardToPotentialSourcePackedAtlas​(AtlasMutationLevel level, java.util.function.Function<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Optional<org.openstreetmap.atlas.geography.atlas.Atlas>> fetcher)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • getAlternateSubFolderOutput

        public static java.lang.String getAlternateSubFolderOutput​(java.lang.String output,
                                                           java.lang.String name)

      • assignedToConcatenatedFeatureChanges

        protected static org.apache.spark.api.java.function.Function2<java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> assignedToConcatenatedFeatureChanges​(AtlasMutationLevel level)
        Parameters:
        level - The level at which the merging is being done
        Returns:
        A function that merges two lists of FeatureChange into one single list. It applies merge rules on all the FeatureChange objects that collide.

      • assignedToShardAppliedFeatureChanges

        protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> assignedToShardAppliedFeatureChanges​(AtlasMutationLevel level)
        Parameters:
        level - The level at which the merging is being done
        Returns:
        A function that merges two lists of FeatureChange into one single list. It applies merge rules on all the FeatureChange objects that collide.

      • changeAndShardToAtlasMapToAtlas

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​scala.Tuple2<org.openstreetmap.atlas.geography.atlas.change.Change,​org.apache.spark.api.java.Optional<java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>>>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.utilities.tuples.Tuple<org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas,​org.openstreetmap.atlas.geography.atlas.change.Change>> changeAndShardToAtlasMapToAtlas​(AtlasMutationLevel level)
        Parameters:
        level - The mutation level definition
        Returns:
        A function that takes a tuple of shard and map of expansion atlases, and returns all the FeatureChange objects generated by that expanded shard.

      • changeToAtlas

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.change.Change>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.utilities.tuples.Tuple<org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas,​org.openstreetmap.atlas.geography.atlas.change.Change>> changeToAtlas​(AtlasMutationLevel level)
        Parameters:
        level - The level at which the application is being done.
        Returns:
        A function that applies a change to the right Atlas and returns a cloned PackedAtlas ready to save.

      • changeToAtlas

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.change.Change>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.utilities.tuples.Tuple<org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas,​org.openstreetmap.atlas.geography.atlas.change.Change>> changeToAtlas​(AtlasMutationLevel level,
                                                                                                                                                                                                                                                                                                                                                                                                                                                                    java.util.function.BiFunction<AtlasMutationLevel,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.dynamic.policy.DynamicAtlasPolicy> levelAndShardToPolicyFunction)
        Parameters:
        level - The level at which the application is being done.
        levelAndShardToPolicyFunction - The function that decides what policy to get from the level (simple generation, or generation from RDD)
        Returns:
        A function that applies a change to the right Atlas and returns a cloned PackedAtlas ready to save along with the actually used filtered change object.

      • embedCountryNameInKey

        protected static <I> org.apache.spark.api.java.JavaPairRDD<java.lang.String,​I> embedCountryNameInKey​(java.lang.String logFileName,
                                                                                                           org.apache.spark.api.java.JavaPairRDD<org.openstreetmap.atlas.geography.sharding.CountryShard,​I> atlasRDD)

      • featureChangeListToChange

        protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.change.Change> featureChangeListToChange​(AtlasMutationLevel level)
        Parameters:
        level - The level at which the operation is happening
        Returns:
        A function that groups each list of FeatureChanges into one Change

      • inputDependencyFilteredAtlas

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> inputDependencyFilteredAtlas​(AtlasMutationLevel level,
                                                                                                                                                                                                                                                                                                                                                                            InputDependency inputDependency)
        Filter Mutated Atlas to provide input dependency
        Parameters:
        level - The level at which the operation is happening
        inputDependency - The InputDependency providing the definition for filtering.
        Returns:
        A function that takes a shard and Atlas pair and filters the Atlas according to the definition in the InputDependency

      • shardFeatureChangesToAssignedShardFeatureChanges

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> shardFeatureChangesToAssignedShardFeatureChanges​(AtlasMutationLevel level)
        Parameters:
        level - The level which can provide the sharding tree.
        Returns:
        A function that assigns every FeatureChange to all the shards where it can be applied.

      • shardToAtlas

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> shardToAtlas​(AtlasMutationLevel level)
        Parameters:
        level - The level of interest
        Returns:
        A function that transforms a CountryShard into the corresponding PackedAtlas from the level's parent's source folder.

      • shardToAtlasMapToFeatureChanges

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> shardToAtlasMapToFeatureChanges​(AtlasMutationLevel level)
        Parameters:
        level - The mutation level definition
        Returns:
        A function that takes a tuple of shard and map of expansion atlases, and returns all the FeatureChange objects generated by that expanded shard.

      • shardToFeatureChanges

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> shardToFeatureChanges​(AtlasMutationLevel level)
        Parameters:
        level - The mutation level definition
        Returns:
        A function that takes a shard and returns all the FeatureChange objects generated by that expanded shard.

      • shardToFeatureChanges

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.List<org.openstreetmap.atlas.geography.atlas.change.FeatureChange>> shardToFeatureChanges​(AtlasMutationLevel level,
                                                                                                                                                                                                                                                                                                        java.util.function.BiFunction<AtlasMutationLevel,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.dynamic.policy.DynamicAtlasPolicy> levelAndShardToPolicyFunction)
        Parameters:
        level - The mutation level definition
        levelAndShardToPolicyFunction - The function that decides what policy to get from the level (simple generation, or generation from RDD)
        Returns:
        A function that takes a shard and returns all the FeatureChange objects generated by that expanded shard.

      • untouchedShardAndMapToPotentialSourcePackedAtlas

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Map<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas>>,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> untouchedShardAndMapToPotentialSourcePackedAtlas​(AtlasMutationLevel level)

      • untouchedShardToPotentialSourcePackedAtlas

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> untouchedShardToPotentialSourcePackedAtlas​(AtlasMutationLevel level)
        Parameters:
        level - The level of interest
        Returns:
        Return a function that provides the original source atlas given a shard, or an empty list if that atlas does not exist.

      • untouchedShardToPotentialSourcePackedAtlas

        protected static org.apache.spark.api.java.function.PairFlatMapFunction<org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.sharding.CountryShard,​org.openstreetmap.atlas.geography.atlas.packed.PackedAtlas> untouchedShardToPotentialSourcePackedAtlas​(AtlasMutationLevel level,
                                                                                                                                                                                                                                                                                                           java.util.function.Function<org.openstreetmap.atlas.geography.sharding.CountryShard,​java.util.Optional<org.openstreetmap.atlas.geography.atlas.Atlas>> fetcher)