org.openstreetmap.atlas.generator

Class AtlasGeneratorHelper

java.lang.Object

org.openstreetmap.atlas.generator.AtlasGeneratorHelper

All Implemented Interfaces:

java.io.Serializable

public final class AtlasGeneratorHelper
extends java.lang.Object
implements java.io.Serializable

Utility class for AtlasGenerator.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	AtlasGeneratorHelper.NamedAtlasStatistics

Method Summary

Modifier and Type	Method and Description
protected static java.util.function.Function<org.openstreetmap.atlas.geography.sharding.Shard,java.util.Optional<org.openstreetmap.atlas.geography.atlas.Atlas>>	atlasFetcher(HadoopAtlasFileCache atlasCache, java.lang.String country, java.util.Set<org.openstreetmap.atlas.geography.sharding.Shard> validShards)
protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>,java.lang.String,org.openstreetmap.atlas.geography.atlas.delta.AtlasDelta>	computeAtlasDelta(java.util.Map<java.lang.String,java.lang.String> sparkContext, java.lang.String previousOutputForDelta)
protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>,java.lang.String,org.openstreetmap.atlas.geography.atlas.statistics.AtlasStatistics>	generateAtlasStatistics(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.sharding.Sharding> sharding)
protected static org.apache.spark.api.java.function.PairFunction<org.openstreetmap.atlas.generator.AtlasGenerationTask,java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>	generateRawAtlas(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries, java.util.Map<java.lang.String,java.lang.String> sparkContext, org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,java.lang.String>> loadingOptions, PbfContext pbfContext, SlippyTilePersistenceScheme atlasScheme)
protected static org.apache.spark.api.java.function.Function2<AtlasGeneratorHelper.NamedAtlasStatistics,AtlasGeneratorHelper.NamedAtlasStatistics,AtlasGeneratorHelper.NamedAtlasStatistics>	reduceAtlasStatistics()
protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>,java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>	sectionRawAtlas(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries, org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.sharding.Sharding> sharding, java.util.Map<java.lang.String,java.lang.String> sparkContext, org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,java.lang.String>> loadingOptions, java.lang.String slicedRawAtlasPath, SlippyTilePersistenceScheme atlasScheme, java.util.List<org.openstreetmap.atlas.generator.AtlasGenerationTask> tasks)
protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>,java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>	sliceRawAtlas(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

atlasFetcher

protected static java.util.function.Function<org.openstreetmap.atlas.geography.sharding.Shard,java.util.Optional<org.openstreetmap.atlas.geography.atlas.Atlas>> atlasFetcher(HadoopAtlasFileCache atlasCache,
                                                                                                                                                                              java.lang.String country,
                                                                                                                                                                              java.util.Set<org.openstreetmap.atlas.geography.sharding.Shard> validShards)

Parameters:

atlasCache - The cache object for the Atlas files

country - The country to look for

validShards - All available shards for given country, to avoid fetching shards that do not exist

Returns:

A function that returns an Atlas given a Shard

computeAtlasDelta

protected static org.apache.spark.api.java.function.PairFlatMapFunction<scala.Tuple2<java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>,java.lang.String,org.openstreetmap.atlas.geography.atlas.delta.AtlasDelta> computeAtlasDelta(java.util.Map<java.lang.String,java.lang.String> sparkContext,
                                                                                                                                                                                                                                                  java.lang.String previousOutputForDelta)

Parameters:

sparkContext - Spark context (or configuration) as a key-value map

previousOutputForDelta - Previous Atlas generation delta output location

Returns:

A Spark PairFlatMapFunction that takes a tuple of a country shard name and atlas file and returns all the AtlasDelta for the country

generateAtlasStatistics

protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>,java.lang.String,org.openstreetmap.atlas.geography.atlas.statistics.AtlasStatistics> generateAtlasStatistics(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.sharding.Sharding> sharding)

Parameters:

sharding - The sharding tree

Returns:

a Spark PairFunction that processes a shard to Atlas tuple, and constructs a AtlasStatistics for each shard.

generateRawAtlas

protected static org.apache.spark.api.java.function.PairFunction<org.openstreetmap.atlas.generator.AtlasGenerationTask,java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas> generateRawAtlas(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries,
                                                                                                                                                                                                        java.util.Map<java.lang.String,java.lang.String> sparkContext,
                                                                                                                                                                                                        org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,java.lang.String>> loadingOptions,
                                                                                                                                                                                                        PbfContext pbfContext,
                                                                                                                                                                                                        SlippyTilePersistenceScheme atlasScheme)

Parameters:

boundaries - The CountryBoundaryMap to use for pbf to atlas generation

sparkContext - Spark context (or configuration) as a key-value map

loadingOptions - The basic required properties to create an AtlasLoadingOption

pbfContext - The context explaining where to find the PBFs

atlasScheme - The folder structure of the output atlas

Returns:

a Spark PairFunction that processes an AtlasGenerationTask, loads the PBF for the task's shard, generates the raw atlas for the shard and outputs a shard name to raw atlas tuple.

reduceAtlasStatistics

protected static org.apache.spark.api.java.function.Function2<AtlasGeneratorHelper.NamedAtlasStatistics,AtlasGeneratorHelper.NamedAtlasStatistics,AtlasGeneratorHelper.NamedAtlasStatistics> reduceAtlasStatistics()

sectionRawAtlas

protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>,java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas> sectionRawAtlas(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries,
                                                                                                                                                                                                                              org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.sharding.Sharding> sharding,
                                                                                                                                                                                                                              java.util.Map<java.lang.String,java.lang.String> sparkContext,
                                                                                                                                                                                                                              org.apache.spark.broadcast.Broadcast<java.util.Map<java.lang.String,java.lang.String>> loadingOptions,
                                                                                                                                                                                                                              java.lang.String slicedRawAtlasPath,
                                                                                                                                                                                                                              SlippyTilePersistenceScheme atlasScheme,
                                                                                                                                                                                                                              java.util.List<org.openstreetmap.atlas.generator.AtlasGenerationTask> tasks)

Parameters:

boundaries - The CountryBoundaryMap required to create an AtlasLoadingOption

sharding - The Sharding strategy

sparkContext - Spark context (or configuration) as a key-value map

loadingOptions - The basic required properties to create an AtlasLoadingOption

slicedRawAtlasPath - The path where the sliced raw atlas files were saved

atlasScheme - The folder structure of the output atlas

tasks - The list of AtlasGenerationTasks used to grab all possible Shards for a country

Returns:

a Spark PairFunction that processes a tuple of shard-name and sliced raw atlas, sections the sliced raw atlas and returns the final sectioned (and sliced) raw atlas for that shard name.

sliceRawAtlas

protected static org.apache.spark.api.java.function.PairFunction<scala.Tuple2<java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas>,java.lang.String,org.openstreetmap.atlas.geography.atlas.Atlas> sliceRawAtlas(org.apache.spark.broadcast.Broadcast<org.openstreetmap.atlas.geography.boundary.CountryBoundaryMap> boundaries)

Parameters:

boundaries - The CountryBoundaryMap to use for slicing

Returns:

a Spark PairFunction that processes a tuple of shard-name and raw atlas, slices the raw atlas and returns the sliced raw atlas for that shard name.