Package org.openstreetmap.atlas.generator.tools.spark.sharded

Class ShardedSparkJob

java.lang.Object

org.openstreetmap.atlas.utilities.runtime.Command

org.openstreetmap.atlas.generator.tools.spark.SparkJob

org.openstreetmap.atlas.generator.tools.spark.sharded.ShardedSparkJob

All Implemented Interfaces:

java.io.Serializable

public abstract class ShardedSparkJob
extends SparkJob

Spark job that takes care of reading the sharding and country list and starts with a JavaPairRDD of CountryName to Shard

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.openstreetmap.atlas.utilities.runtime.Command

org.openstreetmap.atlas.utilities.runtime.Command.Flag, org.openstreetmap.atlas.utilities.runtime.Command.Optionality, org.openstreetmap.atlas.utilities.runtime.Command.Switch<T extends java.lang.Object>, org.openstreetmap.atlas.utilities.runtime.Command.SwitchList

Field Summary

Fields

Modifier and Type	Field	Description
static org.openstreetmap.atlas.utilities.runtime.Command.Switch<org.openstreetmap.atlas.utilities.collections.StringList>	COUNTRIES
static org.openstreetmap.atlas.utilities.runtime.Command.Switch<java.lang.String>	COUNTRY_SHAPES
static org.openstreetmap.atlas.utilities.runtime.Command.Switch<java.lang.String>	SHARDING_TYPE

Fields inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

ADDITIONAL_SPARK_OPTIONS, CLUSTER, COMPRESS_OUTPUT, FAILED_FILE, INPUT, OUTPUT, SAVING_SEPARATOR, SPARK_CONTEXT_PROVIDER, SPARK_OPTIONS, SUCCESS_FILE

Constructor Summary

Constructors

Constructor	Description
ShardedSparkJob()

Method Summary

Modifier and Type	Method	Description
void	start(org.openstreetmap.atlas.utilities.runtime.CommandMap command)	The spark Job
abstract void	start2(org.openstreetmap.atlas.utilities.runtime.CommandMap command, org.openstreetmap.atlas.geography.sharding.Sharding sharding, org.openstreetmap.atlas.utilities.maps.MultiMap<java.lang.String,org.openstreetmap.atlas.geography.sharding.Shard> countryToShardMap)
org.openstreetmap.atlas.utilities.runtime.Command.SwitchList	switches()

Methods inherited from class org.openstreetmap.atlas.generator.tools.spark.SparkJob

configuration, configurationMap, copyToOutput, getAlternateParallelFolderOutput, getAlternateSubFolderOutput, getContext, getName, input, onRun, output, outputToClean, resource, resource, setContext, splitAndSaveAsHadoopFile

Methods inherited from class org.openstreetmap.atlas.utilities.runtime.Command

commandSummary, getCommandMap, lastRawCommand, run, runWithoutQuitting

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COUNTRIES

public static final org.openstreetmap.atlas.utilities.runtime.Command.Switch<org.openstreetmap.atlas.utilities.collections.StringList> COUNTRIES

SHARDING_TYPE

public static final org.openstreetmap.atlas.utilities.runtime.Command.Switch<java.lang.String> SHARDING_TYPE

COUNTRY_SHAPES

public static final org.openstreetmap.atlas.utilities.runtime.Command.Switch<java.lang.String> COUNTRY_SHAPES

Constructor Detail

ShardedSparkJob

public ShardedSparkJob()

Method Detail

start

public void start(org.openstreetmap.atlas.utilities.runtime.CommandMap command)

Description copied from class: SparkJob

The spark Job

Specified by:

start in class SparkJob

Parameters:

command - The arguments passed to the main method

start2

public abstract void start2(org.openstreetmap.atlas.utilities.runtime.CommandMap command,
                            org.openstreetmap.atlas.geography.sharding.Sharding sharding,
                            org.openstreetmap.atlas.utilities.maps.MultiMap<java.lang.String,org.openstreetmap.atlas.geography.sharding.Shard> countryToShardMap)

switches

public org.openstreetmap.atlas.utilities.runtime.Command.SwitchList switches()

Overrides:

switches in class SparkJob