org.openstreetmap.atlas.generator.tools.spark.input

Class SparkInput

java.lang.Object

org.openstreetmap.atlas.generator.tools.spark.input.SparkInput

public final class SparkInput
extends java.lang.Object

Utility tools to load RDDs from elastic file systems (S3 for example), skipping the slowness of the SparkContext textFile and sequenceFile functions. This is inspired from

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.api.java.JavaRDD<org.openstreetmap.atlas.geography.atlas.Atlas>	atlasFiles(org.apache.spark.api.java.JavaSparkContext context, java.lang.String path) Load Atlas files from an input path
static org.apache.spark.api.java.JavaPairRDD<java.lang.String,org.apache.spark.input.PortableDataStream>	binaryFile(org.apache.spark.api.java.JavaSparkContext context, java.lang.String path) Get an RDD from a binary file input
static <K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable> org.apache.spark.api.java.JavaPairRDD<K,V>	sequenceFile(org.apache.spark.api.java.JavaSparkContext context, java.lang.String path, java.lang.Class<K> sequenceKeyClass, java.lang.Class<V> sequenceValueClass) Get an RDD from a sequence file input
protected static void	setFileSystemCreator(FileSystemCreator fileSystemCreator)
static org.apache.spark.api.java.JavaRDD<java.lang.String>	textFile(org.apache.spark.api.java.JavaSparkContext context, java.lang.String path) Get an RDD from a Text file input

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

atlasFiles

public static org.apache.spark.api.java.JavaRDD<org.openstreetmap.atlas.geography.atlas.Atlas> atlasFiles(org.apache.spark.api.java.JavaSparkContext context,
                                                                                                          java.lang.String path)

Load Atlas files from an input path

Parameters:

context - The context from Spark

path - The path to a set of atlas files

Returns:

An RDD of Atlas objects. One Atlas object per slice.

binaryFile

public static org.apache.spark.api.java.JavaPairRDD<java.lang.String,org.apache.spark.input.PortableDataStream> binaryFile(org.apache.spark.api.java.JavaSparkContext context,
                                                                                                                           java.lang.String path)

Get an RDD from a binary file input

Parameters:

context - The context from Spark

path - The path to a set of binary files

Returns:

An RDD made of pairs of names to values from the binary files read.

sequenceFile

public static <K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable> org.apache.spark.api.java.JavaPairRDD<K,V> sequenceFile(org.apache.spark.api.java.JavaSparkContext context,
                                                                                                                                                        java.lang.String path,
                                                                                                                                                        java.lang.Class<K> sequenceKeyClass,
                                                                                                                                                        java.lang.Class<V> sequenceValueClass)

Get an RDD from a sequence file input

Type Parameters:

K - The key type in the sequence file

V - The value type in the sequence file

Parameters:

context - The context from Spark

path - The path to a set of sequence files

sequenceKeyClass - The class to expect for the sequence file keys

sequenceValueClass - The class to expect for the sequence file values

Returns:

An RDD made of pairs of keys to values from the sequence files read.

textFile

public static org.apache.spark.api.java.JavaRDD<java.lang.String> textFile(org.apache.spark.api.java.JavaSparkContext context,
                                                                           java.lang.String path)

Get an RDD from a Text file input

Parameters:

context - The context from Spark

path - The path to open

Returns:

A RDD with one String line per item.

setFileSystemCreator

protected static void setFileSystemCreator(FileSystemCreator fileSystemCreator)