io.warp10.hadoop

Class WarpScriptInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,java.lang.Object>

io.warp10.hadoop.WarpScriptInputFormat

public class WarpScriptInputFormat
extends org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,java.lang.Object>

This InputFormat wraps another InputFormat and creates a RecordReader which returns K/V processed by some WarpScript™ code.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	PATH_CONFIG_KEY Configuration key we will set to the current path if the split if a FileSplit
static java.lang.String	WARPSCRIPT_INPUTFORMAT_CLASS Class of the wrapped InputFormat
static java.lang.String	WARPSCRIPT_INPUTFORMAT_CONF_SUFFIX Suffix to remove from configuration keys to override or create specific configuration entries
static java.lang.String	WARPSCRIPT_INPUTFORMAT_SCRIPT WarpScript™ code fragment to apply
static java.lang.String	WARPSCRIPT_INPUTFORMAT_SUFFIX Suffix to use for the configuration

Constructor Summary

Constructors

Constructor and Description
WarpScriptInputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<java.lang.Object,java.lang.Object>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)
java.util.List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context)
java.lang.String	getSuffix()
WarpScriptExecutor	getWarpScriptExecutor(org.apache.hadoop.conf.Configuration conf, java.lang.String code) Return the actual WarpScript code executor given the script which was passed as parameter.
java.io.InputStream	getWarpScriptInputStream(java.lang.String originalFilePath) Create an InputStream from a file path.
org.apache.hadoop.mapreduce.RecordReader	getWrappedRecordReader()
void	init(org.apache.hadoop.conf.Configuration conf)
java.lang.String	parseWarpScript(java.lang.String filepath)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PATH_CONFIG_KEY

public static final java.lang.String PATH_CONFIG_KEY

Configuration key we will set to the current path if the split if a FileSplit

See Also:

Constant Field Values

WARPSCRIPT_INPUTFORMAT_SUFFIX

public static final java.lang.String WARPSCRIPT_INPUTFORMAT_SUFFIX

Suffix to use for the configuration

See Also:

Constant Field Values

WARPSCRIPT_INPUTFORMAT_CLASS

public static final java.lang.String WARPSCRIPT_INPUTFORMAT_CLASS

Class of the wrapped InputFormat

See Also:

Constant Field Values

WARPSCRIPT_INPUTFORMAT_SCRIPT

public static final java.lang.String WARPSCRIPT_INPUTFORMAT_SCRIPT

WarpScript™ code fragment to apply

See Also:

Constant Field Values

WARPSCRIPT_INPUTFORMAT_CONF_SUFFIX

public static final java.lang.String WARPSCRIPT_INPUTFORMAT_CONF_SUFFIX

Suffix to remove from configuration keys to override or create specific configuration entries

See Also:

Constant Field Values

Constructor Detail

WarpScriptInputFormat

public WarpScriptInputFormat()

Method Detail

getSplits

public java.util.List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context)
                                                                 throws java.io.IOException,
                                                                        java.lang.InterruptedException

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,java.lang.Object>

Throws:

java.io.IOException

java.lang.InterruptedException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<java.lang.Object,java.lang.Object> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                                                                      org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                               throws java.io.IOException,
                                                                                                      java.lang.InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,java.lang.Object>

Throws:

java.io.IOException

java.lang.InterruptedException

getWarpScriptExecutor

public WarpScriptExecutor getWarpScriptExecutor(org.apache.hadoop.conf.Configuration conf,
                                                java.lang.String code)
                                         throws java.io.IOException,
                                                WarpScriptException

Return the actual WarpScript code executor given the script which was passed as parameter.

Throws:

java.io.IOException

WarpScriptException

parseWarpScript

public java.lang.String parseWarpScript(java.lang.String filepath)
                                 throws java.io.IOException

Throws:

java.io.IOException

getWarpScriptInputStream

public java.io.InputStream getWarpScriptInputStream(java.lang.String originalFilePath)
                                             throws java.io.IOException

Create an InputStream from a file path. This method can be overridden if custom loading is needed. In Spark for example SparkFiles#get could be called.

Throws:

java.io.IOException

getSuffix

public java.lang.String getSuffix()

init

public void init(org.apache.hadoop.conf.Configuration conf)

getWrappedRecordReader

public org.apache.hadoop.mapreduce.RecordReader getWrappedRecordReader()