io.warp10.hadoop

Class WarpScriptInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,java.lang.Object>

io.warp10.hadoop.WarpScriptInputFormat

public class WarpScriptInputFormat
extends org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,java.lang.Object>

This InputFormat wraps another InputFormat and creates a RecordReader which returns K/V processed by some WarpScript code.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	PATH_CONFIG_KEY Configuration key we will set to the current path if the split if a FileSplit
static java.lang.String	WARPSCRIPT_INPUTFORMAT_CLASS Class of the wrapped InputFormat
static java.lang.String	WARPSCRIPT_INPUTFORMAT_CONF_SUFFIX Suffix to remove from configuration keys to override or create specific configuration entries
static java.lang.String	WARPSCRIPT_INPUTFORMAT_MACRO WarpScript code fragment producing a macro which will be applied
static java.lang.String	WARPSCRIPT_INPUTFORMAT_SCRIPT WarpScript code fragment to apply, this code will be wrapped in a macro <% ...
static java.lang.String	WARPSCRIPT_INPUTFORMAT_SUFFIX Suffix to use for the configuration

Constructor Summary

Constructors

Constructor and Description
WarpScriptInputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<java.lang.Object,java.lang.Object>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)
java.util.List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context)
java.lang.String	getSuffix()
WarpScriptExecutor	getWarpScriptExecutor(org.apache.hadoop.conf.Configuration conf, java.lang.String code, boolean isMacro) Return the actual WarpScript code executor given the script which was passed as parameter.
java.io.InputStream	getWarpScriptInputStream(java.lang.String originalFilePath) Create an InputStream from a file path.
org.apache.hadoop.mapreduce.RecordReader	getWrappedRecordReader()
void	init(org.apache.hadoop.conf.Configuration conf)
java.lang.String	parseWarpScript(java.lang.String filepath)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PATH_CONFIG_KEY

public static final java.lang.String PATH_CONFIG_KEY

Configuration key we will set to the current path if the split if a FileSplit

See Also:

Constant Field Values

WARPSCRIPT_INPUTFORMAT_SUFFIX

public static final java.lang.String WARPSCRIPT_INPUTFORMAT_SUFFIX

Suffix to use for the configuration

See Also:

Constant Field Values

WARPSCRIPT_INPUTFORMAT_CLASS

public static final java.lang.String WARPSCRIPT_INPUTFORMAT_CLASS

Class of the wrapped InputFormat

See Also:

Constant Field Values

WARPSCRIPT_INPUTFORMAT_SCRIPT

public static final java.lang.String WARPSCRIPT_INPUTFORMAT_SCRIPT

WarpScript code fragment to apply, this code will be wrapped in a macro <% ... %> If code starts by @ or % then the rest of its content is assumed to be a file path to load. If code starts by @ the content of the file will be wrapped in <% ... %>

See Also:

Constant Field Values

WARPSCRIPT_INPUTFORMAT_MACRO

public static final java.lang.String WARPSCRIPT_INPUTFORMAT_MACRO

WarpScript code fragment producing a macro which will be applied

See Also:

Constant Field Values

WARPSCRIPT_INPUTFORMAT_CONF_SUFFIX

public static final java.lang.String WARPSCRIPT_INPUTFORMAT_CONF_SUFFIX

Suffix to remove from configuration keys to override or create specific configuration entries

See Also:

Constant Field Values

Constructor Detail

WarpScriptInputFormat

public WarpScriptInputFormat()

Method Detail

getSplits

public java.util.List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context)
                                                                 throws java.io.IOException,
                                                                        java.lang.InterruptedException

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,java.lang.Object>

Throws:

java.io.IOException

java.lang.InterruptedException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<java.lang.Object,java.lang.Object> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                                                                      org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                               throws java.io.IOException,
                                                                                                      java.lang.InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<java.lang.Object,java.lang.Object>

Throws:

java.io.IOException

java.lang.InterruptedException

getWarpScriptExecutor

public WarpScriptExecutor getWarpScriptExecutor(org.apache.hadoop.conf.Configuration conf,
                                                java.lang.String code,
                                                boolean isMacro)
                                         throws java.io.IOException,
                                                WarpScriptException

Return the actual WarpScript code executor given the script which was passed as parameter. If isMacro is true, the script in 'code' is expected to produce a macro.

Throws:

java.io.IOException

WarpScriptException

parseWarpScript

public java.lang.String parseWarpScript(java.lang.String filepath)
                                 throws java.io.IOException

Throws:

java.io.IOException

getWarpScriptInputStream

public java.io.InputStream getWarpScriptInputStream(java.lang.String originalFilePath)
                                             throws java.io.IOException

Create an InputStream from a file path. This method can be overridden if custom loading is needed. In Spark for example SparkFiles#get could be called.

Throws:

java.io.IOException

getSuffix

public java.lang.String getSuffix()

init

public void init(org.apache.hadoop.conf.Configuration conf)

getWrappedRecordReader

public org.apache.hadoop.mapreduce.RecordReader getWrappedRecordReader()