com.pivotal.gemfirexd.hadoop.mapreduce

Class RowInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<Key,Row>

com.pivotal.gemfirexd.hadoop.mapreduce.RowInputFormat

public class RowInputFormat
extends org.apache.hadoop.mapreduce.InputFormat<Key,Row>

This class extends from InputFormat to provide table data reading ability.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CHECKPOINT_MODE In this mode checkpoint data is provided to job.
protected org.apache.hadoop.conf.Configuration	conf
static String	END_TIME_MILLIS If this filter is set, an event will be provided to map reduce job if it happened before this timestamp in milliseconds
static String	HOME_DIR The home directory for gemfirexd data in HDFS.
static String	INPUT_TABLE The name of the table to process, for example APP.CUSTOMERS.
static String	PROPERTY_PREFIX A valid property starting with this prefix will be used to tune input format's behavior.
static String	START_TIME_MILLIS If this filter is set, an event will be provided to map reduce job if it happened after this timestamp in milliseconds

Constructor Summary

Constructors

Constructor and Description
RowInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<Key,Row>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)
List<org.apache.hadoop.mapreduce.InputSplit>	createSplits(Collection<org.apache.hadoop.fs.FileStatus> hoplogs)
static String	getFullyQualifiedTableName(String name)
Collection<org.apache.hadoop.fs.FileStatus>	getHoplogs(org.apache.hadoop.conf.Configuration job) Identifies filters provided in the job configuration and creates a list of sorted hoplogs satisfying the filters.
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext job)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HOME_DIR

public static final String HOME_DIR

The home directory for gemfirexd data in HDFS. This should match the HOMEDIR setting that was used with the CREATE HDFSSTORE in statement.

See Also:

Constant Field Values

INPUT_TABLE

public static final String INPUT_TABLE

The name of the table to process, for example APP.CUSTOMERS. This should match the table name used with the CREATE TABLE statement.

See Also:

Constant Field Values

START_TIME_MILLIS

public static final String START_TIME_MILLIS

If this filter is set, an event will be provided to map reduce job if it happened after this timestamp in milliseconds

See Also:

Constant Field Values

END_TIME_MILLIS

public static final String END_TIME_MILLIS

If this filter is set, an event will be provided to map reduce job if it happened before this timestamp in milliseconds

See Also:

Constant Field Values

CHECKPOINT_MODE

public static final String CHECKPOINT_MODE

In this mode checkpoint data is provided to job. Mapper, instead of getting all the events that happened in the system, which also includes old values of a key, gets the latest value known at the time of creating the checkpoint only. Checkpoint may not include latest data present in the system when the job is executed. In the checkpoint mode, time filters are ignored

See Also:

Constant Field Values

PROPERTY_PREFIX

public static final String PROPERTY_PREFIX

A valid property starting with this prefix will be used to tune input format's behavior.

See Also:

Constant Field Values

conf

protected org.apache.hadoop.conf.Configuration conf

Constructor Detail

RowInputFormat

public RowInputFormat()

Method Detail

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<Key,Row> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                                   org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                     throws IOException,
                                                                            InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<Key,Row>

Throws:

IOException

InterruptedException

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext job)
                                                       throws IOException,
                                                              InterruptedException

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<Key,Row>

Throws:

IOException

InterruptedException

getHoplogs

public Collection<org.apache.hadoop.fs.FileStatus> getHoplogs(org.apache.hadoop.conf.Configuration job)
                                                       throws IOException

Identifies filters provided in the job configuration and creates a list of sorted hoplogs satisfying the filters. If there are no sorted hoplogs, checks if sequential hoplogs are present and executes the same process for sequential hoplogs

Returns:

list of hoplogs

Throws:

IOException

getFullyQualifiedTableName

public static String getFullyQualifiedTableName(String name)

createSplits

public List<org.apache.hadoop.mapreduce.InputSplit> createSplits(Collection<org.apache.hadoop.fs.FileStatus> hoplogs)
                                                          throws IOException

Throws:

IOException