com.pivotal.gemfirexd.hadoop.mapreduce

Class RowOutputFormat<VALUE>

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<Key,VALUE>

com.pivotal.gemfirexd.hadoop.mapreduce.RowOutputFormat<VALUE>

Type Parameters:

VALUE - User's data class containing setter methods for table columns

Direct Known Subclasses:

RowOutputFormat

public class RowOutputFormat<VALUE>
extends org.apache.hadoop.mapreduce.OutputFormat<Key,VALUE>

Map Reduce OutputFormat implementation for Gemfirexd. This class uses client connection to populate records in the DB. User is responsible for knowing name and type of the table columns. User provides a data class which contains setter methods for each column which needs to be populated. This class calls each setter method to populate the insert query.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	RowOutputFormat.GfxdRecordWriter RecordWriter implementation for Gemfirexd.

Field Summary

Fields

Modifier and Type	Field and Description
static String	OUTPUT_BATCH_SIZE By default row insert commands are executed in batch mode.
static int	OUTPUT_BATCH_SIZE_DEFAULT
static String	OUTPUT_TABLE The name of the table to be populated, for example APP.CUSTOMERS.
static String	OUTPUT_URL System URL where the output table is existing.

Constructor Summary

Constructors

Constructor and Description
RowOutputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext context)
protected static String	getDriver(org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.mapreduce.OutputCommitter	getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
org.apache.hadoop.mapreduce.RecordWriter<Key,VALUE>	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
protected static void	validateConfiguration(org.apache.hadoop.conf.Configuration conf) Validates correctness and completeness of job's output configuration.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OUTPUT_TABLE

public static final String OUTPUT_TABLE

The name of the table to be populated, for example APP.CUSTOMERS. This should match the table name used with the CREATE TABLE statement.

See Also:

Constant Field Values

OUTPUT_URL

public static final String OUTPUT_URL

System URL where the output table is existing. All output will be written into this table

See Also:

Constant Field Values

OUTPUT_BATCH_SIZE

public static final String OUTPUT_BATCH_SIZE

By default row insert commands are executed in batch mode. The batch size can be configured using this property. The default value is 10K commands. If the batch size is equal or less than 0, then the batch is when close operation is executed

See Also:

Constant Field Values

OUTPUT_BATCH_SIZE_DEFAULT

public static final int OUTPUT_BATCH_SIZE_DEFAULT

See Also:

Constant Field Values

Constructor Detail

RowOutputFormat

public RowOutputFormat()

Method Detail

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext context)
                      throws IOException,
                             InterruptedException

Specified by:

checkOutputSpecs in class org.apache.hadoop.mapreduce.OutputFormat<Key,VALUE>

Throws:

IOException

InterruptedException

validateConfiguration

protected static void validateConfiguration(org.apache.hadoop.conf.Configuration conf)
                                     throws org.apache.hadoop.mapred.InvalidJobConfException

Validates correctness and completeness of job's output configuration. Job configuration must contain url, table and schema name

Parameters:

conf - job conf

Throws:

org.apache.hadoop.mapred.InvalidJobConfException

getDriver

protected static String getDriver(org.apache.hadoop.conf.Configuration conf)

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter<Key,VALUE> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                    throws IOException

Specified by:

getRecordWriter in class org.apache.hadoop.mapreduce.OutputFormat<Key,VALUE>

Throws:

IOException

getOutputCommitter

public org.apache.hadoop.mapreduce.OutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                               throws IOException,
                                                                      InterruptedException

Specified by:

getOutputCommitter in class org.apache.hadoop.mapreduce.OutputFormat<Key,VALUE>

Throws:

IOException

InterruptedException