eu.stratosphere.hadoopcompatibility

Class HadoopDataSink<K,V>

java.lang.Object

eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Nothing>

eu.stratosphere.api.common.operators.base.GenericDataSinkBase<eu.stratosphere.types.Record>

eu.stratosphere.api.java.record.operators.GenericDataSink

eu.stratosphere.hadoopcompatibility.HadoopDataSink<K,V>

All Implemented Interfaces:

eu.stratosphere.compiler.contextcheck.Validatable, eu.stratosphere.util.Visitable<eu.stratosphere.api.common.operators.Operator<?>>

public class HadoopDataSink<K,V>
extends GenericDataSink
implements eu.stratosphere.compiler.contextcheck.Validatable

The HadoopDataSink is a generic wrapper for all Hadoop OutputFormats. Example usage:

                HadoopDataSink out = new HadoopDataSink(new org.apache.hadoop.mapred.TextOutputFormat(), new JobConf(), "Hadoop TextOutputFormat",reducer, Text.class,IntWritable.class);
                org.apache.hadoop.mapred.TextOutputFormat.setOutputPath(out.getJobConf(), new Path(output));
 

Note that it is possible to provide custom data type converter. The HadoopDataSink provides a default converter: DefaultStratosphereTypeConverter

Field Summary

Fields inherited from class eu.stratosphere.api.common.operators.base.GenericDataSinkBase

formatWrapper, input

Fields inherited from class eu.stratosphere.api.common.operators.Operator

compilerHints, name, operatorInfo, parameters

Constructor Summary

Constructors

Constructor and Description
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, List<eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record>> input, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record> input, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, String name, List<eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record>> input, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, String name, List<eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record>> input, StratosphereTypeConverter<K,V> conv, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, String name, eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record> input, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, org.apache.hadoop.mapred.JobConf jobConf, String name, eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record> input, StratosphereTypeConverter<K,V> conv, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, List<eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record>> input, Class<K> keyClass, Class<V> valueClass)
HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat, eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record> input, Class<K> keyClass, Class<V> valueClass)

Method Summary

Modifier and Type	Method and Description
void	check()
org.apache.hadoop.mapred.JobConf	getJobConf()

Methods inherited from class eu.stratosphere.api.java.record.operators.GenericDataSink

addInput, addInputs, setInputs, setInputs

Methods inherited from class eu.stratosphere.api.common.operators.base.GenericDataSinkBase

accept, getDataDistribution, getFormatWrapper, getInput, getLocalOrder, getPartitionOrdering, getUserCodeWrapper, setGlobalOrder, setGlobalOrder, setInput, setLocalOrder, setRangePartitioned, setRangePartitioned, toString

Methods inherited from class eu.stratosphere.api.common.operators.Operator

createUnionCascade, createUnionCascade, createUnionCascade, getCompilerHints, getDegreeOfParallelism, getName, getOperatorInfo, getParameters, setDegreeOfParallelism, setName, setParameter, setParameter, setParameter

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
                      org.apache.hadoop.mapred.JobConf jobConf,
                      String name,
                      eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record> input,
                      StratosphereTypeConverter<K,V> conv,
                      Class<K> keyClass,
                      Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
                      org.apache.hadoop.mapred.JobConf jobConf,
                      String name,
                      eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record> input,
                      Class<K> keyClass,
                      Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
                      org.apache.hadoop.mapred.JobConf jobConf,
                      eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record> input,
                      Class<K> keyClass,
                      Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
                      eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record> input,
                      Class<K> keyClass,
                      Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
                      org.apache.hadoop.mapred.JobConf jobConf,
                      String name,
                      List<eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record>> input,
                      StratosphereTypeConverter<K,V> conv,
                      Class<K> keyClass,
                      Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
                      org.apache.hadoop.mapred.JobConf jobConf,
                      String name,
                      List<eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record>> input,
                      Class<K> keyClass,
                      Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
                      org.apache.hadoop.mapred.JobConf jobConf,
                      List<eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record>> input,
                      Class<K> keyClass,
                      Class<V> valueClass)

HadoopDataSink

public HadoopDataSink(org.apache.hadoop.mapred.OutputFormat<K,V> hadoopFormat,
                      List<eu.stratosphere.api.common.operators.Operator<eu.stratosphere.types.Record>> input,
                      Class<K> keyClass,
                      Class<V> valueClass)

Method Detail

getJobConf

public org.apache.hadoop.mapred.JobConf getJobConf()

check

public void check()

Specified by:

check in interface eu.stratosphere.compiler.contextcheck.Validatable