cascading.tuple.hadoop.collect

Class HadoopSpillableTupleList

java.lang.Object

cascading.tuple.collect.SpillableTupleList

cascading.tuple.hadoop.collect.HadoopSpillableTupleList

All Implemented Interfaces:

cascading.tuple.collect.Spillable, java.lang.Iterable<cascading.tuple.Tuple>, java.util.Collection<cascading.tuple.Tuple>

public class HadoopSpillableTupleList
extends cascading.tuple.collect.SpillableTupleList

SpillableTupleList is a simple Iterable object that can store an unlimited number of Tuple instances by spilling excess to a temporary disk file.

Spills will automatically be compressed using the defaultCodecs values. To disable compression or change the codecs, see SpillableProps.SPILL_COMPRESS and SpillableProps.SPILL_CODECS.

It is recommended to add Lzo if available. "org.apache.hadoop.io.compress.LzoCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec"

Nested Class Summary

Nested classes/interfaces inherited from interface cascading.tuple.collect.Spillable

cascading.tuple.collect.Spillable.SpillListener, cascading.tuple.collect.Spillable.SpillStrategy

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	defaultCodecs

Constructor Summary

Constructors

Constructor and Description
HadoopSpillableTupleList(int threshold, CompressionCodec codec, Configuration configuration) Constructor SpillableTupleList creates a new SpillableTupleList instance using the given threshold value, and the first available compression codec, if any.
HadoopSpillableTupleList(int threshold, TupleSerialization tupleSerialization, CompressionCodec codec)

Method Summary

Modifier and Type	Method and Description
protected cascading.tuple.io.TupleInputStream	createTupleInputStream(java.io.File file)
protected cascading.tuple.io.TupleOutputStream	createTupleOutputStream(java.io.File file)
static CompressionCodec	getCodec(cascading.flow.FlowProcess<? extends Configuration> flowProcess, java.lang.String defaultCodecs)

Methods inherited from class cascading.tuple.collect.SpillableTupleList

add, addAll, clear, contains, containsAll, getCodecClass, getGrouping, getThreshold, isEmpty, iterator, remove, removeAll, retainAll, setGrouping, setSpillListener, setSpillStrategy, size, spillCount, toArray, toArray

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Collection

equals, hashCode, parallelStream, removeIf, spliterator, stream

Methods inherited from interface java.lang.Iterable

forEach

Field Detail

defaultCodecs

public static final java.lang.String defaultCodecs

See Also:

Constant Field Values

Constructor Detail

HadoopSpillableTupleList

public HadoopSpillableTupleList(int threshold,
                                CompressionCodec codec,
                                Configuration configuration)

Constructor SpillableTupleList creates a new SpillableTupleList instance using the given threshold value, and the first available compression codec, if any.

Parameters:

threshold - of type long

codec - of type CompressionCodec

HadoopSpillableTupleList

public HadoopSpillableTupleList(int threshold,
                                TupleSerialization tupleSerialization,
                                CompressionCodec codec)

Method Detail

getCodec

public static CompressionCodec getCodec(cascading.flow.FlowProcess<? extends Configuration> flowProcess,
                                        java.lang.String defaultCodecs)

createTupleOutputStream

protected cascading.tuple.io.TupleOutputStream createTupleOutputStream(java.io.File file)

Specified by:

createTupleOutputStream in class cascading.tuple.collect.SpillableTupleList

createTupleInputStream

protected cascading.tuple.io.TupleInputStream createTupleInputStream(java.io.File file)

Specified by:

createTupleInputStream in class cascading.tuple.collect.SpillableTupleList