cascading.tap.hadoop

Class PartitionTap

java.lang.Object

cascading.tap.Tap<Config,Input,Output>

cascading.tap.partition.BasePartitionTap<Configuration,RecordReader,OutputCollector>

cascading.tap.hadoop.PartitionTap

All Implemented Interfaces:

cascading.flow.FlowElement, cascading.flow.planner.ScopedElement, cascading.tap.type.FileType<Configuration>, cascading.util.Traceable, java.io.Serializable

public class PartitionTap
extends cascading.tap.partition.BasePartitionTap<Configuration,RecordReader,OutputCollector>

Class PartitionTap can be used to write tuple streams out to files and sub-directories based on the values in the current Tuple instance.

The constructor takes a Hfs Tap and a Partition implementation. This allows Tuple values at given positions to be used as directory names during write operations, and directory names as data during read operations.

The key value here is that there is no need to duplicate data values in the directory names and inside the data files.

So only values declared in the parent Tap will be read or written to the underlying file system files. But fields declared by the Partition will only be read or written to the directory names. That is, the PartitionTap instance will sink or source the partition fields, plus the parent Tap fields. The partition fields and parent Tap fields do not need to have common field names.

Note that Hadoop can only sink to directories, and all files in those directories are "part-xxxxx" files.

openWritesThreshold limits the number of open files to be output to. This value defaults to 300 files. Each time the threshold is exceeded, 10% of the least recently used open files will be closed.

PartitionTap will populate a given partition without regard to case of the values being used. Thus the resulting paths 2012/June/ and 2012/june/ will likely result in two open files into the same location. Forcing the case to be consistent with a custom Partition implementation or an upstream Function is recommended, see cascading.operation.expression.ExpressionFunction.

Though Hadoop has no mechanism to prevent simultaneous writes to a directory from multiple jobs, it doesn't mean its safe to do so. Same is true with the PartitionTap. Interleaving writes to a common parent (root) directory across multiple flows will very likely lead to data loss.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class cascading.tap.partition.BasePartitionTap

cascading.tap.partition.BasePartitionTap.Counters, cascading.tap.partition.BasePartitionTap.PartitionCollector, cascading.tap.partition.BasePartitionTap.PartitionScheme<Config,Input,Output>

Field Summary

Fields inherited from class cascading.tap.partition.BasePartitionTap

keepParentOnDelete, OPEN_WRITES_THRESHOLD_DEFAULT, openWritesThreshold, parent, partition, sourcePartitionFilters

Fields inherited from interface cascading.tap.type.FileType

CASCADING_SOURCE_PATH

Constructor Summary

Constructors

Constructor and Description
PartitionTap(Hfs parent, cascading.tap.partition.Partition partition) Constructor PartitionTap creates a new PartitionTap instance using the given parent Hfs Tap as the base path and default Scheme, and the partition.
PartitionTap(Hfs parent, cascading.tap.partition.Partition partition, int openWritesThreshold) Constructor PartitionTap creates a new PartitionTap instance using the given parent Hfs Tap as the base path and default Scheme, and the partition.
PartitionTap(Hfs parent, cascading.tap.partition.Partition partition, cascading.tap.SinkMode sinkMode) Constructor PartitionTap creates a new PartitionTap instance using the given parent Hfs Tap as the base path and default Scheme, and the partition.
PartitionTap(Hfs parent, cascading.tap.partition.Partition partition, cascading.tap.SinkMode sinkMode, boolean keepParentOnDelete) Constructor PartitionTap creates a new PartitionTap instance using the given parent Hfs Tap as the base path and default Scheme, and the partition.
PartitionTap(Hfs parent, cascading.tap.partition.Partition partition, cascading.tap.SinkMode sinkMode, boolean keepParentOnDelete, int openWritesThreshold) Constructor PartitionTap creates a new PartitionTap instance using the given parent Hfs Tap as the base path and default Scheme, and the partition.

Method Summary

Modifier and Type	Method and Description
boolean	commitResource(Configuration conf)
protected cascading.tuple.TupleEntrySchemeCollector	createTupleEntrySchemeCollector(cascading.flow.FlowProcess<? extends Configuration> flowProcess, cascading.tap.Tap parent, java.lang.String path, long sequence)
protected cascading.tuple.TupleEntrySchemeIterator	createTupleEntrySchemeIterator(cascading.flow.FlowProcess<? extends Configuration> flowProcess, cascading.tap.Tap parent, java.lang.String path, RecordReader recordReader)
protected java.lang.String	getCurrentIdentifier(cascading.flow.FlowProcess<? extends Configuration> flowProcess)
cascading.tuple.TupleEntryIterator	openForRead(cascading.flow.FlowProcess<? extends Configuration> flowProcess, RecordReader input)
void	sourceConfInit(cascading.flow.FlowProcess<? extends Configuration> flowProcess, Configuration conf)

Methods inherited from class cascading.tap.partition.BasePartitionTap

addSourcePartitionFilter, castFileType, createResource, deleteResource, equals, getChildIdentifiers, getChildIdentifiers, getChildIdentifiers, getChildIdentifiers, getChildPartitionIdentifiers, getFilteredPartitionIdentifiers, getIdentifier, getModifiedTime, getOpenWritesThreshold, getParent, getPartition, getSize, getSize, hashCode, isDirectory, isDirectory, openForWrite, prepareResourceForRead, prepareResourceForWrite, resourceExists, rollbackResource, toString

Methods inherited from class cascading.tap.Tap

createResource, deleteResource, entryStream, entryStream, entryStreamCopy, entryStreamCopy, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForReadUnchecked, openForWrite, outgoingScopeFor, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, setScheme, sinkConfInit, spliterator, splititerator, taps, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

PartitionTap

@ConstructorProperties(value={"parent","partition"})
public PartitionTap(Hfs parent,
                                                                         cascading.tap.partition.Partition partition)

Constructor PartitionTap creates a new PartitionTap instance using the given parent Hfs Tap as the base path and default Scheme, and the partition.

Parameters:

parent - of type Tap

partition - of type Partition

PartitionTap

@ConstructorProperties(value={"parent","partition","openWritesThreshold"})
public PartitionTap(Hfs parent,
                                                                                               cascading.tap.partition.Partition partition,
                                                                                               int openWritesThreshold)

Constructor PartitionTap creates a new PartitionTap instance using the given parent Hfs Tap as the base path and default Scheme, and the partition.

openWritesThreshold limits the number of open files to be output to.

Parameters:

parent - of type Hfs

partition - of type Partition

openWritesThreshold - of type int

PartitionTap

@ConstructorProperties(value={"parent","partition","sinkMode"})
public PartitionTap(Hfs parent,
                                                                                    cascading.tap.partition.Partition partition,
                                                                                    cascading.tap.SinkMode sinkMode)

Constructor PartitionTap creates a new PartitionTap instance using the given parent Hfs Tap as the base path and default Scheme, and the partition.

Parameters:

parent - of type Tap

partition - of type String

sinkMode - of type SinkMode

PartitionTap

@ConstructorProperties(value={"parent","partition","sinkMode","keepParentOnDelete"})
public PartitionTap(Hfs parent,
                                                                                                         cascading.tap.partition.Partition partition,
                                                                                                         cascading.tap.SinkMode sinkMode,
                                                                                                         boolean keepParentOnDelete)

Constructor PartitionTap creates a new PartitionTap instance using the given parent Hfs Tap as the base path and default Scheme, and the partition.

keepParentOnDelete, when set to true, prevents the parent Tap from being deleted when BasePartitionTap.deleteResource(Object) is called, typically an issue when used inside a Cascade.

Parameters:

parent - of type Tap

partition - of type Partition

sinkMode - of type SinkMode

keepParentOnDelete - of type boolean

PartitionTap

@ConstructorProperties(value={"parent","partition","sinkMode","keepParentOnDelete","openWritesThreshold"})
public PartitionTap(Hfs parent,
                                                                                                                               cascading.tap.partition.Partition partition,
                                                                                                                               cascading.tap.SinkMode sinkMode,
                                                                                                                               boolean keepParentOnDelete,
                                                                                                                               int openWritesThreshold)

Constructor PartitionTap creates a new PartitionTap instance using the given parent Hfs Tap as the base path and default Scheme, and the partition.

keepParentOnDelete, when set to true, prevents the parent Tap from being deleted when BasePartitionTap.deleteResource(Object) is called, typically an issue when used inside a Cascade.

openWritesThreshold limits the number of open files to be output to.

Parameters:

parent - of type Tap

partition - of type Partition

sinkMode - of type SinkMode

keepParentOnDelete - of type boolean

openWritesThreshold - of type int

Method Detail

createTupleEntrySchemeCollector

protected cascading.tuple.TupleEntrySchemeCollector createTupleEntrySchemeCollector(cascading.flow.FlowProcess<? extends Configuration> flowProcess,
                                                                                    cascading.tap.Tap parent,
                                                                                    java.lang.String path,
                                                                                    long sequence)
                                                                             throws java.io.IOException

Specified by:

createTupleEntrySchemeCollector in class cascading.tap.partition.BasePartitionTap<Configuration,RecordReader,OutputCollector>

Throws:

java.io.IOException

createTupleEntrySchemeIterator

protected cascading.tuple.TupleEntrySchemeIterator createTupleEntrySchemeIterator(cascading.flow.FlowProcess<? extends Configuration> flowProcess,
                                                                                  cascading.tap.Tap parent,
                                                                                  java.lang.String path,
                                                                                  RecordReader recordReader)
                                                                           throws java.io.IOException

Specified by:

createTupleEntrySchemeIterator in class cascading.tap.partition.BasePartitionTap<Configuration,RecordReader,OutputCollector>

Throws:

java.io.IOException

getCurrentIdentifier

protected java.lang.String getCurrentIdentifier(cascading.flow.FlowProcess<? extends Configuration> flowProcess)

Specified by:

getCurrentIdentifier in class cascading.tap.partition.BasePartitionTap<Configuration,RecordReader,OutputCollector>

sourceConfInit

public void sourceConfInit(cascading.flow.FlowProcess<? extends Configuration> flowProcess,
                           Configuration conf)

Overrides:

sourceConfInit in class cascading.tap.Tap<Configuration,RecordReader,OutputCollector>

openForRead

public cascading.tuple.TupleEntryIterator openForRead(cascading.flow.FlowProcess<? extends Configuration> flowProcess,
                                                      RecordReader input)
                                               throws java.io.IOException

Overrides:

openForRead in class cascading.tap.partition.BasePartitionTap<Configuration,RecordReader,OutputCollector>

Throws:

java.io.IOException

commitResource

public boolean commitResource(Configuration conf)
                       throws java.io.IOException

Overrides:

commitResource in class cascading.tap.partition.BasePartitionTap<Configuration,RecordReader,OutputCollector>

Throws:

java.io.IOException