cascading.tap.hadoop

Class DistCacheTap

java.lang.Object

cascading.tap.Tap<Config,Input,Output>

cascading.tap.DecoratorTap<java.lang.Void,Configuration,RecordReader,OutputCollector>

cascading.tap.hadoop.BaseDistCacheTap

cascading.tap.hadoop.DistCacheTap

All Implemented Interfaces:

cascading.flow.FlowElement, cascading.flow.planner.ScopedElement, cascading.util.Traceable, java.io.Serializable

public class DistCacheTap
extends cascading.tap.hadoop.BaseDistCacheTap

Class DistCacheTap is a Tap decorator for Hfs and can be used to move a file to the DistributedCache on read when accessed cluster side.

This is useful for HashJoins.

The distributed cache is only used when the Tap is used as a source. If the DistCacheTap is used as a sink, it will delegate to the provided parent instance and not use the DistributedCache.

See Also:

Serialized Form

Field Summary

Fields inherited from class cascading.tap.DecoratorTap

metaInfo, original

Constructor Summary

Constructors

Constructor and Description
DistCacheTap(cascading.tap.hadoop.Hfs parent) Constructs a new DistCacheTap instance with the given Hfs.

Method Summary

Modifier and Type	Method and Description
protected void	addLocalCacheFiles(Configuration conf, java.net.URI uri)
protected Path[]	getLocalCacheFiles(cascading.flow.FlowProcess<? extends Configuration> flowProcess)

Methods inherited from class cascading.tap.hadoop.BaseDistCacheTap

openForRead, sourceConfInit

Methods inherited from class cascading.tap.DecoratorTap

commitResource, createResource, createResource, deleteResource, deleteResource, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getIdentifier, getMetaInfo, getModifiedTime, getModifiedTime, getNodeConfigDef, getOriginal, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hasNodeConfigDef, hasStepConfigDef, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForWrite, openForWrite, outgoingScopeFor, prepareResourceForRead, prepareResourceForWrite, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, resourceExists, retrieveSinkFields, retrieveSourceFields, rollbackResource, setMetaInfo, setOriginal, sinkConfInit, toString

Methods inherited from class cascading.tap.Tap

entryStream, entryStream, entryStreamCopy, entryStreamCopy, equals, hashCode, id, openForReadUnchecked, setScheme, spliterator, splititerator, taps, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

DistCacheTap

public DistCacheTap(cascading.tap.hadoop.Hfs parent)

Constructs a new DistCacheTap instance with the given Hfs.

Parameters:

parent - an Hfs or GlobHfs instance representing a small file.

Method Detail

getLocalCacheFiles

protected Path[] getLocalCacheFiles(cascading.flow.FlowProcess<? extends Configuration> flowProcess)
                             throws java.io.IOException

Specified by:

getLocalCacheFiles in class cascading.tap.hadoop.BaseDistCacheTap

Throws:

java.io.IOException

addLocalCacheFiles

protected void addLocalCacheFiles(Configuration conf,
                                  java.net.URI uri)

Specified by:

addLocalCacheFiles in class cascading.tap.hadoop.BaseDistCacheTap