cascading.tap.hadoop

Class GlobHfs

java.lang.Object

cascading.tap.Tap<Config,Input,java.lang.Void>

cascading.tap.SourceTap<Config,Input>

cascading.tap.MultiSourceTap<Hfs,Configuration,RecordReader>

cascading.tap.hadoop.GlobHfs

All Implemented Interfaces:

cascading.flow.FlowElement, cascading.flow.planner.ScopedElement, cascading.tap.CompositeTap<Hfs>, cascading.util.Traceable, java.io.Serializable

public class GlobHfs
extends cascading.tap.MultiSourceTap<Hfs,Configuration,RecordReader>

Class GlobHfs is a type of MultiSourceTap that accepts Hadoop style 'file globing' expressions so multiple files that match the given pattern may be used as the input sources for a given Flow.

See FileSystem#globStatus(org.apache.hadoop.fs.Path) for details on the globing syntax. But in short it is similar to standard regular expressions except alternation is done via {foo,bar} instead of (foo|bar).

Note that a Flow sourcing from GlobHfs is not currently compatible with the Cascade scheduler. GlobHfs expects the files and paths to exist so the wildcards can be resolved into concrete values so that the scheduler can order the Flows properly.

Note that globing can match files or directories. It may consume less resources to match directories and let Hadoop include all sub-files immediately contained in the directory instead of enumerating every individual file. Ending the glob path with a / should match only directories.

See Also:

Hfs, MultiSourceTap, FileSystem, Serialized Form

Field Summary

Fields inherited from class cascading.tap.MultiSourceTap

commonPrefix, prefixMap, taps

Constructor Summary

Constructors

Constructor and Description
GlobHfs(cascading.scheme.Scheme<Configuration,RecordReader,?,?,?> scheme, java.lang.String pathPattern) Constructor GlobHfs creates a new GlobHfs instance.
GlobHfs(cascading.scheme.Scheme<Configuration,RecordReader,?,?,?> scheme, java.lang.String pathPattern, PathFilter pathFilter) Constructor GlobHfs creates a new GlobHfs instance.

Method Summary

Modifier and Type	Method and Description
boolean	equals(java.lang.Object object)
java.lang.String	getIdentifier()
protected Hfs[]	getTaps()
int	hashCode()
void	sourceConfInit(cascading.flow.FlowProcess<? extends Configuration> process, Configuration conf)
java.lang.String	toString()

Methods inherited from class cascading.tap.MultiSourceTap

findMatchingTap, getChildTaps, getModifiedTime, getNumChildTaps, getScheme, getTapPrefixMap, getTapsCommonPrefix, isReplace, openForRead, resourceExists, unwrap

Methods inherited from class cascading.tap.SourceTap

commitResource, createResource, deleteResource, getSinkFields, isSink, openForWrite, prepareResourceForWrite, rollbackResource, sinkConfInit

Methods inherited from class cascading.tap.Tap

createResource, deleteResource, entryStream, entryStream, entryStreamCopy, entryStreamCopy, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getModifiedTime, getNodeConfigDef, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isSource, isTemporary, isUpdate, openForRead, openForReadUnchecked, openForWrite, outgoingScopeFor, prepareResourceForRead, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, setScheme, spliterator, splititerator, taps, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

GlobHfs

@ConstructorProperties(value={"scheme","pathPattern"})
public GlobHfs(cascading.scheme.Scheme<Configuration,RecordReader,?,?,?> scheme,
                                                                      java.lang.String pathPattern)

Constructor GlobHfs creates a new GlobHfs instance.

Parameters:

scheme - of type Scheme

pathPattern - of type String

GlobHfs

@ConstructorProperties(value={"scheme","pathPattern","pathFilter"})
public GlobHfs(cascading.scheme.Scheme<Configuration,RecordReader,?,?,?> scheme,
                                                                                   java.lang.String pathPattern,
                                                                                   PathFilter pathFilter)

Constructor GlobHfs creates a new GlobHfs instance.

Parameters:

scheme - of type Scheme

pathPattern - of type String

pathFilter - of type PathFilter

Method Detail

getIdentifier

public java.lang.String getIdentifier()

Overrides:

getIdentifier in class cascading.tap.MultiSourceTap<Hfs,Configuration,RecordReader>

getTaps

protected Hfs[] getTaps()

Overrides:

getTaps in class cascading.tap.MultiSourceTap<Hfs,Configuration,RecordReader>

sourceConfInit

public void sourceConfInit(cascading.flow.FlowProcess<? extends Configuration> process,
                           Configuration conf)

Overrides:

sourceConfInit in class cascading.tap.MultiSourceTap<Hfs,Configuration,RecordReader>

equals

public boolean equals(java.lang.Object object)

Overrides:

equals in class cascading.tap.MultiSourceTap<Hfs,Configuration,RecordReader>

hashCode

public int hashCode()

Overrides:

hashCode in class cascading.tap.MultiSourceTap<Hfs,Configuration,RecordReader>

toString

public java.lang.String toString()

Overrides:

toString in class cascading.tap.MultiSourceTap<Hfs,Configuration,RecordReader>