cascading.local.tap.aws.s3

Class S3Tap

java.lang.Object

cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>

cascading.local.tap.aws.s3.S3Tap

All Implemented Interfaces:

cascading.flow.FlowElement, cascading.flow.planner.ScopedElement, cascading.tap.type.FileType<java.util.Properties>, cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>, cascading.util.Traceable, java.io.Serializable

public class S3Tap
extends cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>
implements cascading.tap.type.FileType<java.util.Properties>, cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>

Class S3Tap is a Cascading local-mode Tap providing read and write access to data stored in Amazon S3 buckets.

This Tap is not intended to be used with any of the other Cascading planners unless they specify they are local-mode compatible.

S3Tap can read a single key, all objects underneath a key-prefix, or all objects under a key-prefix that match a given globbing pattern.

See the various constructors for the available access parametrizations. Of note are the constructors that take a URI instance. The URI should be in the following format: s3://[bucket]/<key|key-prefix><?glob>

Where bucket is the only required value. The key references a single object, the key-prefix is used to access a set of objects with a common prefix value. The glob value is use to further narrow the resulting object set.

The globbing pattern is specified by the FileSystem.getPathMatcher(java.lang.String) method.

This Tap was designed to allow applications to effectively poll an S3 bucket for new keys to be processed.

When used with the S3FileCheckpointer class, a map of keys last consumed by each bucket will be tracked on disk, with the map surviving JVM restarts allowing for applications to exit and restart safely without retrieving duplicate data.

The S3Checkpointer.commit() method is only called during a graceful shutdown of the Flow or JVM, but every consumed key is passed to the S3Checkpointer, so custom implementations can choose to persist the key more frequently.

AWS Credentials are handled by DefaultAWSCredentialsProviderChain.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_DELIMITER Field DEFAULT_DELIMITER
static java.lang.String	MIME_DIRECTORY Field MIME_DIRECTORY
static java.lang.String	SEQUENCE_TOKEN Field SEQUENCE_TOKEN

Fields inherited from interface cascading.tap.type.FileType

CASCADING_SOURCE_PATH

Constructor Summary

Constructors

Constructor and Description
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, S3Checkpointer checkpointer, java.lang.String bucketName) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, S3Checkpointer checkpointer, java.lang.String bucketName, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, java.util.function.Predicate<java.lang.String> filter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, java.util.function.Predicate<java.lang.String> filter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, S3Checkpointer checkpointer, java.net.URI identifier) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, S3Checkpointer checkpointer, java.net.URI identifier, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String bucketName) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String bucketName, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String bucketName, java.lang.String key) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String bucketName, java.lang.String key, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, java.util.function.Predicate<java.lang.String> filter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, java.util.function.Predicate<java.lang.String> filter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.net.URI identifier) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.net.URI identifier, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, java.util.function.Predicate<java.lang.String> filter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, java.util.function.Predicate<java.lang.String> filter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, java.util.function.Predicate<java.lang.String> filter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, java.util.function.Predicate<java.lang.String> filter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, java.util.function.Predicate<java.lang.String> filter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, java.util.function.Predicate<java.lang.String> filter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.net.URI identifier) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, S3Checkpointer checkpointer, java.net.URI identifier, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, java.util.function.Predicate<java.lang.String> filter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, java.util.function.Predicate<java.lang.String> filter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, java.lang.String key) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, java.lang.String key, java.util.function.Predicate<java.lang.String> filter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, java.lang.String key, java.util.function.Predicate<java.lang.String> filter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, java.lang.String key, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, java.util.function.Predicate<java.lang.String> filter) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, java.util.function.Predicate<java.lang.String> filter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.net.URI identifier) Constructor S3Tap creates a new S3Tap instance.
S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, java.net.URI identifier, cascading.tap.SinkMode sinkMode) Constructor S3Tap creates a new S3Tap instance.

Method Summary

Modifier and Type	Method and Description
protected java.lang.String	cleanKey(java.net.URI identifier)
protected void	commitMarker()
protected cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>	create(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.lang.String bucketName, java.lang.String key, java.lang.String delimiter, java.util.function.Predicate<java.lang.String> filter, cascading.tap.SinkMode sinkMode)
protected cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>	create(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme, com.amazonaws.services.s3.AmazonS3 s3Client, java.net.URI identifier, cascading.tap.SinkMode sinkMode)
boolean	createResource(java.util.Properties conf)
protected com.amazonaws.services.s3.transfer.Upload	createUpload(java.lang.String key, com.amazonaws.services.s3.transfer.TransferManager transferManager, com.amazonaws.services.s3.model.PutObjectRequest request)
boolean	deleteResource(java.util.Properties conf)
java.lang.String	getBucketName() Method getBucketName returns the bucketName of this S3Tap object.
protected java.lang.String	getBucketNameFor(java.net.URI identifier)
S3Checkpointer	getCheckpointer() Method getCheckpointer returns the checkpointer of this S3Tap object.
java.lang.String[]	getChildIdentifiers(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess)
java.lang.String[]	getChildIdentifiers(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, int depth, boolean fullyQualified)
java.lang.String[]	getChildIdentifiers(java.util.Properties conf)
java.lang.String[]	getChildIdentifiers(java.util.Properties conf, int depth, boolean fullyQualified)
java.lang.String	getDelimiter() Method getDelimiter returns the delimiter of this S3Tap object.
java.util.function.Predicate<java.lang.String>	getFilter() Method getFilter returns the filter of this S3Tap object.
java.lang.String	getFullIdentifier(java.util.Properties conf)
java.lang.String	getIdentifier()
java.lang.String	getKey() Method getKey returns the key of this S3Tap object.
protected java.lang.String	getMarker()
long	getModifiedTime(java.util.Properties conf)
protected com.amazonaws.services.s3.model.ObjectMetadata	getObjectMetadata(java.util.Properties conf)
protected com.amazonaws.services.s3.AmazonS3	getS3Client(java.util.Properties properties)
long	getSize(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess)
long	getSize(java.util.Properties conf)
protected com.amazonaws.services.s3.model.AmazonS3Exception	handleException(com.amazonaws.services.s3.AmazonS3 s3Client, com.amazonaws.services.s3.model.AmazonS3Exception exception)
protected void	handleResult(com.amazonaws.services.s3.transfer.Upload upload, com.amazonaws.services.s3.transfer.model.UploadResult uploadResult, java.lang.String loggableIdentifier)
boolean	isDirectory(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess)
boolean	isDirectory(java.util.Properties conf)
protected java.lang.String	makePath(java.util.Iterator<com.amazonaws.services.s3.model.S3ObjectSummary> iterator, boolean fullyQualified)
protected static java.lang.String	makeStringIdentifier(java.lang.String bucketName, java.lang.String keyPrefix)
static java.net.URI	makeURI(java.lang.String bucketName, java.lang.String keyPrefix) Method makeURI creates a new S3 URI from the given parameters.
static java.net.URI	makeURI(java.lang.String bucketName, java.lang.String keyPrefix, java.lang.String glob) Method makeURI creates a new S3 URI from the given parameters.
cascading.tuple.TupleEntryIterator	openForRead(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, java.io.InputStream input)
cascading.tuple.TupleEntryCollector	openForWrite(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, java.io.OutputStream outputStream)
protected java.io.InputStream	openInputStream(com.google.common.io.ByteSource byteSource, java.lang.String loggableIdentifier)
protected java.lang.String	resolveKey(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess, java.lang.String key)
boolean	resourceExists(java.util.Properties conf)
protected void	setLastMarker(java.lang.String marker)
cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>	withChildIdentifier(java.lang.String identifier)
cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>	withScheme(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme)
cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>	withSinkMode(cascading.tap.SinkMode sinkMode)

Methods inherited from class cascading.tap.Tap

commitResource, createResource, deleteResource, entryStream, entryStream, entryStreamCopy, entryStreamCopy, equals, flowConfInit, getConfigDef, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForReadUnchecked, openForWrite, outgoingScopeFor, prepareResourceForRead, prepareResourceForWrite, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, rollbackResource, setScheme, sinkConfInit, sourceConfInit, spliterator, splititerator, taps, toString, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface cascading.tap.type.TapWith

asTap

Field Detail

SEQUENCE_TOKEN

public static final java.lang.String SEQUENCE_TOKEN

Field SEQUENCE_TOKEN

See Also:

Constant Field Values

MIME_DIRECTORY

public static final java.lang.String MIME_DIRECTORY

Field MIME_DIRECTORY

See Also:

Constant Field Values

DEFAULT_DELIMITER

public static final java.lang.String DEFAULT_DELIMITER

Field DEFAULT_DELIMITER

See Also:

Constant Field Values

Constructor Detail

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             java.lang.String key)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

key - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

key - of String

delimiter - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             java.util.function.Predicate<java.lang.String> filter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

filter - of Predicate

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             java.lang.String key,
             java.util.function.Predicate<java.lang.String> filter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

key - of String

filter - of Predicate

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             java.util.function.Predicate<java.lang.String> filter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

key - of String

delimiter - of String

filter - of Predicate

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             java.lang.String bucketName)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

bucketName - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             java.lang.String bucketName,
             java.lang.String key)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

bucketName - of String

key - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

bucketName - of String

key - of String

delimiter - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             java.util.function.Predicate<java.lang.String> filter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

bucketName - of String

key - of String

delimiter - of String

filter - of Predicate

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

delimiter - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.util.function.Predicate<java.lang.String> filter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

filter - of Predicate

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             java.util.function.Predicate<java.lang.String> filter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

filter - of Predicate

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             java.util.function.Predicate<java.lang.String> filter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

delimiter - of String

filter - of Predicate

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             S3Checkpointer checkpointer,
             java.lang.String bucketName)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

checkpointer - of S3Checkpointer

bucketName - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

delimiter - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             java.util.function.Predicate<java.lang.String> filter)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

delimiter - of String

filter - of Predicate

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             java.lang.String key,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

key - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

key - of String

delimiter - of String

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             java.util.function.Predicate<java.lang.String> filter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

filter - of Predicate

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             java.lang.String key,
             java.util.function.Predicate<java.lang.String> filter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

key - of String

filter - of Predicate

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             java.util.function.Predicate<java.lang.String> filter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

bucketName - of String

key - of String

delimiter - of String

filter - of Predicate

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             java.lang.String bucketName,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

bucketName - of String

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             java.lang.String bucketName,
             java.lang.String key,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

bucketName - of String

key - of String

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

bucketName - of String

key - of String

delimiter - of String

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             java.util.function.Predicate<java.lang.String> filter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

bucketName - of String

key - of String

delimiter - of String

filter - of Predicate

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

delimiter - of String

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.util.function.Predicate<java.lang.String> filter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

filter - of Predicate

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             java.util.function.Predicate<java.lang.String> filter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

filter - of Predicate

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             java.util.function.Predicate<java.lang.String> filter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

delimiter - of String

filter - of Predicate

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

checkpointer - of S3Checkpointer

bucketName - of String

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

delimiter - of String

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             S3Checkpointer checkpointer,
             java.lang.String bucketName,
             java.lang.String key,
             java.lang.String delimiter,
             java.util.function.Predicate<java.lang.String> filter,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

checkpointer - of S3Checkpointer

bucketName - of String

key - of String

delimiter - of String

filter - of Predicate

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.net.URI identifier)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

identifier - of URI

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             java.net.URI identifier)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

identifier - of URI

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.net.URI identifier)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

identifier - of URI

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             S3Checkpointer checkpointer,
             java.net.URI identifier)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

checkpointer - of S3Checkpointer

identifier - of URI

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             java.net.URI identifier,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

identifier - of URI

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             java.net.URI identifier,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

identifier - of URI

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             S3Checkpointer checkpointer,
             java.net.URI identifier,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

checkpointer - of S3Checkpointer

identifier - of URI

sinkMode - of SinkMode

S3Tap

public S3Tap(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
             com.amazonaws.services.s3.AmazonS3 s3Client,
             S3Checkpointer checkpointer,
             java.net.URI identifier,
             cascading.tap.SinkMode sinkMode)

Constructor S3Tap creates a new S3Tap instance.

Parameters:

scheme - of Scheme

s3Client - of AmazonS3

checkpointer - of S3Checkpointer

identifier - of URI

sinkMode - of SinkMode

Method Detail

makeURI

public static java.net.URI makeURI(java.lang.String bucketName,
                                   java.lang.String keyPrefix)

Method makeURI creates a new S3 URI from the given parameters.

Parameters:

bucketName - the S3 bucket name

keyPrefix - the S3 object key or key-prefix

Returns:

an URI instance

makeURI

public static java.net.URI makeURI(java.lang.String bucketName,
                                   java.lang.String keyPrefix,
                                   java.lang.String glob)

Method makeURI creates a new S3 URI from the given parameters.

Parameters:

bucketName - the S3 bucket name

keyPrefix - the S3 object key or key-prefix

glob - the globbing pattern to apply to the keys

Returns:

an URI instance

getBucketNameFor

protected java.lang.String getBucketNameFor(java.net.URI identifier)

withScheme

public cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream> withScheme(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme)

Specified by:

withScheme in interface cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>

withChildIdentifier

public cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream> withChildIdentifier(java.lang.String identifier)

Specified by:

withChildIdentifier in interface cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>

create

protected cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream> create(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
                                                                                                           com.amazonaws.services.s3.AmazonS3 s3Client,
                                                                                                           java.net.URI identifier,
                                                                                                           cascading.tap.SinkMode sinkMode)

withSinkMode

public cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream> withSinkMode(cascading.tap.SinkMode sinkMode)

Specified by:

withSinkMode in interface cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream>

create

protected cascading.tap.type.TapWith<java.util.Properties,java.io.InputStream,java.io.OutputStream> create(cascading.scheme.Scheme<java.util.Properties,java.io.InputStream,java.io.OutputStream,?,?> scheme,
                                                                                                           com.amazonaws.services.s3.AmazonS3 s3Client,
                                                                                                           java.lang.String bucketName,
                                                                                                           java.lang.String key,
                                                                                                           java.lang.String delimiter,
                                                                                                           java.util.function.Predicate<java.lang.String> filter,
                                                                                                           cascading.tap.SinkMode sinkMode)

cleanKey

protected java.lang.String cleanKey(java.net.URI identifier)

getS3Client

protected com.amazonaws.services.s3.AmazonS3 getS3Client(java.util.Properties properties)

getCheckpointer

public S3Checkpointer getCheckpointer()

Method getCheckpointer returns the checkpointer of this S3Tap object.

Returns:

the checkpointer (type S3Checkpointer) of this S3Tap object.

getBucketName

public java.lang.String getBucketName()

Method getBucketName returns the bucketName of this S3Tap object.

Returns:

the bucketName (type String) of this S3Tap object.

getKey

public java.lang.String getKey()

Method getKey returns the key of this S3Tap object.

Returns:

the key (type String) of this S3Tap object.

getMarker

protected java.lang.String getMarker()

setLastMarker

protected void setLastMarker(java.lang.String marker)

commitMarker

protected void commitMarker()

getFilter

public java.util.function.Predicate<java.lang.String> getFilter()

Method getFilter returns the filter of this S3Tap object.

Returns:

the filter (type Predicate) of this S3Tap object.

getDelimiter

public java.lang.String getDelimiter()

Method getDelimiter returns the delimiter of this S3Tap object.

Returns:

the delimiter (type String) of this S3Tap object.

getIdentifier

public java.lang.String getIdentifier()

Specified by:

getIdentifier in class cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>

getFullIdentifier

public java.lang.String getFullIdentifier(java.util.Properties conf)

Overrides:

getFullIdentifier in class cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>

deleteResource

public boolean deleteResource(java.util.Properties conf)
                       throws java.io.IOException

Specified by:

deleteResource in class cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>

Throws:

java.io.IOException

createResource

public boolean createResource(java.util.Properties conf)
                       throws java.io.IOException

Specified by:

createResource in class cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>

Throws:

java.io.IOException

getObjectMetadata

protected com.amazonaws.services.s3.model.ObjectMetadata getObjectMetadata(java.util.Properties conf)

openForRead

public cascading.tuple.TupleEntryIterator openForRead(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                                                      java.io.InputStream input)
                                               throws java.io.IOException

Specified by:

openForRead in class cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>

Throws:

java.io.IOException

openForWrite

public cascading.tuple.TupleEntryCollector openForWrite(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                                                        java.io.OutputStream outputStream)
                                                 throws java.io.IOException

Specified by:

openForWrite in class cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>

Throws:

java.io.IOException

handleResult

protected void handleResult(com.amazonaws.services.s3.transfer.Upload upload,
                            com.amazonaws.services.s3.transfer.model.UploadResult uploadResult,
                            java.lang.String loggableIdentifier)

openInputStream

protected java.io.InputStream openInputStream(com.google.common.io.ByteSource byteSource,
                                              java.lang.String loggableIdentifier)

createUpload

protected com.amazonaws.services.s3.transfer.Upload createUpload(java.lang.String key,
                                                                 com.amazonaws.services.s3.transfer.TransferManager transferManager,
                                                                 com.amazonaws.services.s3.model.PutObjectRequest request)

resolveKey

protected java.lang.String resolveKey(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                                      java.lang.String key)

resourceExists

public boolean resourceExists(java.util.Properties conf)
                       throws java.io.IOException

Specified by:

resourceExists in class cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>

Throws:

java.io.IOException

handleException

protected com.amazonaws.services.s3.model.AmazonS3Exception handleException(com.amazonaws.services.s3.AmazonS3 s3Client,
                                                                            com.amazonaws.services.s3.model.AmazonS3Exception exception)

getModifiedTime

public long getModifiedTime(java.util.Properties conf)
                     throws java.io.IOException

Specified by:

getModifiedTime in class cascading.tap.Tap<java.util.Properties,java.io.InputStream,java.io.OutputStream>

Throws:

java.io.IOException

isDirectory

public boolean isDirectory(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess)
                    throws java.io.IOException

Specified by:

isDirectory in interface cascading.tap.type.FileType<java.util.Properties>

Throws:

java.io.IOException

isDirectory

public boolean isDirectory(java.util.Properties conf)
                    throws java.io.IOException

Specified by:

isDirectory in interface cascading.tap.type.FileType<java.util.Properties>

Throws:

java.io.IOException

getChildIdentifiers

public java.lang.String[] getChildIdentifiers(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess)
                                       throws java.io.IOException

Specified by:

getChildIdentifiers in interface cascading.tap.type.FileType<java.util.Properties>

Throws:

java.io.IOException

getChildIdentifiers

public java.lang.String[] getChildIdentifiers(java.util.Properties conf)
                                       throws java.io.IOException

Specified by:

getChildIdentifiers in interface cascading.tap.type.FileType<java.util.Properties>

Throws:

java.io.IOException

getChildIdentifiers

public java.lang.String[] getChildIdentifiers(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess,
                                              int depth,
                                              boolean fullyQualified)
                                       throws java.io.IOException

Specified by:

getChildIdentifiers in interface cascading.tap.type.FileType<java.util.Properties>

Throws:

java.io.IOException

getChildIdentifiers

public java.lang.String[] getChildIdentifiers(java.util.Properties conf,
                                              int depth,
                                              boolean fullyQualified)
                                       throws java.io.IOException

Specified by:

getChildIdentifiers in interface cascading.tap.type.FileType<java.util.Properties>

Throws:

java.io.IOException

makePath

protected java.lang.String makePath(java.util.Iterator<com.amazonaws.services.s3.model.S3ObjectSummary> iterator,
                                    boolean fullyQualified)

getSize

public long getSize(cascading.flow.FlowProcess<? extends java.util.Properties> flowProcess)
             throws java.io.IOException

Specified by:

getSize in interface cascading.tap.type.FileType<java.util.Properties>

Throws:

java.io.IOException

getSize

public long getSize(java.util.Properties conf)
             throws java.io.IOException

Specified by:

getSize in interface cascading.tap.type.FileType<java.util.Properties>

Throws:

java.io.IOException

makeStringIdentifier

protected static java.lang.String makeStringIdentifier(java.lang.String bucketName,
                                                       java.lang.String keyPrefix)