cascading.local.tap.aws.s3

Class S3FileCheckpointer

java.lang.Object

cascading.local.tap.aws.s3.S3FileCheckpointer

All Implemented Interfaces:

S3Checkpointer

public class S3FileCheckpointer
extends java.lang.Object
implements S3Checkpointer

Class S3FileCheckpointer persists a map of bucket names and last seen keys to disk so multiple starts of a given application or new Flow instances relying on a S3Tap will re-retrieve the same data as previous executions.

By default this class will write all checkpoints to the directory [user.home]/.cascading/s3tap-checkpoints in files named [bucket-name].txt.

A Function can be supplied to override the filename creation.

Constructor Summary

Constructors

Constructor and Description
S3FileCheckpointer() Constructor S3FileCheckpointer creates a new S3FileCheckpointer instance.
S3FileCheckpointer(java.util.function.Function<java.lang.String,java.lang.String> filename) Constructor S3FileCheckpointer creates a new S3FileCheckpointer instance.
S3FileCheckpointer(java.nio.file.Path path) Constructor S3FileCheckpointer creates a new S3FileCheckpointer instance.
S3FileCheckpointer(java.nio.file.Path path, java.util.function.Function<java.lang.String,java.lang.String> filename) Constructor S3FileCheckpointer creates a new S3FileCheckpointer instance.
S3FileCheckpointer(java.lang.String path) Constructor S3FileCheckpointer creates a new S3FileCheckpointer instance.

Method Summary

Modifier and Type	Method and Description
void	commit() Method commit notifies the instance that no more keys will be handled.
static java.nio.file.Path	currentDir() Method currentDir uses the System property user.dir to retrieve the user's current working directory
java.lang.String	getLastKey(java.lang.String bucketName) Method getLastKey returns the last known key or null for the requested bucket.
static java.nio.file.Path	homeDir() Method homeDir uses the System property user.home to retrieve the user's home directory
static java.nio.file.Path	makeHidden(java.nio.file.Path path) Method makeHidden will append .cascading/s3tap-checkpoints to the given Path instance.
void	setLastKey(java.lang.String bucketName, java.lang.String key) Method setLastKey stores the last seen and consumed key for the given bucket.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

S3FileCheckpointer

public S3FileCheckpointer()

Constructor S3FileCheckpointer creates a new S3FileCheckpointer instance.

S3FileCheckpointer

public S3FileCheckpointer(java.lang.String path)

Constructor S3FileCheckpointer creates a new S3FileCheckpointer instance.

Parameters:

path - of String

S3FileCheckpointer

public S3FileCheckpointer(java.nio.file.Path path)

Constructor S3FileCheckpointer creates a new S3FileCheckpointer instance.

Parameters:

path - of Path

S3FileCheckpointer

public S3FileCheckpointer(java.util.function.Function<java.lang.String,java.lang.String> filename)

Constructor S3FileCheckpointer creates a new S3FileCheckpointer instance.

Parameters:

filename - of Function

S3FileCheckpointer

public S3FileCheckpointer(java.nio.file.Path path,
                          java.util.function.Function<java.lang.String,java.lang.String> filename)

Constructor S3FileCheckpointer creates a new S3FileCheckpointer instance.

Parameters:

path - of Path

filename - of Function

Method Detail

homeDir

public static java.nio.file.Path homeDir()

Method homeDir uses the System property user.home to retrieve the user's home directory

Returns:

Path

currentDir

public static java.nio.file.Path currentDir()

Method currentDir uses the System property user.dir to retrieve the user's current working directory

Returns:

Path

makeHidden

public static java.nio.file.Path makeHidden(java.nio.file.Path path)

Method makeHidden will append .cascading/s3tap-checkpoints to the given Path instance.

Parameters:

path - of Path

Returns:

Path

getLastKey

public java.lang.String getLastKey(java.lang.String bucketName)

Description copied from interface: S3Checkpointer

Method getLastKey returns the last known key or null for the requested bucket.

Specified by:

getLastKey in interface S3Checkpointer

Parameters:

bucketName - of String

Returns:

String

setLastKey

public void setLastKey(java.lang.String bucketName,
                       java.lang.String key)

Description copied from interface: S3Checkpointer

Method setLastKey stores the last seen and consumed key for the given bucket.

Specified by:

setLastKey in interface S3Checkpointer

Parameters:

bucketName - of String

key - of String

commit

public void commit()

Description copied from interface: S3Checkpointer

Method commit notifies the instance that no more keys will be handled.

Specified by:

commit in interface S3Checkpointer