io.lakefs

Class LakeFSFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

io.lakefs.LakeFSFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

public class LakeFSFileSystem
extends org.apache.hadoop.fs.FileSystem

A dummy implementation of the core lakeFS Filesystem. This class implements a LakeFSFileSystem that can be registered to Spark and support limited write and read actions.

Configure Spark to use lakeFS filesystem by property: spark.hadoop.fs.lakefs.impl=io.lakefs.LakeFSFileSystem.

Configure the application or the filesystem application by properties: fs.lakefs.endpoint=http://localhost:8000/api/v1 fs.lakefs.access.key=AKIAIOSFODNN7EXAMPLE fs.lakefs.secret.key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	LakeFSFileSystem.ClientFactory

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static String	LAKEFS_DELETE_BULK_SIZE
static org.slf4j.Logger	LOG
static org.slf4j.Logger	OPERATIONS_LOG

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
LakeFSFileSystem()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path path, int i, org.apache.hadoop.util.Progressable progressable)
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Called on a file write Spark/Hadoop action.
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
boolean	delete(org.apache.hadoop.fs.Path path, boolean recursive)
boolean	exists(org.apache.hadoop.fs.Path path)
long	getDefaultBlockSize()
long	getDefaultBlockSize(org.apache.hadoop.fs.Path path)
LakeFSFileStatus	getFileStatus(org.apache.hadoop.fs.Path path) Return a file status object that represents the path.
String	getScheme() Return the protocol scheme for the FileSystem.
URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory()
void	initialize(URI name, org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listFiles(org.apache.hadoop.fs.Path f, boolean recursive)
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path path)
boolean	mkdirs(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission fsPermission) Make the given path and all non-existent parents into directories.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path path, int bufSize)
ObjectLocation	pathToObjectLocation(org.apache.hadoop.fs.Path path) Returns Location with repository, ref and path used by lakeFS based on filesystem path.
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Rename, behaving similarly to the POSIX "mv" command, but non-atomically.
void	setWorkingDirectory(org.apache.hadoop.fs.Path path)
protected <R> R	withFileSystemAndTranslatedPhysicalPath(String physicalAddress, io.lakefs.LakeFSFileSystem.BiFunctionWithIOException<org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.Path,R> f)

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, close, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setReplication, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

OPERATIONS_LOG

public static final org.slf4j.Logger OPERATIONS_LOG

LAKEFS_DELETE_BULK_SIZE

public static final String LAKEFS_DELETE_BULK_SIZE

See Also:

Constant Field Values

Constructor Detail

LakeFSFileSystem

public LakeFSFileSystem()

Method Detail

getUri

public URI getUri()

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

initialize

public void initialize(URI name,
                       org.apache.hadoop.conf.Configuration conf)
                throws IOException

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

withFileSystemAndTranslatedPhysicalPath

protected <R> R withFileSystemAndTranslatedPhysicalPath(String physicalAddress,
                                                        io.lakefs.LakeFSFileSystem.BiFunctionWithIOException<org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.Path,R> f)
                                                 throws URISyntaxException,
                                                        IOException

Returns:

FileSystem suitable for the translated physical address

Throws:

URISyntaxException

IOException

getDefaultBlockSize

public long getDefaultBlockSize(org.apache.hadoop.fs.Path path)

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

getDefaultBlockSize

public long getDefaultBlockSize()

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path path,
                                                   int bufSize)
                                            throws IOException

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

listFiles

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFiles(org.apache.hadoop.fs.Path f,
                                                                                             boolean recursive)
                                                                                      throws FileNotFoundException,
                                                                                             IOException

Overrides:

listFiles in class org.apache.hadoop.fs.FileSystem

Throws:

FileNotFoundException

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path path,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Called on a file write Spark/Hadoop action. This method writes the content of the file in path into stdout.

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path path,
                                                      int i,
                                                      org.apache.hadoop.util.Progressable progressable)
                                               throws IOException

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws IOException

Rename, behaving similarly to the POSIX "mv" command, but non-atomically. 1. Rename is only supported for uncommitted data on the same branch. 2. The following rename scenarios are supported: file -> existing-file-name: rename(src.txt, existing-dst.txt) -> existing-dst.txt, existing-dst.txt is overridden file -> existing-directory-name: rename(src.txt, existing-dstdir) -> existing-dstdir/src.txt file -> non-existing dst: in case of non-existing rename target, false is return. note that empty directory is considered an existing directory and rename will move the directory/file into that folder. directory -> existing directory: rename(srcDir(containing srcDir/a.txt), existing-dstdir) -> existing-dstdir/a.txt 3. Rename dst path can be an uncommitted file, that will be overridden as a result of the rename operation. 4. The 'mtime' of the src object is not preserved.

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path path,
                      boolean recursive)
               throws IOException

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path path)
                                             throws FileNotFoundException,
                                                    IOException

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Throws:

FileNotFoundException

IOException

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path path)

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path path,
                      org.apache.hadoop.fs.permission.FsPermission fsPermission)
               throws IOException

Make the given path and all non-existent parents into directories. We use the same technic as S3A implementation, an object size 0, without a name with delimiter ('/') that keeps the directory exists. When we write an object into the directory - we can delete the marker.

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to create

fsPermission - to apply (passing to the underlying filesystem)

Returns:

an IOException that corresponds to the translated API exception

Throws:

IOException

getFileStatus

public LakeFSFileStatus getFileStatus(org.apache.hadoop.fs.Path path)
                               throws IOException

Return a file status object that represents the path.

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

path - to a file or directory

Returns:

a LakeFSFileStatus object

Throws:

FileNotFoundException - when the path does not exist; IOException API call or underlying filesystem exceptions

IOException

getScheme

public String getScheme()

Return the protocol scheme for the FileSystem.

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

Returns:

lakefs scheme

exists

public boolean exists(org.apache.hadoop.fs.Path path)
               throws IOException

Overrides:

exists in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

pathToObjectLocation

@Nonnull
public ObjectLocation pathToObjectLocation(org.apache.hadoop.fs.Path path)

Returns Location with repository, ref and path used by lakeFS based on filesystem path.

Parameters:

path - to extract information from.

Returns:

lakeFS Location with repository, ref and path

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path path,
                                                                  org.apache.hadoop.fs.permission.FsPermission permission,
                                                                  EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

IOException