alluxio.hadoop

Class AbstractFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

alluxio.hadoop.AbstractFileSystem

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

Direct Known Subclasses:

FileSystem

@NotThreadSafe
public abstract class AbstractFileSystem
extends org.apache.hadoop.fs.FileSystem

Base class for Apache Hadoop based Alluxio FileSystem. This class really just delegates to FileSystem for most operations. All implementing classes must define isZookeeperMode() which states if fault tolerant is used and getScheme() for Hadoop's ServiceLoader support.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
protected AlluxioConfiguration	mAlluxioConf
protected FileSystem	mFileSystem

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractFileSystem() Constructs a new AbstractFileSystem instance.
protected	AbstractFileSystem(FileSystem fileSystem) Constructs a new AbstractFileSystem instance with specified a FileSystem handler for tests.

Method Summary

Modifier and Type	Method and Description
void	access(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsAction mode)
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path path, int bufferSize, org.apache.hadoop.util.Progressable progress)
protected java.net.URI	canonicalizeUri(java.net.URI uri)
void	close()
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Attempts to create a file.
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) Deprecated. API only for 0.20-append
boolean	delete(org.apache.hadoop.fs.Path path) Deprecated. Use delete(Path, boolean) instead.
boolean	delete(org.apache.hadoop.fs.Path path, boolean recursive) Attempts to delete the file or directory with the specified path.
protected abstract AlluxioURI	getAlluxioPath(org.apache.hadoop.fs.Path path) Used to convert hadoop path to Alluxio path.
java.lang.String	getCanonicalServiceName()
protected abstract java.util.Map<java.lang.String,java.lang.Object>	getConfigurationFromUri(java.net.URI uri, org.apache.hadoop.conf.Configuration conf) Gets the connection configuration from the input uri.
long	getDefaultBlockSize()
protected int	getDefaultPort()
short	getDefaultReplication()
org.apache.hadoop.fs.BlockLocation[]	getFileBlockLocations(org.apache.hadoop.fs.FileStatus file, long start, long len)
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path path) If the file does not exist in Alluxio, query it from HDFS.
protected abstract org.apache.hadoop.fs.Path	getFsPath(java.lang.String fsUriHeader, URIStatus fileStatus) Used to get FS native path from Alluxio file status.
protected abstract java.lang.String	getFsScheme(java.net.URI fsUri) Used to get FS scheme.
abstract java.lang.String	getScheme() Gets the URI scheme that maps to the FileSystem.
java.net.URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory()
void	initialize(java.net.URI uri, org.apache.hadoop.conf.Configuration conf) Sets up a lazy connection to Alluxio through mFileSystem.
void	initialize(java.net.URI uri, org.apache.hadoop.conf.Configuration conf, AlluxioConfiguration alluxioConfiguration) Initialize the FileSystem.
protected abstract boolean	isZookeeperMode() Deprecated. in 1.6.0, directly infer the value from PropertyKey.ZOOKEEPER_ENABLED configuration value. Determines if zookeeper should be used for the FileSystem. This method should only be used for initialize(java.net.URI, org.apache.hadoop.conf.Configuration).
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path path)
boolean	mkdirs(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission) Attempts to create a folder with the specified path.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path path, int bufferSize) Attempts to open the specified file for reading.
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	setOwner(org.apache.hadoop.fs.Path path, java.lang.String username, java.lang.String groupname) Changes owner or group of a path (i.e.
void	setPermission(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission) Changes permission of a path.
boolean	setReplication(org.apache.hadoop.fs.Path path, short replication)
void	setWorkingDirectory(org.apache.hadoop.fs.Path path)
protected abstract void	validateFsUri(java.net.URI fsUri) Validates given FS base URI for scheme and authority.

Methods inherited from class org.apache.hadoop.fs.FileSystem

append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, checkPath, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createDataInputStreamBuilder, createDataInputStreamBuilder, createDataOutputStreamBuilder, createFile, createMultipartUploader, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, hasPathCapability, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFile, openFile, openFileWithOptions, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setQuota, setQuotaByStorageType, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, collectDelegationTokens

Field Detail

mAlluxioConf

protected AlluxioConfiguration mAlluxioConf

mFileSystem

protected FileSystem mFileSystem

Constructor Detail

AbstractFileSystem

protected AbstractFileSystem(FileSystem fileSystem)

Constructs a new AbstractFileSystem instance with specified a FileSystem handler for tests.

Parameters:

fileSystem - handler to file system

AbstractFileSystem

protected AbstractFileSystem()

Constructs a new AbstractFileSystem instance.

Method Detail

access

public void access(org.apache.hadoop.fs.Path path,
                   org.apache.hadoop.fs.permission.FsAction mode)
            throws java.io.IOException

Overrides:

access in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path path,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws java.io.IOException

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Overrides:

close in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path path,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws java.io.IOException

Attempts to create a file. Overwrite will not succeed if the path exists and is a folder.

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to create

permission - permissions of the created file/folder

overwrite - overwrite if file exists

bufferSize - the size in bytes of the buffer to be used

replication - under filesystem replication factor, this is ignored

blockSize - block size in bytes

progress - queryable progress

Returns:

an FSDataOutputStream created at the indicated path of a file

Throws:

java.io.IOException

createNonRecursive

@Deprecated
public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path path,
                                                                              org.apache.hadoop.fs.permission.FsPermission permission,
                                                                              boolean overwrite,
                                                                              int bufferSize,
                                                                              short replication,
                                                                              long blockSize,
                                                                              org.apache.hadoop.util.Progressable progress)
                                                                       throws java.io.IOException

Deprecated. API only for 0.20-append

Opens an FSDataOutputStream at the indicated Path with write-progress reporting. Same as FileSystem.create(Path, boolean, int, short, long, Progressable), except fails if parent directory doesn't already exist. TODO(hy): We need to refactor this method after having a new internal API support (ALLUXIO-46).

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Parameters:

path - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used

replication - required block replication for the file

blockSize - the size in bytes of the buffer to be used

progress - queryable progress

Throws:

java.io.IOException

See Also:

setPermission(Path, FsPermission)

delete

@Deprecated
public boolean delete(org.apache.hadoop.fs.Path path)
                           throws java.io.IOException

Deprecated. Use delete(Path, boolean) instead.

Attempts to delete the file or directory with the specified path.

Overrides:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to delete

Returns:

true if one or more files/directories were deleted; false otherwise

Throws:

java.io.IOException

delete

public boolean delete(org.apache.hadoop.fs.Path path,
                      boolean recursive)
               throws java.io.IOException

Attempts to delete the file or directory with the specified path.

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to delete

recursive - if true, will attempt to delete all children of the path

Returns:

true if one or more files/directories were deleted; false otherwise

Throws:

java.io.IOException

getDefaultPort

protected int getDefaultPort()

Overrides:

getDefaultPort in class org.apache.hadoop.fs.FileSystem

canonicalizeUri

protected java.net.URI canonicalizeUri(java.net.URI uri)

Overrides:

canonicalizeUri in class org.apache.hadoop.fs.FileSystem

getDefaultBlockSize

public long getDefaultBlockSize()

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

getCanonicalServiceName

public java.lang.String getCanonicalServiceName()

Specified by:

getCanonicalServiceName in interface org.apache.hadoop.security.token.DelegationTokenIssuer

Overrides:

getCanonicalServiceName in class org.apache.hadoop.fs.FileSystem

getFileBlockLocations

@Nullable
public org.apache.hadoop.fs.BlockLocation[] getFileBlockLocations(org.apache.hadoop.fs.FileStatus file,
                                                                            long start,
                                                                            long len)
                                                                     throws java.io.IOException

Overrides:

getFileBlockLocations in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

getDefaultReplication

public short getDefaultReplication()

Overrides:

getDefaultReplication in class org.apache.hadoop.fs.FileSystem

setReplication

public boolean setReplication(org.apache.hadoop.fs.Path path,
                              short replication)
                       throws java.io.IOException

Overrides:

setReplication in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path path)
                                              throws java.io.IOException

If the file does not exist in Alluxio, query it from HDFS.

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

setOwner

public void setOwner(org.apache.hadoop.fs.Path path,
                     java.lang.String username,
                     java.lang.String groupname)
              throws java.io.IOException

Changes owner or group of a path (i.e. a file or a directory). If username is null, the original username remains unchanged. Same as groupname. If username and groupname are non-null, both of them will be changed.

Overrides:

setOwner in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to set owner or group

username - username to be set

groupname - groupname to be set

Throws:

java.io.IOException

setPermission

public void setPermission(org.apache.hadoop.fs.Path path,
                          org.apache.hadoop.fs.permission.FsPermission permission)
                   throws java.io.IOException

Changes permission of a path.

Overrides:

setPermission in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to set permission

permission - permission set to path

Throws:

java.io.IOException

getScheme

public abstract java.lang.String getScheme()

Gets the URI scheme that maps to the FileSystem. This was introduced in Hadoop 2.x as a means to make loading new FileSystem s simpler.

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

Returns:

scheme hadoop should map to

See Also:

FileSystem.createFileSystem(java.net.URI, org.apache.hadoop.conf.Configuration)

getUri

public java.net.URI getUri()

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

initialize

public void initialize(java.net.URI uri,
                       org.apache.hadoop.conf.Configuration conf)
                throws java.io.IOException

Sets up a lazy connection to Alluxio through mFileSystem. This must be called before client operations. If it is called twice on the same object concurrently, one of the thread will do the initialization work, the other will wait until initialization is done. If it is called after initialized, then this is a noop.

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

initialize

public void initialize(java.net.URI uri,
                       org.apache.hadoop.conf.Configuration conf,
                       @Nullable
                       AlluxioConfiguration alluxioConfiguration)
                throws java.io.IOException

Initialize the FileSystem.

Parameters:

uri - file system Uri

conf - hadoop configuration

alluxioConfiguration - [optional] alluxio configuration

Throws:

java.io.IOException

isZookeeperMode

@Deprecated
protected abstract boolean isZookeeperMode()

Deprecated. in 1.6.0, directly infer the value from PropertyKey.ZOOKEEPER_ENABLED configuration value. Determines if zookeeper should be used for the FileSystem. This method should only be used for initialize(java.net.URI, org.apache.hadoop.conf.Configuration).

Returns:

true if zookeeper should be used

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path path)
                                             throws java.io.IOException

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path path,
                      org.apache.hadoop.fs.permission.FsPermission permission)
               throws java.io.IOException

Attempts to create a folder with the specified path. Parent directories will be created.

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to create

permission - permissions to grant the created folder

Returns:

true if the indicated folder is created successfully or already exists

Throws:

java.io.IOException

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path path,
                                                   int bufferSize)
                                            throws java.io.IOException

Attempts to open the specified file for reading.

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Parameters:

path - the file name to open

bufferSize - stream buffer size in bytes, currently unused

Returns:

an FSDataInputStream at the indicated path of a file

Throws:

java.io.IOException

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws java.io.IOException

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Throws:

java.io.IOException

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path path)

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

getConfigurationFromUri

protected abstract java.util.Map<java.lang.String,java.lang.Object> getConfigurationFromUri(java.net.URI uri,
                                                                                            org.apache.hadoop.conf.Configuration conf)

Gets the connection configuration from the input uri.

Parameters:

uri - an Alluxio Uri that may contain connection configuration

validateFsUri

protected abstract void validateFsUri(java.net.URI fsUri)
                               throws java.io.IOException,
                                      java.lang.IllegalArgumentException

Validates given FS base URI for scheme and authority.

Parameters:

fsUri - FS Uri

Throws:

java.io.IOException

java.lang.IllegalArgumentException

getFsScheme

protected abstract java.lang.String getFsScheme(java.net.URI fsUri)

Used to get FS scheme.

Parameters:

fsUri - file system base URI

Returns:

file system scheme

getAlluxioPath

protected abstract AlluxioURI getAlluxioPath(org.apache.hadoop.fs.Path path)

Used to convert hadoop path to Alluxio path.

Parameters:

path - the input path

Returns:

the Alluxio path

getFsPath

protected abstract org.apache.hadoop.fs.Path getFsPath(java.lang.String fsUriHeader,
                                                       URIStatus fileStatus)

Used to get FS native path from Alluxio file status.

Parameters:

fsUriHeader - FS URI header -> "scheme://authority"

fileStatus - Alluxio file status

Returns:

FS native path