org.apache.hadoop.hive.shims

Class Hadoop23Shims

java.lang.Object

org.apache.hadoop.hive.shims.HadoopShimsSecure

org.apache.hadoop.hive.shims.Hadoop23Shims

All Implemented Interfaces:

HadoopShims

public class Hadoop23Shims
extends HadoopShimsSecure

Implemention of shims against Hadoop 0.23.0.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	Hadoop23Shims.Hadoop23FileStatus
class	Hadoop23Shims.HdfsEncryptionShim
class	Hadoop23Shims.KerberosNameShim Shim for KerberosName
class	Hadoop23Shims.MiniDFSShim MiniDFSShim.
class	Hadoop23Shims.MiniMrShim Shim for MiniMrCluster
class	Hadoop23Shims.MiniSparkShim Shim for MiniSparkOnYARNCluster
class	Hadoop23Shims.MiniTezShim Shim for MiniTezCluster
static class	Hadoop23Shims.StoragePolicyShim

Nested classes/interfaces inherited from class org.apache.hadoop.hive.shims.HadoopShimsSecure

HadoopShimsSecure.CombineFileInputFormatShim<K,V>, HadoopShimsSecure.CombineFileRecordReader<K,V>, HadoopShimsSecure.InputSplitShim

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.shims.HadoopShims

HadoopShims.ByteBufferPoolShim, HadoopShims.DirectCompressionType, HadoopShims.DirectDecompressorShim, HadoopShims.HCatHadoopShims, HadoopShims.HdfsFileStatus, HadoopShims.JobTrackerState, HadoopShims.NoopHdfsEncryptionShim, HadoopShims.StoragePolicyValue, HadoopShims.WebHCatJTShim, HadoopShims.ZeroCopyReaderShim

Field Summary

Fields

Modifier and Type	Field and Description
protected static Method	accessMethod
protected static Method	getPasswordMethod

Constructor Summary

Constructors

Constructor and Description
Hadoop23Shims()

Method Summary

Methods

Modifier and Type	Method and Description
void	addDelegationTokens(FileSystem fs, org.apache.hadoop.security.Credentials cred, String uname)
void	checkFileAccess(FileSystem fs, FileStatus stat, org.apache.hadoop.fs.permission.FsAction action)
HadoopShims.HdfsEncryptionShim	createHdfsEncryptionShim(FileSystem fs, org.apache.hadoop.conf.Configuration conf)
FileSystem	createProxyFileSystem(FileSystem fs, URI uri)
HadoopShims.CombineFileInputFormatShim	getCombineFileInputFormat()
org.apache.hadoop.conf.Configuration	getConfiguration(org.apache.hadoop.mapreduce.JobContext context)
Path	getCurrentTrashPath(org.apache.hadoop.conf.Configuration conf, FileSystem fs)
long	getDefaultBlockSize(FileSystem fs, Path path)
short	getDefaultReplication(FileSystem fs, Path path)
HadoopShims.DirectDecompressorShim	getDirectDecompressor(HadoopShims.DirectCompressionType codec)
HadoopShims.HdfsFileStatus	getFullFileStatus(org.apache.hadoop.conf.Configuration conf, FileSystem fs, Path file)
Map<String,String>	getHadoopConfNames()
HadoopShims.HCatHadoopShims	getHCatShim()
org.apache.hadoop.mapred.JobConf	getJobConf(org.apache.hadoop.mapred.JobContext context)
String	getJobLauncherHttpAddress(org.apache.hadoop.conf.Configuration conf)
String	getJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf)
HadoopShims.JobTrackerState	getJobTrackerState(org.apache.hadoop.mapred.ClusterStatus clusterStatus)
Hadoop23Shims.KerberosNameShim	getKerberosNameShim(String name) Returns a shim to wrap KerberosName
BlockLocation[]	getLocations(FileSystem fs, FileStatus status)
TreeMap<Long,BlockLocation>	getLocationsWithOffset(FileSystem fs, FileStatus status)
Comparator<org.apache.hadoop.io.LongWritable>	getLongComparator()
void	getMergedCredentials(org.apache.hadoop.mapred.JobConf jobConf)
HadoopShims.MiniDFSShim	getMiniDfs(org.apache.hadoop.conf.Configuration conf, int numDataNodes, boolean format, String[] racks)
Hadoop23Shims.MiniMrShim	getMiniMrCluster(org.apache.hadoop.conf.Configuration conf, int numberOfTaskTrackers, String nameNode, int numDir) Returns a shim to wrap MiniMrCluster
Hadoop23Shims.MiniMrShim	getMiniSparkCluster(org.apache.hadoop.conf.Configuration conf, int numberOfTaskTrackers, String nameNode, int numDir) Returns a shim to wrap MiniSparkOnYARNCluster
Hadoop23Shims.MiniMrShim	getMiniTezCluster(org.apache.hadoop.conf.Configuration conf, int numberOfTaskTrackers, String nameNode, int numDir) Returns a shim to wrap MiniMrTez
FileSystem	getNonCachedFileSystem(URI uri, org.apache.hadoop.conf.Configuration conf)
String	getPassword(org.apache.hadoop.conf.Configuration conf, String name)
Path	getPathWithoutSchemeAndAuthority(Path path)
HadoopShims.StoragePolicyShim	getStoragePolicyShim(FileSystem fs)
String	getTaskAttemptLogUrl(org.apache.hadoop.mapred.JobConf conf, String taskTrackerHttpAddress, String taskAttemptId)
HadoopShims.WebHCatJTShim	getWebHCatShim(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.security.UserGroupInformation ugi)
HadoopShims.ZeroCopyReaderShim	getZeroCopyReader(FSDataInputStream in, HadoopShims.ByteBufferPoolShim pool)
boolean	hasStickyBit(org.apache.hadoop.fs.permission.FsPermission permission)
void	hflush(FSDataOutputStream stream)
boolean	isDirectory(FileStatus fileStatus)
protected boolean	isExtendedAclEnabled(org.apache.hadoop.conf.Configuration conf)
static boolean	isHdfsEncryptionSupported()
boolean	isLocalMode(org.apache.hadoop.conf.Configuration conf)
List<FileStatus>	listLocatedStatus(FileSystem fs, Path path, PathFilter filter)
void	mergeCredentials(org.apache.hadoop.mapred.JobConf dest, org.apache.hadoop.mapred.JobConf src)
boolean	moveToAppropriateTrash(FileSystem fs, Path path, org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.mapreduce.JobContext	newJobContext(org.apache.hadoop.mapreduce.Job job)
org.apache.hadoop.mapreduce.TaskAttemptContext	newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.util.Progressable progressable)
org.apache.hadoop.mapreduce.TaskAttemptID	newTaskAttemptID(org.apache.hadoop.mapreduce.JobID jobId, boolean isMap, int taskId, int id)
int	readByteBuffer(FSDataInputStream file, ByteBuffer dest)
void	refreshDefaultQueue(org.apache.hadoop.conf.Configuration conf, String userName) Load the fair scheduler queue for given user if available.
boolean	runDistCp(Path src, Path dst, org.apache.hadoop.conf.Configuration conf)
void	setFullFileStatus(org.apache.hadoop.conf.Configuration conf, HadoopShims.HdfsFileStatus sourceStatus, FileSystem fs, Path target)
void	setJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf, String val)
void	setTotalOrderPartitionFile(org.apache.hadoop.mapred.JobConf jobConf, Path partitionFile)
void	startPauseMonitor(org.apache.hadoop.conf.Configuration conf)
boolean	supportStickyBit()
boolean	supportTrashFeature()

Methods inherited from class org.apache.hadoop.hive.shims.HadoopShimsSecure

run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

accessMethod

protected static final Method accessMethod

getPasswordMethod

protected static final Method getPasswordMethod

Constructor Detail

Hadoop23Shims

public Hadoop23Shims()

Method Detail

getCombineFileInputFormat

public HadoopShims.CombineFileInputFormatShim getCombineFileInputFormat()

getTaskAttemptLogUrl

public String getTaskAttemptLogUrl(org.apache.hadoop.mapred.JobConf conf,
                          String taskTrackerHttpAddress,
                          String taskAttemptId)
                            throws MalformedURLException

Throws:

MalformedURLException

getJobTrackerState

public HadoopShims.JobTrackerState getJobTrackerState(org.apache.hadoop.mapred.ClusterStatus clusterStatus)
                                               throws Exception

Specified by:

getJobTrackerState in interface HadoopShims

Specified by:

getJobTrackerState in class HadoopShimsSecure

Throws:

Exception

newTaskAttemptContext

public org.apache.hadoop.mapreduce.TaskAttemptContext newTaskAttemptContext(org.apache.hadoop.conf.Configuration conf,
                                                                   org.apache.hadoop.util.Progressable progressable)

Specified by:

newTaskAttemptContext in interface HadoopShims

Specified by:

newTaskAttemptContext in class HadoopShimsSecure

newTaskAttemptID

public org.apache.hadoop.mapreduce.TaskAttemptID newTaskAttemptID(org.apache.hadoop.mapreduce.JobID jobId,
                                                         boolean isMap,
                                                         int taskId,
                                                         int id)

newJobContext

public org.apache.hadoop.mapreduce.JobContext newJobContext(org.apache.hadoop.mapreduce.Job job)

Specified by:

newJobContext in interface HadoopShims

Specified by:

newJobContext in class HadoopShimsSecure

startPauseMonitor

public void startPauseMonitor(org.apache.hadoop.conf.Configuration conf)

isLocalMode

public boolean isLocalMode(org.apache.hadoop.conf.Configuration conf)

Specified by:

isLocalMode in interface HadoopShims

Specified by:

isLocalMode in class HadoopShimsSecure

getJobLauncherRpcAddress

public String getJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf)

Specified by:

getJobLauncherRpcAddress in interface HadoopShims

Specified by:

getJobLauncherRpcAddress in class HadoopShimsSecure

setJobLauncherRpcAddress

public void setJobLauncherRpcAddress(org.apache.hadoop.conf.Configuration conf,
                            String val)

Specified by:

setJobLauncherRpcAddress in interface HadoopShims

Specified by:

setJobLauncherRpcAddress in class HadoopShimsSecure

getJobLauncherHttpAddress

public String getJobLauncherHttpAddress(org.apache.hadoop.conf.Configuration conf)

Specified by:

getJobLauncherHttpAddress in interface HadoopShims

Specified by:

getJobLauncherHttpAddress in class HadoopShimsSecure

isExtendedAclEnabled

protected boolean isExtendedAclEnabled(org.apache.hadoop.conf.Configuration conf)

getDefaultBlockSize

public long getDefaultBlockSize(FileSystem fs,
                       Path path)

Specified by:

getDefaultBlockSize in interface HadoopShims

Specified by:

getDefaultBlockSize in class HadoopShimsSecure

getDefaultReplication

public short getDefaultReplication(FileSystem fs,
                          Path path)

Specified by:

getDefaultReplication in interface HadoopShims

Specified by:

getDefaultReplication in class HadoopShimsSecure

moveToAppropriateTrash

public boolean moveToAppropriateTrash(FileSystem fs,
                             Path path,
                             org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Specified by:

moveToAppropriateTrash in interface HadoopShims

Specified by:

moveToAppropriateTrash in class HadoopShimsSecure

Throws:

IOException

setTotalOrderPartitionFile

public void setTotalOrderPartitionFile(org.apache.hadoop.mapred.JobConf jobConf,
                              Path partitionFile)

getLongComparator

public Comparator<org.apache.hadoop.io.LongWritable> getLongComparator()

refreshDefaultQueue

public void refreshDefaultQueue(org.apache.hadoop.conf.Configuration conf,
                       String userName)
                         throws IOException

Load the fair scheduler queue for given user if available.

Throws:

IOException

getMiniMrCluster

public Hadoop23Shims.MiniMrShim getMiniMrCluster(org.apache.hadoop.conf.Configuration conf,
                                        int numberOfTaskTrackers,
                                        String nameNode,
                                        int numDir)
                                          throws IOException

Returns a shim to wrap MiniMrCluster

Throws:

IOException

getMiniTezCluster

public Hadoop23Shims.MiniMrShim getMiniTezCluster(org.apache.hadoop.conf.Configuration conf,
                                         int numberOfTaskTrackers,
                                         String nameNode,
                                         int numDir)
                                           throws IOException

Returns a shim to wrap MiniMrTez

Throws:

IOException

getMiniSparkCluster

public Hadoop23Shims.MiniMrShim getMiniSparkCluster(org.apache.hadoop.conf.Configuration conf,
                                           int numberOfTaskTrackers,
                                           String nameNode,
                                           int numDir)
                                             throws IOException

Returns a shim to wrap MiniSparkOnYARNCluster

Throws:

IOException

getMiniDfs

public HadoopShims.MiniDFSShim getMiniDfs(org.apache.hadoop.conf.Configuration conf,
                                 int numDataNodes,
                                 boolean format,
                                 String[] racks)
                                   throws IOException

Throws:

IOException

getHCatShim

public HadoopShims.HCatHadoopShims getHCatShim()

getWebHCatShim

public HadoopShims.WebHCatJTShim getWebHCatShim(org.apache.hadoop.conf.Configuration conf,
                                       org.apache.hadoop.security.UserGroupInformation ugi)
                                         throws IOException

Throws:

IOException

listLocatedStatus

public List<FileStatus> listLocatedStatus(FileSystem fs,
                                 Path path,
                                 PathFilter filter)
                                   throws IOException

Throws:

IOException

getLocations

public BlockLocation[] getLocations(FileSystem fs,
                           FileStatus status)
                             throws IOException

Throws:

IOException

getLocationsWithOffset

public TreeMap<Long,BlockLocation> getLocationsWithOffset(FileSystem fs,
                                                 FileStatus status)
                                                   throws IOException

Throws:

IOException

hflush

public void hflush(FSDataOutputStream stream)
            throws IOException

Throws:

IOException

getFullFileStatus

public HadoopShims.HdfsFileStatus getFullFileStatus(org.apache.hadoop.conf.Configuration conf,
                                           FileSystem fs,
                                           Path file)
                                             throws IOException

Throws:

IOException

setFullFileStatus

public void setFullFileStatus(org.apache.hadoop.conf.Configuration conf,
                     HadoopShims.HdfsFileStatus sourceStatus,
                     FileSystem fs,
                     Path target)
                       throws IOException

Throws:

IOException

createProxyFileSystem

public FileSystem createProxyFileSystem(FileSystem fs,
                               URI uri)

Specified by:

createProxyFileSystem in interface HadoopShims

Specified by:

createProxyFileSystem in class HadoopShimsSecure

getHadoopConfNames

public Map<String,String> getHadoopConfNames()

getZeroCopyReader

public HadoopShims.ZeroCopyReaderShim getZeroCopyReader(FSDataInputStream in,
                                               HadoopShims.ByteBufferPoolShim pool)
                                                 throws IOException

Throws:

IOException

getDirectDecompressor

public HadoopShims.DirectDecompressorShim getDirectDecompressor(HadoopShims.DirectCompressionType codec)

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration(org.apache.hadoop.mapreduce.JobContext context)

getJobConf

public org.apache.hadoop.mapred.JobConf getJobConf(org.apache.hadoop.mapred.JobContext context)

getNonCachedFileSystem

public FileSystem getNonCachedFileSystem(URI uri,
                                org.apache.hadoop.conf.Configuration conf)
                                  throws IOException

Specified by:

getNonCachedFileSystem in interface HadoopShims

Specified by:

getNonCachedFileSystem in class HadoopShimsSecure

Throws:

IOException

getMergedCredentials

public void getMergedCredentials(org.apache.hadoop.mapred.JobConf jobConf)
                          throws IOException

Throws:

IOException

mergeCredentials

public void mergeCredentials(org.apache.hadoop.mapred.JobConf dest,
                    org.apache.hadoop.mapred.JobConf src)
                      throws IOException

Throws:

IOException

checkFileAccess

public void checkFileAccess(FileSystem fs,
                   FileStatus stat,
                   org.apache.hadoop.fs.permission.FsAction action)
                     throws IOException,
                            AccessControlException,
                            Exception

Specified by:

checkFileAccess in interface HadoopShims

Overrides:

checkFileAccess in class HadoopShimsSecure

Throws:

IOException

AccessControlException

Exception

getPassword

public String getPassword(org.apache.hadoop.conf.Configuration conf,
                 String name)
                   throws IOException

Throws:

IOException

supportStickyBit

public boolean supportStickyBit()

hasStickyBit

public boolean hasStickyBit(org.apache.hadoop.fs.permission.FsPermission permission)

supportTrashFeature

public boolean supportTrashFeature()

getCurrentTrashPath

public Path getCurrentTrashPath(org.apache.hadoop.conf.Configuration conf,
                       FileSystem fs)

getKerberosNameShim

public Hadoop23Shims.KerberosNameShim getKerberosNameShim(String name)
                                                   throws IOException

Returns a shim to wrap KerberosName

Throws:

IOException

isDirectory

public boolean isDirectory(FileStatus fileStatus)

getStoragePolicyShim

public HadoopShims.StoragePolicyShim getStoragePolicyShim(FileSystem fs)

runDistCp

public boolean runDistCp(Path src,
                Path dst,
                org.apache.hadoop.conf.Configuration conf)
                  throws IOException

Throws:

IOException

isHdfsEncryptionSupported

public static boolean isHdfsEncryptionSupported()

createHdfsEncryptionShim

public HadoopShims.HdfsEncryptionShim createHdfsEncryptionShim(FileSystem fs,
                                                      org.apache.hadoop.conf.Configuration conf)
                                                        throws IOException

Throws:

IOException

getPathWithoutSchemeAndAuthority

public Path getPathWithoutSchemeAndAuthority(Path path)

readByteBuffer

public int readByteBuffer(FSDataInputStream file,
                 ByteBuffer dest)
                   throws IOException

Throws:

IOException

addDelegationTokens

public void addDelegationTokens(FileSystem fs,
                       org.apache.hadoop.security.Credentials cred,
                       String uname)
                         throws IOException

Specified by:

addDelegationTokens in interface HadoopShims

Specified by:

addDelegationTokens in class HadoopShimsSecure

Throws:

IOException