org.apache.hadoop.hive.ql.txn.compactor
Class Worker

java.lang.Object
  java.lang.Thread
      org.apache.hadoop.hive.ql.txn.compactor.Worker

All Implemented Interfaces:

Runnable, MetaStoreThread

public class Worker
extends Thread

A class to do compactions. This will run in a separate thread. It will spin on the compaction queue and look for new work to do.

Nested Class Summary

Nested classes/interfaces inherited from class java.lang.Thread

Thread.State, Thread.UncaughtExceptionHandler

Nested classes/interfaces inherited from interface org.apache.hadoop.hive.metastore.MetaStoreThread

MetaStoreThread.BooleanPointer

Field Summary

protected HiveConf	conf
protected RawStore	rs
protected MetaStoreThread.BooleanPointer	stop
protected int	threadId
protected CompactionTxnHandler	txnHandler

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

Worker()

Method Summary

protected String	findUserToRunAs(String location, Table t) Determine which user to run an operation as, based on the owner of the directory to be compacted.
static String	hostname() Get the hostname that this worker is run on.
void	init(MetaStoreThread.BooleanPointer stop)
protected Partition	resolvePartition(CompactionInfo ci) Get the partition being compacted.
protected StorageDescriptor	resolveStorageDescriptor(Table t, Partition p) Get the storage descriptor for a compaction.
protected Table	resolveTable(CompactionInfo ci) Find the table being compacted
void	run()
protected boolean	runJobAsSelf(String owner) Determine whether to run this job as the current user or whether we need a doAs to switch users.
void	setHiveConf(HiveConf conf)
void	setThreadId(int threadId)
protected String	tableName(Table t)

Methods inherited from class java.lang.Thread

activeCount, checkAccess, clone, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hive.metastore.MetaStoreThread

start

Field Detail

conf

protected HiveConf conf

txnHandler

protected CompactionTxnHandler txnHandler

rs

protected RawStore rs

threadId

protected int threadId

stop

protected MetaStoreThread.BooleanPointer stop

Constructor Detail

Worker

public Worker()

Method Detail

hostname

public static String hostname()

Get the hostname that this worker is run on. Made static and public so that other classes can use the same method to know what host their worker threads are running on.

Returns:

hostname

run

public void run()

Specified by:

run in interface Runnable

Overrides:

run in class Thread

init

public void init(MetaStoreThread.BooleanPointer stop)
          throws MetaException

Specified by:

init in interface MetaStoreThread

Throws:

MetaException

setHiveConf

public void setHiveConf(HiveConf conf)

Specified by:

setHiveConf in interface MetaStoreThread

setThreadId

public void setThreadId(int threadId)

Specified by:

setThreadId in interface MetaStoreThread

resolveTable

protected Table resolveTable(CompactionInfo ci)
                      throws MetaException

Find the table being compacted

Parameters:

ci - compaction info returned from the compaction queue

Returns:

metastore table

Throws:

MetaException - if the table cannot be found.

resolvePartition

protected Partition resolvePartition(CompactionInfo ci)
                              throws Exception

Get the partition being compacted.

Parameters:

ci - compaction info returned from the compaction queue

Returns:

metastore partition, or null if there is not partition in this compaction info

Throws:

Exception - if underlying calls throw, or if the partition name resolves to more than one partition.

resolveStorageDescriptor

protected StorageDescriptor resolveStorageDescriptor(Table t,
                                                     Partition p)

Get the storage descriptor for a compaction.

Parameters:

t - table from resolveTable(org.apache.hadoop.hive.metastore.txn.CompactionInfo)

p - table from resolvePartition(org.apache.hadoop.hive.metastore.txn.CompactionInfo)

Returns:

metastore storage descriptor.

findUserToRunAs

protected String findUserToRunAs(String location,
                                 Table t)
                          throws IOException,
                                 InterruptedException

Determine which user to run an operation as, based on the owner of the directory to be compacted. It is asserted that either the user running the hive metastore or the table owner must be able to stat the directory and determine the owner.

Parameters:

location - directory that will be read or written to.

t - metastore table object

Returns:

username of the owner of the location.

Throws:

IOException - if neither the hive metastore user nor the table owner can stat the location.

InterruptedException

runJobAsSelf

protected boolean runJobAsSelf(String owner)

Determine whether to run this job as the current user or whether we need a doAs to switch users.

Parameters:

owner - of the directory we will be working in, as determined by findUserToRunAs(String, org.apache.hadoop.hive.metastore.api.Table)

Returns:

true if the job should run as the current user, false if a doAs is needed.

tableName

protected String tableName(Table t)