io.dstream.tez.utils

Class HadoopUtils

java.lang.Object

io.dstream.tez.utils.HadoopUtils

public class HadoopUtils
extends java.lang.Object

Utility functions related to variety of tasks to be performed in HADOOP such as setting up LocalResource, provisioning classpath etc.

Constructor Summary

Constructors

Constructor and Description
HadoopUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.yarn.api.records.LocalResource	createLocalResource(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path provisionedResourcePath) Creates a single LocalResource for the provisioned resource identified with Path
static java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource>	createLocalResources(org.apache.hadoop.fs.FileSystem fs, java.lang.String classPathDir) Creates LocalResources based on the current user's classpath
static org.apache.hadoop.fs.FileSystem	getFileSystem(org.apache.hadoop.conf.Configuration configuration)
static org.apache.hadoop.fs.Path	provisionResourceToFs(java.io.File localResource, org.apache.hadoop.fs.FileSystem fs, java.lang.String applicationName) Provisions resource represented as File to the FileSystem for a given application

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopUtils

public HadoopUtils()

Method Detail

getFileSystem

public static org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration configuration)

Parameters:

configuration -

Returns:

createLocalResources

public static java.util.Map<java.lang.String,org.apache.hadoop.yarn.api.records.LocalResource> createLocalResources(org.apache.hadoop.fs.FileSystem fs,
                                                                                                                    java.lang.String classPathDir)

Creates LocalResources based on the current user's classpath

Parameters:

fs -

classPathDir -

Returns:

provisionResourceToFs

public static org.apache.hadoop.fs.Path provisionResourceToFs(java.io.File localResource,
                                                              org.apache.hadoop.fs.FileSystem fs,
                                                              java.lang.String applicationName)
                                                       throws java.lang.Exception

Provisions resource represented as File to the FileSystem for a given application

Parameters:

localResource -

fs -

applicationName -

Returns:

Throws:

java.lang.Exception

createLocalResource

public static org.apache.hadoop.yarn.api.records.LocalResource createLocalResource(org.apache.hadoop.fs.FileSystem fs,
                                                                                   org.apache.hadoop.fs.Path provisionedResourcePath)

Creates a single LocalResource for the provisioned resource identified with Path

Parameters:

fs -

provisionedResourcePath -

Returns: