cascading.tap.hadoop.util

Class Hadoop18TapUtil

java.lang.Object

cascading.tap.hadoop.util.Hadoop18TapUtil

public class Hadoop18TapUtil
extends java.lang.Object

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	TEMPORARY_PATH The Hadoop temporary path used to prevent collisions

Constructor Summary

Constructors

Constructor and Description
Hadoop18TapUtil()

Method Summary

Modifier and Type	Method and Description
static void	cleanupJob(Configuration conf) May only be called once.
static void	cleanupTapMetaData(Configuration conf, cascading.tap.Tap tap) Called from flow step to remove temp dirs
static boolean	commitTask(Configuration conf) copies all files from the taskoutputpath to the outputpath
protected static boolean	isOutputWriteDirect(Configuration conf, FileSystem fs)
static void	makeTempPath(Configuration conf)
static boolean	needsTaskCommit(Configuration conf)
static void	setupJob(Configuration conf) should only be called if not in a Flow
static void	setupTask(Configuration conf)
static void	writeSuccessMarker(Configuration conf)
static void	writeSuccessMarker(Configuration conf, Path outputPath)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEMPORARY_PATH

public static final java.lang.String TEMPORARY_PATH

The Hadoop temporary path used to prevent collisions

See Also:

Constant Field Values

Constructor Detail

Hadoop18TapUtil

public Hadoop18TapUtil()

Method Detail

setupJob

public static void setupJob(Configuration conf)
                     throws java.io.IOException

should only be called if not in a Flow

Parameters:

conf -

Throws:

java.io.IOException

setupTask

public static void setupTask(Configuration conf)
                      throws java.io.IOException

Throws:

java.io.IOException

needsTaskCommit

public static boolean needsTaskCommit(Configuration conf)
                               throws java.io.IOException

Throws:

java.io.IOException

commitTask

public static boolean commitTask(Configuration conf)
                          throws java.io.IOException

copies all files from the taskoutputpath to the outputpath

Parameters:

conf -

Throws:

java.io.IOException

cleanupTapMetaData

public static void cleanupTapMetaData(Configuration conf,
                                      cascading.tap.Tap tap)
                               throws java.io.IOException

Called from flow step to remove temp dirs

Parameters:

conf -

Throws:

java.io.IOException

writeSuccessMarker

public static void writeSuccessMarker(Configuration conf)
                               throws java.io.IOException

Throws:

java.io.IOException

writeSuccessMarker

public static void writeSuccessMarker(Configuration conf,
                                      Path outputPath)
                               throws java.io.IOException

Throws:

java.io.IOException

cleanupJob

public static void cleanupJob(Configuration conf)
                       throws java.io.IOException

May only be called once. should only be called if not in a flow

Parameters:

conf -

Throws:

java.io.IOException

makeTempPath

public static void makeTempPath(Configuration conf)
                         throws java.io.IOException

Throws:

java.io.IOException

isOutputWriteDirect

protected static boolean isOutputWriteDirect(Configuration conf,
                                             FileSystem fs)