cascading.tap.hadoop

Class HfsProps

java.lang.Object

cascading.property.Props

cascading.tap.hadoop.HfsProps

public class HfsProps
extends cascading.property.Props

Class HfsProps is a fluent helper for setting various Hadoop FS level properties that some Flow may or may not be required to have set. These properties are typically passed to a Flow via a FlowConnector.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	COMBINE_INPUT_FILES Field COMBINE_INPUT_FILES
static java.lang.String	COMBINE_INPUT_FILES_SAFE_MODE Field COMBINE_INPUT_FILES_SAFEMODE
static java.lang.String	COMBINE_INPUT_FILES_SIZE_MAX Field COMBINE_INPUT_FILES_SIZE_MAX
protected java.lang.Long	combinedInputMaxSize
protected java.lang.Boolean	combinedInputSafeMode
static java.lang.String	LOCAL_MODE_SCHEME Fields LOCAL_MODE_SCHEME *
protected java.lang.String	localModeScheme
static java.lang.String	TEMPORARY_DIRECTORY Field TEMPORARY_DIRECTORY
protected java.lang.String	temporaryDirectory
protected java.lang.Boolean	useCombinedInput

Constructor Summary

Constructors

Constructor and Description
HfsProps()

Method Summary

Modifier and Type	Method and Description
protected void	addPropertiesTo(java.util.Properties properties)
java.lang.Long	getCombinedInputMaxSize()
java.lang.String	getLocalModeScheme()
java.lang.String	getTemporaryDirectory()
static HfsProps	hfsProps() Creates a new HfsProps instance.
boolean	isUseCombinedInput()
boolean	isUseCombinedInputSafeMode()
HfsProps	setCombinedInputMaxSize(long combinedInputMaxSize) Method setCombinedInputMaxSize sets the maximum input split size to be used.
static void	setCombinedInputMaxSize(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.Long size) Method setCombinedInputMaxSize sets the maximum input split size to be used.
static void	setLocalModeScheme(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String scheme) Method setLocalModeScheme provides a means to change the scheme value used to detect when a MapReduce job should be run in Hadoop local mode.
HfsProps	setLocalModeScheme(java.lang.String localModeScheme) Method setLocalModeScheme provides a means to change the scheme value used to detect when a MapReduce job should be run in Hadoop local mode.
static void	setTemporaryDirectory(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String temporaryDirectory) Method setTemporaryDirectory sets the temporary directory on the given properties object.
HfsProps	setTemporaryDirectory(java.lang.String temporaryDirectory) Method setTemporaryDirectory sets the temporary directory for use on the underlying filesystem.
HfsProps	setUseCombinedInput(boolean useCombinedInput) Method setUseCombinedInput provides a means to indicate whether to leverage org.apache.hadoop.mapred.lib.CombineFileInputFormat for the input format.
static void	setUseCombinedInput(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.Boolean combine) Method setUseCombinedInput provides a means to indicate whether to leverage org.apache.hadoop.mapred.lib.CombineFileInputFormat for the input format.
HfsProps	setUseCombinedInputSafeMode(boolean combinedInputSafeMode) Method setUseCombinedInputSafeMode toggles safe mode when using org.apache.hadoop.mapred.lib.CombineFileInputFormat.
static void	setUseCombinedInputSafeMode(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.Boolean safeMode) Method setUseCombinedInputSafeMode toggles safe mode when using org.apache.hadoop.mapred.lib.CombineFileInputFormat.

Methods inherited from class cascading.property.Props

buildProperties, buildProperties, buildProperties, buildProperties, setProperties, setProperties

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEMPORARY_DIRECTORY

public static final java.lang.String TEMPORARY_DIRECTORY

Field TEMPORARY_DIRECTORY

See Also:

Constant Field Values

LOCAL_MODE_SCHEME

public static final java.lang.String LOCAL_MODE_SCHEME

Fields LOCAL_MODE_SCHEME *

See Also:

Constant Field Values

COMBINE_INPUT_FILES

public static final java.lang.String COMBINE_INPUT_FILES

Field COMBINE_INPUT_FILES

See Also:

Constant Field Values

COMBINE_INPUT_FILES_SAFE_MODE

public static final java.lang.String COMBINE_INPUT_FILES_SAFE_MODE

Field COMBINE_INPUT_FILES_SAFEMODE

See Also:

Constant Field Values

COMBINE_INPUT_FILES_SIZE_MAX

public static final java.lang.String COMBINE_INPUT_FILES_SIZE_MAX

Field COMBINE_INPUT_FILES_SIZE_MAX

See Also:

Constant Field Values

temporaryDirectory

protected java.lang.String temporaryDirectory

localModeScheme

protected java.lang.String localModeScheme

useCombinedInput

protected java.lang.Boolean useCombinedInput

combinedInputMaxSize

protected java.lang.Long combinedInputMaxSize

combinedInputSafeMode

protected java.lang.Boolean combinedInputSafeMode

Constructor Detail

HfsProps

public HfsProps()

Method Detail

setTemporaryDirectory

public static void setTemporaryDirectory(java.util.Map<java.lang.Object,java.lang.Object> properties,
                                         java.lang.String temporaryDirectory)

Method setTemporaryDirectory sets the temporary directory on the given properties object.

Parameters:

properties - of type Map

temporaryDirectory - of type String

setLocalModeScheme

public static void setLocalModeScheme(java.util.Map<java.lang.Object,java.lang.Object> properties,
                                      java.lang.String scheme)

Method setLocalModeScheme provides a means to change the scheme value used to detect when a MapReduce job should be run in Hadoop local mode. By default the value is "file", set to "none" to disable entirely.

Parameters:

properties - of type Map

scheme - a String

setUseCombinedInput

public static void setUseCombinedInput(java.util.Map<java.lang.Object,java.lang.Object> properties,
                                       java.lang.Boolean combine)

Method setUseCombinedInput provides a means to indicate whether to leverage org.apache.hadoop.mapred.lib.CombineFileInputFormat for the input format. By default it is false.

Use setCombinedInputMaxSize(long) to set the max split/combined input size. Other specific properties must be specified directly if needed. Specifically "mapred.min.split.size.per.node" and "mapred.min.split.size.per.rack", which are 0 by default.

Parameters:

properties - of type Map

combine - a boolean

setUseCombinedInputSafeMode

public static void setUseCombinedInputSafeMode(java.util.Map<java.lang.Object,java.lang.Object> properties,
                                               java.lang.Boolean safeMode)

Method setUseCombinedInputSafeMode toggles safe mode when using org.apache.hadoop.mapred.lib.CombineFileInputFormat. Safe mode will throw an exception if the underlying InputFormat is not of type org.apache.hadoop.mapred.FileInputFormat. If safeMode is off a warning will be logged instead. safeMode is on by default.

Setting this property when not setting setUseCombinedInput(boolean) to true has no effect.

Parameters:

properties - of type Map

safeMode - a boolean

setCombinedInputMaxSize

public static void setCombinedInputMaxSize(java.util.Map<java.lang.Object,java.lang.Object> properties,
                                           java.lang.Long size)

Method setCombinedInputMaxSize sets the maximum input split size to be used.

This property is an alias for the Hadoop property "mapred.max.split.size".

Parameters:

properties - of type Map

size - of type long

hfsProps

public static HfsProps hfsProps()

Creates a new HfsProps instance.

Returns:

HfsProps instance

getTemporaryDirectory

public java.lang.String getTemporaryDirectory()

setTemporaryDirectory

public HfsProps setTemporaryDirectory(java.lang.String temporaryDirectory)

Method setTemporaryDirectory sets the temporary directory for use on the underlying filesystem.

Parameters:

temporaryDirectory - of type String

Returns:

returns this instance

getLocalModeScheme

public java.lang.String getLocalModeScheme()

setLocalModeScheme

public HfsProps setLocalModeScheme(java.lang.String localModeScheme)

Method setLocalModeScheme provides a means to change the scheme value used to detect when a MapReduce job should be run in Hadoop local mode. By default the value is "file", set to "none" to disable entirely.

Parameters:

localModeScheme - of type String

Returns:

returns this instance

isUseCombinedInput

public boolean isUseCombinedInput()

setUseCombinedInput

public HfsProps setUseCombinedInput(boolean useCombinedInput)

Method setUseCombinedInput provides a means to indicate whether to leverage org.apache.hadoop.mapred.lib.CombineFileInputFormat for the input format. By default it is false.

Parameters:

useCombinedInput - boolean

Returns:

returns this instance

getCombinedInputMaxSize

public java.lang.Long getCombinedInputMaxSize()

setCombinedInputMaxSize

public HfsProps setCombinedInputMaxSize(long combinedInputMaxSize)

Method setCombinedInputMaxSize sets the maximum input split size to be used.

This value is not honored unless setUseCombinedInput(boolean) is true.

Parameters:

combinedInputMaxSize - of type long

Returns:

returns this instance

isUseCombinedInputSafeMode

public boolean isUseCombinedInputSafeMode()

setUseCombinedInputSafeMode

public HfsProps setUseCombinedInputSafeMode(boolean combinedInputSafeMode)

Method setUseCombinedInputSafeMode toggles safe mode when using org.apache.hadoop.mapred.lib.CombineFileInputFormat. Safe mode will throw an exception if the underlying InputFormat is not of type org.apache.hadoop.mapred.FileInputFormat. If safeMode is off a warning will be logged instead. safeMode is on by default.

Setting this property when not setting setUseCombinedInput(boolean) to true has no effect.

Parameters:

combinedInputSafeMode - boolean

Returns:

returns this instance

addPropertiesTo

protected void addPropertiesTo(java.util.Properties properties)

Specified by:

addPropertiesTo in class cascading.property.Props