eu.stratosphere.core.testing.io

Class BinaryInputFormat<T>

java.lang.Object

eu.stratosphere.api.common.io.FileInputFormat<T>

eu.stratosphere.core.testing.io.BinaryInputFormat<T>

All Implemented Interfaces:

eu.stratosphere.api.common.io.InputFormat<T,eu.stratosphere.core.fs.FileInputSplit>, Serializable

Direct Known Subclasses:

SequentialInputFormat

public abstract class BinaryInputFormat<T>
extends eu.stratosphere.api.common.io.FileInputFormat<T>

Base class for all input formats that use blocks of fixed size. The input splits are aligned to these blocks. Without configuration, these block sizes equal the native block sizes of the HDFS.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	BinaryInputFormat.AbstractConfigBuilder<T> Abstract builder used to set parameters to the input format's configuration in a fluent way.
static class	BinaryInputFormat.ConfigBuilder A builder used to set parameters to the input format's configuration in a fluent way.

Nested classes/interfaces inherited from class eu.stratosphere.api.common.io.FileInputFormat

eu.stratosphere.api.common.io.FileInputFormat.FileBaseStatistics, eu.stratosphere.api.common.io.FileInputFormat.InputSplitOpenThread

Field Summary

Fields

Modifier and Type	Field and Description
static long	NATIVE_BLOCK_SIZE Use block size from FileSystem.

Fields inherited from class eu.stratosphere.api.common.io.FileInputFormat

filePath, minSplitSize, numSplits, openTimeout, splitLength, splitStart, stream

Constructor Summary

Constructors

Constructor and Description
BinaryInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(eu.stratosphere.configuration.Configuration parameters)
static BinaryInputFormat.ConfigBuilder	configureBinaryFormat(eu.stratosphere.configuration.Configuration config) Creates a configuration builder that can be used to set the input format's parameters to the config in a fluent fashion.
static BinaryInputFormat.ConfigBuilder	configureBinaryFormat(eu.stratosphere.api.common.operators.GenericDataSource<?> target) Creates a configuration builder that can be used to set the input format's parameters to the config in a fluent fashion.
protected BlockInfo	createBlockInfo() Creates the block info.
eu.stratosphere.core.fs.FileInputSplit[]	createInputSplits(int minNumSplits)
protected eu.stratosphere.core.testing.io.BinaryInputFormat.SequentialStatistics	createStatistics(List<eu.stratosphere.core.fs.FileStatus> files, eu.stratosphere.api.common.io.FileInputFormat.FileBaseStatistics stats) Fill in the statistics.
protected abstract void	deserialize(T record, eu.stratosphere.core.memory.DataInputView dataInput) Deserializes the given record from the DataInputView.
long	getBlockSize() Returns the blockSize.
eu.stratosphere.core.testing.io.BinaryInputFormat.SequentialStatistics	getStatistics(eu.stratosphere.api.common.io.statistics.BaseStatistics cachedStats)
boolean	nextRecord(T record)
void	open(eu.stratosphere.core.fs.FileInputSplit split)
boolean	reachedEnd()
void	setBlockSize(long blockSize) Sets the blockSize to the specified value.

Methods inherited from class eu.stratosphere.api.common.io.FileInputFormat

close, configureFileFormat, getFilePath, getFileStats, getInputSplitType, getMinSplitSize, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, setFilePath, setFilePath, setMinSplitSize, setNumSplits, setOpenTimeout, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

NATIVE_BLOCK_SIZE

public static final long NATIVE_BLOCK_SIZE

Use block size from FileSystem.

See Also:

Constant Field Values

Constructor Detail

BinaryInputFormat

public BinaryInputFormat()

Method Detail

configure

public void configure(eu.stratosphere.configuration.Configuration parameters)

Specified by:

configure in interface eu.stratosphere.api.common.io.InputFormat<T,eu.stratosphere.core.fs.FileInputSplit>

Overrides:

configure in class eu.stratosphere.api.common.io.FileInputFormat<T>

createInputSplits

public eu.stratosphere.core.fs.FileInputSplit[] createInputSplits(int minNumSplits)
                                                           throws IOException

Specified by:

createInputSplits in interface eu.stratosphere.api.common.io.InputFormat<T,eu.stratosphere.core.fs.FileInputSplit>

Overrides:

createInputSplits in class eu.stratosphere.api.common.io.FileInputFormat<T>

Throws:

IOException

getBlockSize

public long getBlockSize()

Returns the blockSize.

Returns:

the blockSize

getStatistics

public eu.stratosphere.core.testing.io.BinaryInputFormat.SequentialStatistics getStatistics(eu.stratosphere.api.common.io.statistics.BaseStatistics cachedStats)

Specified by:

getStatistics in interface eu.stratosphere.api.common.io.InputFormat<T,eu.stratosphere.core.fs.FileInputSplit>

Overrides:

getStatistics in class eu.stratosphere.api.common.io.FileInputFormat<T>

nextRecord

public boolean nextRecord(T record)
                   throws IOException

Throws:

IOException

open

public void open(eu.stratosphere.core.fs.FileInputSplit split)
          throws IOException

Specified by:

open in interface eu.stratosphere.api.common.io.InputFormat<T,eu.stratosphere.core.fs.FileInputSplit>

Overrides:

open in class eu.stratosphere.api.common.io.FileInputFormat<T>

Throws:

IOException

reachedEnd

public boolean reachedEnd()
                   throws IOException

Throws:

IOException

setBlockSize

public void setBlockSize(long blockSize)

Sets the blockSize to the specified value.

Parameters:

blockSize - the blockSize to set

createBlockInfo

protected BlockInfo createBlockInfo()

Creates the block info. May be hooked for additional information.

createStatistics

protected eu.stratosphere.core.testing.io.BinaryInputFormat.SequentialStatistics createStatistics(List<eu.stratosphere.core.fs.FileStatus> files,
                                                                                      eu.stratosphere.api.common.io.FileInputFormat.FileBaseStatistics stats)
                                                                                           throws IOException

Fill in the statistics. The last modification time and the total input size are prefilled.

Parameters:

files - The files that are associated with this block input format.

stats - The pre-filled statistics.

Throws:

IOException

deserialize

protected abstract void deserialize(T record,
               eu.stratosphere.core.memory.DataInputView dataInput)
                             throws IOException

Deserializes the given record from the DataInputView.

Throws:

IOException

configureBinaryFormat

public static BinaryInputFormat.ConfigBuilder configureBinaryFormat(eu.stratosphere.configuration.Configuration config)

Creates a configuration builder that can be used to set the input format's parameters to the config in a fluent fashion.

Returns:

A config builder for setting parameters.

configureBinaryFormat

public static BinaryInputFormat.ConfigBuilder configureBinaryFormat(eu.stratosphere.api.common.operators.GenericDataSource<?> target)

Creates a configuration builder that can be used to set the input format's parameters to the config in a fluent fashion.

Returns:

A config builder for setting parameters.