org.calrissian.accumulorecipes.commons.hadoop

Class GroupedKeyRangePartitioner

java.lang.Object

org.apache.hadoop.mapreduce.Partitioner<GroupedKey,org.apache.hadoop.io.Writable>

org.calrissian.accumulorecipes.commons.hadoop.GroupedKeyRangePartitioner

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable

public class GroupedKeyRangePartitioner
extends org.apache.hadoop.mapreduce.Partitioner<GroupedKey,org.apache.hadoop.io.Writable>
implements org.apache.hadoop.conf.Configurable

Hadoop partitioner that uses ranges, and optionally sub-bins based on hashing. This range partitioner will use multiple groups to determine the partition, therefore allowing several reducers to represent several different writes to files for different tables. It could be used, for instance, for a multi-table bulk ingest.

Constructor Summary

Constructors

Constructor and Description
GroupedKeyRangePartitioner()

Method Summary

Methods

Modifier and Type	Method and Description
static void	addSplitFile(org.apache.hadoop.mapreduce.JobContext job, String group, String file) Sets the hdfs file name to use, containing a newline separated list of Base64 encoded split points that represent ranges for partitioning
org.apache.hadoop.conf.Configuration	getConf()
int	getPartition(GroupedKey key, org.apache.hadoop.io.Writable value, int numPartitions)
void	setConf(org.apache.hadoop.conf.Configuration conf)
static void	setNumSubBins(org.apache.hadoop.mapreduce.JobContext job, int num) Sets the number of random sub-bins per range

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GroupedKeyRangePartitioner

public GroupedKeyRangePartitioner()

Method Detail

addSplitFile

public static void addSplitFile(org.apache.hadoop.mapreduce.JobContext job,
                String group,
                String file)

Sets the hdfs file name to use, containing a newline separated list of Base64 encoded split points that represent ranges for partitioning

setNumSubBins

public static void setNumSubBins(org.apache.hadoop.mapreduce.JobContext job,
                 int num)

Sets the number of random sub-bins per range

getPartition

public int getPartition(GroupedKey key,
               org.apache.hadoop.io.Writable value,
               int numPartitions)

Specified by:

getPartition in class org.apache.hadoop.mapreduce.Partitioner<GroupedKey,org.apache.hadoop.io.Writable>

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface org.apache.hadoop.conf.Configurable

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable