org.kairosdb.bigqueue

Class BigArrayImpl

java.lang.Object

org.kairosdb.bigqueue.BigArrayImpl

All Implemented Interfaces:

Closeable, AutoCloseable, IBigArray

public class BigArrayImpl
extends Object
implements IBigArray

A big array implementation supporting sequential append and random read. Main features: 1. FAST : close to the speed of direct memory access, extremely fast in append only and sequential read modes, sequential append and read are close to O(1) memory access, random read is close to O(1) memory access if data is in cache and is close to O(1) disk access if data is not in cache. 2. MEMORY-EFFICIENT : automatic paging and swapping algorithm, only most-recently accessed data is kept in memory. 3. THREAD-SAFE : multiple threads can concurrently read/append the array without data corruption. 4. PERSISTENT - all array data is persisted on disk, and is crash resistant. 5. BIG(HUGE) - the total size of the array data is only limited by the available disk space.

Author:

bulldog

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_DATA_PAGE_SIZE
static int	MINIMUM_DATA_PAGE_SIZE

Fields inherited from interface org.kairosdb.bigqueue.IBigArray

NOT_FOUND

Constructor Summary

Constructors

Constructor and Description
BigArrayImpl(String arrayDir, String arrayName) A big array implementation supporting sequential write and random read, use default back data file size per page, see DEFAULT_DATA_PAGE_SIZE.
BigArrayImpl(String arrayDir, String arrayName, int pageSize) A big array implementation supporting sequential write and random read.

Method Summary

Modifier and Type	Method and Description
long	append(byte[] data) Append the data into the head of the array
void	close()
long	findClosestIndex(long timestamp) Find an index closest to the specific timestamp when the corresponding item was appended
void	flush() Force to persist newly appended data, normally, you don't need to flush explicitly since: 1.) BigArray will automatically flush a cached page when it is replaced out, 2.) BigArray uses memory mapped file technology internally, and the OS will flush the changes even your process crashes, call this periodically only if you need transactional reliability and you are aware of the cost to performance.
byte[]	get(long index) Get the data at specific index
String	getArrayDirectory()
long	getBackFileSize() Get total size of back files(index and data files) of the big array
int	getDataPageSize() Get the back data file size per page.
long	getHeadIndex() The head of the array.
int	getItemLength(long index) Get the data item length at specific index
long	getTailIndex() The tail of the array.
long	getTimestamp(long index) Get the timestamp of data at specific index, this is the timestamp when the data was appended.
boolean	isEmpty() Check if the array is empty or not
boolean	isFull() Check if the ring space of java long type has all been used up.
void	limitBackFileSize(long sizeLimit) limit the back file size, truncate back file and advance array tail index accordingly, Note, this is a best effort call, exact size limit can't be guaranteed
void	removeAll() Remove all data in this array, this will empty the array and delete all back page files.
void	removeBefore(long timestamp) Remove all data before specific timestamp, this will advance the array tail and delete back page files accordingly.
void	removeBeforeIndex(long index) Remove all data before specific index, this will advance the array tail to index and delete back page files before index.
void	setClock(Clock clock) Used for changing the clock for unit tests
long	size() The total number of items has been appended into the array

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_DATA_PAGE_SIZE

public static final int DEFAULT_DATA_PAGE_SIZE

See Also:

Constant Field Values

MINIMUM_DATA_PAGE_SIZE

public static final int MINIMUM_DATA_PAGE_SIZE

See Also:

Constant Field Values

Constructor Detail

BigArrayImpl

public BigArrayImpl(String arrayDir,
                    String arrayName)
             throws IOException

A big array implementation supporting sequential write and random read, use default back data file size per page, see DEFAULT_DATA_PAGE_SIZE.

Parameters:

arrayDir - directory for array data store

arrayName - the name of the array, will be appended as last part of the array directory

Throws:

IOException - exception throws during array initialization

BigArrayImpl

public BigArrayImpl(String arrayDir,
                    String arrayName,
                    int pageSize)
             throws IOException

A big array implementation supporting sequential write and random read.

Parameters:

arrayDir - directory for array data store

arrayName - the name of the array, will be appended as last part of the array directory

pageSize - the back data file size per page in bytes, see minimum allowed MINIMUM_DATA_PAGE_SIZE.

Throws:

IOException - exception throws during array initialization

Method Detail

setClock

public void setClock(Clock clock)

Used for changing the clock for unit tests

Parameters:

clock - Clock instance to use

getArrayDirectory

public String getArrayDirectory()

removeAll

public void removeAll()
               throws IOException

Description copied from interface: IBigArray

Remove all data in this array, this will empty the array and delete all back page files.

Specified by:

removeAll in interface IBigArray

Throws:

IOException - exception thrown if IO error occurs durring remove all

removeBeforeIndex

public void removeBeforeIndex(long index)
                       throws IOException

Description copied from interface: IBigArray

Remove all data before specific index, this will advance the array tail to index and delete back page files before index.

Specified by:

removeBeforeIndex in interface IBigArray

Parameters:

index - an index

Throws:

IOException - exception thrown if there was any IO error during the removal operation

removeBefore

public void removeBefore(long timestamp)
                  throws IOException

Description copied from interface: IBigArray

Remove all data before specific timestamp, this will advance the array tail and delete back page files accordingly.

Specified by:

removeBefore in interface IBigArray

Parameters:

timestamp - a timestamp

Throws:

IOException - exception thrown if there was any IO error during the removal operation

append

public long append(byte[] data)
            throws IOException

Append the data into the head of the array

Specified by:

append in interface IBigArray

Parameters:

data - binary data to append

Returns:

appended index

Throws:

IOException - if there is any IO error

flush

public void flush()

Description copied from interface: IBigArray

Force to persist newly appended data, normally, you don't need to flush explicitly since: 1.) BigArray will automatically flush a cached page when it is replaced out, 2.) BigArray uses memory mapped file technology internally, and the OS will flush the changes even your process crashes, call this periodically only if you need transactional reliability and you are aware of the cost to performance.

Specified by:

flush in interface IBigArray

get

public byte[] get(long index)
           throws IOException

Description copied from interface: IBigArray

Get the data at specific index

Specified by:

get in interface IBigArray

Parameters:

index - valid data index

Returns:

binary data if the index is valid

Throws:

IOException - if there is any IO error

getTimestamp

public long getTimestamp(long index)
                  throws IOException

Description copied from interface: IBigArray

Get the timestamp of data at specific index, this is the timestamp when the data was appended.

Specified by:

getTimestamp in interface IBigArray

Parameters:

index - valid data index

Returns:

timestamp when the data was appended

Throws:

IOException - if there is any IO error

size

public long size()

Description copied from interface: IBigArray

The total number of items has been appended into the array

Specified by:

size in interface IBigArray

Returns:

total number

getHeadIndex

public long getHeadIndex()

Description copied from interface: IBigArray

The head of the array. This is the next to append index, the index of the last appended data is [headIndex - 1] if the array is not empty.

Specified by:

getHeadIndex in interface IBigArray

Returns:

an index

getTailIndex

public long getTailIndex()

Description copied from interface: IBigArray

The tail of the array. The is the index of the first appended data

Specified by:

getTailIndex in interface IBigArray

Returns:

an index

isEmpty

public boolean isEmpty()

Description copied from interface: IBigArray

Check if the array is empty or not

Specified by:

isEmpty in interface IBigArray

Returns:

true if empty false otherwise

isFull

public boolean isFull()

Description copied from interface: IBigArray

Check if the ring space of java long type has all been used up. can always assume false, if true, the world is end:)

Specified by:

isFull in interface IBigArray

Returns:

array full or not

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getDataPageSize

public int getDataPageSize()

Description copied from interface: IBigArray

Get the back data file size per page.

Specified by:

getDataPageSize in interface IBigArray

Returns:

size per page

findClosestIndex

public long findClosestIndex(long timestamp)
                      throws IOException

Description copied from interface: IBigArray

Find an index closest to the specific timestamp when the corresponding item was appended

Specified by:

findClosestIndex in interface IBigArray

Parameters:

timestamp - when the corresponding item was appended

Returns:

an index

Throws:

IOException - exception thrown if there was any IO error during the getClosestIndex operation

getBackFileSize

public long getBackFileSize()
                     throws IOException

Description copied from interface: IBigArray

Get total size of back files(index and data files) of the big array

Specified by:

getBackFileSize in interface IBigArray

Returns:

total size of back files

Throws:

IOException - exception thrown if there was any IO error during the getBackFileSize operation

limitBackFileSize

public void limitBackFileSize(long sizeLimit)
                       throws IOException

Description copied from interface: IBigArray

limit the back file size, truncate back file and advance array tail index accordingly, Note, this is a best effort call, exact size limit can't be guaranteed

Specified by:

limitBackFileSize in interface IBigArray

Parameters:

sizeLimit - the size to limit

Throws:

IOException - exception thrown if there was any IO error during the limitBackFileSize operation

getItemLength

public int getItemLength(long index)
                  throws IOException

Description copied from interface: IBigArray

Get the data item length at specific index

Specified by:

getItemLength in interface IBigArray

Parameters:

index - valid data index

Returns:

the length of binary data if the index is valid

Throws:

IOException - if there is any IO error