net.metanotion.io.appenddb
Class AppendBlockIO

java.lang.Object
  net.metanotion.io.appenddb.AppendBlockIO

All Implemented Interfaces:

Closeable, Flushable, ByteGet, BlockIO<Integer,byte[],Transaction>

public final class AppendBlockIO
extends Object
implements BlockIO<Integer,byte[],Transaction>, ByteGet

This implements a block device tied to a random access file. This is the "third" version of this idea I have implemented. Also, this is an evolution of ideas expressed in the low level SQLite file format. The chief advancement this represents over my previous attempts is to use an "immutable" in memory structure and file writes that only append to the file. "Atomic Appends" has allowed me to dispense entirely with the notion of "fixed" page sizes and makes all "block" numbers a logical address, rather than physical. Also, this style makes transactions easier to implement, as there is no "write-ahead-logging". Instead, a cache of the "last" version of a block is stored in a B-Tree(also immutable) stored at the end of the file. In the event that this tree is not there, it is assumed that the file needs to be recovered, and it is "replayed" to get the current state. The only real issue with this structure is that failure recovery could be time-consuming (as the whole file has to be replayed), and that the file will always grow. The growing file issue can be solved with multiple AppendBlockIO files. Basically, a compaction is started on the current file into the new file, with a reset transaction counter. This functionality has not been implemented yet.

Field Summary

protected RandomAccess	appendFile Handle to the file backing this device.
protected	cache Block cache.
protected Object	commitSync Synchronzie's transaction commits.
protected AtomicBoolean	isOpen
static long	MAGIC Magic bytes identifying the file format.
protected AtomicInteger	nextFreeBlock
protected AtomicInteger	nextXID
protected Map<Long,BlockKey>	offsetToBlocks
protected AtomicReference<net.metanotion.io.appenddb.BlockRootMap>	root

Constructor Summary

AppendBlockIO(RandomAccess appendFile, long maxBytes)
Create a n Append BlockIO device with given file and transaction log.

Method Summary

void	abort(Transaction t) Abandon a transaction in progress.
Integer	allocBlock()
Integer	allocBlock(Transaction t) Allocate a block # for the application to use.
Transaction	begin() Begin a transaction.
Transaction	begin(Transaction t) Start a nested transaction.
void	close()
void	commit(Transaction t) Attempt to commit the transaction.
void	compact(RandomAccess outfile) Compact this block device to outfile.
void	flush()
void	freeBlock(Integer block)
void	freeBlock(Integer block, Transaction t)
BlockHandle<byte[]>	getBlock(Integer block)
BlockHandle<byte[]>	getBlock(Integer block, Transaction t)
byte[]	getBytes(int block)
boolean	hasTransactions() This device supports transactions, and will always return true.
protected net.metanotion.io.appenddb.Block	load(long offset) Load a block from the file.
void	setBytes(int block, byte[] d)
protected long	writeBlock(net.metanotion.io.appenddb.Block block)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAGIC

public static final long MAGIC

Magic bytes identifying the file format. Although this is unstructured, it will probably be incremented by 1 each time the format changes. Whether that is the case or not, it will be changed if there is a change in format. My only regret is that I can't spell madcow in hex.

See Also:

Constant Field Values

appendFile

protected final RandomAccess appendFile

Handle to the file backing this device.

cache

protected final  cache

Block cache. Size is determined in constructor

offsetToBlocks

protected final Map<Long,BlockKey> offsetToBlocks

isOpen

protected final AtomicBoolean isOpen

nextFreeBlock

protected final AtomicInteger nextFreeBlock

nextXID

protected final AtomicInteger nextXID

root

protected final AtomicReference<net.metanotion.io.appenddb.BlockRootMap> root

commitSync

protected final Object commitSync

Synchronzie's transaction commits.

Constructor Detail

AppendBlockIO

public AppendBlockIO(RandomAccess appendFile,
                     long maxBytes)
              throws RecoveryException

Create a n Append BlockIO device with given file and transaction log.

Parameters:

appendFile - File to store blocks in.

maxBytes - Size of read/write cache in bytes(this is approximate).

Throws:

RecoveryException - to indicate the need to run "recovery mode" on the index. Please see the RecoverDB.recover static method.

Method Detail

load

protected net.metanotion.io.appenddb.Block load(long offset)

Load a block from the file. This delegates the decoding work net.metanotion.io.blockdevices.Block. This looks for the block in the cache first.

Parameters:

offset - The file offset to load the block from.

getBytes

public byte[] getBytes(int block)

Specified by:

getBytes in interface ByteGet

setBytes

public void setBytes(int block,
                     byte[] d)

Specified by:

setBytes in interface ByteGet

writeBlock

protected long writeBlock(net.metanotion.io.appenddb.Block block)

flush

public void flush()

Specified by:

flush in interface Flushable

Specified by:

flush in interface BlockIO<Integer,byte[],Transaction>

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface BlockIO<Integer,byte[],Transaction>

compact

public void compact(RandomAccess outfile)

Compact this block device to outfile.

Parameters:

outfile - File to store the compacted file into.

allocBlock

public Integer allocBlock(Transaction t)

Allocate a block # for the application to use.

Specified by:

allocBlock in interface BlockIO<Integer,byte[],Transaction>

Parameters:

t - Transaction Handle.

freeBlock

public void freeBlock(Integer block,
                      Transaction t)

Specified by:

freeBlock in interface BlockIO<Integer,byte[],Transaction>

getBlock

public BlockHandle<byte[]> getBlock(Integer block,
                                    Transaction t)

Specified by:

getBlock in interface BlockIO<Integer,byte[],Transaction>

allocBlock

public Integer allocBlock()

Specified by:

allocBlock in interface BlockIO<Integer,byte[],Transaction>

getBlock

public BlockHandle<byte[]> getBlock(Integer block)

Specified by:

getBlock in interface BlockIO<Integer,byte[],Transaction>

freeBlock

public void freeBlock(Integer block)

Specified by:

freeBlock in interface BlockIO<Integer,byte[],Transaction>

hasTransactions

public boolean hasTransactions()

This device supports transactions, and will always return true. The effectiveness of its implementation does depend somewhat on the JVM, and host operating system in order to enforce meaningful guarantees. However, since the file is immutable(append only operation) it is reasonably likely it will be consistent and correct under most circumstances, despite OS and JVM issues.

Specified by:

hasTransactions in interface BlockIO<Integer,byte[],Transaction>

begin

public Transaction begin()

Begin a transaction. This is a non-blocking operation and will always succeed, unless the transaction counter wraps(2^31-1 transactions, so... good luck with that). The transaction counter can be reset by performing a compaction.

Specified by:

begin in interface BlockIO<Integer,byte[],Transaction>

begin

public Transaction begin(Transaction t)
                  throws UnsupportedOperationException

Start a nested transaction. Nested transactions are unsupported by this device.

Specified by:

begin in interface BlockIO<Integer,byte[],Transaction>

Parameters:

t - Parent transaction.

Throws:

UnsupportedOperationException - to indicate this operation is not supported.

abort

public void abort(Transaction t)

Abandon a transaction in progress. This is a non-blocking operation, and will pretty much always succeed.

Specified by:

abort in interface BlockIO<Integer,byte[],Transaction>

Parameters:

t - Transaction handle to abandon. This handle will be unusable after this call.

commit

public void commit(Transaction t)

Attempt to commit the transaction. This is a potentially blocking operation, and obtains a lock. If this call succeeds, the transaction can still be lost if the blocks that are a part of the transaction are not flushed from the cache before a failure mode. If you really need durable, guaranteed transactions, the proper pattern is to call .flush() immediately after the .commit(t). Flush will block until the OS and JVM report that the dirty cache pages have been written to disk(note: you can still suffer from the OS write-cache).

Specified by:

commit in interface BlockIO<Integer,byte[],Transaction>

Parameters:

t - Transaction Handle to commit.

Throws:

CommitException - if the transaction was aborted.