org.apache.spark.streaming.util

Class WriteAheadLog

Object

org.apache.spark.streaming.util.WriteAheadLog

public abstract class WriteAheadLog
extends Object

This abstract class represents a write ahead log (aka journal) that is used by Spark Streaming to save the received data (by receivers) and associated metadata to a reliable storage, so that they can be recovered after driver failures. See the Spark documentation for more information on how to plug in your own custom implementation of a write ahead log.

Constructor Summary

Constructors

Constructor and Description
WriteAheadLog()

Method Summary

Methods

Modifier and Type	Method and Description
abstract void	clean(long threshTime, boolean waitForCompletion) Clean all the records that are older than the threshold time.
abstract void	close() Close this log and release any resources.
abstract java.nio.ByteBuffer	read(WriteAheadLogRecordHandle handle) Read a written record based on the given record handle.
abstract java.util.Iterator<java.nio.ByteBuffer>	readAll() Read and return an iterator of all the records that have been written but not yet cleaned up.
abstract WriteAheadLogRecordHandle	write(java.nio.ByteBuffer record, long time) Write the record to the log and return a record handle, which contains all the information necessary to read back the written record.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WriteAheadLog

public WriteAheadLog()

Method Detail

write

public abstract WriteAheadLogRecordHandle write(java.nio.ByteBuffer record,
                              long time)

Write the record to the log and return a record handle, which contains all the information necessary to read back the written record. The time is used to the index the record, such that it can be cleaned later. Note that implementations of this abstract class must ensure that the written data is durable and readable (using the record handle) by the time this function returns.

read

public abstract java.nio.ByteBuffer read(WriteAheadLogRecordHandle handle)

Read a written record based on the given record handle.

readAll

public abstract java.util.Iterator<java.nio.ByteBuffer> readAll()

Read and return an iterator of all the records that have been written but not yet cleaned up.

clean

public abstract void clean(long threshTime,
         boolean waitForCompletion)

Clean all the records that are older than the threshold time. It can wait for the completion of the deletion.

close

public abstract void close()

Close this log and release any resources.