org.apache.spark.streaming.dstream

Class FileInputDStream.FileInputDStreamCheckpointData

Object

org.apache.spark.streaming.dstream.DStreamCheckpointData<scala.Tuple2<K,V>>

org.apache.spark.streaming.dstream.FileInputDStream.FileInputDStreamCheckpointData

All Implemented Interfaces:

java.io.Serializable, Logging

Enclosing class:

FileInputDStream<K,V,F extends org.apache.hadoop.mapreduce.InputFormat<K,V>>

public class FileInputDStream.FileInputDStreamCheckpointData
extends DStreamCheckpointData<scala.Tuple2<K,V>>

A custom version of the DStreamCheckpointData that stores names of Hadoop files as checkpoint data.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
FileInputDStream.FileInputDStreamCheckpointData()

Method Summary

Methods

Modifier and Type	Method and Description
void	cleanup(Time time) Cleanup old checkpoint data.
scala.collection.mutable.HashMap<Time,String[]>	hadoopFiles()
void	restore() Restore the checkpoint data.
String	toString()
void	update(Time time) Updates the checkpoint data of the DStream.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

FileInputDStream.FileInputDStreamCheckpointData

public FileInputDStream.FileInputDStreamCheckpointData()

Method Detail

hadoopFiles

public scala.collection.mutable.HashMap<Time,String[]> hadoopFiles()

update

public void update(Time time)

Description copied from class: DStreamCheckpointData

Updates the checkpoint data of the DStream. This gets called every time the graph checkpoint is initiated. Default implementation records the checkpoint files to which the generate RDDs of the DStream has been saved.

Overrides:

update in class DStreamCheckpointData<scala.Tuple2<K,V>>

cleanup

public void cleanup(Time time)

Description copied from class: DStreamCheckpointData

Cleanup old checkpoint data. This gets called after a checkpoint of time has been written to the checkpoint directory.

Overrides:

cleanup in class DStreamCheckpointData<scala.Tuple2<K,V>>

restore

public void restore()

Description copied from class: DStreamCheckpointData

Restore the checkpoint data. This gets called once when the DStream graph (along with its DStreams) are being restored from a graph checkpoint file. Default implementation restores the RDDs from their checkpoint files.

Overrides:

restore in class DStreamCheckpointData<scala.Tuple2<K,V>>

toString

public String toString()

Overrides:

toString in class DStreamCheckpointData<scala.Tuple2<K,V>>