org.apache.spark.sql.execution.streaming
OffsetSeqLog
Related Docs:
object OffsetSeqLog
| package streaming
org.apache.spark.sql.execution.streaming
Store the metadata for the specified batchId and return true if successful.
Store the metadata for the specified batchId and return true if successful. If the batchId's
metadata has already been stored, this method will return false.
A PathFilter to filter only batch files
A PathFilter to filter only batch files
Return metadata for batches between startId (inclusive) and endId (inclusive).
Return metadata for batches between startId (inclusive) and endId (inclusive). If startId is
None, just return all batches before endId (inclusive).
Return the metadata for the specified batchId if it's stored.
Return the metadata for the specified batchId if it's stored. Otherwise, return None.
the deserialized metadata in a batch file, or None if file not exist.
IllegalArgumentException when path does not point to a batch file.
Return the latest batch Id and its metadata if exist.
Return the latest batch Id and its metadata if exist.
Get an array of [FileStatus] referencing batch files.
Get an array of [FileStatus] referencing batch files. The array is sorted by most recent batch file first to oldest batch file.
Removes all the log entry earlier than thresholdBatchId (exclusive).
Removes all the log entry earlier than thresholdBatchId (exclusive).
This class is used to log offsets to persistent files in HDFS. Each file corresponds to a specific batch of offsets. The file format contain a version string in the first line, followed by a the JSON string representation of the offsets separated by a newline character. If a source offset is missing, then that line will contain a string value defined in the SERIALIZED_VOID_OFFSET variable in OffsetSeqLog companion object. For instance, when dealing with LongOffset types: v1 // version 1 metadata {0} // LongOffset 0 {3} // LongOffset 3