org.apache.spark.sql.execution.streaming
Derives a compact interval from the latest compact batch id and a default compact interval.
Returns all necessary logs before batchId
(inclusive).
Returns all necessary logs before batchId
(inclusive). If batchId
is a compaction, just
return itself. Otherwise, it will find the previous compaction batch and return all batches
between it and batchId
.
Returns all valid batches before the specified compactionBatchId
.
Returns all valid batches before the specified compactionBatchId
. They contain all logs we
need to do a new compaction.
E.g., if compactInterval
is 3 and compactionBatchId
is 5, this method should returns
Seq(2, 3, 4)
(Note: it includes the previous compaction batch 2).
Returns if this is a compaction batch.
Returns if this is a compaction batch. FileStreamSinkLog will compact old logs every
compactInterval
commits.
E.g., if compactInterval
is 3, then 2, 5, 8, ... are all compaction batches.
Returns the next compaction batch id after batchId
.