org.apache.spark.sql.execution.columnar.impl
ColumnDelta
Related Docs:
object ColumnDelta
| package impl
org.apache.spark.sql.execution.columnar.impl
first delta update for a column will be put as is into the region
first delta update for a column will be put as is into the region
<invalid inheritdoc annotation>
<invalid inheritdoc annotation>
This keeps track of whether the buffer is compressed or not.
This keeps track of whether the buffer is compressed or not. In addition it keeps a count of how many times compression was done on the buffer without intervening decompression, and if it exceeds ColumnFormatEntry.MAX_CONSECUTIVE_COMPRESSIONS and no one is using the decompressed buffer, then replace columnBuffer with compressed version.
A negative value indicates that the buffer is not compressible (too small or not enough compression can be achieved), a zero indicates a compressed buffer while a positive count indicates a decompressed buffer and number of times compression was done.
Return the data as a ByteBuffer.
Return the data as a ByteBuffer. Should be invoked only after a retain or getValueRetain call.
The returned buffer is guaranteed to have its order as ByteOrder.LITTLE_ENDIAN.
Callers of this method should have a corresponding release method for eager release to work else off-heap object may keep around occupying system RAM until the next GC cycle.
Callers of this method should have a corresponding release method for eager release to work else off-heap object may keep around occupying system RAM until the next GC cycle. Callers may decide whether to keep the release method in a finally block to ensure its invocation, or do it only in normal paths because JVM reference collector will eventually clean it in any case.
Calls to this specific class are guaranteed to always return buffers which have position as zero so callers can make simplifying assumptions about the same.
The returned buffer is guaranteed to have its order as ByteOrder.LITTLE_ENDIAN.
Encapsulates a delta for update to be applied to column table and also is stored in the region. The key for a delta is a negative columnIndex evaluated as (
ColumnFormatEntry.DELTA_STATROW_COL_INDEX - 1 + MAX_DEPTH * -columnIndex) wherecolumnIndexis the 0-based index of the underlying table column.Note that this delta is for carrying the delta update and applying on existing delta, if any, while the actual value that is stored in the region is a ColumnFormatValue. This is to ensure clean working of the delta mechanism where store-layer code checks the type of object for Delta and makes assumptions about it (like it being a temporary value that should not go into region etc).
For a description of column delta format see the class comments in org.apache.spark.sql.execution.columnar.encoding.ColumnDeltaEncoder.