org.apache.spark.sql.execution.columnar
impl
This class acts as a DataSource provider for column format tables provided Snappy.
Base trait for iterators that are capable of reading and returning the entire set of columns of a column batch.
Base trait for iterators that are capable of reading and returning the entire set of columns of a column batch. These can be local region iterators or those fetching entries from remote nodes.
Encapsulates a delta for update to be applied to column table and also is stored in the region.
Encapsulates a delta for update to be applied to column table and also
is stored in the region. The key for a delta is a negative columnIndex
evaluated as (ColumnFormatEntry.DELTA_STATROW_COL_INDEX - 1 + MAX_DEPTH * -columnIndex)
where columnIndex is the 0-based index of the underlying table column.
Note that this delta is for carrying the delta update and applying on existing delta, if any, while the actual value that is stored in the region is a ColumnFormatValue. This is to ensure clean working of the delta mechanism where store-layer code checks the type of object for Delta and makes assumptions about it (like it being a temporary value that should not go into region etc).
For a description of column delta format see the class comments in org.apache.spark.sql.execution.columnar.encoding.ColumnDeltaEncoder.
A RowEncoder implementation for ColumnFormatValue and child classes.
A customized iterator for column store tables that projects out the required columns and returns those column batches first that have all their columns in the memory.
A customized iterator for column store tables that projects out the required columns and returns those column batches first that have all their columns in the memory. Further this will make use of DiskBlockSortManager to allow for concurrent partition iterators to do cross-partition disk block sorting and fault-in for best disk read performance (SNAP-2012).
Key object in the column store.
Value object in the column store simply encapsulates binary data as a ByteBuffer.
Value object in the column store simply encapsulates binary data as a ByteBuffer. This can be either a direct buffer or a heap buffer depending on the system off-heap configuration. The reason for a separate type is to easily store data off-heap without any major changes to engine otherwise as well as efficiently serialize/deserialize them directly to Oplog/socket.
This class extends SerializedDiskBuffer to avoid a copy when reading/writing from Oplog. Consequently it writes the serialization header itself (typeID + classID + size) into stream as would be written by DataSerializer.writeObject. This helps it avoid additional byte writes when transferring data to the channels.
Partition resolver for the column store.
Result of compaction of a column batch added to transaction pre-commit results.
Result of compaction of a column batch added to transaction pre-commit results.
NOTE: if the layout of this class or ColumnFormatKey changes, then update the regex pattern in SnapshotConnectionListener.parseCompactionResult that parses the toString() of this class
Column tables don't support any extensions over regular Spark schema syntax, but the support for ExternalSchemaRelationProvider has been added as a workaround to allow for specifying schema in a CREATE TABLE AS SELECT statement.
Column tables don't support any extensions over regular Spark schema syntax, but the support for ExternalSchemaRelationProvider has been added as a workaround to allow for specifying schema in a CREATE TABLE AS SELECT statement.
Normally Spark does not allow specifying schema in a CTAS statement for DataSources (except its special "hive" provider), so schema is passed here as string which is parsed locally in the CreatableRelationProvider implementation.
Currently this is same as ColumnFormatRelation but has kept it as a separate class to allow adding of any index specific functionality in future.
Column Store implementation for GemFireXD.
A ClusteredColumnIterator that fetches entries from a remote bucket.
A ClusteredColumnIterator that fetches entries from a remote bucket.
TODO: PERF: instead of fetching using getAll, this should instead open a named ColumnFormatIterator on the remote node hosting the bucket, then step through the iterator to fetch batch (or batches) at a time using Function/GfxdFunctionMessage invocations. As of now, the getAll invocation does not honour ordered disk reads, proper fault-in etc.
Provides a ColumnBatchIterator over a single column batch for ColumnTableScan.
The type of the generated class used by column stats check for a column batch.
Compact column batches, if required, and insert new compacted column batches, or if they are too small then push into row delta buffer.
Utility methods for column format storage keys and values.
This class acts as a DataSource provider for column format tables provided Snappy. It uses GemFireXD as actual datastore to physically locate the tables. Column tables can be used for storing data in columnar compressed format. A example usage is given below.
val data = Seq(Data(1, 2, 3), Data(7, 8, 9), Data(9, 2, 3), Data(4, 2, 3), Data(5, 6, 7)) val dataDF = snc.createDataset(data)(Encoders.product) snc.createTable(tableName, "column", dataDF.schema, props) dataDF.write.insertInto(tableName)
This provider scans underlying tables in parallel and is aware of the data partition. It does not introduces a shuffle if simple table query is fired. One can insert a single or multiple rows into this table as well as do a bulk insert by a Spark DataFrame. Bulk insert example is shown above.