CatalogTable

A table defined in the catalog.

Note that Hive's metastore also tracks skewed columns. We should consider adding that in the future once we have a better understanding of how we want to handle skewed columns.

provider: the name of the data source provider for this table, e.g. parquet, json, etc. Can be None if this table is a View, should be "hive" for hive serde tables.
unsupportedFeatures: is a list of string descriptions of features that are used by the underlying table but not supported by Spark SQL yet.
tracksPartitionsInCatalog: whether this table's partition metadata is stored in the catalog. If false, it is inferred automatically based on file structure.
schemaPreservesCase: Whether or not the schema resolved for this table is case-sensitive. When using a Hive Metastore, this flag is set to false if a case- sensitive schema was unable to be read from the table properties. Used to trigger case-sensitive schema inference at query time, when configured.

Linear Supertypes

Serializable, Serializable, Product, Equals, AnyRef, Any

Instance Constructors

new CatalogTable(identifier: TableIdentifier, tableType: CatalogTableType, storage: CatalogStorageFormat, schema: StructType, provider: Option[String] = None, partitionColumnNames: Seq[String] = Seq.empty, bucketSpec: Option[BucketSpec] = None, owner: String = "", createTime: Long = System.currentTimeMillis, lastAccessTime: Long = 1, properties: Map[String, String] = Map.empty, stats: Option[Statistics] = None, viewOriginalText: Option[String] = None, viewText: Option[String] = None, comment: Option[String] = None, unsupportedFeatures: Seq[String] = Seq.empty, tracksPartitionsInCatalog: Boolean = false, schemaPreservesCase: Boolean = true)

provider
the name of the data source provider for this table, e.g. parquet, json, etc. Can be None if this table is a View, should be "hive" for hive serde tables.
unsupportedFeatures
is a list of string descriptions of features that are used by the underlying table but not supported by Spark SQL yet.
tracksPartitionsInCatalog
whether this table's partition metadata is stored in the catalog. If false, it is inferred automatically based on file structure.
schemaPreservesCase
Whether or not the schema resolved for this table is case-sensitive. When using a Hive Metastore, this flag is set to false if a case- sensitive schema was unable to be read from the table properties. Used to trigger case-sensitive schema inference at query time, when configured.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
val bucketSpec: Option[BucketSpec]
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val comment: Option[String]
val createTime: Long
def database: String

Return the database this table was specified to belong to, assuming it exists.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
val identifier: TableIdentifier
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val lastAccessTime: Long
def location: String

Return the table location, assuming it is specified.
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
val owner: String
val partitionColumnNames: Seq[String]
def partitionSchema: StructType

schema of this table's partition columns
val properties: Map[String, String]
val provider: Option[String]

the name of the data source provider for this table, e.g.
the name of the data source provider for this table, e.g. parquet, json, etc. Can be None if this table is a View, should be "hive" for hive serde tables.
def qualifiedName: String

Return the fully qualified name of this table, assuming the database was specified.
val schema: StructType
val schemaPreservesCase: Boolean

Whether or not the schema resolved for this table is case-sensitive.
Whether or not the schema resolved for this table is case-sensitive. When using a Hive Metastore, this flag is set to false if a case- sensitive schema was unable to be read from the table properties. Used to trigger case-sensitive schema inference at query time, when configured.
val stats: Option[Statistics]
val storage: CatalogStorageFormat
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
val tableType: CatalogTableType
def toString(): String

Definition Classes
CatalogTable → AnyRef → Any
val tracksPartitionsInCatalog: Boolean

whether this table's partition metadata is stored in the catalog.
whether this table's partition metadata is stored in the catalog. If false, it is inferred automatically based on file structure.
val unsupportedFeatures: Seq[String]

is a list of string descriptions of features that are used by the underlying table but not supported by Spark SQL yet.
val viewOriginalText: Option[String]
val viewText: Option[String]
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
def withNewStorage(locationUri: Option[String] = storage.locationUri, inputFormat: Option[String] = storage.inputFormat, outputFormat: Option[String] = storage.outputFormat, compressed: Boolean = false, serde: Option[String] = storage.serde, properties: Map[String, String] = storage.properties): CatalogTable

Syntactic sugar to update a field in storage.

Related Doc: package catalog

Instance Constructors

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

val bucketSpec: Option[BucketSpec]

def clone(): AnyRef

val comment: Option[String]

val createTime: Long

def database: String

final def eq(arg0: AnyRef): Boolean

def finalize(): Unit

final def getClass(): Class[_]

val identifier: TableIdentifier

final def isInstanceOf[T0]: Boolean

val lastAccessTime: Long

def location: String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val owner: String

val partitionColumnNames: Seq[String]

def partitionSchema: StructType

val properties: Map[String, String]

val provider: Option[String]

def qualifiedName: String

val schema: StructType

val schemaPreservesCase: Boolean

val stats: Option[Statistics]

val storage: CatalogStorageFormat

final def synchronized[T0](arg0: ⇒ T0): T0

val tableType: CatalogTableType

def toString(): String

val tracksPartitionsInCatalog: Boolean

val unsupportedFeatures: Seq[String]

val viewOriginalText: Option[String]

val viewText: Option[String]

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped