Catalog

Instance Constructors

new Catalog()

Abstract Value Members

abstract def cacheTable(tableName: String): Unit

Caches the specified table in-memory.
Caches the specified table in-memory.

Since
2.0.0
abstract def clearCache(): Unit

Removes all cached tables from the in-memory cache.
Removes all cached tables from the in-memory cache.

Since
2.0.0
abstract def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

:: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options.
:: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Annotations
@Experimental() @Evolving()
Since
2.0.0
abstract def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

:: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options.
:: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Annotations
@Experimental() @Evolving()
Since
2.0.0
abstract def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

:: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options.
:: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Annotations
@Experimental() @Evolving()
Since
2.0.0
abstract def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

:: Experimental :: Creates an external table from the given path based on a data source and a set of options.
:: Experimental :: Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Annotations
@Experimental() @Evolving()
Since
2.0.0
abstract def createExternalTable(tableName: String, path: String, source: String): DataFrame

:: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.
:: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

Annotations
@Experimental() @Evolving()
Since
2.0.0
abstract def createExternalTable(tableName: String, path: String): DataFrame

:: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame.
:: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

Annotations
@Experimental() @Evolving()
Since
2.0.0
abstract def currentDatabase: String

Returns the current default database in this session.
Returns the current default database in this session.

Since
2.0.0
abstract def databaseExists(dbName: String): Boolean

Check if the database with the specified name exists.
Check if the database with the specified name exists.

Since
2.1.0
abstract def dropGlobalTempView(viewName: String): Boolean

Drops the global temporary view with the given view name in the catalog.
Drops the global temporary view with the given view name in the catalog. If the view has been cached before, then it will also be uncached.
Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application, i.e. it will be automatically dropped when the application terminates. It's tied to a system preserved database _global_temp, and we must use the qualified name to refer a global temp view, e.g. SELECT * FROM _global_temp.view1.
viewName
the name of the view to be dropped.
returns
true if the view is dropped successfully, false otherwise.

Since
2.1.0
abstract def dropTempView(viewName: String): Boolean

Drops the local temporary view with the given view name in the catalog.
Drops the local temporary view with the given view name in the catalog. If the view has been cached before, then it will also be uncached.
Local temporary view is session-scoped. Its lifetime is the lifetime of the session that created it, i.e. it will be automatically dropped when the session terminates. It's not tied to any databases, i.e. we can't use db1.view1 to reference a local temporary view.
Note that, the return type of this method was Unit in Spark 2.0, but changed to Boolean in Spark 2.1.
viewName
the name of the view to be dropped.
returns
true if the view is dropped successfully, false otherwise.

Since
2.0.0
abstract def functionExists(dbName: String, functionName: String): Boolean

Check if the function with the specified name exists in the specified database.
Check if the function with the specified name exists in the specified database.

Since
2.1.0
abstract def functionExists(functionName: String): Boolean

Check if the function with the specified name exists.
Check if the function with the specified name exists. This can either be a temporary function or a function in the current database.

Since
2.1.0
abstract def getDatabase(dbName: String): Database

Get the database with the specified name.
Get the database with the specified name. This throws an AnalysisException when the database cannot be found.

Annotations
@throws( "database does not exist" )
Since
2.1.0
abstract def getFunction(dbName: String, functionName: String): Function

Get the function with the specified name.
Get the function with the specified name. This throws an AnalysisException when the function cannot be found.

Annotations
@throws( ... )
Since
2.1.0
abstract def getFunction(functionName: String): Function

Get the function with the specified name.
Get the function with the specified name. This function can be a temporary function or a function in the current database. This throws an AnalysisException when the function cannot be found.

Annotations
@throws( "function does not exist" )
Since
2.1.0
abstract def getTable(dbName: String, tableName: String): Table

Get the table or view with the specified name in the specified database.
Get the table or view with the specified name in the specified database. This throws an AnalysisException when no Table can be found.

Annotations
@throws( "database or table does not exist" )
Since
2.1.0
abstract def getTable(tableName: String): Table

Get the table or view with the specified name.
Get the table or view with the specified name. This table can be a temporary view or a table/view in the current database. This throws an AnalysisException when no Table can be found.

Annotations
@throws( "table does not exist" )
Since
2.1.0
abstract def isCached(tableName: String): Boolean

Returns true if the table is currently cached in-memory.
Returns true if the table is currently cached in-memory.

Since
2.0.0
abstract def listColumns(dbName: String, tableName: String): Dataset[Column]

Returns a list of columns for the given table in the specified database.
Returns a list of columns for the given table in the specified database.

Annotations
@throws( "database or table does not exist" )
Since
2.0.0
abstract def listColumns(tableName: String): Dataset[Column]

Returns a list of columns for the given table in the current database or the given temporary table.
Returns a list of columns for the given table in the current database or the given temporary table.

Annotations
@throws( "table does not exist" )
Since
2.0.0
abstract def listDatabases(): Dataset[Database]

Returns a list of databases available across all sessions.
Returns a list of databases available across all sessions.

Since
2.0.0
abstract def listFunctions(dbName: String): Dataset[Function]

Returns a list of functions registered in the specified database.
Returns a list of functions registered in the specified database. This includes all temporary functions

Annotations
@throws( "database does not exist" )
Since
2.0.0
abstract def listFunctions(): Dataset[Function]

Returns a list of functions registered in the current database.
Returns a list of functions registered in the current database. This includes all temporary functions

Since
2.0.0
abstract def listTables(dbName: String): Dataset[Table]

Returns a list of tables in the specified database.
Returns a list of tables in the specified database. This includes all temporary tables.

Annotations
@throws( "database does not exist" )
Since
2.0.0
abstract def listTables(): Dataset[Table]

Returns a list of tables in the current database.
Returns a list of tables in the current database. This includes all temporary tables.

Since
2.0.0
abstract def recoverPartitions(tableName: String): Unit

Recover all the partitions in the directory of a table and update the catalog.
Recover all the partitions in the directory of a table and update the catalog.

Since
2.1.1
abstract def refreshByPath(path: String): Unit

Invalidate and refresh all the cached data (and the associated metadata) for any dataframe that contains the given data source path.
Invalidate and refresh all the cached data (and the associated metadata) for any dataframe that contains the given data source path. Path matching is by prefix, i.e. "/" would invalidate everything that is cached.

Since
2.0.0
abstract def refreshTable(tableName: String): Unit

Invalidate and refresh all the cached metadata of the given table.
Invalidate and refresh all the cached metadata of the given table. For performance reasons, Spark SQL or the external data source library it uses might cache certain metadata about a table, such as the location of blocks. When those change outside of Spark SQL, users should call this function to invalidate the cache.
If this table is cached as an InMemoryRelation, drop the original cached version and make the new version cached lazily.

Since
2.0.0
abstract def setCurrentDatabase(dbName: String): Unit

Sets the current default database in this session.
Sets the current default database in this session.

Since
2.0.0
abstract def tableExists(dbName: String, tableName: String): Boolean

Check if the table or view with the specified name exists in the specified database.
Check if the table or view with the specified name exists in the specified database.

Since
2.1.0
abstract def tableExists(tableName: String): Boolean

Check if the table or view with the specified name exists.
Check if the table or view with the specified name exists. This can either be a temporary view or a table/view in the current database.

Since
2.1.0
abstract def uncacheTable(tableName: String): Unit

Removes the specified table from the in-memory cache.
Removes the specified table from the in-memory cache.

Since
2.0.0

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package catalog

abstract class Catalog extends AnyRef

Instance Constructors

new Catalog()

Abstract Value Members

abstract def cacheTable(tableName: String): Unit

abstract def clearCache(): Unit

abstract def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

abstract def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

abstract def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

abstract def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

abstract def createExternalTable(tableName: String, path: String, source: String): DataFrame

abstract def createExternalTable(tableName: String, path: String): DataFrame

abstract def currentDatabase: String

abstract def databaseExists(dbName: String): Boolean

abstract def dropGlobalTempView(viewName: String): Boolean

abstract def dropTempView(viewName: String): Boolean

abstract def functionExists(dbName: String, functionName: String): Boolean

abstract def functionExists(functionName: String): Boolean

abstract def getDatabase(dbName: String): Database

abstract def getFunction(dbName: String, functionName: String): Function

abstract def getFunction(functionName: String): Function

abstract def getTable(dbName: String, tableName: String): Table

abstract def getTable(tableName: String): Table

abstract def isCached(tableName: String): Boolean

abstract def listColumns(dbName: String, tableName: String): Dataset[Column]

abstract def listColumns(tableName: String): Dataset[Column]

abstract def listDatabases(): Dataset[Database]

abstract def listFunctions(dbName: String): Dataset[Function]

abstract def listFunctions(): Dataset[Function]

abstract def listTables(dbName: String): Dataset[Table]

abstract def listTables(): Dataset[Table]

abstract def recoverPartitions(tableName: String): Unit

abstract def refreshByPath(path: String): Unit

abstract def refreshTable(tableName: String): Unit

abstract def setCurrentDatabase(dbName: String): Unit

abstract def tableExists(dbName: String, tableName: String): Boolean

abstract def tableExists(tableName: String): Boolean

abstract def uncacheTable(tableName: String): Unit

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from AnyRef

Inherited from Any

Ungrouped