Class

org.apache.spark.sql.execution.columnar.impl

DefaultSource

Related Doc: package impl

Permalink

final class DefaultSource extends ExternalSchemaRelationProvider with SchemaRelationProvider with CreatableRelationProvider with DataSourceRegister with Logging

Column tables don't support any extensions over regular Spark schema syntax, but the support for ExternalSchemaRelationProvider has been added as a workaround to allow for specifying schema in a CREATE TABLE AS SELECT statement.

Normally Spark does not allow specifying schema in a CTAS statement for DataSources (except its special "hive" provider), so schema is passed here as string which is parsed locally in the CreatableRelationProvider implementation.

Linear Supertypes
Logging, DataSourceRegister, CreatableRelationProvider, SchemaRelationProvider, ExternalSchemaRelationProvider, RelationProvider, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DefaultSource
  2. Logging
  3. DataSourceRegister
  4. CreatableRelationProvider
  5. SchemaRelationProvider
  6. ExternalSchemaRelationProvider
  7. RelationProvider
  8. AnyRef
  9. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DefaultSource()

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  6. def createRelation(sqlContext: SQLContext, mode: SaveMode, options: Map[String, String], data: DataFrame): BaseColumnFormatRelation

    Permalink

    Save the DataFrame to the destination and return a relation with the given parameters based on the contents of the given DataFrame.

    Save the DataFrame to the destination and return a relation with the given parameters based on the contents of the given DataFrame. The mode specifies the expected behavior of createRelation when data already exists. Right now, there are three modes, Append, Overwrite, and ErrorIfExists. Append mode means that when saving a DataFrame to a data source, if data already exists, contents of the DataFrame are expected to be appended to existing data. Overwrite mode means that when saving a DataFrame to a data source, if data already exists, existing data is expected to be overwritten by the contents of the DataFrame. ErrorIfExists mode means that when saving a DataFrame to a data source, if data already exists, an exception is expected to be thrown.

    Definition Classes
    DefaultSource → CreatableRelationProvider
    Since

    1.3.0

  7. def createRelation(sqlContext: SQLContext, options: Map[String, String], schema: StructType): BaseColumnFormatRelation

    Permalink

    Returns a new base relation with the given parameters and user defined schema.

    Returns a new base relation with the given parameters and user defined schema.

    Definition Classes
    DefaultSource → SchemaRelationProvider
    Note

    The parameters' keywords are case insensitive and this insensitivity is enforced by the Map that is passed to the function.

  8. def createRelation(sqlContext: SQLContext, options: Map[String, String]): BaseColumnFormatRelation

    Permalink

    Returns a new base relation with the given parameters.

    Returns a new base relation with the given parameters.

    Definition Classes
    DefaultSource → RelationProvider
    Note

    The parameters' keywords are case insensitive and this insensitivity is enforced by the Map that is passed to the function.

  9. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  10. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  11. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  12. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  13. def getSchemaString(options: Map[String, String]): Option[String]

    Permalink
    Definition Classes
    ExternalSchemaRelationProvider

  14. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  15. def initializeLogIfNecessary(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  16. final def isDebugEnabled: Boolean

    Permalink
    Definition Classes
    Logging

  17. final def isInfoEnabled: Boolean

    Permalink
    Definition Classes
    Logging

  18. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  19. final def isTraceEnabled: Boolean

    Permalink
    Definition Classes
    Logging

  20. final var levelFlags: Int

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  21. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  22. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging

  23. def logDebug(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging

  24. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging

  25. def logError(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging

  26. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging

  27. def logInfo(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging

  28. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  29. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging

  30. def logTrace(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging

  31. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging

  32. def logWarning(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging

  33. final var log_: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  34. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  35. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  36. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  37. def resetLogger(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  38. def shortName(): String

    Permalink

    The string that represents the format that this data source provider uses.

    The string that represents the format that this data source provider uses. This is overridden by children to provide a nice alias for the data source. For example:

    override def shortName(): String = "parquet"
    Definition Classes
    DefaultSource → DataSourceRegister
    Since

    1.5.0

  39. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  40. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  41. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  42. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  43. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from DataSourceRegister

Inherited from CreatableRelationProvider

Inherited from SchemaRelationProvider

Inherited from ExternalSchemaRelationProvider

Inherited from RelationProvider

Inherited from AnyRef

Inherited from Any

Ungrouped