util

Type Members

class AbstractScalaRowIterator[T] extends Iterator[T]

Shim to allow us to implement scala.Iterator in Java.
Shim to allow us to implement scala.Iterator in Java. Scala 2.11+ has an AbstractIterator class for this, but that class is private[scala] in 2.10. We need to explicitly fix this to Row in order to work around a spurious IntelliJ compiler error. This cannot be an abstract class because that leads to compilation errors under Scala 2.11.
class ArrayBasedMapData extends MapData
abstract class ArrayData extends SpecializedGetters with Serializable
class CaseInsensitiveMap[T] extends Map[String, T] with Serializable

Builds a map in which keys are case insensitive.
Builds a map in which keys are case insensitive. Input map can be accessed for cases where case-sensitive information is required. The primary constructor is marked private to avoid nested case-insensitive map creation, otherwise the keys in the original map will become case-insensitive in this scenario.
class GenericArrayData extends ArrayData
abstract class MapData extends Serializable

This is an internal data representation for map type in Spark SQL.
This is an internal data representation for map type in Spark SQL. This should not implement equals and hashCode because the type cannot be used as join keys, grouping keys, or in equality tests. See SPARK-9415 and PR#13847 for the discussions.
class QuantileSummaries extends Serializable

Helper class to compute approximate quantile summary.
Helper class to compute approximate quantile summary. This implementation is based on the algorithm proposed in the paper: "Space-efficient Online Computation of Quantile Summaries" by Greenwald, Michael and Khanna, Sanjeev. (http://dx.doi.org/10.1145/375663.375670)
In order to optimize for speed, it maintains an internal buffer of the last seen samples, and only inserts them after crossing a certain size threshold. This guarantees a near-constant runtime complexity compared to the original algorithm.
final class SerializedArray extends ArrayData with SerializedRowData

An implementation of ArrayData serialized like a SerializedRow that pays attention to the platform endianness and uses a consistent endian format (little-endian) so is suitable for storage.
final class SerializedMap extends MapData with Externalizable with KryoSerializable

An implementation of MapData serialized like SerializedRow elements that pays attention to the platform endianness and uses a consistent endian format (little-endian) so is suitable for storage.
final class SerializedRow extends InternalRow with SerializedRowData

Like UnsafeRow uses raw memory bytes to encode the data in a row instead of objects.
Like UnsafeRow uses raw memory bytes to encode the data in a row instead of objects. The difference is that it pays attention to endianness to use a consistent endian format (little-endian) so is suitable for storage.
trait SerializedRowData extends SpecializedGetters with Externalizable with KryoSerializable

Contains the main code for SerializedRow and SerializedArray.
class StringKeyHashMap[T] extends AnyRef

Value Members

object ArrayBasedMapData extends Serializable
object ArrayData extends Serializable
object CaseInsensitiveMap extends Serializable
object CompressionCodecs
object DateTimeUtils

Helper functions for converting between internal and external date and time representations.
Helper functions for converting between internal and external date and time representations. Dates are exposed externally as java.sql.Date and are represented internally as the number of dates since the Unix epoch (1970-01-01). Timestamps are exposed externally as java.sql.Timestamp and are stored internally as longs, which are capable of storing timestamps with 100 nanosecond precision.
object NumberConverter
object ParseModes
object QuantileSummaries extends Serializable
object StringKeyHashMap

Build a map with String type of key, and it also supports either key case sensitive or insensitive.
object StringUtils
object TypeUtils

Helper functions to check for valid data types.
def benchmark[A](f: ⇒ A): A
def fileToString(file: File, encoding: String = "UTF-8"): String
def quietly[A](f: ⇒ A): A

Silences output to stderr or stdout for the duration of f
def quoteIdentifier(name: String): String
def resourceToBytes(resource: String, classLoader: ClassLoader = Utils.getSparkClassLoader): Array[Byte]
def resourceToString(resource: String, encoding: String = "UTF-8", classLoader: ClassLoader = Utils.getSparkClassLoader): String
def sideBySide(left: Seq[String], right: Seq[String]): Seq[String]
def sideBySide(left: String, right: String): Seq[String]
def stackTraceToString(t: Throwable): String
def stringOrNull(a: AnyRef): String
def stringToFile(file: File, str: String): File
def toPrettySQL(e: Expression): String
def usePrettyExpression(e: Expression): Expression

package util

Type Members

class AbstractScalaRowIterator[T] extends Iterator[T]

class ArrayBasedMapData extends MapData

abstract class ArrayData extends SpecializedGetters with Serializable

class CaseInsensitiveMap[T] extends Map[String, T] with Serializable

class GenericArrayData extends ArrayData

abstract class MapData extends Serializable

class QuantileSummaries extends Serializable

final class SerializedArray extends ArrayData with SerializedRowData

final class SerializedMap extends MapData with Externalizable with KryoSerializable

final class SerializedRow extends InternalRow with SerializedRowData

trait SerializedRowData extends SpecializedGetters with Externalizable with KryoSerializable

class StringKeyHashMap[T] extends AnyRef

Value Members

object ArrayBasedMapData extends Serializable

object ArrayData extends Serializable

object CaseInsensitiveMap extends Serializable

object CompressionCodecs

object DateTimeUtils

object NumberConverter

object ParseModes

object QuantileSummaries extends Serializable

object StringKeyHashMap

object StringUtils

object TypeUtils

def benchmark[A](f: ⇒ A): A

def fileToString(file: File, encoding: String = "UTF-8"): String

def quietly[A](f: ⇒ A): A

def quoteIdentifier(name: String): String

def resourceToBytes(resource: String, classLoader: ClassLoader = Utils.getSparkClassLoader): Array[Byte]

def resourceToString(resource: String, encoding: String = "UTF-8", classLoader: ClassLoader = Utils.getSparkClassLoader): String

def sideBySide(left: Seq[String], right: Seq[String]): Seq[String]

def sideBySide(left: String, right: String): Seq[String]

def stackTraceToString(t: Throwable): String

def stringOrNull(a: AnyRef): String

def stringToFile(file: File, str: String): File

def toPrettySQL(e: Expression): String

def usePrettyExpression(e: Expression): Expression

Inherited from AnyRef

Inherited from Any

Ungrouped