Class/Object

org.apache.spark.sql.execution

StratifiedSampler

Related Docs: object StratifiedSampler | package execution

Permalink

abstract class StratifiedSampler extends Serializable with Cloneable with Logging

Linear Supertypes
Logging, Cloneable, Cloneable, Serializable, Serializable, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StratifiedSampler
  2. Logging
  3. Cloneable
  4. Cloneable
  5. Serializable
  6. Serializable
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new StratifiedSampler(options: SampleOptions)

    Permalink

Type Members

  1. type ReservoirSegment = SegmentMap[Row, StratumReservoir]

    Permalink
  2. final class RowWithWeight extends Row

    Permalink
    Attributes
    protected

Abstract Value Members

  1. abstract def append[U](rows: Iterator[Row], init: U, processFlush: (U, InternalRow) ⇒ U, startBatch: (U, Int) ⇒ U, endBatch: (U) ⇒ U, rowEncoder: ExpressionEncoder[Row], partIndex: Int): Long

    Permalink
  2. abstract def flushReservoir[U](init: U, process: (U, InternalRow) ⇒ U, startBatch: (U, Int) ⇒ U, endBatch: (U) ⇒ U): U

    Permalink
  3. abstract def onTruncate(): Unit

    Permalink
  4. abstract def sample(items: Iterator[InternalRow], rowEncoder: ExpressionEncoder[Row], flush: Boolean): Iterator[InternalRow]

    Permalink
  5. abstract def strataReservoirSize: Int

    Permalink
    Attributes
    protected

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. final def concurrency: Int

    Permalink
  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def foldDrainSegment[U](prevReservoirSize: Int, fullReset: Boolean, process: (U, InternalRow) ⇒ U)(init: U, seg: ReservoirSegment): U

    Permalink
    Attributes
    protected
  11. final def foldReservoir[U](prevReservoirSize: Int, doReset: Boolean, fullReset: Boolean, process: (U, InternalRow) ⇒ U)(bid: Int, sr: StratumReservoir, init: U): U

    Permalink
    Attributes
    protected
  12. def getBucketId(partIndex: Int, primaryBucketIds: IntArrayList = null)(hashValue: Int): Int

    Permalink
    Attributes
    protected
  13. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  14. def getReservoirSegment(newQcs: Array[Int], types: Array[DataType], numColumns: Int, initialCapacity: Int, loadFactor: Double, qcsColHandler: Option[ColumnHandler], segi: Int, nsegs: Int): ReservoirSegment

    Permalink
    Attributes
    protected
  15. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  16. def initializeLogIfNecessary(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  17. def isBucketLocal(partIndex: Int): Boolean

    Permalink
    Attributes
    protected
  18. final def isDebugEnabled: Boolean

    Permalink
    Definition Classes
    Logging
  19. final def isInfoEnabled: Boolean

    Permalink
    Definition Classes
    Logging
  20. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  21. final def isTraceEnabled: Boolean

    Permalink
    Definition Classes
    Logging
  22. def iterator(segmentStart: Int, segmentEnd: Int): Iterator[InternalRow]

    Permalink
  23. def iteratorOnRegion(buckets: Set[Integer]): Iterator[InternalRow]

    Permalink
  24. final var levelFlags: Int

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  25. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  26. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging
  27. def logDebug(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  28. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging
  29. def logError(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  30. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging
  31. def logInfo(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  32. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  33. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging
  34. def logTrace(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  35. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Definition Classes
    Logging
  36. def logWarning(msg: ⇒ String): Unit

    Permalink
    Definition Classes
    Logging
  37. final var log_: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  38. def module: String

    Permalink
  39. final def name: String

    Permalink
  40. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  41. final def newMutableRow(row: Row, rowEncoder: ExpressionEncoder[Row]): UnsafeRow

    Permalink
    Attributes
    protected
  42. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  43. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  44. final val options: SampleOptions

    Permalink
  45. final val pendingBatch: AtomicReference[ArrayBuffer[InternalRow]]

    Permalink

    Store pending values to be flushed in a separate buffer so that we do not end up creating too small ColumnBatches.

    Store pending values to be flushed in a separate buffer so that we do not end up creating too small ColumnBatches.

    Note that this mini-cache is copy-on-write (to avoid copy-on-read for readers) so the buffer inside should never be changed rather the whole buffer replaced if required. This should happen only inside flushCache.

    Attributes
    protected
  46. final def qcs: Array[Int]

    Permalink
  47. final def qcsSparkPlan: Option[(CodeAndComment, ArrayBuffer[Any], Int, Array[DataType])]

    Permalink
  48. def reservoirInRegion: Boolean

    Permalink
  49. def resetLogger(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  50. final val rng: Random

    Permalink

    Random number generator for sampling.

    Random number generator for sampling.

    Attributes
    protected
  51. final def schema: StructType

    Permalink
  52. def setFlushStatus(doFlush: Boolean): Unit

    Permalink
  53. final val strata: ConcurrentSegmentedHashMap[Row, StratumReservoir, ReservoirSegment]

    Permalink

    Map of each stratum key (i.e.

    Map of each stratum key (i.e. a unique combination of values of columns in qcs) to related metadata and reservoir

    Attributes
    protected
  54. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  55. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  56. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  57. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  58. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  59. def waitForSamplers(waitUntil: Int, maxMillis: Long): Unit

    Permalink
    Attributes
    protected

Inherited from Logging

Inherited from Cloneable

Inherited from Cloneable

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped