streaming

Type Members

1. final class DataStreamReader extends internal.Logging

   

   Interface used to load a streaming Dataset from external storage systems (e.g.

   Interface used to load a streaming Dataset from external storage systems (e.g. file systems, key-value stores, etc). Use SparkSession.readStream to access this.

   Annotations

   @Experimental() @Evolving()

   Since

   2.0.0

2. final class DataStreamWriter[T] extends AnyRef

   

   :: Experimental :: Interface used to write a streaming Dataset to external storage systems (e.g.

   :: Experimental :: Interface used to write a streaming Dataset to external storage systems (e.g. file systems, key-value stores, etc). Use Dataset.writeStream to access this.

   Annotations

   @Experimental() @Evolving()

   Since

   2.0.0

3. class DefaultSnappySinkCallback extends SnappySinkCallback with Logging

   

4. final class DirectKafkaStreamRelation extends StreamBaseRelation with Logging with Serializable

   

5. class DirectKafkaStreamSource extends StreamPlanProvider with DataSourceRegister

   

6. final class FileStreamRelation extends StreamBaseRelation

   

7. final class FileStreamSource extends StreamPlanProvider with DataSourceRegister

   

8. class HashTagToRowsConverter extends StreamToRowsConverter with Serializable

   

9. case class LogicalDStreamPlan(output: Seq[Attribute], stream: DStream[InternalRow])(streamingSnappy: SnappyStreamingContext) extends LogicalPlan with MultiInstanceRelation with Product with Serializable

   

10. class MyStreamConverter extends StreamConverter with Serializable

    

11. class OutputMode extends AnyRef

    

12. case class PhysicalDStreamPlan(output: Seq[Attribute], rowStream: DStream[InternalRow]) extends SparkPlan with StreamPlan with Product with Serializable

    

    A PhysicalPlan wrapper of SchemaDStream, inject the validTime and generate an effective RDD of current batchDuration.

13. case class ProcessingTime(intervalMs: Long) extends Trigger with Product with Serializable

    

    :: Experimental :: A trigger that runs a query periodically based on the processing time.

    :: Experimental :: A trigger that runs a query periodically based on the processing time. If interval is 0, the query will run as fast as possible.

    Scala Example:

    df.write.trigger(ProcessingTime("10 seconds"))
    
    import scala.concurrent.duration._
    df.write.trigger(ProcessingTime(10.seconds))

    Java Example:

    df.write.trigger(ProcessingTime.create("10 seconds"))
    
    import java.util.concurrent.TimeUnit
    df.write.trigger(ProcessingTime.create(10, TimeUnit.SECONDS))

    Annotations

    @Experimental() @Evolving()

    Since

    2.0.0

14. trait RabbitMQDecoder[T] extends AnyRef

    

15. final class RabbitMQInputDStream[T, D] extends ReceiverInputDStream[T]

    

16. final class RabbitMQReceiver[T, D] extends Receiver[T] with Logging

    

17. final class RabbitMQStreamRelation extends StreamBaseRelation

    

18. final class RabbitMQStreamSource extends StreamPlanProvider with DataSourceRegister

    

19. final class RabbitMQStringDecoder extends RabbitMQDecoder[String]

    

20. final class RawSocketStreamRelation extends StreamBaseRelation

    

21. final class RawSocketStreamSource extends StreamPlanProvider with DataSourceRegister

    

22. class SchemaDStream extends DStream[Row]

    

    A SQL based DStream with support for schema/Product This class offers the ability to manipulate SQL query on DStreams It is similar to SchemaRDD, which offers the similar functions Internally, RDD of each batch duration is treated as a small table and CQs are evaluated on those small tables Some of the abstraction and code is borrowed from the project: https://github.com/Intel-bigdata/spark-streamingsql

23. class SinkProgress extends Serializable

    

    :: Experimental :: Information about progress made for a sink in the execution of a StreamingQuery during a trigger.

    :: Experimental :: Information about progress made for a sink in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.

    Annotations

    @Experimental() @Evolving()

    Since

    2.1.0

24. trait SnappySinkCallback extends AnyRef

    

    Should be implemented by clients who wants to override default behavior provided by DefaultSnappySinkCallback.

    Should be implemented by clients who wants to override default behavior provided by DefaultSnappySinkCallback.

    In order to override the default callback behavior the qualified name of the implementing class needs to be passed against sinkCallback option while defining stream query.

25. case class SnappyStoreSink(snappySession: SnappySession, parameters: Map[String, String], sinkCallback: SnappySinkCallback) extends Sink with Logging with Product with Serializable

    

26. class SnappyStoreSinkProvider extends StreamSinkProvider with DataSourceRegister

    

27. class SnappyStreamingContextFactory extends SparkContextFactory

    

28. abstract class SnappyStreamingJob extends SparkJobBase

    

29. class SnappyStreamingQueryListener extends StreamingQueryListener

    

30. class SnappyStreamingQueryManager extends StreamingQueryManager

    

31. final class SocketStreamRelation extends StreamBaseRelation

    

32. final class SocketStreamSource extends StreamPlanProvider with DataSourceRegister

    

33. class SourceProgress extends Serializable

    

    :: Experimental :: Information about progress made for a source in the execution of a StreamingQuery during a trigger.

    :: Experimental :: Information about progress made for a source in the execution of a StreamingQuery during a trigger. See StreamingQueryProgress for more information.

    Annotations

    @Experimental() @Evolving()

    Since

    2.1.0

34. class StateOperatorProgress extends Serializable

    

    :: Experimental :: Information about updates made to stateful operators in a StreamingQuery during a trigger.

    :: Experimental :: Information about updates made to stateful operators in a StreamingQuery during a trigger.

    Annotations

    @Experimental() @Evolving()

35. abstract class StreamBaseRelation extends BaseRelation with DestroyRelation with StreamPlan with TableScan with Serializable with Logging

    

36. trait StreamConverter extends Serializable

    

37. trait StreamPlan extends AnyRef

    

38. trait StreamPlanProvider extends SchemaRelationProvider

    

39. trait StreamToRowsConverter extends Serializable

    

40. trait StreamingQuery extends AnyRef

    

    :: Experimental :: A handle to a query that is executing continuously in the background as new data arrives.

    :: Experimental :: A handle to a query that is executing continuously in the background as new data arrives. All these methods are thread-safe.

    Annotations

    @Experimental() @Evolving()

    Since

    2.0.0

41. class StreamingQueryException extends Exception

    

    :: Experimental :: Exception that stopped a StreamingQuery.

    :: Experimental :: Exception that stopped a StreamingQuery. Use cause get the actual exception that caused the failure.

    Annotations

    @Experimental() @Evolving()

    Since

    2.0.0

42. abstract class StreamingQueryListener extends AnyRef

    

    :: Experimental :: Interface for listening to events related to StreamingQueries.

    :: Experimental :: Interface for listening to events related to StreamingQueries.

    Annotations

    @Experimental() @Evolving()

    Since

    2.0.0

    Note

    The methods are not thread-safe as they may be called from different threads.

43. class StreamingQueryManager extends AnyRef

    

    :: Experimental :: A class to manage all the StreamingQuery active on a SparkSession.

    :: Experimental :: A class to manage all the StreamingQuery active on a SparkSession.

    Annotations

    @Experimental() @Evolving()

    Since

    2.0.0

44. class StreamingQueryProgress extends Serializable

    

    :: Experimental :: Information about progress made in the execution of a StreamingQuery during a trigger.

    :: Experimental :: Information about progress made in the execution of a StreamingQuery during a trigger. Each event relates to processing done for a single trigger of the streaming query. Events are emitted even when no new data is available to be processed.

    Annotations

    @Experimental() @Evolving()

    Since

    2.1.0

45. class StreamingQueryStatistics extends AnyRef

    

46. class StreamingQueryStatus extends Serializable

    

    :: Experimental :: Reports information about the instantaneous status of a streaming query.

    :: Experimental :: Reports information about the instantaneous status of a streaming query.

    Annotations

    @Experimental() @Evolving()

    Since

    2.1.0

47. class StreamingRepository extends internal.Logging

    

48. final class TextSocketStreamRelation extends StreamBaseRelation

    

49. final class TextSocketStreamSource extends StreamPlanProvider with DataSourceRegister

    

50. sealed trait Trigger extends AnyRef

    

    :: Experimental :: Used to indicate how often results should be produced by a StreamingQuery.

    :: Experimental :: Used to indicate how often results should be produced by a StreamingQuery.

    Annotations

    @Experimental() @Evolving()

    Since

    2.0.0

51. class TweetToHashtagRow extends StreamToRowsConverter with Serializable

    

52. class TweetToRetweetRow extends StreamToRowsConverter with Serializable

    

53. class TweetToRowsConverter extends StreamToRowsConverter with Serializable

    

54. final class TwitterStreamRelation extends StreamBaseRelation

    

55. final class TwitterStreamSource extends StreamPlanProvider with DataSourceRegister

    

56. case class WindowLogicalPlan(windowDuration: Duration, slideDuration: Option[Duration], child: LogicalPlan, transformed: Boolean = false) extends UnaryNode with Product with Serializable

    

57. case class WindowPhysicalPlan(windowDuration: Duration, slide: Option[Duration], child: SparkPlan) extends SparkPlan with UnaryExecNode with StreamPlan with Product with Serializable