org.apache.spark.streaming.api.java
JavaSnappyStreamingContext
Related Docs:
object JavaSnappyStreamingContext
| package java
org.apache.spark.streaming.api.java
Recreate a JavaSnappyStreamingContext from a checkpoint file using an existing SparkContext.
Recreate a JavaSnappyStreamingContext from a checkpoint file using an existing SparkContext.
Path to the directory that was specified as the checkpoint directory
Existing SparkContext
Recreate a JavaSnappyStreamingContext from a checkpoint file.
Recreate a JavaSnappyStreamingContext from a checkpoint file.
Path to the directory that was specified as the checkpoint directory
Recreate a JavaSnappyStreamingContext from a checkpoint file.
Recreate a JavaSnappyStreamingContext from a checkpoint file.
Path to the directory that was specified as the checkpoint directory
Optional, configuration object if necessary for reading from HDFS compatible filesystems
Create a JavaSnappyStreamingContext by providing the configuration necessary for a new SparkContext.
Create a JavaSnappyStreamingContext by providing the configuration necessary for a new SparkContext.
a org.apache.spark.SparkConf object specifying Spark parameters
the time interval at which streaming data will be divided into batches
Create a JavaSnappyStreamingContext using an existing SparkContext.
Create a JavaSnappyStreamingContext using an existing SparkContext.
existing SparkContext
the time interval at which streaming data will be divided into batches
Create a JavaSnappyStreamingContext using an existing SparkContext.
Create a JavaSnappyStreamingContext using an existing SparkContext.
existing SparkContext
checkpoint directory
the time interval at which streaming data will be divided into batches
Add a org.apache.spark.streaming.scheduler.StreamingListener object for receiving system events related to streaming.
Add a org.apache.spark.streaming.scheduler.StreamingListener object for receiving system events related to streaming.
Wait for the execution to stop.
Wait for the execution to stop. Any exceptions that occurs during the execution will be thrown in this thread.
Wait for the execution to stop.
Wait for the execution to stop. Any exceptions that occurs during the execution will be thrown in this thread.
time to wait in milliseconds
true if it's stopped; or throw the reported error during the execution; or false
if the waiting time elapsed before returning from the method.
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as flat binary files with fixed record lengths, yielding byte arrays
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as flat binary files with fixed record lengths, yielding byte arrays
HDFS directory to monitor for new files
The length at which to split the records
We ensure that the byte array for each record in the resulting RDDs of the DStream has the provided record length.
Sets the context to periodically checkpoint the DStream operations for master fault-tolerance.
Sets the context to periodically checkpoint the DStream operations for master fault-tolerance. The graph will be checkpointed every batch interval.
HDFS-compatible directory where the checkpoint data will be reliably stored
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format. Files must be written to the monitored directory by "moving" them from another location within the same file system. File names starting with . are ignored.
Key type for reading HDFS file
Value type for reading HDFS file
Input format for reading HDFS file
HDFS directory to monitor for new file
class of key for reading HDFS file
class of value for reading HDFS file
class of input format for reading HDFS file
Function to filter paths to process
Should process only new files and ignore existing files in the directory
Hadoop configuration
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format. Files must be written to the monitored directory by "moving" them from another location within the same file system. File names starting with . are ignored.
Key type for reading HDFS file
Value type for reading HDFS file
Input format for reading HDFS file
HDFS directory to monitor for new file
class of key for reading HDFS file
class of value for reading HDFS file
class of input format for reading HDFS file
Function to filter paths to process
Should process only new files and ignore existing files in the directory
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format. Files must be written to the monitored directory by "moving" them from another location within the same file system. File names starting with . are ignored.
Key type for reading HDFS file
Value type for reading HDFS file
Input format for reading HDFS file
HDFS directory to monitor for new file
class of key for reading HDFS file
class of value for reading HDFS file
class of input format for reading HDFS file
:: DeveloperApi ::
:: DeveloperApi ::
Return the current state of the context. The context can be in three possible states -
- StreamingContextState.INITIALIZED - The context has been created, but not been started yet. Input DStreams, transformations and output operations can be created on the context.
- StreamingContextState.ACTIVE - The context has been started, and been not stopped. Input DStreams, transformations and output operations cannot be created on the context.
- StreamingContextState.STOPPED - The context has been stopped and cannot be used any more.
Create an input stream from a queue of RDDs.
Create an input stream from a queue of RDDs. In each batch, it will process either one or all of the RDDs returned by the queue.
Type of objects in the RDD
Queue of RDDs
Whether only one RDD should be consumed from the queue in every interval
Default RDD is returned by the DStream when the queue is empty
1. Changes to the queue after the stream is created will not be recognized.
2. Arbitrary RDDs can be added to queueStream, there is no way to recover data of
those RDDs, so queueStream doesn't support checkpointing.
Create an input stream from a queue of RDDs.
Create an input stream from a queue of RDDs. In each batch, it will process either one or all of the RDDs returned by the queue.
Type of objects in the RDD
Queue of RDDs
Whether only one RDD should be consumed from the queue in every interval
1. Changes to the queue after the stream is created will not be recognized.
2. Arbitrary RDDs can be added to queueStream, there is no way to recover data of
those RDDs, so queueStream doesn't support checkpointing.
Create an input stream from a queue of RDDs.
Create an input stream from a queue of RDDs. In each batch, it will process either one or all of the RDDs returned by the queue.
Type of objects in the RDD
Queue of RDDs
1. Changes to the queue after the stream is created will not be recognized.
2. Arbitrary RDDs can be added to queueStream, there is no way to recover data of
those RDDs, so queueStream doesn't support checkpointing.
Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them.
Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them. This is the most efficient way to receive data.
Type of the objects in the received blocks
Hostname to connect to for receiving data
Port to connect to for receiving data
Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them.
Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them. This is the most efficient way to receive data.
Type of the objects in the received blocks
Hostname to connect to for receiving data
Port to connect to for receiving data
Storage level to use for storing the received objects
Create an input stream with any arbitrary user implemented receiver.
Create an input stream with any arbitrary user implemented receiver. Find more details at: http://spark.apache.org/docs/latest/streaming-custom-receivers.html
Custom implementation of Receiver
Registers and executes given SQL query and returns SchemaDStream to consume the results
Registers and executes given SQL query and returns SchemaDStream to consume the results
Sets each DStreams in this context to remember RDDs it generated in the last given duration.
Sets each DStreams in this context to remember RDDs it generated in the last given duration. DStreams remember RDDs only for a limited duration of duration and releases them for garbage collection. This method allows the developer to specify how long to remember the RDDs ( if the developer wishes to query old data outside the DStream computation).
Minimum duration that each DStream should remember its RDDs
Create an input stream from network source hostname:port.
Create an input stream from network source hostname:port. Data is received using a TCP socket and the receive bytes it interpreted as object using the given converter.
Type of the objects received (after converting bytes to objects)
Hostname to connect to for receiving data
Port to connect to for receiving data
Function to convert the byte stream to objects
Storage level to use for storing the received objects
Create an input stream from network source hostname:port.
Create an input stream from network source hostname:port. Data is received using a TCP socket and the receive bytes is interpreted as UTF8 encoded \n delimited lines. Storage level of the data will be the default StorageLevel.MEMORY_AND_DISK_SER_2.
Hostname to connect to for receiving data
Port to connect to for receiving data
Create an input stream from network source hostname:port.
Create an input stream from network source hostname:port. Data is received using a TCP socket and the receive bytes is interpreted as UTF8 encoded \n delimited lines.
Hostname to connect to for receiving data
Port to connect to for receiving data
Storage level to use for storing the received objects
The underlying SparkContext
The underlying SparkContext
Start the execution of the streams.
Start the execution of the streams. Also registers population of AQP tables from stream tables if present.
IllegalStateException if the JavaSnappyStreamingContext is already stopped
Stop the execution of the streams.
Stop the execution of the streams.
Stop the associated SparkContext or not
Stop gracefully by waiting for the processing of all received data to be completed
Stop the execution of the streams.
Stop the execution of the streams.
Stop the associated SparkContext or not
Stop the execution of the streams.
Stop the execution of the streams. Will stop the associated JavaSparkContext as well.
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as text files (using key as LongWritable, value as Text and input format as TextInputFormat).
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as text files (using key as LongWritable, value as Text and input format as TextInputFormat). Files must be written to the monitored directory by "moving" them from another location within the same file system. File names starting with . are ignored.
HDFS directory to monitor for new file
Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams.
Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams. The order of the JavaRDDs in the transform function parameter will be the same as the order of corresponding DStreams in the list.
For adding a JavaPairDStream in the list of JavaDStreams, convert it to a JavaDStream using org.apache.spark.streaming.api.java.JavaPairDStream.toJavaDStream(). In the transform function, convert the JavaRDD corresponding to that JavaDStream to a JavaPairRDD using org.apache.spark.api.java.JavaPairRDD.fromJavaRDD().
Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams.
Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams. The order of the JavaRDDs in the transform function parameter will be the same as the order of corresponding DStreams in the list.
For adding a JavaPairDStream in the list of JavaDStreams, convert it to a JavaDStream using org.apache.spark.streaming.api.java.JavaPairDStream.toJavaDStream(). In the transform function, convert the JavaRDD corresponding to that JavaDStream to a JavaPairRDD using org.apache.spark.api.java.JavaPairRDD.fromJavaRDD().
Create a unified DStream from multiple DStreams of the same type and same slide duration.
Create a unified DStream from multiple DStreams of the same type and same slide duration.
Create a unified DStream from multiple DStreams of the same type and same slide duration.
Create a unified DStream from multiple DStreams of the same type and same slide duration.